7. Quality assurance
Quality assurance of prediction results¶
In the previous tutorial, we looked at various ways to visualise the results of our model.
These are useful for quality assurance purposes because they allow us to understand how our model works and verify that it is doing something sensible. They can also be useful to identify examples where the model is not performing as expected.
In addition to these spot checks, Splink also has functions to perform more formal accuracy analysis. These functions allow you to understand the likely prevalence of false positives and false negatives in your linkage models.
They rely on the existence of a sample of labelled (ground truth) matches, which may have been produced (for example) by human beings. For the accuracy analysis to be unbiased, the sample should be representative of the overall dataset.
# Rerun our predictions to we're ready to view the charts
from splink.duckdb.duckdb_linker import DuckDBLinker
import pandas as pd
import altair as alt
alt.renderers.enable('mimetype')
df = pd.read_csv("./data/fake_1000.csv")
linker = DuckDBLinker(df)
linker.load_settings_from_json("./demo_settings/saved_model_from_demo.json")
df_predictions = linker.predict(threshold_match_probability=0.2)
df_labels = pd.read_csv("./data/fake_1000_labels.csv")
df_labels.head(5)
labels_table = linker.register_labels_table(df_labels)
linker.roc_chart_from_labels_table(labels_table)
Precision-recall chart¶
An alternative representation of truth space is called a precision recall curve.
This can be plotted as follows:
linker.precision_recall_chart_from_labels_table(labels_table)
Truth table¶
Finally, Splink can also report the underlying table used to construct the ROC and precision recall curves.
roc_table = linker.truth_space_table_from_labels_table(labels_table)
roc_table.as_pandas_dataframe(limit=5)