Methods in Linker.table_management

Register Splink tables against your database backend and manage the Splink cache. Accessed via linker.table_management.

compute_tf_table(column_name)

Compute a term frequency table for a given column and persist to the database

This method is useful if you want to pre-compute term frequency tables e.g. so that real time linkage executes faster, or so that you can estimate various models without having to recompute term frequency tables each time

Examples:

Real time linkage
```py
linker = Linker(df, settings="saved_settings.json", db_api=db_api)
linker.table_management.compute_tf_table("surname")
linker.compare_two_records(record_left, record_right)
```
Pre-computed term frequency tables
```py
linker = Linker(df, db_api)
df_first_name_tf = linker.table_management.compute_tf_table("first_name")
df_first_name_tf.write.parquet("folder/first_name_tf")
>>>
# On subsequent data linking job, read this table rather than recompute
df_first_name_tf = pd.read_parquet("folder/first_name_tf")
df_first_name_tf.createOrReplaceTempView("__splink__df_tf_first_name")
```

Parameters:

Name	Type	Description	Default
column_name	str	The column name in the input table	required

Returns:

Name	Type	Description
SplinkDataFrame	SplinkDataFrame	The resultant table as a splink data frame

invalidate_cache()

Invalidate the Splink cache. Any previously-computed tables will be recomputed. This is useful, for example, if the input data tables have changed.

register_table_input_nodes_concat_with_tf(input_data, overwrite=False)

Register a pre-computed version of the input_nodes_concat_with_tf table that you want to re-use e.g. that you created in a previous run.

This method allows you to register this table in the Splink cache so it will be used rather than Splink computing this table anew.

Parameters:

Name	Type	Description	Default
input_data	AcceptableInputTableType	The data you wish to register. This can be either a dictionary, pandas dataframe, pyarrow table or a spark dataframe.	required
overwrite	bool	Overwrite the table in the underlying database if it exists.	False

Returns:

Name	Type	Description
SplinkDataFrame	SplinkDataFrame	An abstraction representing the table created by the sql pipeline

register_table_predict(input_data, overwrite=False)

Register a pre-computed version of the prediction table for use in Splink.

This method allows you to register a pre-computed prediction table in the Splink cache so it will be used rather than Splink computing the table anew.

Examples:

predict_df = pd.read_parquet("path/to/predict_df.parquet")
predict_as_splinkdataframe = linker.table_management.register_table_predict(predict_df)
clusters = linker.clustering.cluster_pairwise_predictions_at_threshold(
    predict_as_splinkdataframe, threshold_match_probability=0.75
)

Parameters:

Name	Type	Description	Default
input_data	AcceptableInputTableType	The data you wish to register. This can be either a dictionary, pandas dataframe, pyarrow table, or a spark dataframe.	required
overwrite	bool	Overwrite the table in the underlying database if it exists. Defaults to False.	False

Returns:

Name	Type	Description
SplinkDataFrame		An abstraction representing the table created by the SQL pipeline.

register_term_frequency_lookup(input_data, col_name, overwrite=False)

Register a pre-computed term frequency lookup table for a given column.

This method allows you to register a term frequency table in the Splink cache for a specific column. This table will then be used during linkage rather than computing the term frequency table anew from your input data.

Parameters:

Name	Type	Description	Default
input_data	AcceptableInputTableType	The data representing the term frequency table. This can be either a dictionary, pandas dataframe, pyarrow table, or a spark dataframe.	required
col_name	str	The name of the column for which the term frequency lookup table is being registered.	required
overwrite	bool	Overwrite the table in the underlying database if it exists. Defaults to False.	False

Returns:

Name	Type	Description
SplinkDataFrame		An abstraction representing the registered term
		frequency table.

Examples:

tf_table = [
    {"first_name": "theodore", "tf_first_name": 0.012},
    {"first_name": "alfie", "tf_first_name": 0.013},
]
tf_df = pd.DataFrame(tf_table)
linker.table_management.register_term_frequency_lookup(tf_df,
                                                        "first_name")

register_table(input_table, table_name, overwrite=False)

Register a table to your backend database, to be used in one of the splink methods, or simply to allow querying.

Tables can be of type: dictionary, record level dictionary, pandas dataframe, pyarrow table and in the spark case, a spark df.

Examples:

test_dict = {"a": [666,777,888],"b": [4,5,6]}
linker.table_management.register_table(test_dict, "test_dict")
linker.query_sql("select * from test_dict")

Parameters:

Name	Type	Description	Default
input_table	AcceptableInputTableType	The data you wish to register. This can be either a dictionary, pandas dataframe, pyarrow table or a spark dataframe.	required
table_name	str	The name you wish to assign to the table.	required
overwrite	bool	Overwrite the table in the underlying database if it exists	False

Returns:

Name	Type	Description
SplinkDataFrame	SplinkDataFrame	An abstraction representing the table created by the sql pipeline