Skip to content

Splink: data linkage at scale. (Splink logo).

Fast, accurate and scalable probabilistic data linkage

Splink is a Python package for probabilistic record linkage (entity resolution) that allows you to deduplicate and link records from datasets without unique identifiers.

Get Started with Splink

Key Features

⚡ Speed: Capable of linking a million records on a laptop in approximately one minute.
🎯 Accuracy: Full support for term frequency adjustments and user-defined fuzzy matching logic.
🌐 Scalability: Execute linkage jobs in Python (using DuckDB) or big-data backends like AWS Athena or Spark for 100+ million records.
🎓 Unsupervised Learning: No training data is required, as models can be trained using an unsupervised approach.
📊 Interactive Outputs: Provides a wide range of interactive outputs to help users understand their model and diagnose linkage problems.

Splink's core linkage algorithm is based on Fellegi-Sunter's model of record linkage, with various customizations to improve accuracy.

Consider the following records that lack a unique person identifier:

tables showing what Splink does

Splink predicts which rows link together:

tables showing what Splink does

and clusters these links to produce an estimated person ID:

tables showing what Splink does

Before using Splink, input data should be standardised, with consistent column names and formatting (e.g., lowercased, punctuation cleaned up, etc.).

Splink performs best with input data containing multiple columns that are not highly correlated. For instance, if the entity type is persons, you may have columns for full name, date of birth, and city. If the entity type is companies, you could have columns for name, turnover, sector, and telephone number.

High correlation occurs when the value of a column is highly constrained (predictable) from the value of another column. For example, a 'city' field is almost perfectly correlated with 'postcode'. Gender is highly correlated with 'first name'. Correlation is particularly problematic if all of your input columns are highly correlated.

Splink is not designed for linking a single column containing a 'bag of words'. For example, a table with a single 'company name' column, and no other details.

Support

If after reading the documentatation you still have questions, please feel free to post on our discussion forum.

Use Cases

Here is a list of some of our known users and their use cases:

  • Marie Curie have used Splink to build a single customer view on fundraising data which has been a "huge success [...] the tooling is just so much better. [...] The power of being able to select, plug in, configure and train a tool versus writing code. It's just mind boggling actually." Amongst other benefits, the system is expected to "dramatically reduce manual reporting efforts previously required". See also the blog post here.
  • Club Brugge uses Splink to link football players from different data providers to their own database, simplifying and reducing the need for manual linkage labor.
  • GN Group use Splink to deduplicate large volumes of customer records

Sadly, we don't hear about the majority of our users or what they are working on. If you have a use case and it is not shown here please add it to the list!

Awards

🥈 Civil Service Awards 2023: Best Use of Data, Science, and Technology - Runner up

🥇 Analysis in Government Awards 2022: People's Choice Award - Winner

🥈 Analysis in Government Awards 2022: Innovative Methods - Runner up

🥇 Analysis in Government Awards 2020: Innovative Methods - Winner

🥇 Ministry of Justice Data and Analytical Services Directorate (DASD) Awards 2020: Innovation and Impact - Winner

Citation

If you use Splink in your research, we'd be grateful for a citation as follows:

@article{Linacre_Lindsay_Manassis_Slade_Hepworth_2022,
    title        = {Splink: Free software for probabilistic record linkage at scale.},
    author       = {Linacre, Robin and Lindsay, Sam and Manassis, Theodore and Slade, Zoe and Hepworth, Tom and Kennedy, Ross and Bond, Andrew},
    year         = 2022,
    month        = {Aug.},
    journal      = {International Journal of Population Data Science},
    volume       = 7,
    number       = 3,
    doi          = {10.23889/ijpds.v7i3.1794},
    url          = {https://ijpds.org/article/view/1794},
}

Acknowledgements

We are very grateful to ADR UK (Administrative Data Research UK) for providing the initial funding for this work as part of the Data First project.

We are extremely grateful to professors Katie Harron, James Doidge and Peter Christen for their expert advice and guidance in the development of Splink. We are also very grateful to colleagues at the UK's Office for National Statistics for their expert advice and peer review of this work. Any errors remain our own.