record-linkage

Star

Entity resolution (also known as data matching, data linkage, record linkage, and many other terms) is the task of finding entities in a dataset that refer to the same entity across different data sources (e.g., data files, books, websites, and databases). Entity resolution is necessary when joining different data sets based on entities that may or may not share a common identifier (e.g., database key, URI, National identification number), which may be due to differences in record shape, storage location, or curator style or preference.

Here are 126 public repositories matching this topic...

Language:All

Filter by language

All126 Python57 Jupyter Notebook23 HTML8 R8 Java4 Scala4 C3 C#3 JavaScript3 C++2

Sort:Most stars

Sort options

Most stars Fewest stars Most forks Fewest forks Recently updated Least recently updated

dedupeio /dedupe

Star4.2k

🆔 A python library for accurate and scalable fuzzy matching, record deduplication and entity-resolution.

python clustering dedupe record-linkage python-library entity-resolution datamade dedupe-library de-duplicating

UpdatedNov 25, 2024
Python

openvenues /libpostal

Star4.2k

A C library for parsing/normalizing street addresses around the world. Powered by statistical NLP and open geo data.

c nlp machine-learning natural-language-processing record-linkage address address-parser international deduplication deduping

UpdatedFeb 10, 2025
C

moj-analytical-services /splink

Star1.5k

Fast, accurate and scalable probabilistic data linkage with support for multiple SQL backends

data-science spark record-linkage entity-resolution fuzzy-matching deduplication em-algorithm data-matching deduplicate-data duckdb uk-gov-data-science

UpdatedMar 19, 2025
Python

J535D165 /recordlinkage

Sponsor

Star990

A powerful and modular toolkit for record linkage and duplicate detection in Python

python machine-learning privacy dedupe record-linkage python-library entity-resolution similarity string-distance deduplication data-matching utrecht-university

UpdatedFeb 21, 2024
Python

Yomguithereal /talisman

Sponsor

Star709

Straightforward fuzzy matching, information retrieval and NLP building blocks for JavaScript.

machine-learning natural-language-processing information-retrieval clustering record-linkage fuzzy-matching deduplication

UpdatedJun 23, 2024
JavaScript

dedupeio /csvdedupe

Star419

🆔 Command line tool for deduplicating CSV files

cli dedupe record-linkage csv-files entity-resolution

UpdatedMar 31, 2020
Python

dedupeio /dedupe-examples

Star410

🆔 Examples for using the dedupe library

python dedupe record-linkage entity-resolution

UpdatedAug 10, 2024
Python

J535D165 /data-matching-software

Sponsor

Star378

A list of free data matching and record linkage software.

open-source machine-learning awesome record-linkage entity-resolution fuzzy-matching software awesome-list deduplication data-matching

UpdatedFeb 21, 2024

Bergvca /string_grouper

Star367

Super Fast String Matching in Python

record-linkage fuzzy-matching string-matching string-matching-algorithms

UpdatedMar 14, 2025
Python

maxharlow /csvmatch

Star188

🔎 Finds fuzzy matches between CSV files

csv record-linkage entity-resolution fuzzy-matching data-matching

UpdatedFeb 2, 2025
Python

vintasoftware /entity-embed

Star151

PyTorch library for transforming entities like companies, products, etc. into vectors to support scalable Record Linkage / Entity Resolution using Approximate Nearest Neighbors.

python deep-learning record-linkage entity-resolution pytorch embeddings representation-learning deduplication entity-matching data-matching approximate-nearest-neighbors