What is entity resolution?

Entity resolution finds the records that describe the same real thing and joins them into one entity. The entity keeps a link to every record it came from. The thing can be a part, a supplier, a customer or an aircraft. The records can sit in one system or in ten.

The idea goes by many names. A tutorial by Lise Getoor and Ashwin Machanavajjhala defines it as “the problem of extracting, matching and resolving entity mentions in structured and unstructured data”. It lists record linkage, deduplication and reference reconciliation among the names other fields use for it.

Two neighbors are easy to confuse with it. Search finds records that look like a question; entity resolution decides that two records are the same thing. Entity linking connects a mention in text, such as a part number in a ticket, to an entity that already exists. Resolution decides what the entities are in the first place.

A graph answers a question by following edges from one record to the next, and every edge hangs off an identity. Say the parts list writes P-220 rev C and the ticket system writes P220-C. The graph then holds two parts where there is one. A question that starts at the ticket never reaches the supplier notice filed against the part.

Duplicates also break counts. William Winkler of the U.S. Census Bureau notes in his overview of record linkage that duplicates in a list of businesses inflate the estimates for each industry. In a plant the same error makes one supplier look like three small ones, and spreads its open quality issues thin.

What is a knowledge graph? explains why identity matters to a graph. This article is about getting identity right when the systems disagree.

Most entity resolution runs in four steps.

1. Standardize. Bring each field to one form: case, spacing, separators, legal forms such as GmbH, units and dates. P220-C and P-220 rev C become one value once both are written the same way.
2. Block. Compare only the pairs that could match, such as records with the same postcode or the same part family. The Splink documentation gives the reason: “a dataset of 1 million input records would generate around 500 billion pairwise record comparisons.”
3. Compare and score. Check each field of a candidate pair and add up the evidence. Fellegi and Sunter published the classic model for this in 1969, and Winkler’s overview describes it. Each pattern of agreement is weighed by how often it occurs among true matches and among non-matches. Two suppliers in the same city say little. Two with the same tax number say a lot.
4. Decide and merge. Join the pairs that clear a cutoff into one entity, choose its canonical name and fields, and keep a link to each source record.

The cheapest match is a shared key: a vendor number, a serial number, a tax number. When two systems carry the same key for the same thing, the match is exact and needs no score. Start with keys, and only score what they leave unmatched.

Keys fail in their own ways. Each system tends to have its own: the ERP knows vendor 40117, the ticket system only a name. Some keys move. An aircraft’s registration can change when it is sold abroad, while the manufacturer’s serial number stays with the airframe. Choose the key that stays with the thing.

Where no key is shared, fuzzy matching compares what is left: names, addresses, descriptions. It counts how many characters differ, so it absorbs typos and spelling variants. That suits names and does not suit identifiers. Our text search documentation makes the point for ticket numbers: a ticket one character away from T-9182 is a different ticket, not a typo. Part numbers and revisions behave the same way. Standardize an identifier, then match it exactly.

Every cutoff trades two errors. A missed match leaves apart two records that describe one thing, and the graph has a gap: a question about the supplier misses a third of its notices. A false merge joins two different things. That error spreads, because every edge of one now looks like an edge of the other.

The two errors do not cost the same. A missed match costs completeness, and someone who knows the supplier usually notices. A false merge produces confident wrong answers: revision C of a part inherits the deviation raised against revision D, and an assistant cites it. In regulated work, count a false merge as the more expensive error and set the cutoffs for it.

The classic answer is a third zone between the two cutoffs. Winkler’s decision rule for a pair there reads: “designate pair as a possible match and hold for clerical review.” A person decides those pairs, and the rule decides the rest.

• Matching on a field that is not an identity. Two suppliers in different countries can share a name, and two parts can share a description.
• Merges with no way back. A merge that discards the old ids cannot be undone, and an answer can no longer be traced to the record it came from.
• Blocking that hides matches. A block on postcode never compares a supplier with the same supplier at its new address.
• One pass, then never again. New records arrive every day. Resolution that ran once, at load time, drifts away from the sources.
• Merged fields that ignore access. An entity that copies fields from a restricted record can show them to people who may not open that record.

Five questions, asked of a sample of your own records rather than a demo set:

1. Which key identifies each type in each system, and does that key stay with the thing?
2. Check 100 resolved entities by hand. How many are false merges, and how many matches were missed?
3. Where are the cutoffs, and who reviews the pairs between them?
4. Can every entity list the records it was built from, and can a merge be undone?
5. When a new record arrives, how long until it is resolved against the rest?

In Curiosity Studio every node type has a key. The graph model documentation explains that the key decides whether a record updates an existing node or creates a new one. Connectors from more than 70 systems write records by their key. Loading the same record again updates its node instead of adding a second one.

Where a record names an entity in its text, entity linking connects the mention to an existing node with a _Mentions edge. Spotters match a list of canonical names and their aliases. Pattern spotters match identifiers such as ticket numbers. A node can also carry aliases of its own, so it is found under its other names.

When two nodes turn out to be one, code can merge them. The edges move to the node you keep, and the merged node’s key can stay on it as a searchable alias. An administrator can do the same from the data views. Access is synced from the source systems and checked per record at query time. The knowledge graph and data enrichment pages cover the developer side.