Deduplicating and standardising a customer master list
Two records — same person, different spelling, a rearranged address, one stale phone number — are judged to be the same entity or not, and the survivor is written in a standard shape. The model scores each candidate pair; a threshold decides which pairs merge automatically and which go to a human queue.
- Effort
- Weeks of work
- Skill level
- Comfortable with software
- Organisation size
- Small business
- Value
- Quality, Time saved
Tools named for this
- A blocking step that narrows candidate pairs before any model is called
- A language model prompted with the two records to make the match decision
- A review queue holding every pair that lands inside the uncertainty band
What to check before you ship it in India
- A wrong merge does not lose a record, it invents a person. Where data is used to make a decision about someone or is disclosed to another fiduciary, section 8(3) puts the duty of completeness, accuracy and consistency on you, and the merge step is precisely where accuracy is destroyed.
- Configuration does not transfer between datasets. The published evaluation found no single best prompt, and that it has to be tuned for each model and dataset pairing, so settings proven on someone else's list are not proven on yours.
Sources
Every claim on this page traces to one of these, on the date it was read.
- The Digital Personal Data Protection Act, 2023 (No. 22 of 2023) — most obligations commence 13 May 2027 under the DPDP Rules 2025 — s.8(3) · Ministry of Electronics and Information Technology · a rule · read 2026-09-01
- Can Foundation Models Wrangle Your Data? · arXiv (Narayan, Chami, Orr, Arora, Ré — Stanford) · how it is done · read 2026-09-01
- Entity Matching using Large Language Models · arXiv (Peeters, Steiner, Bizer — University of Mannheim) · how it is done · read 2026-09-01