Name matching in Indian languages
Deciding whether two records describe the same person is the hardest step in almost every Indian data project. It is also the step most teams assume a library will handle, which is why so many datasets quietly contain the same person four times.
One person, four records
Take a man from a village in Andhra Pradesh. His bank account says K. Srinivasa Rao. His land record says Kondapalli Srinivasa Rao. The scanned register has his name in Telugu. A scheme database has Sreenivas Rao Kondapalli, S/o Ramaiah.
Four records, one man, and no two strings are equal. Any system that joins on name will treat him as four people — or, worse, will merge him with his neighbour who is also a Srinivasa Rao.
Why Indian names vary so much
Transliteration is not standardised
A Telugu or Tamil name written in English has no single correct spelling. Reddy and Reddi. Lakshmi, Laxmi and Lakshmy. Srinivasa, Srinivas and Sreenivasa. The same clerk will spell it two ways in one afternoon, and both are right.
Initials carry the real information
In much of South India the surname, village or father's name is an initial. K. Srinivasa Rao, Srinivasa Rao Kondapalli and Kondapalli Srinivasa Rao are one person. A matcher that treats the initial as a middle name gets all three wrong.
Name order is not fixed
Surname-first and surname-last both appear, sometimes in the same dataset, depending on which clerk filled which form. Position-based matching breaks immediately.
Patronymics instead of family names
Many records use the father's name as the second element. It changes between generations, and for married women it may change between documents, so it is not a stable key.
Honorifics and titles leak in
Sri, Smt, Dr, Md, Late, S/o, W/o and D/o get typed into the name field. Strip them and matching improves immediately; leave them and identical people look different.
Script mixing in one record
Name in Telugu, father's name transliterated into English, address in a third form. You cannot compare strings until you have brought them into one representation.
OCR noise sits on top
When the source is a scanned register, the variation above is compounded by recognition errors. You are matching noisy versions of already-inconsistent names.
Why standard tools fail here
Soundex and Metaphone encode English phonetics
They were built for English surnames. Indic retroflex and aspirated consonant distinctions — the difference between t and th, d and dh — are exactly where Indian name variants differ, and English phonetic algorithms collapse or ignore them.
Edit distance treats all errors equally
Plain Levenshtein says Reddy to Reddi costs the same as Reddy to Teddy. Linguistically the first is the same name and the second is a different one, so a flat distance threshold either merges too much or too little.
Position-based comparison assumes fixed order
Most libraries compare first name to first name. With surname-first and surname-last mixed in one file, that comparison is meaningless.
Everything-to-everything does not scale
A lakh of records is five billion comparisons. Without blocking — only comparing records that share a cheap signal — the job never finishes.
How to resolve it properly
1. Normalise
Unicode NFC normalisation, strip honorifics and relationship markers, collapse punctuation and whitespace, standardise case. This alone resolves a surprising share of apparent mismatches.
2. Bring scripts together
Transliterate everything into one comparison space using a documented scheme rather than ad-hoc rules, so a Telugu name and its English spelling become comparable while the original is kept for display.
3. Treat the name as a bag of tokens
Split into parts and compare sets rather than positions, so surname-first and surname-last match. Expand initials against full tokens instead of comparing them as letters.
4. Use script-aware similarity
Weight substitutions by how plausible they are in Indic transliteration — i and y, v and w, retroflex and dental pairs cost little; unrelated consonants cost a lot.
5. Block before you compare
Group candidates by a cheap key — a phonetic code, a sorted token prefix, or an attribute like house number or date of birth — then compare only inside each group. This is what makes lakhs of records tractable.
6. Score, do not decide
Produce a confidence score per pair. Auto-merge above a high threshold, auto-reject below a low one, and route the band between them to a human. The middle band is where all the real errors live.
7. Use the other fields
Age, house number, relative's name and locality disambiguate far better than the name alone. Two identical names in one household are usually two people; two spellings at the same address are usually one.
The decision that matters most
Every matching system has a dial between merging too eagerly and splitting too eagerly, and you cannot have both. The right setting depends entirely on what happens next.
If a merge means two people share a bank account or a medical history, set it conservatively and accept more duplicates for review. If a split means paying a subsidy twice, lean the other way. Any vendor who does not ask you this question before building has not thought about your problem.
Where this is needed
| Banks and NBFCs | Customer deduplication across branches and legacy systems, and KYC matching where the same person appears differently in each document. |
| Land records | Linking a chain of owners across deeds, mutations and revenue records written decades apart in different scripts. |
| Hospitals | Preventing a returning patient from getting a second medical record number and a split clinical history. |
| Government schemes | Finding genuine duplicates in beneficiary lists without wrongly removing two different people who share a name. |
| Cooperatives and FPOs | One farmer record across procurement, payments and membership registers. |
| Libraries and archives | Author and subject authority control across catalogues in several scripts. |
Questions
- Why not just use a fuzzy matching library?
- Libraries like fuzzywuzzy or recordlinkage are useful building blocks, and we use them. What they do not give you is Indic transliteration handling, initial expansion, order-independent comparison and sensible blocking. Those are the parts that decide whether the output is trustworthy, and they have to be built for the dataset.
- What accuracy can we expect?
- It depends on how much supporting data each record carries. Name alone is weak. Name plus age plus address is strong. We measure precision and recall on a hand-checked sample from your data and report both, rather than quoting one number — because the two failure modes have very different costs.
- Which error is worse, a false merge or a false split?
- Almost always the false merge, because it silently combines two real people into one record and is very hard to detect later. We tune thresholds to prefer leaving a duplicate for review over merging two strangers, and we say so explicitly in the report.
- Can this run on our own servers?
- Yes. The pipeline is Python and can run entirely inside your infrastructure, which is usually the requirement for banks and government departments.
- How long does a project take?
- A sample and accuracy report in about a week. A full run on a few lakh records is typically days once the rules are tuned, because the compute is cheap and the thinking is the expensive part.
Send us a few thousand messy records
We will run matching on a sample of your own data and send back the duplicate pairs we found, the ones we were unsure about, and measured precision and recall. Then you decide whether it is worth doing properly.