Turning scanned registers into datasets
Land records, court files, hospital admission registers, society ledgers. The information exists, it is just locked in images nobody can query. This is the pipeline that gets it out, the places it breaks, and how to know the result is right.
Why an OCR API is not the answer
Every cloud provider sells optical character recognition, so this looks like a solved problem. Run a thirty-year-old bound register through one and you will get a wall of text with no structure, characters confused wherever a photocopy thinned a stroke, and nothing telling you which values to trust.
That is because reading characters is the easy part. The hard parts are reconstructing the table, handling the script, and proving the output matches the source — and a general-purpose API addresses none of them.
The pipeline, in order
1. Split and triage
Every page is scored for skew, resolution, contrast and whether it is a copy of a copy. Pages that will not read reliably are flagged now, not discovered after a full run.
2. Clean the image
De-skew, de-speckle, remove the dark gutter shadow, normalise contrast. On degraded registers this step moves accuracy more than changing the OCR model does.
3. Find the structure before the text
A register is a grid, not prose. Detect the table, then rows, then cells — because a value read correctly into the wrong column is worse than a value not read at all.
4. Read in the right script
OCR tuned for the script on the page, handling vowel signs that wrap the consonant, conjuncts that stack below the baseline, and English numerals mixed into the same line.
5. Parse into typed fields
Raw text becomes dates, numbers, identifiers and names, each validated against an expected format and a plausible range.
6. Reconcile against the source
Row counts per page and per volume checked against the totals written on the documents themselves. A mismatch halts the run instead of shipping quietly.
The order is the point. Reading text before finding the table is the most common mistake in this work, and it produces output that looks plausible and is wrong.
Where it breaks
The third-generation photocopy
A scan of a photocopy of a print. Thin strokes vanish and the dots that distinguish characters fill in. No model recovers information the page no longer contains — this is why sample testing matters before quoting.
Tables that drift across the page
A page scanned at a slight angle turns straight column boundaries into diagonals. Detect the grid per page rather than assuming one layout for a whole volume.
Columns that change mid-volume
Registers maintained over decades change format. A pipeline that assumes one schema produces confidently mislabelled data for the later years.
Handwritten amendments
An entry struck through with a correction written beside it. Machines read both and produce a contradiction. These pages belong in human review, not in the output.
Bound-margin loss
Text disappearing into the spine of a bound register. Often recoverable only by rescanning, and worth identifying before the project rather than after.
Then comes the harder half
Extraction gives you rows. It does not give you a dataset. The same person, village or plot will appear spelled several ways across volumes, and until that is resolved you cannot count anything reliably.
That step — deciding when two records describe the same entity — is where most projects quietly fail. We wrote it up separately in name matching in Indian languages.
Where this applies
| Land and revenue records | Deeds, mutations, pattadar passbooks and jamabandi — usually the hardest, because they combine old scripts, dense tables and handwriting. |
| Court records | Case files, cause lists and judgments where the structure of the record matters as much as the text. |
| Hospital and clinic registers | Admission and OP registers that nobody can search once the volume is closed. |
| Libraries and archives | Manuscripts, catalogues and periodicals, usually destined for full-text search rather than a database. |
| Cooperative and society records | Membership, procurement and payment registers kept by hand for decades. |
| Education records | Admission, examination and transfer-certificate registers held by schools and universities. |
| Corporate and legacy business records | Ledgers and contract files from before the business had any software. |
When records describe people
Many registers contain personal information, so governance is part of the engineering. We work only from a lawful source, process on our own servers in India, extract only the fields the question needs, report aggregates rather than individuals, and delete raw data on an agreed schedule. We decline work whose output would profile identifiable people.
Questions
- How much does digitisation cost?
- Per page, by script and scan quality, quoted after a free sample. The variables that matter are how degraded the scans are, how complex the table is, and whether handwriting is involved — not the page count, which is the cheapest part.
- Should we rescan the originals first?
- Sometimes, and we will tell you when. If the existing scans are 150 dpi copies of copies, rescanning at a higher resolution costs less than the human review that bad scans force. We assess this on the sample.
- What about handwriting?
- Handwriting is a different and harder problem than print, and results vary by hand and era. We assess it on your sample and say plainly whether it is viable, partially viable, or best handled by typing.
- How do we know the output is correct?
- Reconciliation is part of the deliverable: row counts matched to the totals in the source, field-level validation, confidence scores, and a hand-checked sample used to measure real accuracy. You get the numbers, not an assurance.
- What formats can we get?
- CSV, JSON, a database, validated XML to your schema, or searchable PDF/EPUB. Most archive projects want structured data plus a search interface over it.
- Who owns the output?
- You do — the data, the accuracy report and the pipeline code, so next year's volumes can be processed by your own team.
Send your worst pages, not your best
The photocopies nobody wants to deal with tell us what the project really costs. You get the extracted data and a measured accuracy report before committing to anything.