Custom Fine-Tuned OCR Model

Overview

This project produced a custom optical character recognition model, trained on a client's own document formats, together with an API that returns the extracted fields as structured data. The client is confidential, so we describe the work in general terms. The goal was to replace manual re-keying of information from documents with something that reads them reliably.

The Challenge

General-purpose OCR does well on clean, standard text and struggles with real business documents: unusual layouts, stamps and signatures crossing text, tables, mixed print quality, skewed scans, and fields whose meaning depends on where they sit on the page. The client's documents were exactly that kind of material. Off-the-shelf output produced text but not dependable, structured data, and each error had to be caught and corrected by a person.

Who Uses It and How

The direct users are the client's developers and systems: an internal application submits a scanned or photographed document to the API and receives named fields back. Behind that are operations staff who used to type values in by hand. They now see a review queue containing only documents where the model was unsure, with the suspect field highlighted, and they confirm or correct it. Those corrections are valuable, because they become new labelled examples for the next training round.

Our Approach

We treated it as a data problem first. We worked with the client to collect representative samples, agree exactly which fields were needed and define what a correct result looks like. Labelled examples were prepared from that set, with held-out documents kept aside so the model was judged on pages it had never seen. Rather than training from scratch, we fine-tuned an existing pretrained architecture, which brings strong general reading ability and lets the client's documents shape the final behaviour.

What We Built

The deliverable has three parts. First, the fine-tuned model trained on the client's document types. Second, the pre- and post-processing around it: cleaning and normalising scans, locating regions of interest, and converting raw recognition output into named fields with consistent formats. Third, a service exposing the whole pipeline over HTTP, so existing systems can submit a document and receive structured results. Validation rules flag uncertain readings for human review, so people spend time only on the difficult documents.

Architecture and Data

A request carries the image or PDF. Pre-processing straightens and cleans the page, then the model detects and reads text regions. Post-processing maps regions to fields using layout and content rules, normalises formats such as dates and amounts, and attaches a confidence signal to each field. Cross-field checks help: a total should agree with its line items, a date should be plausible, an identifier should match its expected pattern. The response is typed JSON. Training data, labels and model versions are kept together so any result can be tied to the model that produced it.

Security, Privacy and Quality

Business documents often contain personal or commercially sensitive details, so the service is designed to process files without keeping them longer than needed, to sit behind authenticated access, and to run within the client's chosen environment. Quality is measured per field, not just per page, since one wrong digit in an amount matters more than a misread footer. We verify that low-confidence cases really do reach the review queue.

Technology

Python is the natural home for machine-learning work, and PyTorch gave us the flexibility to fine-tune and evaluate the model precisely. FastAPI serves the model through a typed, documented interface with efficient request handling, which makes integration straightforward and keeps deployment simple.

Outcome

The client has an extraction capability tuned to its own paperwork instead of a generic tool that approximates it. Documents become structured data through an API call, uncertain results are surfaced for review, and the model can improve as the client supplies more examples.

Frequently asked questions

When is a custom OCR model better than a generic OCR service?

When your documents have unusual layouts, stamps, tables or poor print quality, and you need named fields rather than a block of text. If a generic tool already reads your documents accurately, it is usually the cheaper choice.

How much does it cost to build a fine-tuned OCR model?

The main cost drivers are the number and variety of document types, how many labelled examples you can supply, how much accuracy the business requires, the review workflow, and where the model must be hosted. Data preparation often takes more effort than training.

How much sample data is needed to train an OCR model?

It depends on how varied the documents are and how large the pretrained starting point is. Fine-tuning needs far fewer examples than training from scratch, but the samples must be representative and correctly labelled.