How to Benchmark a Bank Statement Converter — Accurate Bank Statement Converter

The best bank statement converter is the one that performs on your banks, and no published percentage can tell you that. This is a method for measuring it yourself in an afternoon: how to choose a sample, what to score, how to compute a defensible number, and what a vendor's own benchmark must disclose to be worth reading.

Key takeaways

An accuracy figure without a stated method is unfalsifiable. Method first, number second.
Sample design decides the result. An easy sample produces a high number that predicts nothing.
Score cells, not documents — a "99% accurate" document can still be unusable.
Balance reconciliation is the cheapest high-value metric and should be reported separately.

Why can't you just compare vendors' accuracy claims?

Because the same output can honestly be described as 99% accurate or 62% accurate depending on what you count.

Consider a statement with 40 transactions and 5 fields each — 200 cells. Suppose extraction gets every date, description and balance right, but misassigns debit versus credit on 15 rows.

Cell accuracy: 185/200 = 92.5%
Row accuracy (a row is correct only if all fields are): 25/40 = 62.5%
Document accuracy (a document is correct only if every row is): 0/1 = 0%

Every one of those is a defensible number computed from the same output. Without knowing which was used, the percentage carries no information. Add the choice of which banks were sampled, whether inputs were digital or scanned, and who selected the files, and two vendors' figures become entirely incomparable.

This is why the useful question is not "whose number is higher" but "what happens on my files".

Designing a sample that predicts anything

Sample design is where most informal comparisons go wrong. People test the statement they have handy, which is usually a clean single-page digital PDF that every tool handles.

Cover your actual banks. Weight the sample by the banks your clients actually use. Three statements from each of your top four banks is worth more than twenty from banks you never see.

Include the hard cases deliberately. A sample without these will overstate every tool:

Case	Why it matters
Multi-page statement (4+ pages)	Continuity, repeated headers, carried-forward lines
Wrapped descriptions	Geometrically identical to a row with no amount
Section-based debit/credit	Direction carried by heading, not by row
Refunds and reversals	Sign inversions
Scanned or photographed	Adds the OCR error class
Foreign currency or FX rows	Extra columns, different formatting
Month with very high volume	Density stresses row segmentation

Aim for 10–20 statements. Below ten, one bad file dominates. Above twenty, manual ground-truth becomes a bigger job than the decision warrants.

Fix the sample before you test. Choose the files first, then run the tools. Choosing after you have seen results is how benchmarks become marketing.

Building ground truth

This is the tedious part and there is no shortcut. For each statement, key the correct values into a spreadsheet: date, description, debit, credit, balance, in document order.

Three practices make it reliable:

Transcribe from the PDF, not from any tool's output. Correcting a tool's output biases ground truth toward that tool.
Reconcile your ground truth. Apply your transcribed transactions to the opening balance and check against the printed closing balance. If your own transcription does not reconcile, fix it before testing anything.
Normalise formatting decisions up front. Date format, whether to keep or strip a trailing reference number in descriptions, how to represent an empty cell. Write these rules down, because you will apply them dozens of times.

For a 15-statement sample this is a few hours. It is also reusable — the same ground truth tests every future tool and every future version.

What to measure

Score five things. Each answers a different question.

1. Cell accuracy. Correct cells ÷ total cells. The headline number, and the one to state alongside any claim.

2. Row accuracy. Rows where every field is correct ÷ total rows. Always lower than cell accuracy, and closer to what using the output feels like.

3. Transaction count error. Extracted rows minus true rows, per statement. Catches merged and split rows, which cell accuracy can hide.

4. Reconciliation pass rate. Proportion of statements whose extracted transactions reproduce the printed closing balance. The cheapest metric to compute and arguably the most decision-relevant: it is the check you would run in production anyway.

5. Cleanup time. Minutes to make each output importable. This converts accuracy into money. A tool that is two points less accurate but needs no manual fixing may be the better buy at your volume.

Report per-bank as well as overall. An 85% average can hide 98% on three banks and 45% on a fourth — and if that fourth is your largest client's bank, the average actively misled you.

Comparing description text fairly

Descriptions rarely match character for character, and scoring them as binary right/wrong is unfair to every tool.

Pick one rule and apply it to all candidates:

Normalised exact match — lowercase, collapse whitespace, strip punctuation, then compare. Strict but consistent.
Token overlap — proportion of shared words. More forgiving of truncation.
Substring containment — correct if the true description contains the extracted one or vice versa. Most forgiving; useful when tools truncate differently.

State which you used. A description score means nothing without it.

What a credible published benchmark must disclose

If you are reading a vendor's benchmark rather than running your own, these are the disclosures that make it checkable:

Which metric — cell, row or document accuracy, defined explicitly.
Sample composition — how many statements, from which banks, in what proportion.
Digital versus scanned split, since these perform very differently.
How ground truth was produced and by whom.
The description matching rule.
Failure cases, named. A benchmark with no reported failures was not measuring hard enough.
Date and version, since both statement formats and tools change.
Reproducibility — enough detail that someone else could run it.

A benchmark missing items 1, 2 or 6 is a marketing artefact. That is not necessarily dishonest, but it is not evidence.

Our position on this

We have not published a benchmark meeting the standard above, so treat our accuracy as unproven and test it yourself. Signup includes 50 free credits at one credit per page, which covers a 15-statement sample comfortably — enough to run this method end to end before spending anything.

The buyer's checklist covers the non-accuracy criteria that also matter: format coverage, output schema, batch handling and data retention. And whatever you conclude, the reconciliation check in step 4 is worth adopting permanently, because it is the difference between output you trust and output you have verified.

Run your sample through it.

Frequently asked questions

What accuracy should I expect from a bank statement converter?

There is no meaningful general figure, because results vary enormously between clean digital statements and poor scans, between simple and complex layouts, and between banks. Any single number quoted without stating the metric, the sample and the input type is not comparable to any other number.

How many statements do I need to test a converter properly?

Ten to twenty, weighted toward the banks you actually handle and deliberately including hard cases: multi-page files, wrapped descriptions, refunds, and at least one scan. Fewer than ten lets a single unusual file dominate; more than twenty makes building ground truth disproportionate to the decision.

What is the difference between cell accuracy and row accuracy?

Cell accuracy counts individual correct fields; row accuracy counts rows where every field is correct. Row accuracy is always lower and better reflects usability, since a row with one wrong field still needs manual correction. Quoting cell accuracy alone makes any tool look considerably better than it feels to use.

Is reconciliation enough on its own to test a converter?

It is the strongest single check and catches most amount and sign errors, but it is not complete. A statement can reconcile while descriptions are truncated or dates are misformatted, since neither affects the arithmetic. Use reconciliation as the primary gate and check descriptions and dates separately.

How often should a benchmark be repeated?

At least annually, and whenever a bank you rely on changes its statement layout or a tool ships a significant update. Statement templates change periodically, and a benchmark result attaches to a specific combination of tool version and format generation rather than being permanent.