← All briefings
Enterprise AIDocument AIPilot evaluation

China Document AI: A Matching Total Can Hide Wrong Line Items

An invoice can add up correctly while item amounts are swapped or a line is missing. Use a row-by-row fixture to test what the total check cannot establish.

China AI Tour Research5 min read
Conceptual close-up of a magnifying glass over a paper table, with a pencil and an exclamation-mark card.
Conceptual close-up of a magnifying glass over a paper table, with a pencil and an exclamation-mark card.

An extracted invoice can match the printed total and still contain the wrong line items. During a China document-AI evaluation, ask for a row-by-row reconciliation alongside the arithmetic check. A balanced total is useful evidence of consistency; it does not establish that every item survived extraction correctly.

Use fabricated documents in a host-approved demonstration environment. The exercise below tests whether the exported table preserves the source rows.

Make a balanced error visible

Prepare a simple synthetic invoice with three clearly legible lines. Give each line a unique item code and use one currency. For this fixture, assume there are no taxes, discounts, shipping charges or rounding adjustments.

Source lineExpected amountCandidate extracted amount
Part A120120
Part B80100
Part C10080
Total300300

The candidate table passes a sum-to-total check: both columns add to 300. Two of its three item amounts are nevertheless wrong. Part B has gained 20 and Part C has lost 20. This is a deliberately constructed test case, not a reported product failure.

Ask the supplier to show both checks independently. One compares the sum of extracted amounts with the extracted or printed total. The other matches each extracted item and amount to the source document. A system that reports only a green total check has not answered the second question.

Match rows before comparing values

Preserve the source document version, page number and row location for each extracted line. Retain the raw text as well as the normalized value. If the application sorts rows, changes a unit label or combines repeated items, the reviewer needs to see that transformation.

For the simple fixture, the unique item code can anchor the match. In operational documents, an item code may occur more than once. Agree how separate occurrences are identified before treating an apparent duplicate as an error. Page and row references can help a reviewer distinguish repeated purchases from the same line extracted twice.

Record unmatched source rows and unmatched output rows explicitly. Do not silently force each output into the nearest-looking source row. Where a source line is ambiguous, a review state is more informative than an invented match. The unreadable-page guide covers that different problem: deciding what to do when the source itself does not support a clear value.

Include failures that arithmetic will miss

Run a small set of controlled variants after establishing a correct baseline. For each fixture, keep its expected source table fixed and document exactly which candidate output changed. For the equal-value omission test, prepare a separate source with Part A at 120, Part B at 80 and Part C at 80, totaling 280. A candidate that omits C and repeats B still has three lines totaling 280, but does not preserve all three source items. Supplying a fabricated output to the reconciliation component tests that component; it does not measure the extraction model's error rate.

VariantWhy the total check is insufficientAdditional evidence to request
Swap two different item amountsThe sum stays unchangedAmount-to-item matches for both lines
Omit one line and duplicate a different equal-value lineThe sum and row count can both stay unchangedA one-to-one account of source and output occurrences
Return the right amounts under the wrong currency labelNumeric arithmetic may still balanceDocument currency and exported currency fields
Preserve line totals but change quantity and unit priceTheir product can stay unchangedSeparate comparisons of quantity, unit, price and amount

For the last variant, a synthetic line with quantity 2 at unit price 50 and an output of quantity 1 at unit price 100 both show an amount of 100. Which fields matter depends on the receiving workflow. Agree the required fields before the test rather than judging only the easiest values afterward.

Follow the corrected table into the export

After the reviewer corrects a mismatch, inspect the actual exported file or receiving test record. Confirm that it contains the approved row values and document version. A corrected screen does not prove that a previously generated export was replaced.

Keep separate counts of expected source rows, correctly matched rows, missing rows, extra rows and rows with a required-field mismatch. Report how many complete documents meet the agreed acceptance rule as well. A high proportion of correct fields can coexist with several documents that still require review.

NIST AI RMF 1.0 (2023), MEASURE 2.1 calls for documenting test sets, metrics and evaluation tools. Retain the synthetic source pages, expected row table, candidate outputs and reconciliation results together so another evaluator can reproduce this exercise.

Leave with an explanation of which checks are automated, which mismatches require a person and which approved values reach the receiving system. Matching the total is one check in that evidence, not a substitute for preserving the document's contents.