China Document AI: A Matching Total Can Hide Wrong Line Items
An invoice can add up correctly while item amounts are swapped or a line is missing. Use a row-by-row fixture to test what the total check cannot establish.

An extracted invoice can match the printed total and still contain the wrong line items. During a China document-AI evaluation, ask for a row-by-row reconciliation alongside the arithmetic check. A balanced total is useful evidence of consistency; it does not establish that every item survived extraction correctly.
Use fabricated documents in a host-approved demonstration environment. The exercise below tests whether the exported table preserves the source rows.
Make a balanced error visible
Prepare a simple synthetic invoice with three clearly legible lines. Give each line a unique item code and use one currency. For this fixture, assume there are no taxes, discounts, shipping charges or rounding adjustments.
| Source line | Expected amount | Candidate extracted amount |
|---|---|---|
| Part A | 120 | 120 |
| Part B | 80 | 100 |
| Part C | 100 | 80 |
| Total | 300 | 300 |
The candidate table passes a sum-to-total check: both columns add to 300. Two of its three item amounts are nevertheless wrong. Part B has gained 20 and Part C has lost 20. This is a deliberately constructed test case, not a reported product failure.
Ask the supplier to show both checks independently. One compares the sum of extracted amounts with the extracted or printed total. The other matches each extracted item and amount to the source document. A system that reports only a green total check has not answered the second question.
Match rows before comparing values
Preserve the source document version, page number and row location for each extracted line. Retain the raw text as well as the normalized value. If the application sorts rows, changes a unit label or combines repeated items, the reviewer needs to see that transformation.
For the simple fixture, the unique item code can anchor the match. In operational documents, an item code may occur more than once. Agree how separate occurrences are identified before treating an apparent duplicate as an error. Page and row references can help a reviewer distinguish repeated purchases from the same line extracted twice.
Record unmatched source rows and unmatched output rows explicitly. Do not silently force each output into the nearest-looking source row. Where a source line is ambiguous, a review state is more informative than an invented match. The unreadable-page guide covers that different problem: deciding what to do when the source itself does not support a clear value.
Include failures that arithmetic will miss
Run a small set of controlled variants after establishing a correct baseline. For each fixture, keep its expected source table fixed and document exactly which candidate output changed. For the equal-value omission test, prepare a separate source with Part A at 120, Part B at 80 and Part C at 80, totaling 280. A candidate that omits C and repeats B still has three lines totaling 280, but does not preserve all three source items. Supplying a fabricated output to the reconciliation component tests that component; it does not measure the extraction model's error rate.
| Variant | Why the total check is insufficient | Additional evidence to request |
|---|---|---|
| Swap two different item amounts | The sum stays unchanged | Amount-to-item matches for both lines |
| Omit one line and duplicate a different equal-value line | The sum and row count can both stay unchanged | A one-to-one account of source and output occurrences |
| Return the right amounts under the wrong currency label | Numeric arithmetic may still balance | Document currency and exported currency fields |
| Preserve line totals but change quantity and unit price | Their product can stay unchanged | Separate comparisons of quantity, unit, price and amount |
For the last variant, a synthetic line with quantity 2 at unit price 50 and an output of quantity 1 at unit price 100 both show an amount of 100. Which fields matter depends on the receiving workflow. Agree the required fields before the test rather than judging only the easiest values afterward.
Follow the corrected table into the export
After the reviewer corrects a mismatch, inspect the actual exported file or receiving test record. Confirm that it contains the approved row values and document version. A corrected screen does not prove that a previously generated export was replaced.
Keep separate counts of expected source rows, correctly matched rows, missing rows, extra rows and rows with a required-field mismatch. Report how many complete documents meet the agreed acceptance rule as well. A high proportion of correct fields can coexist with several documents that still require review.
NIST AI RMF 1.0 (2023), MEASURE 2.1 calls for documenting test sets, metrics and evaluation tools. Retain the synthetic source pages, expected row table, candidate outputs and reconciliation results together so another evaluator can reproduce this exercise.
Leave with an explanation of which checks are automated, which mismatches require a person and which approved values reach the receiving system. Matching the total is one check in that evidence, not a substitute for preserving the document's contents.