China AI Pilots: Did Accuracy Rise Because the System Answered Less?
Compare accuracy with the share of tasks answered automatically. A worked threshold example shows which errors disappear and how much work moves to human review.
An enterprise AI supplier raises its reported accuracy from 90% to 95%. Before treating that as an improvement in the work your team can automate, ask how many requests the system answered at each setting. A system can become more accurate on the answers it releases while sending substantially more work to people.
During a China AI evaluation, request three counts from the same set of tasks: correct automatic answers, incorrect automatic answers and tasks withheld from automatic delivery. Keep all three visible when comparing the supplier's settings.
Hold the task set and candidate answers fixed
Use an authorized evaluation set with reference answers established independently of the system's confidence scores. For a document assistant, the unit might be one requested field from one document. Specify how partial answers, missing fields and unreadable inputs are classified before scoring. A ten-field document should not silently become one task in one report and ten in another.
For a controlled threshold comparison, retain each candidate answer and its score, then apply two release thresholds to those same saved outputs. The stricter setting releases a subset of the answers released by the looser setting. This isolates the effect of withholding answers. If the supplier changes the model, prompt or retrieval at the same time, evaluate that configuration separately.
An abstention means the system withholds an automatic answer under the agreed rule. Track technical failures separately in the actual ledger. The example below assumes no technical failures and routes every withheld task to human review.
Read accuracy alongside coverage
Suppose the evaluation contains 100 tasks. These are assumed counts for a worked example.
| Outcome on the same 100 tasks | Looser release threshold | Stricter release threshold |
|---|---|---|
| Correct automatic answers | 81 | 57 |
| Incorrect automatic answers | 9 | 3 |
| Tasks withheld for review | 10 | 40 |
| Automatic answers released | 90 | 60 |
| Accuracy among released answers | 81 / 90 = 90% | 57 / 60 = 95% |
| Coverage, or fraction answered automatically | 90 / 100 = 90% | 60 / 100 = 60% |
The stricter setting removes 30 answers from automatic delivery. Of those, 24 were correct and six were wrong. It leaves three wrong answers in automatic delivery. The higher reported accuracy therefore accompanies fewer correct automatic answers and more review work.
That trade-off can be worthwhile when an incorrect answer has a high consequence. It still needs an operating decision. If each additional review takes an assumed five active minutes, the extra 30 reviews require 150 reviewer-minutes. Record actual review durations and outcomes during a pilot. Routing a task to a person does not establish that it was resolved correctly or within the required time. The human review queue guide explains how to assess that capacity.
Inspect the cases that changed route
Ask for the IDs of the 30 tasks withheld only by the stricter setting. Group them by the business consequences of a wrong answer and by the work required to resolve them. A mistaken delivery address and a harmless formatting error should not receive the same importance merely because each counts as one error.
Keep the original ten withheld tasks visible too. Otherwise, the comparison understates the total review load. Preserve the three remaining automatic errors and their severity. A setting that removes many minor mistakes can still release the error your workflow most needs to prevent.
Require the supplier to define its confidence score. A displayed score of 0.95 should only be interpreted as a 95% probability of correctness if that interpretation has been supported by appropriate calibration evidence for the relevant task population. For this threshold exercise, the score is simply the value used to decide which saved answers are released.
Agree a release rule before the final evaluation
Choose settings using development data, then freeze the rule and evaluate it on a separately held task set. Report the task count, coverage, conditional accuracy, incorrect automatic answers by consequence, technical failures and unresolved review cases. Preserve task-level records so the team can reconstruct every denominator.
NIST AI RMF 1.0 (2023), MEASURE 2.1 calls for documenting test sets, metrics and evaluation tools. Use that documentation principle to request both the accuracy calculation and the work the system withheld. Agree acceptable automatic-error consequences and available review capacity with the people who own the workflow before selecting a threshold.