China AI Pilots: Same Score, Different Failed Cases
Two runs can both score 80% while different cases fail. Use a case-by-run matrix to distinguish repeatable acceptance from an unchanged headline score.

An enterprise AI pilot can score 80% twice without handling the same cases successfully twice. Before treating a repeated headline score as evidence of stable behavior, match the results by case identifier and inspect which cases changed outcome.
For a team evaluating a Chinese AI supplier, this is a practical follow-up to a demonstration: rerun the same authorized task set under a recorded configuration and retain every output. The question is whether the workflow can meet its acceptance rule repeatedly, not whether a presenter can find one satisfactory answer.
Repeat the workflow the user will receive
Agree the input set, acceptance rubric and number of repetitions before testing. Use synthetic or appropriately authorized material. Give every case a stable identifier and every execution a separate run identifier. Retain failures, missing responses and timeouts according to a predetermined scoring rule.
Freeze the model version, prompts, tool permissions, retrieval collection and generation settings as far as the evaluation permits. Record dependencies that cannot be fixed, including external search results and live service versions. If those dependencies change, the exercise measures the variability of that operating workflow; it cannot attribute the change solely to the model.
Start each execution from the same intended initial state. For a one-shot document task, that may mean a fresh conversation and a reset test workspace. For a workflow intended to retain memory, specify the starting memory and reset to that state. A second execution that inherits the first answer is not an equivalent repeat.
Use the real retry and selection policy if these are part of the proposed product, and record their resource use. Do not introduce extra attempts only for the cases that disappoint the evaluator. The winning-attempt guide explains how selection changes the result a user receives.
Join the results by case, not just by total
Consider this hypothetical exercise with 100 cases, each executed twice. An evaluator applies the same acceptance rule to all 200 outputs.
| Outcome across the two executions | Cases |
|---|---|
| Acceptable in both | 70 |
| Acceptable only in the first | 10 |
| Acceptable only in the second | 10 |
| Unacceptable in both | 10 |
| Total | 100 |
Each execution passes 80 cases: 70 + 10. Both therefore report 80% success. Across individual outputs, 160 of 200 pass, also 80%.
But only 70 of the 100 cases pass both observed executions. Twenty cases change acceptance outcome, giving an observed outcome-disagreement rate of 20%. Ninety cases pass at least once. That last figure does not describe the result of a single ordinary execution or a workflow that can reliably select the passing answer.
These are different denominators and different questions. Keep all three case-level groups visible: consistently acceptable in this exercise, mixed, and consistently unacceptable. Here, “consistently” refers only to the executions observed; two passes do not establish a guarantee for the next attempt.
Identical pass/fail labels can also conceal different answers. Two unacceptable outputs may fail for different reasons, while two acceptable outputs may use different wording that the task legitimately permits. Record the failed requirement and the business consequence alongside the verdict. Exact text equality is usually a separate requirement from acceptable task completion.
Separate output variation from scoring variation
Conceal run labels from evaluators where practical and apply one written rubric. Where judgment is disputed, use the agreed adjudication process and preserve both the original labels and the resolution. Otherwise, changing reviewers or standards can look like changing system behavior.
Open each mixed-outcome case. Compare inputs, retrieved material, tool responses, final output and any intervention. Mark the cause unresolved when the available trace cannot distinguish generation variation from a changed dependency. Do not assume that repeated calls are statistically independent simply because they have different run identifiers.
NIST AI RMF 1.0 (2023), MEASURE 2.1 calls for documenting test sets, metrics and evaluation tools. For this repeatability exercise, the useful record is a case-by-run matrix connected to the original outputs and scoring decisions.
Set a repeatability rule that fits the decision
Define critical case categories before seeing results. A team might require every observed repetition of a mandatory approval check to pass, while reviewing lower-impact drafting variation through a different criterion. Have the responsible process owner justify those rules and the amount of testing needed. The illustrative two-run exercise is a way to expose disagreement, not a universal sample-size recommendation.
Adding repetitions may reveal failures that were not seen earlier. Report the number of cases and repetitions together; do not compare an all-runs-pass percentage from two repeats directly with one from ten as though the test were unchanged. Repeating one case many times also does not expand coverage to new document types, users or operating conditions.
Use the mixed-outcome cases to investigate and improve the workflow, then preserve separate evaluation material for the next decision. If comparing different model versions, use the update regression ledger as well. A version-to-version comparison and variability within one configuration answer separate questions.
Leave the supplier discussion with the matrix, the unresolved failure categories and an agreed repeat plan. “The score was the same twice” should be the start of that discussion, not its conclusion.