China AI Demos: Who Picks the Winning Attempt?
An acceptable answer among three attempts may never reach the user. Compare first attempts, available candidates and the workflow’s actual selections.
A demonstration shows an excellent answer after the presenter generates three versions and picks one. Before using that result to evaluate an enterprise AI workflow, ask who will choose the answer in ordinary operation and what information that person or system will have.
During a China AI evaluation, retain every attempt for each task. Score the first answer, whether any answer in the allowed set meets the requirement, and the answer the proposed workflow actually selects. Those measures answer different operating questions.
Freeze the generation and selection rules
Choose a permitted task set and agree the acceptance criteria before generating answers. Record the model version, prompt, input, tools, maximum attempts and deadline. Keep a task identifier and a numbered record of each attempt, including failures and timeouts.
For a controlled exercise, generate three candidates per task using fixed settings. Require the selector to choose one before the independent evaluator reveals the reference verdicts. The selector may be a model, a rule or an authorized employee. Give it only the information it would have in the intended workflow.
If a person rewrites the prompt, supplies the correct fact or edits an answer, record that intervention separately. It changes the process being measured. Keep the original candidates so the evaluator can distinguish selecting an existing answer from repairing one.
Compare three results on the same tasks
Consider a hypothetical evaluation of 20 tasks, each with three generated candidates. The independent evaluator finds the following outcomes.
| Task group | Number of tasks |
|---|---|
| The first candidate meets the acceptance rule | 10 |
| The first candidate fails, but a later candidate meets the rule | 5 |
| None of the three candidates meets the rule | 5 |
First-attempt success is 10 / 20 = 50%. At least one acceptable candidate exists for 15 / 20 = 75% of tasks. That second figure uses the evaluator's verdicts after generation. It does not establish that the operating workflow can identify the acceptable candidate.
Suppose the selector chooses an acceptable answer for 12 of those 15 tasks. For the other three, it picks an unacceptable candidate despite having an acceptable one available. The five tasks with no acceptable candidate also fail. The selected-answer success rate is therefore 12 / 20 = 60%, with eight unacceptable selected answers.
The exercise generated 20 × 3 = 60 candidates. Reporting only the 12 accepted selections would conceal most of the generated work. Keep the costs and elapsed time of all attempts and the selection step. The cost-per-accepted-task guide explains how to connect that work to the delivered result.
Investigate selection failures separately
The three tasks with an acceptable candidate available but a bad selection deserve their own review. Did the selector prefer a fluent answer with the wrong fact? Did it miss a required field? Did its ranking rule reward a feature that the acceptance policy did not value?
Preserve the selector's inputs, chosen candidate identifier and any recorded rationale. Avoid giving it reference answers during a replay unless those answers will genuinely be available in operation. Otherwise, the replay tests access to the answer key rather than the intended selection process.
The five tasks with no acceptable candidate need a different response. More careful ranking cannot select an answer that was never generated. A workflow may route those tasks to human assistance, but its ability to recognize that condition must also be tested. The evaluator's knowledge that all three failed is not automatically available to the system.
Compare suppliers under the same attempt budget
Use the same eligible tasks, acceptance policy, generation budget and deadline for the proposed system and the baseline. Report first-attempt and final selected-answer outcomes together. If one workflow uses a different budget, show that as a separate operating configuration with its own resource and time requirements.
A production process that stops early needs an observable stopping rule. Re-run that actual rule rather than assuming the system knows when it has produced a correct answer. Likewise, three attempts generated in parallel and three generated sequentially can have different elapsed times even when both consume three calls.
NIST AI RMF 1.0 (2023), MEASURE 2.1 calls for documenting test sets, metrics and evaluation tools. For this exercise, include the selector and attempt budget in that record. Leave the demonstration with a reproducible answer to one question: which result would the ordinary user receive, after how much work, without access to the evaluator's verdicts?