← All briefings
Enterprise AIModel evaluationPilot evaluation

China AI Comparisons: Swap the Sides Before Trusting the Score

Replay the same two AI answers in reversed positions. Track whether reviewers keep the same answer or the same side before using their preferences to rank suppliers.

China AI Tour Research5 min read
Conceptual print illustration of three machines processing matching cards into different shapes, with hands comparing outputs through a magnifying glass.
Conceptual print illustration of three machines processing matching cards into different shapes, with hands comparing outputs through a magnifying glass.

A reviewer chooses the answer on the left. When the same two answers change places, the reviewer again chooses the left. The chosen answer has changed even though its content has not. Before using a pairwise preference score to select an AI system, check whether the decision follows the answer or its display position.

For a China enterprise-AI evaluation, use saved responses to approved or synthetic tasks. Replaying fixed text isolates the scoring exercise from fresh model generation. The question is whether the review procedure gives stable preferences for these answers, not whether a second generation happened to produce better text.

Preserve identity while changing position

Assign each task and each saved response a permanent identifier. Keep the underlying model version, prompt and generation settings in a separate record. Show reviewers neutral labels and the same task context, acceptance instructions and reference material.

For each pair, save the exact displayed text and the mapping from response identifier to left or right position. Swapping the positions must not edit the wording, truncate one answer or change the available evidence. If formatting is part of the product being evaluated, define which formatting is retained and apply that rule consistently.

Let reviewers choose left, right, tie, neither acceptable or insufficient evidence. A preferred answer is not necessarily an acceptable answer: both may be wrong. Preserve a separate acceptance verdict when the operating decision requires it.

The model-comparison framework covers task choice and operating fit. This exercise checks the presentation and scoring layer before its preferences become part of that comparison.

Retain both observations

Prepare a second presentation with the two responses reversed. Counterbalance initial positions across the task set, so a particular model does not always appear first or on the left. Randomize task order and document the assignment. Collect judgments independently, without revealing other reviewers' choices or supplier identities.

A repeated presentation to the same person can be affected by recognition, learning or fatigue. Separate the presentations where practical, record their order and ask reviewers to flag recognized pairs. A changed choice is a reason to investigate the review procedure; this small replay alone cannot establish that position caused it. Independent reviewers assigned different presentation orders can provide additional evidence, with differences between reviewers retained rather than hidden.

Keep the original judgments. Do not replace a changed preference with a negotiated final answer before counting it. Adjudication can be recorded in another field after the initial comparison.

Translate side choices back to answer identities

Consider a hypothetical illustration of twelve pairs assessed twice by the same reviewer. To make the table easy to follow, response X is on the left in every first presentation and on the right in every reversed presentation. This simplified layout is for the arithmetic example; the actual evaluation should counterbalance the initial positions as described above. Every judgment in this example chooses one answer, so there are no ties or unresolved cases.

First choiceChoice after reversalAnswer-level interpretationPairs
Left: XRight: XX preferred both times5
Right: YLeft: YY preferred both times3
Left: XLeft: YPreference changed; left chosen twice3
Right: YRight: XPreference changed; right chosen twice1

Eight of twelve pairs retain the same answer-level preference. Four of twelve change. X receives eight preferences in the first presentation and six after reversal. That difference would be invisible if the record retained only the first score, or if the second set of left/right choices were mapped using the first layout.

These twelve pairs are an illustrative fixture, not evidence about any vendor or a population estimate. In a real scorecard, report ties, neither-acceptable judgments, missing reviews and recognized repeats separately. State which pairs enter each denominator. Do not pool every presentation into an apparently larger set of independent tasks.

Find what changed in the judgment

For the changed pairs, ask the reviewer to identify the task requirement and exact passage that supports the preference. Inspect whether one answer was truncated, whether the reference material changed, and whether the reviewer recognized the pair. Preserve these explanations alongside the display mapping.

If the disagreement concerns the acceptance rule, clarify that rule and evaluate a fresh set. The reviewer-agreement guide shows how to retain original decisions while recording an adjudicated interpretation. A clarification does not retroactively make the initial procedure consistent.

NIST AI RMF 1.0 (2023), MEASURE 2.1 calls for documenting test sets, metrics and evaluation tools. For this exercise, include the presentation mapping, scoring options and repeated-pair treatment in that record.

Before accepting a supplier ranking, request answer-level results that survive correct identity mapping and expose changed judgments. The useful deliverable is a reproducible scorecard with its unresolved cases, not a preference percentage detached from how the answers were shown.