China AI Pilots: When Reviewers Block Different Replies
Two reviewers agree on 92 of 100 replies but would block different messages. Reconstruct their decisions before using the labels to select an assistant.
Two reviewers score the same 100 answers from a customer-service assistant. They agree on 92. Before using their labels to choose a model, ask for the cases each reviewer would block. A high overall agreement rate can coexist with very different decisions about which answers are fit to send.
Give both reviewers the same decision
For a China enterprise AI pilot, use a specific decision such as whether a draft reply can be sent without correcting its proposed remedy. Give both reviewers the same synthetic customer requests, assistant replies, policy version and scoring instructions. Have them score independently before discussing difficult cases. Preserve both original labels.
Record missing evidence separately. If a transcript lacks the facts needed to judge the remedy, give it an unresolved status and report its count. The example below assumes that all 100 replies contain enough information for both people to assign a binary label.
Count the disagreements that matter to the operator
| Independent decisions on the same reply | Reviewer B would send | Reviewer B would block |
|---|---|---|
| Reviewer A would send | 90 | 4 |
| Reviewer A would block | 4 | 2 |
Overall agreement is (90 + 2) / 100 = 92%. Each person would block six replies. Yet only two of those replies appear on both block lists. Across the ten distinct replies flagged by either person, they agree to block 2 / 10 = 20%. Eight replies would receive different treatment depending on who was on duty.
That 20% measures the overlap between the two block lists. It does not tell you which reviewer applied the policy correctly. Both could also overlook the same error in the 90 replies they would send. Preserve a sample from that cell for independent checking alongside the eight disagreements.
Find the sentence that changed the decision
Take a synthetic refund request whose policy allows an agent to offer a credit but reserves approval for a supervisor. The assistant writes, “Your credit has been approved.” Ask each reviewer to identify the sentence in the reply, the policy clause and the decision they made. A difference over whether the promise requires correction points to a specific rule the service owner can clarify.
Have the authorized policy owner resolve the disputed interpretation and record the reason. Keep the adjudicated label in a third column alongside the original two. Replacing both original labels with the agreed result would erase the evidence about how reliably the original instructions could be applied.
Test the revised rule on fresh replies
Version the clarified instructions and give both reviewers a fresh, unseen set. Include ordinary replies as well as examples near the policy boundary. If the team intentionally selects extra difficult cases, report that group separately so its block rate is interpretable. Keep the same four-cell table and report unresolved counts for each group.
MEASURE 2.1 in NIST's AI Risk Management Framework 1.0 (2023) calls for documenting test sets, metrics and evaluation tools. Keep the policy version, scoring sheet and case selection alongside the results. The contact-centre sampling guide covers which interactions enter a review. Bring the independent labels to the next pilot decision so the operations lead can see exactly where a change of reviewer would change a customer's outcome.