← All briefings
Enterprise AIPilot evaluationOperating model

China AI Pilots: Separate AI Gains from Extra Staffing

A pilot adds an AI assistant and another operator at the same time. Use a four-condition comparison to identify which improvement the evidence can support.

China AI Tour Research5 min read
Conceptual print illustration of hands passing a red folder across a gap, beside inspection, pause and return symbols.
Conceptual print illustration of hands passing a red folder across a gap, beside inspection, pause and return symbols.

An enterprise AI pilot increases accepted output from 40 to 75 cases per hour. During the same pilot, the operating team grows from two people to three. The combined workflow improved by 87.5%, but that comparison cannot tell the buyer how much improvement came from AI.

For a China enterprise AI evaluation, ask for the missing comparison: what would the same staffing arrangement achieve without the assistant? Keep the task mix and acceptance rule fixed while examining the two changes separately.

Put staffing and AI in separate columns

Consider a hypothetical back-office team checking synthetic product records. A case counts as accepted only after the same independent quality check and completion deadline. Assume comparable task batches, equal observation periods, sufficient incoming work and staff with comparable preparation in every condition. The figures below are invented for this evaluation exercise.

Available staffWithout AI assistanceWith AI assistance
Two people40 accepted cases/hour50 accepted cases/hour
Three people60 accepted cases/hour75 accepted cases/hour

The diagonal comparison is 75 / 40 - 1 = 87.5%. It changes both staffing and AI access. Label it as the observed difference between two whole operating arrangements.

At two people, the AI comparison is 50 - 40 = 10 additional accepted cases per hour, or 25%. At three people, it is 75 - 60 = 15 additional cases, also 25%. The non-AI staffing comparison is 60 - 40 = 20 additional cases per hour, or 50%.

These assumed values illustrate why the original 87.5% cannot be described as the assistant's measured gain. They also show that a percentage effect and an absolute effect need not behave alike. AI adds ten cases at one staffing level and fifteen at the other, even though the relative increase is the same.

Do not add 50% and 25% to reconstruct the diagonal percentage. In this particular table, the relative factors multiply: 1.5 × 1.25 = 1.875. Other measured tables may have a different pattern. Preserve all four results rather than assuming AI and staffing combine in this way.

Make the comparison credible before running it

The table is a plan for the observations needed, not proof that four convenient hourly totals identify a causal effect. A stronger pilot needs repeated observation blocks and a defensible way to allocate work and people.

Use authorized synthetic or appropriately anonymized work. Define the eligible case population, quality rubric, deadline and treatment of unfinished cases before collecting results. Keep difficult cases in the records. Report rejected, late and unresolved work alongside accepted output, so a condition cannot appear faster by quietly relaxing quality or moving work beyond the observation window.

Randomize comparable task batches across conditions where practical. Balance task classes and observation times. Give staff adequate preparation, record experience and prevent one condition from always receiving the strongest team or the quietest hour. Keep software, instructions and access permissions fixed except for the intended AI change.

Check for carryover. Staff who see an AI answer may remember it when handling the same case later without assistance. Reusing those cases can contaminate the comparison. Use distinct comparable batches, and record learning or shared-work effects that remain. Do not assume that simply switching AI off resets the team to its earlier state.

Retain results for each observation block, not just the four averages. Examine variation and have the evaluation owner justify the amount of evidence needed before making a rollout decision. A single demonstration block is useful for checking the workflow and logging, but cannot establish a stable operating gain.

Choose the operating arrangement you can sustain

Record usable staff time as well as headcount. Three names on a rota do not equal three people continuously available to this workflow. Capture interruptions, specialist help, rework and any supplier support supplied during the pilot.

The three-person AI arrangement has the highest assumed hourly output in the table. That alone does not make it the preferred arrangement. Compare required service volume, quality, unresolved workload and the resources each condition consumes. Use the cost per accepted task worksheet for the costing boundary and the human review queue guide for availability and waiting-time constraints.

NIST AI RMF 1.0 (2023), MEASURE 2.1 calls for documenting test sets, metrics and evaluation tools. Apply that documentation principle by retaining the assigned condition, task batch, usable staffing time and acceptance record for every observation block.

Leave the supplier discussion with a precise missing comparison. If the pilot changed both AI access and staffing, identify which same-staffing result is still absent and what controlled observation would supply it.