China Robot Pilots: Compare the Same Mix of Stations

A robot supplier shows two picking configurations. One completed 85 of 100 tasks, the other 55 of 100. Before choosing either for a second site, ask how many tasks each ran at each station. The station mix alone can reverse the ranking.

Rebuild the totals from the station logs

Consider a hypothetical pilot with two stations. At the first, items arrive aligned in a fixture. At the second, the robot picks loosely packed items from a bin. Both configurations face the same success rule within each station. A first attempt must deliver the specified item to the destination within the agreed time without assistance. Each planned task has a recorded result.

Station	Configuration A	Configuration B
Fixture	81 of 90 (90%)	19 of 20 (95%)
Loose bin	4 of 10 (40%)	36 of 80 (45%)
All observed tasks	85 of 100 (85%)	55 of 100 (55%)

B finished five percentage points ahead at each station. Its lower pooled score comes from running 80% of its tasks at the harder station. A ran only 10% there. The two headline percentages accurately describe the demonstrations. They answer different questions about the workload.

Give both configurations the receiving site's mix

Suppose the receiving site's order records show that half of its eligible picks go through each station. Apply those same weights to both configurations. A's estimated completion rate becomes 0.5 × 90% + 0.5 × 40% = 65%. B's becomes 0.5 × 95% + 0.5 × 45% = 70%. Retain the original counts beside these estimates. Weighting ten bin trials more heavily does not create more observations.

Ask the site owner for counts of eligible tasks by station over a representative period. Use incoming tasks to estimate the mix. Counting only successful picks would give difficult work less weight precisely because more attempts failed. Keep retries linked to their original task. If a site will add a new gripper or a third station, mark the corresponding workload as untested until the team obtains comparable evidence.

Decide what the difference warrants testing

In these assumed figures B leads within both stations, so any common blend of the two observed station rates preserves its five-point lead. That arithmetic does not establish a five-point advantage on the next shift. A's loose-bin result comes from ten tasks. Item shape, operator help, run order or a configuration change could also differ within a station.

For a follow-up trial, freeze the two configurations, choose representative task classes at both stations, and assign comparable items to each configuration within each class. Alternate or randomize run order where practical. Record failed attempts, timeouts and interventions alongside successes. Agree how to handle unresolved outcomes before collecting results. Review uncertainty within each station before using its observed rate in a purchasing decision.

MEASURE 2.3 in NIST's AI Risk Management Framework 1.0 (2023) calls for documented performance measures under conditions similar to deployment. For a China robotics evaluation, bring the receiving site's workload split to the demonstration and ask the host to reconstruct one published or presented total from its station counts. Use the dataset audit guide to check which tasks entered the test. Leave with the number of additional trials needed at the least-tested station recorded as an open design question.