China Robot Demos: What Zero Failures Can Tell You

A robot completes 30 demonstration tasks without a failure. The result is encouraging, but does it support a pilot requirement of less than 1% task failure? Under a simple fixed-sample model, it does not: the one-sided 95% upper confidence limit after 30 failure-free trials is about 9.5%.

When evaluating an AI-enabled robot in China, bring the proposed failure-rate requirement and the number of observed trials into the same discussion. Zero observed failures is a count. The strength of the inference depends on how the trials were chosen and how many there were.

Define the task and the sampling plan

Choose a binary outcome before collecting results. For a synthetic picking evaluation, success might mean one unassisted first attempt places the specified item at its destination within an agreed time. A retry or human correction would fail that first-attempt definition even if the request is eventually completed.

Freeze the robot configuration, task definition and planned number of trials. Record every scheduled result, including unsuccessful or unresolved runs, with the conditions and any exclusions. An unresolved outcome needs investigation; it cannot silently become a success.

The calculation below assumes independent trials drawn from the same target workload, with a constant failure probability for that configuration. Use permitted test records or an agreed controlled evaluation. A demonstration selected for presentation does not automatically meet that sampling model.

Why 30 successes can coexist with a 1% failure rate

For this hypothetical calculation, suppose each independent task has a 1% failure probability. Its success probability is 0.99. The probability of seeing no failures in 30 tasks is:

0.99^30 = 0.7397, or approximately 73.97%.

That means a system with this assumed failure rate would often produce a flawless 30-task sample. The observation alone offers little basis for concluding that its failure rate is below 1%.

This example describes an assumed statistical model. It is not a measured result for a supplier, customer or robot platform.

Calculate the zero-failure upper limit

With zero failures in a fixed sample of n independent, identically distributed binary trials, the exact one-sided 95% upper confidence limit for failure probability is:

Upper limit = 1 − 0.05^(1/n).

The expression comes from solving (1 − upper limit)^n = 0.05. At that candidate failure probability, obtaining a failure-free sample of this size has a 5% probability.

Planned trials, all with zero failures	Upper failure-probability limit
30	9.503%
100	2.951%
298	1.00024%
299	0.99691%

For an illustrative requirement that this upper limit be below 1%, the smallest whole-number sample is 299 trials with zero failures. The unrounded sample-size threshold is ln(0.05) / ln(0.99) = 298.073. Rounding the 298-trial limit to “1.00%” would conceal that it remains above the target.

The confidence level refers to a repeated-sampling procedure whose upper limits cover the true failure probability at least 95% of the time under the stated model. The displayed expression is its zero-failure case. It does not promise a failure-free next shift or assign a 95% probability to a particular fixed failure rate being below the observed limit.

Check whether more repetitions add independent evidence

Three hundred cycles on the same easy object can leave important operating variation untested. Ask how packages, positions, lighting, replenishment and other relevant conditions enter the sample. If the intended workload contains several task types, record the sampling method and the results by type as well as any overall result.

Trials may also share conditions that make their outcomes dependent. A sequence from one prepared batch, a robot that adapts during the evaluation, or repeated attempts that benefit from an earlier correction can violate the simple model. Increasing the displayed count does not repair that assumption.

Ask the evaluation owner to document how the chosen method handles those dependencies. Where the trial model is unsupported, report the observed counts and the unresolved uncertainty rather than applying this bound automatically.

The MEASURE section of NIST's AI Risk Management Framework 1.0 (2023) calls for performance assessment with associated measures of uncertainty. Here, preserve the trial definition and sampling assumptions alongside the calculated limit so another reviewer can judge what it supports.

Agree the decision rule before observing the final sample

Have the process owner choose the task-failure threshold, confidence level, relevant conditions and response to a failed trial before the final evaluation. The 1% target above is an example for the calculation, not a requirement for every application.

Use a fixed final sample plan. Do not tune the configuration on the evaluation cases and retain only the successful reruns. If the configuration changes, preserve the earlier record and define the evaluation for the revised version. If failures occur, use a method that includes their actual count; the zero-failure formula no longer applies to that sample.

Keep the acceptance question separate from output and staffing questions. A low observed task-failure rate does not state how many good units a cell delivers per scheduled hour. Use the robot-cell output guide for that calculation and the dataset-boundary guide to inspect selection and exclusions.

Bring a proposed sampling plan and acceptance rule to an industry-focused China AI program. They make a flawless demonstration the starting point for a test your team can interpret.