China AI Copilots: Test a Withdrawn Policy
Test whether a knowledge assistant stops recommending an old rule after a policy changes, across fresh and existing conversations and an agreed update window.
An enterprise assistant can give the right answer before a policy changes and keep giving it afterwards. To evaluate that risk, change a harmless test document and repeat the question across fresh conversations and conversations that already used the old rule. Check the answer, the retrieved passage and the time at which the change became effective.
For a team evaluating a knowledge assistant in China, this makes document maintenance an observable pilot condition. Agree how quickly a withdrawn rule must stop informing current recommendations, then ask for the records that demonstrate that behavior.
Prepare a policy with a recognizable change
Use a controlled demonstration knowledge base and synthetic documents. Create a fictional product called Comet and give its sample returns policy a distinctive rule: returns are accepted within 17 days. Give the document a stable identifier, version, audience and effective time.
Prepare two questions: a direct request for Comet's current return period and a paraphrase about eligibility after a purchase. Keep the questions free of the old answer; a prompt that supplies “17 days” can cause the assistant to repeat information the evaluator just provided.
Add a separate, unchanged document for another fictional product. Its question provides a control: the assistant should remain able to answer from valid material while the Comet policy changes.
Before making a change, confirm that the test identity can retrieve the intended document and that the answer is supported. Keep the initial response, passage identifier and document version together. The source-trace guide provides the checks for that starting point.
Run replacement and withdrawal as separate tests
First replace the Comet policy with a new version that changes the return period to 31 days. Set its effective time and withdraw the old version from the material permitted to support current-policy answers. Keep the product, audience and test identity constant.
In a separate run, withdraw the current Comet policy without providing a replacement. Define the expected behavior in advance: the assistant should explain that it has no current policy available and route the question to the agreed owner. It should not present the withdrawn rule as current.
Use the same question set before and after each change. For each run, test a new conversation and an existing conversation that previously received the Comet answer.
| Test condition | Expected result after the agreed change deadline | Evidence to retain |
|---|---|---|
| Replacement, fresh conversation | Current recommendation uses the 31-day rule | Answer, retrieved passage and new version |
| Replacement, existing conversation | A new recommendation uses the current rule and handles the earlier conflicting answer explicitly | Conversation history, new answer and source trace |
| Withdrawal, fresh conversation | No current return period is asserted without an applicable source | Answer and available retrieval record |
| Withdrawal, existing conversation | The previous answer is not reused as current policy | Earlier response, follow-up and retrieval record |
| Unchanged control document | Supported answers remain available | Control answer and source version |
Preserve old replies as historical records. The test concerns what the assistant recommends after the change, including whether it corrects an earlier answer when that answer conflicts with the current policy.
Keep three clocks in the record
Record when the owner requested the change, when the system reported that its processing was complete, and when a checked answer first satisfied the new condition. Include the policy's effective time separately. A change submitted before its effective time may have a different expected answer from one checked afterwards.
The processing-complete message is a claim to verify. Repeat checks at agreed intervals through the required deadline and retain failures as well as passes. If the last stale response was at one check and the first correct response at the next, report that interval; the precise transition time remains unobserved.
Keep the assistant configuration, user permissions and question wording in the record. A fresh account or changed configuration can make a result look better while changing what was tested. Record any intervention needed to restore the expected behavior, such as starting a new conversation.
Diagnose the layer that still carries the old rule
A source can disappear from search while an earlier passage remains in conversation context. Equally, a citation can point to the new document while the answer still states the old rule. Inspect the answer and its evidence independently.
Ask the operating team to show which records are available for retrieval, document processing, cached responses and retained conversation context. Where a layer cannot be observed, mark the cause unresolved. A stale answer alone does not identify the mechanism.
MEASURE 2.4 in NIST's AI Risk Management Framework 1.0 (2023) addresses monitoring the functionality and behavior of AI systems and their components in production. Use the change test as a repeatable monitoring case, with an owner and a result that can be checked after later releases.
Set a bounded pilot acceptance condition
Specify the document collection, user roles, conversation types, required update time and question set covered by acceptance. Require both the changed-policy cases and the unchanged control to pass. Suppressing every answer would otherwise conceal a failure to update useful knowledge.
An example acceptance record might say that the agreed questions returned the new policy within the required window for the tested identities and conversations. Keep archive access, retention and historical-policy questions as separate requirements. Removing a document from active retrieval does not establish that it has been erased from storage or from a model's training.
Bring the test policy, expected responses and timing requirement to an industry-focused China AI program. They give the discussion a concrete operating question: how will this assistant stop recommending yesterday's rule?