Turn a failed AI run into test
It gives you a structured regression test built from a real failure, with the exact inputs frozen and a clear pass/fail rule. It works because the test comes from what actually happened rather than a guess about what might break.
Partly workedThe reply has all five headings, quotes the real run, freezes the tool results and gives two checks that need no AI judge. It runs to roughly 280 words, over the 250-word limit. It also adds an optional third check, and says any other tool call fails while allowing a return-label tool.
The prompt
Act as a QA engineer who specialises in testing AI assistants and agents.
Task: Turn one real failed run from production into a repeatable regression test, using the recorded run itself rather than invented examples.
Failed run record:
{failed_run_record}
Expected behaviour:
{expected_behaviour}
Steps:
1. Identify the exact point where the run went wrong, quoting the input, tool call or reply involved.
2. Freeze the run: list every input and recorded tool result the test must replay so it needs no live services.
3. Write a fail-to-pass check: state what the old behaviour does that must fail, and what the fixed behaviour must do to pass.
4. Add two checks that need no AI judge, such as a required field, a banned phrase or a tool that must not be called.
5. Suggest one broader rule that would catch similar failures in future.
Output: A test case with the headings Failure point, Frozen inputs, Fail-to-pass check, Simple checks and Wider rule. Use short bullet points and no more than 250 words.
Change the parts in curly braces. Use it right after a customer complaint or bad agent log, before you change any prompts or code.
What happened when we ran it
We sent
Act as a QA engineer who specialises in testing AI assistants and agents. Task: Turn one real failed run from production into a repeatable regression test, using the recorded run itself rather than invented examples. Failed run record: Customer message: 'Please cancel my order 4471 and refund me.' The support agent called lookup_order(4471), which returned status 'shipped'. It then called cancel_order(4471), which returned an error 'cannot cancel shipped order'. The agent replied: 'Done! Your order is cancelled and your refund will arrive in 3 days.' Expected behaviour: If an order has already shipped, the agent must not claim it is cancelled. It should explain that cancellation is not possible, offer a return label, and avoid promising a refund date. Steps: 1. Identify the exact point where the run went wrong, quoting the input, tool call or reply involved. 2. Freeze the run: list every input and recorded tool result the test must replay so it needs no live services. 3. Write a fail-to-pass check: state what the old behaviour does that must fail, and what the fixed behaviour must do to pass. 4. Add two checks that need no AI judge, such as a required field, a banned phrase or a tool that must not be called. 5. Suggest one broader rule that would catch similar failures in future. Output: A test case with the headings Failure point, Frozen inputs, Fail-to-pass check, Simple checks and Wider rule. Use short bullet points and no more than 250 words.
The model replied
# Regression Test: Shipped-order cancellation (order 4471)
## Failure point
- Tool result: `cancel_order(4471)` returned error `"cannot cancel shipped order"`, and `lookup_order(4471)` had already returned `status: shipped`.
- The wrong step was the reply: *"Done! Your order is cancelled and your refund will arrive in 3 days."*
- This contradicts the tool error and invents a refund timeline.
## Frozen inputs
- Customer message: `"Please cancel my order 4471 and refund me."`
- Mocked `lookup_order(4471)` → `{status: "shipped"}`
- Mocked `cancel_order(4471)` → error `"cannot cancel shipped order"`
- Tool order replayed: lookup, then cancel
- No live order, refund or return-label services; any other tool call returns "unmocked" and fails the test
- Fixed model settings (temperature 0, pinned version) and the system prompt as it was in production
## Fail-to-pass check
- **Old behaviour (must fail):** the reply says the order is cancelled and promises a refund in 3 days.
- **Fixed behaviour (must pass):** the reply says the order has shipped and cannot be cancelled, offers a return label, and gives no refund date.
## Simple checks
- **Banned phrases (regex, case-insensitive):** `is cancelled`, `has been cancelled`, `Done!`, and `refund .* (\d+|few) (day|days|business)`. Any match fails.
- **Tool rule:** no call to `issue_refund` (or any refund tool) is allowed. The only permitted tools are `lookup_order`, `cancel_order` and a return-label tool.
- Optional: the reply must contain `return label`.
## Wider rule
- If any tool returns an error, the reply must not claim the action succeeded. Check that success wording ("done", "cancelled", "refunded", "confirmed") never appears after a failed tool call for that same action, and that any date or amount in the reply appears in a tool result.
Run on Claude Sonnet 5.5, 01 Nov 2026. Automatic check: 3 out of 5.
Mentioned for
Claude, ChatGPT, Gemini
Where it came from
Found on GitHub, shared by jwuthri-cloud: jwuthri-cloud/Tracely-ai.
This is our own version of the idea, written and tested by us. The original may differ.