ForkFree

Prompts / Agents and automation

Find Silent Errors in Agent Traces

You get a structured list of hidden mistakes in an agent run, even when every tool call succeeded. It works because the AI checks each step against the stated goal instead of only the final answer.

Partly workedIt found the real fault (a one-stop £48 flight booked instead of direct LM88 at £61) and its goal and step list were fine. But step 4 of the task said to group findings that share a cause, and it gave four separate JSON items for one root cause, only grouping them in prose afterwards. It also added extra prose around the JSON array.

The prompt

Act as a careful reviewer of AI agent run logs.

Task: Inspect the agent trace below and find mistakes that did not cause any visible failure, judging each step against what the agent was trying to do.

Agent goal: {agent_goal}

Trace:
{agent_trace}

Steps:
1. State the agent's goal in one sentence, then list each step by number with a few words describing it.
2. Search the steps for ones that look wrong against the goal, such as the wrong item chosen, a tool result ignored, or a claim not backed by the evidence. Read those steps in full.
3. Do not judge by the final answer alone, and ignore anything that is merely odd but harmless.
4. Group findings that share the same cause.

Output: A JSON array. Each item has "step", "error_type", "evidence" (a short quote from the trace), "impact" (low, medium or high) and "suggested_fix". If you find nothing, return an empty array followed by one sentence explaining why.

Change the parts in curly braces. Use it after a support, booking or research agent run that looked successful but produced a customer complaint.

What happened when we ran it

We sent

Act as a careful reviewer of AI agent run logs.

Task: Inspect the agent trace below and find mistakes that did not cause any visible failure, judging each step against what the agent was trying to do.

Agent goal: Book the cheapest direct flight from Manchester to Edinburgh for Friday 14 March for one passenger, and email the confirmation to the customer.

Trace:
Step 1 (LLM): Plan: search flights MAN to EDI on 14 March, pick cheapest direct, book, email. Step 2 (tool search_flights): returned 4 results. Direct: BA1420 £92, Loganair LM88 £61. One stop via Dublin: Aer Lingus EI3 £48. Step 3 (LLM): The cheapest option is the Aer Lingus flight at £48, so I will book it. Step 4 (tool book_flight): status 200, booking ref QX81Z for EI3. Step 5 (tool send_email): status 200, sent to customer with ref QX81Z. Step 6 (LLM): I have booked the cheapest direct flight and emailed your confirmation.

Steps:
1. State the agent's goal in one sentence, then list each step by number with a few words describing it.
2. Search the steps for ones that look wrong against the goal, such as the wrong item chosen, a tool result ignored, or a claim not backed by the evidence. Read those steps in full.
3. Do not judge by the final answer alone, and ignore anything that is merely odd but harmless.
4. Group findings that share the same cause.

Output: A JSON array. Each item has "step", "error_type", "evidence" (a short quote from the trace), "impact" (low, medium or high) and "suggested_fix". If you find nothing, return an empty array followed by one sentence explaining why.

The model replied

**Goal:** Book the cheapest direct MAN→EDI flight on Fri 14 March for one passenger and email the confirmation to the customer. **Steps:** 1. Plan: search, pick cheapest direct, book, email 2. search_flights: 4 results (2 direct, 1 one-stop shown, others not itemised) 3. Choose cheapest option (EI3 £48) 4. book_flight: EI3, ref QX81Z 5. send_email: confirmation sent 6. Final summary claims cheapest direct booked **Findings:** ```json [ { "step": 3, "error_type": "constraint_violation / wrong_item_selected", "evidence": "The cheapest option is the Aer Lingus flight at £48, so I will book it.", "impact": "high", "suggested_fix": "Filter results to direct flights before comparing prices. The correct choice was Loganair LM88 at £61. Add a check that the selected itinerary has zero stops before calling book_flight." }, { "step": 4, "error_type": "wrong_action_from_bad_decision", "evidence": "status 200, booking ref QX81Z for EI3", "impact": "high", "suggested_fix": "Validate the booking against the goal constraints (direct, cheapest direct) before committing. Cancel or rebook the one-stop EI3 and book LM88. A 200 status only shows the call succeeded, not that the right flight was booked." }, { "step": 5, "error_type": "propagated_error", "evidence": "status 200, sent to customer with ref QX81Z", "impact": "medium", "suggested_fix": "Hold the confirmation email until the booking is verified as meeting the requirements. The customer has now been sent a confirmation for a non-direct flight and may rely on it. Send a correction after fixing the booking." }, { "step": 6, "error_type": "unsupported_claim", "evidence": "I have booked the cheapest direct flight and emailed your confirmation.", "impact": "high", "suggested_fix": "Make the final summary reflect the actual tool results (booked EI3, one stop via Dublin, £48). Check claims against the booking details before reporting, and disclose any deviation from the request." } ] ``` **Grouping:** All four findings share one root cause: at step 3 the agent ignored the "direct" constraint and compared prices across all results. Steps 4–6 are downstream effects (booking, email, and a false summary). Fixing step 3 would resolve the rest, though step 6 also shows the agent doesn't verify its claims against evidence. **Minor note:** The search reported 4 results but only 3 were itemised. The agent never examined the fourth, so it could not be sure the cheapest direct flight was LM88. This is a low-impact gap, but worth checking.

Run on Claude Sonnet 5.5, 09 Nov 2026. Automatic check: 3 out of 5.

Mentioned for

Claude, ChatGPT, Gemini

Where it came from

Found on Hacker News, shared by skull8888888: Launch HN: Laminar (YC S24) – RL-trained model that finds errors in agent traces.

This is our own version of the idea, written and tested by us. The original may differ.

More agents and automation prompts

Agents and automation Partly worked
Rewrite Agent Rules With Explicit Exceptions

You get a cleaner set of agent instructions where risky actions have a clear approval gate instead of a vague ban, plus progress updates and a cap on helpers. It works because models tend to treat "never" as absolute, so spelling out the exception avoids both blocked work and unsafe guesses.

Found on Web, shared by Anthropic (official docs)
Agents and automation Worked in our test
Make the AI Finish Every Task

You get every item on a list finished properly, instead of a few done and the rest left as suggestions. It works because it sets the depth of thinking up front, defines "done" as real deliverables, and makes the model count its work before replying.

Found on Web, shared by Moritz Kremb (@moritzkremb)
Agents and automation Worked in our test
Audit Subagents Before Folding Them

You get a verdict for each subagent, backed by evidence and a test, plus a safe order for making changes. It works because it forces a measured decision per subagent instead of a blanket rewrite.

Found on GitHub, shared by anthropics