Prompts / Agents and automation
Find Silent Errors in Agent Traces
You get a structured list of hidden mistakes in an agent run, even when every tool call succeeded. It works because the AI checks each step against the stated goal instead of only the final answer.
The prompt
Act as a careful reviewer of AI agent run logs.
Task: Inspect the agent trace below and find mistakes that did not cause any visible failure, judging each step against what the agent was trying to do.
Agent goal: {agent_goal}
Trace:
{agent_trace}
Steps:
1. State the agent's goal in one sentence, then list each step by number with a few words describing it.
2. Search the steps for ones that look wrong against the goal, such as the wrong item chosen, a tool result ignored, or a claim not backed by the evidence. Read those steps in full.
3. Do not judge by the final answer alone, and ignore anything that is merely odd but harmless.
4. Group findings that share the same cause.
Output: A JSON array. Each item has "step", "error_type", "evidence" (a short quote from the trace), "impact" (low, medium or high) and "suggested_fix". If you find nothing, return an empty array followed by one sentence explaining why.
Change the parts in curly braces. Use it after a support, booking or research agent run that looked successful but produced a customer complaint.
What happened when we ran it
We sent
Act as a careful reviewer of AI agent run logs. Task: Inspect the agent trace below and find mistakes that did not cause any visible failure, judging each step against what the agent was trying to do. Agent goal: Book the cheapest direct flight from Manchester to Edinburgh for Friday 14 March for one passenger, and email the confirmation to the customer. Trace: Step 1 (LLM): Plan: search flights MAN to EDI on 14 March, pick cheapest direct, book, email. Step 2 (tool search_flights): returned 4 results. Direct: BA1420 £92, Loganair LM88 £61. One stop via Dublin: Aer Lingus EI3 £48. Step 3 (LLM): The cheapest option is the Aer Lingus flight at £48, so I will book it. Step 4 (tool book_flight): status 200, booking ref QX81Z for EI3. Step 5 (tool send_email): status 200, sent to customer with ref QX81Z. Step 6 (LLM): I have booked the cheapest direct flight and emailed your confirmation. Steps: 1. State the agent's goal in one sentence, then list each step by number with a few words describing it. 2. Search the steps for ones that look wrong against the goal, such as the wrong item chosen, a tool result ignored, or a claim not backed by the evidence. Read those steps in full. 3. Do not judge by the final answer alone, and ignore anything that is merely odd but harmless. 4. Group findings that share the same cause. Output: A JSON array. Each item has "step", "error_type", "evidence" (a short quote from the trace), "impact" (low, medium or high) and "suggested_fix". If you find nothing, return an empty array followed by one sentence explaining why.
The model replied
Run on Claude Sonnet 5.5, 09 Nov 2026. Automatic check: 3 out of 5.
Mentioned for
Claude, ChatGPT, Gemini
Where it came from
Found on Hacker News, shared by skull8888888: Launch HN: Laminar (YC S24) – RL-trained model that finds errors in agent traces.
This is our own version of the idea, written and tested by us. The original may differ.