Prompts / Research and analysis
Fact-Check an AI Answer Against Sources
You get a claim-by-claim audit of an AI answer, showing exactly which statements are backed by your source text and which were made up. It works because the model must quote evidence for every claim and is barred from using outside knowledge.
Worked in our testIt split the answer into numbered claims, labelled each correctly with accurate quotes, gave a 2/5 score with one-sentence reasoning and added a corrected answer. It was not fully strict, though: the table has an extra "#" column, and the "Supported claims only" rewrite adds reference details (30 days, tags, postage rules) that were not Supported claims in the original answer.
The prompt
Act as a meticulous fact-checking reviewer who judges AI answers strictly against the evidence supplied.
Task: Decide whether the answer below is fully backed by the reference material, and flag anything invented.
Question asked: {customer_question}
Reference material: {reference_material}
Answer to check: {ai_answer}
Steps:
1. Split the answer into separate factual claims and number them.
2. For each claim, find the sentence in the reference material that supports or contradicts it, and quote a short extract.
3. Label each claim Supported, Contradicted or Unsupported (not mentioned in the reference). Use no outside knowledge.
4. Score the answer from 1 to 5 for faithfulness to the reference and give one sentence of reasoning.
Output: A table with the columns Claim, Label and Evidence quote. Beneath it, a line reading "Faithfulness score: X/5" with your reason, then a corrected version of the answer that uses only Supported claims.
Change the parts in curly braces. Use it before publishing chatbot replies, support macros or AI summaries that must stick to your policy documents.
What happened when we ran it
We sent
Act as a meticulous fact-checking reviewer who judges AI answers strictly against the evidence supplied. Task: Decide whether the answer below is fully backed by the reference material, and flag anything invented. Question asked: Can I return a bike helmet I bought online, and how long do refunds take? Reference material: Unused items can be returned within 30 days of delivery for a full refund. Refunds are issued to the original payment method within 5 to 7 working days of us receiving the parcel. Return postage is free for faulty items but customers pay for change-of-mind returns. Helmets must be returned with the original tags attached. Answer to check: Yes, you can return your helmet within 60 days for a full refund. Refunds arrive within 5 to 7 working days after we receive it. Return postage is always free, and you will also get a 10% discount code on your next order. Steps: 1. Split the answer into separate factual claims and number them. 2. For each claim, find the sentence in the reference material that supports or contradicts it, and quote a short extract. 3. Label each claim Supported, Contradicted or Unsupported (not mentioned in the reference). Use no outside knowledge. 4. Score the answer from 1 to 5 for faithfulness to the reference and give one sentence of reasoning. Output: A table with the columns Claim, Label and Evidence quote. Beneath it, a line reading "Faithfulness score: X/5" with your reason, then a corrected version of the answer that uses only Supported claims.
The model replied
**Claims and verdicts**
| # | Claim | Label | Evidence quote |
|---|-------|-------|----------------|
| 1 | You can return your helmet for a full refund | Supported | "Unused items can be returned within 30 days of delivery for a full refund." |
| 2 | The return window is 60 days | Contradicted | "within 30 days of delivery" |
| 3 | Refunds arrive within 5 to 7 working days after the parcel is received | Supported | "Refunds are issued to the original payment method within 5 to 7 working days of us receiving the parcel." |
| 4 | Return postage is always free | Contradicted | "Return postage is free for faulty items but customers pay for change-of-mind returns." |
| 5 | You get a 10% discount code on your next order | Unsupported | No mention of any discount code in the reference. |
**Faithfulness score: 2/5.** Only the refund eligibility and the 5 to 7 working day timing match the reference, while the 60-day window and "always free" postage are contradicted and the 10% discount code is invented.
**Corrected answer (Supported claims only):**
Yes, you can return your helmet within 30 days of delivery for a full refund, provided it is unused and still has its original tags attached. Refunds are issued to your original payment method within 5 to 7 working days of us receiving the parcel. Return postage is free if the helmet is faulty, but you will pay for the return if it is a change-of-mind return.
Run on Claude Sonnet 5.5, 27 Oct 2026. Automatic check: 4 out of 5.
Mentioned for
ChatGPT, Claude, Gemini
Where it came from
Found on GitHub, shared by comet-ml: comet-ml/opik.
This is our own version of the idea, written and tested by us. The original may differ.