How to stop an AI chatbot giving wrong answers

A support bot goes live on a Tuesday. By Thursday somebody forwards a screenshot where it told a customer the return window is sixty days. It is thirty. Nobody wrote sixty anywhere, the model was not trained on your policy, and the sentence reads exactly as confidently as the twenty correct answers before it.

The instinct at that point is to edit the prompt. Add a line telling it to be accurate, or to only use company information, and watch the next hundred conversations. Sometimes that appears to work, which is worse than if it plainly did not, because the mechanism has not been touched and the failure will come back on a question nobody tested.

What follows is the order of work that actually reduces this, why prompt wording sits fourth rather than first, and the one failure mode that is more dangerous than a wrong answer.

It has a name, and the name is useful

Calling it hallucination makes it sound like an occasional glitch. The term used by NIST is more precise and more useful to anyone building. In the Generative AI Profile of its AI Risk Management Framework, published in July 2024, NIST lists confabulation as the second of the risks novel to or exacerbated by generative AI, and defines it as the production of confidently stated but erroneous or false content by which users may be misled or deceived.

The expanded definition adds two cases people forget to test for. Confabulations include outputs that diverge from the prompt or other input, and outputs that contradict statements the system itself made earlier in the same conversation. If your evaluation only checks single questions in isolation, it cannot see the second kind at all.

The most important sentence in that section is the one about cause. NIST describes confabulations as a natural result of the way generative models are designed, because they generate outputs that approximate the statistical distribution of their training data, predicting the next token in a sequence. That is not a bug report. It means no amount of instruction makes the behaviour go away, and anyone who tells you their prompt eliminated it is describing a test set, not a property.

NIST also names the conditions where it gets worse: open-ended prompts asking for long-form responses, and domains needing highly contextual or specialist expertise. Both describe a support bot answering policy questions in free text, which is what most businesses build first.

Grounding comes before prompt wording

If the model is predicting plausible continuations, the fix is to make the correct answer the most available thing in front of it. That means retrieval: the question goes to your own documents first, the relevant passages are pulled, and the model is asked to answer from those passages rather than from memory.

This is where most of the available improvement lives, and most of it is unglamorous retrieval engineering rather than anything to do with the model. If the right passage is not retrieved, no instruction saves the answer, because the model is now being asked to write about something it cannot see. Chunk sizes that split a policy across two fragments, a search that matches on wording rather than meaning, and documents that contradict each other because nobody retired the 2024 version are the three causes we hit most often.

That last one is organisational rather than technical, and the one clients resist. A bot grounded in a document set containing both the old and the new returns policy will answer correctly some of the time, and the inconsistency will look like a model problem. We went through this in more detail in AI search over company documents, and where it still invents, and the conclusion there holds: the retrieval layer inherits whatever disorder exists in the source material.

The invented citation is the dangerous failure

There is a failure mode worse than a wrong answer, and NIST names it directly: outputs may include confabulated logic or citations that purport to justify or explain the answer, which can further mislead people into inappropriately trusting it. The document notes that models sometimes lay out the logical steps by which they reached an answer even when the answer itself is incorrect.

This matters because citations are the control most teams reach for. Make it cite its sources, the thinking goes, and a human can check. That works only if the citation is guaranteed to be a real retrieved passage rather than something generated alongside the answer. If the model writes the reference the same way it writes the prose, you have added an authority signal to a wrong answer and made it harder to catch, not easier.

The engineering requirement follows from that. A citation must be a pointer to a document that was actually retrieved for that query, rendered by your code from the retrieval result, not a string the model composed. If you cannot click a citation in your own logs and land on the passage the model was shown, the citations are decoration.

Refusal has to be built, not requested

A bot that says it does not know is more valuable than one that is right slightly more often, because the first can be safely put in front of customers and the second cannot. But refusal is not a natural behaviour you can ask for politely. It needs a condition in your code: if retrieval returns nothing above a relevance threshold, the model is not asked the question at all, and the user gets a handover instead.

Deciding what sits above and below that line is a business decision, and the same one as scoping any other support automation. Our note on what to trust a support chatbot with works through where to draw it, and AI email triage and the human check covers the handover mechanics when it goes to a person.

One rule we apply without exception: anything that commits the business to money, a date or an entitlement does not get answered generatively. Prices, refund eligibility, delivery promises and anything legal are looked up and rendered, or escalated. The model can be the interface to that lookup. It should not be the source.

Evaluation is the step that gets cut

Nearly every chatbot project we have been brought in to fix had no evaluation set. Correctness was judged by whoever built it trying a dozen questions they already knew, which selects for the cases that work.

The survey by Kenthapadi, Sameki and Taly, Grounding and Evaluation for Large Language Models, accompanying their KDD 2024 tutorial, makes the broader argument well: evaluation of these systems has to reach past accuracy to cover robustness, security and interpretability, and it has to continue after deployment rather than ending at a pre-launch check. That second half is the one that gets dropped. A system evaluated once, at launch, is unmeasured from the first document change onward.

In practice the useful artefact is boring: a spreadsheet of real questions from your support inbox, each with the correct answer and its source document. Fifty is enough to be informative. Run it on every change to the prompt, the retrieval configuration or the document set, and record which items moved. Without it, every change is a guess and nobody can tell whether last week's fix cost you something elsewhere.

An order of work that does not waste a month

• Build the evaluation set first, from real questions in your support history, with correct answers and their source documents. Fifty items beats a week of prompt tuning.
• Fix the document set before touching the model. Retire superseded versions and resolve contradictions, because retrieval cannot choose between two policies that both look current.
• Check retrieval on its own. For each evaluation question, confirm the right passage is in what gets retrieved, before asking whether the answer was right.
• Render citations from retrieval results in your own code, so a citation cannot be something the model composed.
• Add a relevance threshold and a refusal path, and route anything about money, dates or entitlements to a lookup or a person.
• Only then tune the prompt, and measure each change against the evaluation set rather than against the last conversation somebody noticed.
• Keep running the set after launch, because a document change is a silent change in behaviour.

What none of this fixes

Grounding reduces confabulation. It does not end it. A model given the correct passage can still summarise it wrongly, merge two passages that disagree without saying so, or answer confidently when the documents simply do not address the question. NIST is candid on the wider point, noting that the downstream scale and impact of confabulations is difficult to estimate, and that remains true of your deployment too.

The honest target is not a bot that is never wrong, but one whose wrong answers are rare, visible in your logs, and confined to subjects where being wrong costs an apology rather than money. If the use case cannot tolerate that, the answer is not a better prompt. It is a narrower bot, or a lookup with a conversational interface and no generation in the answer path at all.

Whether it is worth building at all is a separate question from whether it can be made accurate, and the two get conflated constantly. Khaled Badr's note on how the maths on a small business chatbot changed in 2025 is a good counterweight to read first if you have not yet committed, and our own where a website chatbot loses the lead covers the commercial side of the same decision.