AI Hallucinations

Where does it come from?

• The model learns to predict text, not to verify facts. When it doesn't know the answer, it generates something that merely looks like an answer.
• OpenAI showed in the paper Why language models hallucinate (2025) that model tests reward guessing more than saying "I don't know". The model thus learns to shoot in the dark.
• It invents details most frequently: dates, numbers, names, paper titles, and URLs.

How does our lab handle it?

• Code checks numbers, not the model. In reports, the model writes the text, but a program compares every number against the data. A number outside the dataset causes the text to fail.
• The chat responds with a source or says "I don't know". Every sentence includes a citation to an article. When the database has nothing, the chat says so directly. When it answers from the model's general knowledge, it displays that in a gray line below the response.
• Our own lesson: the AI assistant we are building the lab with once "found" a security flaw that didn't exist. The description made it into the documentation, and subsequent sessions repeated it as fact until three independent checks debunked it. An invention written into a document becomes "knowledge".
• Chat tests (promptfoo): 52/52 cases with hard rules, including "no source = error".

What does our database show?

The lab database contains 90 texts on hallucinations: from OpenAI's explanations, through public model hallucination leaderboards, to papers on detecting fabrications from inside the model before an answer is given.

How to limit hallucinations?

• Provide models with sources (RAG) and make them cite them.
• Allow the model to say "I don't know", and even encourage it to do so.
• Verify what can be verified using code: numbers, dates, addresses.
• For critical decisions, a human approves. In our lab, every publication awaits a click.