RAG: what it is and how chatbots answer from your documents

RAG (Retrieval-Augmented Generation) is a method of building a chatbot where the AI model first searches for matching excerpts in your documents before answering, and then replies strictly based on them. The model does not need to "remember" your company. It just needs to be given the right pages to read.

How does it work?

Preparation (once): documents are split into chunks, each is converted into a vector (embedding) and saved in a vector database.
Question: the user's question is converted into a vector using the same model.
Retrieval: the database returns the chunks closest in meaning.
Answer: the model receives the question along with the chunks and the instruction "answer only using this."
Sources: the answer indicates which chunk each sentence comes from.

How does our lab do it?

The chat in the corner of this page is a RAG over ~20k texts about AI. Every question goes through the same pipeline:

Data masking: Polish ID numbers (PESEL), emails, and phone numbers are replaced with [PESEL], [EMAIL], [PHONE] before the question proceeds further.
Question vector is computed by local EmbeddingGemma on our server for 0 PLN.
12 nearest chunks from the database (PostgreSQL + pgvector).
Topic gate: if the best chunk does not exceed the 0.764 threshold, the chat states directly that the database does not contain this information.
6 best chunks are sent to the response model. This costs about 0.08 gr per answer.
Output gate checks the answer before sending: no links in the text, no code, no prompt leakage.
Sources as footnotes [1], [2] with article URLs.

When is it worth it and when not?

Worth it: company website chat, employee knowledge base, answers from terms of service, price lists, and documentation that change frequently. Document change = recalculating a single chunk, without retraining the model.
Not worth it: when the model needs to learn style or format (fine-tuning is better here) or when all documents fit into a single prompt to the model.

Most common pitfalls

The chat answers everything. Without a similarity threshold, the model gets weak chunks and makes up the rest from memory.
Missing content looks like a bad chat. In our case, the question "what are vector databases?" failed because the database only had 3 titles about vectors. This dictionary's cards were created partly for this reason.
Lack of sources. An answer without citations cannot be verified, so no one trusts it.
Personal data in questions. People type emails and phone numbers. Mask them before saving anything.

Questions and answers

RAG or fine-tuning?

RAG gives the model knowledge, fine-tuning changes its behavior. For a company chat, RAG is almost always enough.

How much does a RAG chatbot cost?

For us, about 0.08 gr per answer plus your own server. On a client's Cloudflare account, it can fit within the free tier with low traffic.

Does RAG eliminate hallucinations?

It reduces them, not eliminates them. Thresholds, citations, and output gates help.

Sources

Lewis et al.: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, arXiv:2005.11401
Lab measurements: threshold calibration (21.10.2026), promptfoo chat tests (27–28.10.2026)

RAG