What Is RAG? Retrieval-Augmented Generation Explained

RAG, short for retrieval-augmented generation, is a way to make a language model answer from your own documents. Before the model writes anything, the system searches your data for passages that match the question and adds them to the prompt. The model then answers from those passages instead of from what it memorized in training.

The term comes from a 2020 research paper by a team at Facebook AI Research. It is now the usual way to build chat over company documents, support assistants, and internal search.

How RAG works

A RAG system has two phases. The first runs ahead of time, whenever your documents change. The second runs on every question.

1. Ingest. Split each document into chunks of a few hundred words. Turn each chunk into an embedding, a list of numbers that places similar meanings close together, and store the chunks and embeddings in an index. We use Postgres with pgvector for this.
2. Retrieve. Embed the user's question the same way and pull the closest chunks. Production systems usually run vector search and keyword search together and merge the results, because vectors miss exact identifiers like error codes and product names.
3. Rerank. A second model scores each retrieved chunk against the question and keeps the best few. This adds latency and removes many near-misses.
4. Generate. Put the best chunks in the prompt with an instruction to answer only from them and to cite which chunk each claim came from.

Why teams use it

• Your data stays outside the model. To change an answer, you update the document and re-index it. There is no retraining.
• Answers can point to their sources, so a reader can check them.
• Access control can happen at retrieval, so a user only gets passages from documents they are allowed to see.
• It works with private and recent data that no public model was trained on.

What RAG does not fix

RAG reduces made-up answers. It does not remove them. The model can ignore the passages it was given, and the retriever can return the wrong ones. When retrieval misses, the model often answers anyway, with confidence.

• Bad chunking. A table split across two chunks, or a paragraph cut away from its heading, loses the context that made it answerable.
• Stale or conflicting documents. If two policies disagree, the model picks one without telling you.
• No refusal path. A system that always answers will eventually answer from nothing.

We cover these failure modes and the fix in why RAG demos break in production, including the evidence check that makes a system refuse instead of guess.

When RAG is the right tool

Use RAG when the answer lives in a body of documents that changes, and users need to see where it came from. Support knowledge bases, internal policy search, contract and report Q&A, and product documentation assistants all fit.

It is the wrong tool for questions that need math over a whole dataset, such as total revenue by region. Those belong in SQL, with the model writing or calling the query.

It is also the wrong tool for changing how a model writes or formats its output. That is a job for prompting or fine-tuning, which we compare in RAG vs fine-tuning.

What we build around retrieval

The retrieval step is most of the work. On Ragix, our production RAG platform, we combine vector and keyword search, rerank the results, and add an evidence gate that refuses to answer when the retrieved context is too thin. Each tenant's data is isolated in the database itself, using Postgres row-level security, so a missed filter in application code cannot leak documents between customers.

If you are scoping a RAG feature, start with a list of real questions your users ask and the documents that answer them. Run retrieval on that list before you write any prompt. If the right passage does not come back, no prompt change will fix the answer.