RAG Explained for Developers Who Don't Want a PhD in RAG

After building a small RAG app, the concept became much easier to explain.

RAG stands for retrieval-augmented generation, but the name makes it sound more complicated than it is.

Imagine I have a large document and ask a language model one question about it. I have three basic options.

I can send the entire document with every question, which becomes expensive and eventually exceeds context limits.

I can train a model on my document, which is usually massive overkill.

Or I can first retrieve the small part of the document that is probably relevant and send only that part.

That third option is RAG.

The retrieval step usually uses embeddings. An embedding represents a piece of text as numbers in a way that roughly preserves semantic meaning. Texts about similar things end up closer together in that mathematical space.

So when a user asks a question, the system can look for document chunks whose embeddings are closest to the question embedding.

The model then gets something like:

Here are the relevant passages:
[chunk 1]
[chunk 2]
[chunk 3]

Question:
...

and generates the answer from there.

The important thing is that RAG does not magically make the model know my data. It is a system for choosing useful context before calling the model.

That also explains many of its failure modes. If the document is split badly, retrieval may return the wrong chunks. If too many irrelevant chunks are included, the model gets distracted. If the correct information is never retrieved, the model cannot reliably answer from it.

So when a RAG application gives a bad answer, the prompt is not always the first thing I should blame.

Sometimes the real problem happened before the model ever saw the question.