What Is Retrieval-Augmented Generation (RAG)?

A general-purpose language model is trained on a broad slice of public text, frozen at some point in time. It doesn't know your internal documentation, your product catalog, your support tickets, or anything that happened after its training cutoff. Ask it a question that depends on that information and it will either say it doesn't know, or — worse — guess convincingly.

Retrieval-augmented generation (RAG) is the standard fix. Instead of relying on the model's frozen internal knowledge, you retrieve the relevant information from your own data at the moment of the question, and hand it to the model as context before it answers.

How it actually works

A RAG pipeline has three moving parts. First, your source documents — PDFs, wiki pages, database records, whatever — get split into chunks and converted into vector embeddings, numerical representations that capture meaning rather than exact wording. Those embeddings get stored in a vector database.

Second, when a user asks a question, that question also gets embedded, and the system searches the vector database for the chunks whose embeddings are closest in meaning — this is the 'retrieval' step. Third, those retrieved chunks get inserted into the prompt alongside the original question, and the language model generates an answer grounded in that specific context — the 'generation' step.

Why this matters more than fine-tuning, most of the time

Fine-tuning changes the model's weights to bias it toward certain patterns — useful for tone or task format, but a poor fit for keeping a model up to date with fast-changing facts, since every update means retraining. RAG sidesteps that entirely: update the underlying documents, and the next query automatically retrieves the new version. No retraining, no model versioning to manage.

RAG also gives you something fine-tuning can't: traceability. Because the model's answer is built from specific retrieved chunks, you can show users exactly which source document backed a given answer — which matters a lot when the cost of a wrong answer is high.

Where RAG systems actually break

The failure mode people don't expect is rarely the language model itself — it's retrieval. If the wrong chunks get retrieved, the model will confidently generate a fluent, well-formed answer from irrelevant context. Chunking strategy, embedding model choice, and how you handle queries that don't match anything well are where most of the real engineering effort goes.

• check_circleChunk size that's too large dilutes relevance; too small loses context
• check_circleEmbedding models tuned for general text can underperform on domain-specific jargon
• check_circleNo fallback behavior when nothing in the index is actually relevant
• check_circleStale indexes when the underlying documents change but the vector store isn't re-synced