codesbyjit.dev
← cd ../blog
#RAG#LLM#AI Engineering#Vector Databases#Machine Learning#System Design

What is RAG? How Retrieval-Augmented Generation Actually Works

acodesbyjit·JUL 18, 2026·1 min read
What is RAG? How Retrieval-Augmented Generation Actually Works

What is RAG? How Retrieval-Augmented Generation Actually Works

Large language models are impressively fluent, but fluency isn't the same as truth. Ask an LLM a question outside its training data — or even something inside it, if the details are obscure enough — and it will often answer with total confidence anyway. That's the core problem Retrieval-Augmented Generation (RAG) was built to solve.

The Problem: LLMs Don't Know What They Don't Know

A base LLM has no built-in mechanism to say "I'm not sure" or "this isn't in my training data." It generates the statistically most likely next tokens, whether or not those tokens are grounded in fact. This is called hallucination, and it's the single biggest reason companies hesitate to put raw LLMs in front of real users for anything factual — legal, medical, internal documentation, customer support.

RAG doesn't fix hallucination by making the model smarter. It fixes it by changing what the model is allowed to answer from.

The Core Idea

Instead of asking an LLM to answer purely from memory, a RAG system:

  1. Retrieves relevant passages from a trusted knowledge source (your documents, a database, a wiki) based on the user's question
  2. Injects those passages into the model's context window alongside the question
  3. Generates an answer that's explicitly grounded in the retrieved text, not just the model's internal weights

The model is no longer answering "what do I remember about this topic" — it's answering "given these specific passages, what's the answer." That distinction is everything.

How Retrieval Actually Works

The retrieval step relies on embeddings — numerical representations of text where semantically similar content ends up close together in vector space. A typical pipeline looks like this:

  • Chunking: Documents get split into smaller pieces (often 300–800 characters, with some overlap so context isn't cut mid-thought)
  • Embedding: Each chunk is converted into a vector using an embedding model
  • Indexing: Vectors are stored in a vector database (Chroma, Pinecone, Weaviate, FAISS, etc.) optimized for fast similarity search
  • Query time: The user's question gets embedded the same way, and the system finds the top-k most similar chunks via cosine similarity or a comparable metric
  • Augmentation: Those chunks are stitched into the prompt sent to the LLM

Why Citations Matter More Than the Answer Itself

A good RAG system doesn't just return an answer — it shows its work. Surfacing which exact passage supported which claim lets a user verify the answer instead of just trusting it. This is the difference between a system you can deploy in a regulated industry and one you can't.

It also enables something more interesting: automated evaluation. If every claim in an answer is supposed to trace back to a retrieved passage, you can programmatically check whether it actually does — scoring faithfulness (is the answer supported?), relevance (did it actually address the question?), and retrieval precision (did the system pull the right passages in the first place?). That turns "does this work?" from a vibe check into a measurable, testable property of the system.

Where RAG Systems Actually Fail

RAG isn't magic, and most of the failure modes aren't about the LLM — they're about retrieval:

  • Bad chunking — passages cut off mid-sentence lose meaning, and chunks that are too large dilute relevance
  • Retrieval miss — if the top-k search doesn't surface the right passage, the LLM will still confidently answer from nothing
  • Over-trusting the model — even with good context, LLMs can still ignore it and hallucinate on top of retrieved passages, which is exactly why a separate evaluation layer matters
  • Stale embeddings — if your documents change but your vector index doesn't get rebuilt, you're retrieving outdated ground truth

The Bigger Picture

RAG has become the default architecture for building trustworthy, document-grounded AI applications because it sidesteps the two hardest problems in LLM deployment — hallucination and staleness — without needing to retrain the model at all. Swap out the documents, and the system's "knowledge" updates instantly.

The next frontier isn't just retrieving and generating — it's verifying. Systems that can score their own groundedness, flag low-confidence answers, and cite sources transparently are what separates a RAG demo from a RAG system you can actually trust in production.

share: