Skip to content
Home/Resources/AI & ML/What is RAG?
AI explained ยท AI & ML

What Is RAG? Retrieval-Augmented Generation Explained, With a Toy You Can Run Offline

By Younus FardeenUpdated 11 Nov 202614 min read
Short answer

RAG (retrieval-augmented generation) makes an AI answer from documents you supply. The system splits documents into chunks, finds the chunks closest to the question, puts them into the prompt and asks an LLM to answer only from them, with citations. It cuts made-up answers on private or fresh data but does not remove them.

Most RAG explainers jump straight to vector databases. This page starts with why RAG exists, shows the whole pipeline in a script you can run offline, and then lists what real systems add and where they still fail. It is for students who want to build one or answer interview questions about it.

Facts about papers, vendor test results and tools were read from the original sources on 11 November 2026. The Python script is a toy: it scores keywords with TF-IDF instead of a neural embedding model, and it builds the prompt without calling an LLM.

  1. 1Chunksplit the documents
  2. 2Embedturn chunks into vectors
  3. 3Retrievefind the closest chunks
  4. 4Generatethe LLM answers from them
  5. 5Citeshow sources or say not found
The RAG pipeline: five steps from documents to a cited answer.

What RAG is, why it exists and its five steps

An LLM only knows what was in its training data and what you put in the prompt. Ask it about your college's hostel timings, or a rule that changed last week, and it will refuse or guess. Retrieval-augmented generation (RAG) closes the gap by looking the facts up first and then asking the model to answer from what was found. Think of an open-book exam: the model still does the writing, but the book is on the desk.

The name comes from a 2020 paper by Patrick Lewis and colleagues. It paired a pretrained text-generation model with an index of Wikipedia passages searched by a neural retriever, and reported more specific and factual text than a model without retrieval. Today the word covers any system that retrieves text and adds it to the prompt. For the model side of the story, read what is a large language model.

RAG does not retrain the model and does not make it smarter. It changes what the model can see on one request. Here are the five steps and where each one tends to break.

StepWhat happensCommon toolWhat goes wrong
1. ChunkSplit documents into passages, often a few sentences eachPlain code or a library text splitterA fact is cut in half, or chunks are so big they bury it
2. EmbedTurn each chunk into a vector, a list of numbers, so similar meanings sit close togetherAn embedding model from an API, or one you run locallyThe model handles your jargon, language or numbers badly
3. RetrieveEmbed the question and return the few closest chunksA vector index, keyword search, or bothThe right chunk exists but is not in the top results
4. GeneratePut the chunks and the question in a prompt; the LLM writes the answerAny LLM APIThe model ignores the chunks or adds claims they do not contain
5. CiteShow which chunk supports each claim, or say 'not found'Prompt rules plus a check in codeA citation that looks right but does not support the claim

A RAG toy you can run offline

The script below is a whole RAG pipeline in about 70 lines of plain Python, with no install and no API key. The document is a made-up college help-desk text. Chunking groups whole sentences, 'embedding' is TF-IDF (word counts weighted so rare words count more), retrieval is cosine similarity, and generation is replaced by printing the prompt you would send to an LLM. It is a toy. A real system uses a neural embedding model that knows 'fee' and 'costs' are related. This one does not.

import math
import re
from collections import Counter

document = (
    "Students can borrow 4 books for 14 days. A late return costs 2 rupees per day. "
    "The reading room is open from 8 am to 10 pm on weekdays. "
    "The hostel mess closes at 9 pm and the hostel gate closes at 10 pm. "
    "Visitors are not allowed in rooms after 6 pm. "
    "The merit scholarship form opens on 1 July and closes on 31 July. "
    "Attach the previous semester marksheet and an income certificate."
)

MIN_SCORE = 0.15  # below this, the bot should say it does not know

STOP = {"the", "a", "an", "is", "are", "of", "for", "on", "at", "to", "and", "in", "from", "does", "do", "how", "what", "when", "much"}

def tokens(text):
    return [w for w in re.findall(r"[a-z0-9]+", text.lower()) if w not in STOP]

def split_by_words(text, size):
    words = text.split()
    return [" ".join(words[i:i + size]) for i in range(0, len(words), size)]

def split_by_sentences(text, max_words=24):
    chunks, current = [], []
    for s in re.split(r"(?<=\.)\s+", text.strip()):
        if current and len(" ".join(current + [s]).split()) > max_words:
            chunks.append(" ".join(current))
            current = []
        current.append(s)
    return chunks + [" ".join(current)]

def build_retriever(chunks):
    n = len(chunks)
    df = Counter(w for c in chunks for w in set(tokens(c)))

    def embed(text):  # TF-IDF vector as a dict: a stand-in for a real embedding model
        return {w: c * (math.log((1 + n) / (1 + df.get(w, 0))) + 1) for w, c in Counter(tokens(text)).items()}

    def cosine(a, b):
        dot = sum(v * b.get(w, 0.0) for w, v in a.items())
        norm = math.sqrt(sum(v * v for v in a.values())) * math.sqrt(sum(v * v for v in b.values()))
        return dot / norm if norm else 0.0

    vectors = [embed(c) for c in chunks]

    def retrieve(question, k=1):
        q = embed(question)
        ranked = sorted(((cosine(q, v), i) for i, v in enumerate(vectors)), reverse=True)
        return [(round(s, 3), chunks[i]) for s, i in ranked[:k]]

    return retrieve

def build_prompt(question, chunk):
    return f"Answer only from this source and cite it as [1].\n[1] {chunk}\nQuestion: {question}"

sentences = build_retriever(split_by_sentences(document))
for q in [
    "When does the hostel gate close?",
    "What is the late return charge per day?",
    "How much is the late fee for a library book?",
    "Who won the cricket match?",
]:
    score, chunk = sentences(q)[0]
    print(f"{score:.3f}  {q}")
    print("   ->", chunk if score >= MIN_SCORE else "below the cut-off, so the bot says: I could not find this.")

print()
print(build_prompt("When does the hostel gate close?", sentences("When does the hostel gate close?")[0][1]))

print()
tiny = build_retriever(split_by_words(document, 6))
print("Chunks cut every 6 words:", tiny("What is the late return charge per day?")[0][1])

Output:

0.328  When does the hostel gate close?
   -> The hostel mess closes at 9 pm and the hostel gate closes at 10 pm. Visitors are not allowed in rooms after 6 pm.
0.442  What is the late return charge per day?
   -> Students can borrow 4 books for 14 days. A late return costs 2 rupees per day.
0.104  How much is the late fee for a library book?
   -> below the cut-off, so the bot says: I could not find this.
0.000  Who won the cricket match?
   -> below the cut-off, so the bot says: I could not find this.

Answer only from this source and cite it as [1].
[1] The hostel mess closes at 9 pm and the hostel gate closes at 10 pm. Visitors are not allowed in rooms after 6 pm.
Question: When does the hostel gate close?

Chunks cut every 6 words: 2 rupees per day. The reading

What the output teaches

  • Retrieval works when words overlap. 'When does the hostel gate close?' finds the hostel chunk with a score of 0.328, and only that chunk goes into the prompt.
  • It fails on synonyms. 'How much is the late fee for a library book?' scores 0.104 against the right chunk, because the text says 'late return costs 2 rupees' and never says 'fee'. With a 0.15 cut-off the bot refuses although the answer is in the document. Neural embeddings and hybrid search exist to fix misses like this.
  • A cut-off lets the system say 'not found'. The cricket question scores 0.000, so nothing is sent to the model. A bot with no cut-off always hands the LLM something, and the LLM will happily answer from it.
  • Chunk size matters. With chunks cut every six words, the best chunk for the late-return question is '2 rupees per day. The reading'. It holds the number but not the rule the number belongs to.

What changes in production, and where RAG still fails

What teams add

  • An embedding model and a vector index. Chunks are stored as vectors. A list in memory or a file is enough for a small project. At thousands of chunks, a vector database or an extension such as pgvector, which its README describes as vector similarity search for Postgres with HNSW and IVFFlat indexes, saves you from comparing against everything. Types of databases shows where vector stores fit.
  • Hybrid search and reranking. Anthropic's September 2024 write-up on contextual retrieval reports that adding context to each chunk and mixing embeddings with BM25 keyword scoring cut failed top-20 retrievals from 5.7% to 2.9%, and a reranking step cut them to 1.9%. These are the vendor's own test numbers: a direction, not a promise for your data.
  • Evals. A fixed list of questions with expected answers, run after every change. Score retrieval (did the right chunk come back?) separately from generation (did the answer stick to the chunk?). Without this you cannot tell whether a change helped.
  • Access control and safe handling. Retrieved text is untrusted input. OWASP's 2025 list of LLM application risks includes prompt injection (LLM01) and vector and embedding weaknesses (LLM08). Apply each user's permissions before retrieval, so one student's marks never reach another student's prompt.

Limits you cannot engineer away

  • RAG reduces hallucination but does not remove it. A 2024 academic study (Magesh and colleagues) tested commercial legal research tools built on RAG and found they still hallucinated between 17% and 33% of the time, which the authors say shows that vendor claims of RAG 'eliminating' hallucinations are overstated.
  • Models use long input unevenly. The 'Lost in the Middle' paper found models do best when the relevant text sits at the start or end of the input and worse in the middle. A few well-chosen chunks are safer than stuffing in many.
  • Guessing is rewarded. A 2025 paper by Kalai and colleagues argues that training and evaluation procedures reward guessing over admitting uncertainty. So 'say you do not know when the sources are silent' must be in your prompt and in your tests.

RAG vs fine-tuning vs long context vs prompting

ApproachUse it whenWeak spot
Prompt engineeringThe model already knows the facts and you need a better format, tone or steps. OpenAI's guide calls it typically the best place to startCannot add facts the model lacks
RAGAnswers depend on documents that are private, large or change oftenRetrieval can miss; needs chunking, evals and upkeep
Fine-tuningYou need consistent behaviour: a fixed format, tone or reasoning patternOpenAI's guide frames it as improving consistency of behaviour, not as a way to add facts; changed documents mean retraining
Long context (paste it all)The whole knowledge base is small. In 2024 Anthropic advised that below 200,000 tokens, about 500 pages, you can put it all in the promptYou pay for every token on every call, and facts in the middle can be missed

Context windows have grown since 2024. Anthropic's model overview lists a 1M-token window for its current models as of 11 November 2026. A whole handbook can now be pasted in, but input tokens are billed on every call: sending 200,000 tokens instead of 3,000 multiplies that call's input cost by about 67. The cost sums are in what is a large language model. Prompt caching narrows the gap for a fixed document: Anthropic lists cache reads at 5% of the base input price for Claude Sonnet 5.5.

A sensible order is to write the prompt, build an eval set, try long context if the documents are small, add RAG when they are not, and consider fine-tuning only for behaviour you cannot get from prompts. OpenAI's guide calls RAG and fine-tuning additive, not exclusive. For the prompt side, see what is prompt engineering.

Build one for free, and the mistakes to avoid

  1. Pick 10 to 20 pages of a real document, such as your college handbook, a syllabus or a product manual.
  2. Write 20 to 30 questions with the expected answers before you write code. This is your eval set.
  3. Run the toy above on your text and note which questions it misses.
  4. Swap TF-IDF for an embedding model on a free tier. The list of free LLM APIs for developers shows what each offers.
  5. Keep the vectors in a file, or in Postgres with pgvector. Free backends and databases for AI apps compares hosts.
  6. Add citations and a 'not found' answer, re-run your questions and write down the score.

A full build with Gemini and Streamlit is in RAG chatbot with Gemini and Streamlit, and more ideas are in AI project ideas. Agents often use RAG as one of their tools, which what is an AI agent explains.

Common mistakes

  • Building before testing: no eval set, so no way to know whether a change helped.
  • No cut-off score, so unrelated questions still get an answer.
  • Trusting citations without checking that the cited chunk says what the answer claims.
  • Indexing documents that nobody should see together, such as one student's marks beside another's.
  • Using RAG to fix a behaviour problem that a better prompt would fix.

Masai's own guides cover the same ground from a career angle: Prompt Engineering vs RAG vs Fine-Tuning and What is RAG in AI.

RAG interview questions with short answers

QuestionShort answer
What is RAG and why use it?Retrieve relevant text first, then let the LLM answer from it. It handles private, large or changing knowledge without retraining.
RAG or fine-tuning?RAG for missing or changing knowledge. Fine-tuning for consistent behaviour such as format or tone. They can be combined.
What is chunking and how do you pick a size?Splitting documents into passages. There is no universal size: split on natural boundaries such as sentences or headings, then test sizes on your own questions.
Why use hybrid search?Keywords catch exact terms and names. Embeddings catch paraphrases. Combining them misses fewer relevant chunks.
How do you evaluate a RAG system?With a fixed question set. Score whether the right chunk was retrieved, and separately whether the answer is supported by it.
Can RAG still hallucinate?Yes. Retrieval can miss, and the model can misread or add to what it retrieves. Published tests of RAG-based legal tools found 17% to 33% hallucination.
What is prompt injection in RAG?A retrieved document contains instructions aimed at the model. Treat retrieved text as data, never as instructions, and limit what the model's output can trigger.

Frequently asked questions

What is RAG in simple words?

RAG, or retrieval-augmented generation, means the AI looks up relevant text from your documents before it answers. The system finds the best-matching passages, adds them to the prompt and tells the LLM to answer only from them. It works like an open-book exam: the model writes the answer, but the facts come from the book you supply.

What is the difference between RAG and fine-tuning?

RAG gives the model facts at question time by retrieving them from documents, so updating knowledge means updating the documents. Fine-tuning changes the model's weights through extra training, and OpenAI's guidance frames it as a way to get consistent behaviour such as format or tone. The two can be combined. For knowledge problems, try prompts and RAG first.

What is the difference between RAG and prompt engineering?

Prompt engineering improves the instructions you give the model. RAG improves the information in the prompt by retrieving documents and inserting them. In practice RAG is a way of building the prompt automatically, so prompts still matter: you must tell the model to answer only from the sources and to say so when they do not contain the answer.

Does RAG stop hallucinations?

No. It lowers them by grounding answers in documents, but it does not remove them. A 2024 study of commercial legal research tools built on RAG measured hallucination rates between 17% and 33%. Retrieval can miss the right passage, and the model can misread or add to what it retrieves, so test with your own questions.

Do I need a vector database for RAG?

Not to start. A few hundred chunks can sit in a Python list or a file, as the toy on this page shows. As the data grows, use a vector index such as pgvector in Postgres, or a dedicated vector database. Choose one when search speed or filtering becomes a real problem, not before.

Is RAG still needed with million-token context windows?

Often yes. A small handbook can simply be pasted into a 1M-token window, which Anthropic lists for its current models as of 11 November 2026. But input tokens are billed on every call, facts in the middle of a long input can be missed, and large or changing document sets still need retrieval. Try the simple option first and measure.

How can I build a RAG chatbot for free?

Use a free LLM API tier for embeddings and answers, keep the vectors in a file or in free Postgres with pgvector, and host the app on a free tier. Start with 10 to 20 pages of text and 20 test questions. The linked Gemini and Streamlit guide walks through it, and the free LLM API list shows current limits.

Sources checked

Continue learning

Concept

What is an LLM?

LLMs in plain English: a table of terms, a Python toy, costs and limits.

Open →
Explainer

What is prompt engineering?

What works in 2026, myths to drop and sources for every claim.

Open →
AI explained

What is an AI agent?

The agent loop, agent vs chatbot, real agents to try and a demo

Open →