RAG Step by Step
RAG lets a model answer questions about your documents by finding the right parts first and giving them to the model with the question.
What RAG Means
RAG stands for Retrieval-Augmented Generation:
- Retrieval: find the pieces of your documents that relate to the question
- Augmented: add those pieces to the prompt
- Generation: the model writes an answer using them
The model never "learns" your documents. It reads the relevant parts fresh on every question. That is why RAG works for facts that change, like timings, prices and policies.
The Chatbots Tutorial shows a short RAG example in JavaScript in Adding Your Own Knowledge. Here we build it slowly in Python, step by step, and then turn it into a real program.
The Five Steps
| Step | Done when | What happens |
|---|---|---|
| 1. Collect | Once | Gather the documents: handbook, FAQ, price list |
| 2. Chunk | Once | Split them into small pieces |
| 3. Index | Once | Store each chunk (with its embedding) so it can be searched |
| 4. Retrieve | Every question | Find the top few chunks for the question |
| 5. Generate | Every question | Send the chunks and the question to the model |
Step 2: Chunking
Chunks should be small enough to be specific, but big enough to make sense alone. A common start is a few hundred words per chunk, with a small overlap so a sentence cut in half still appears whole in one chunk. Here is a tiny version with very small chunks so you can see the overlap:
Example
def chunk_words(text, size=12, overlap=4):
words = text.split()
step = size - overlap
return [" ".join(words[i:i + size]) for i in range(0, max(len(words) - overlap, 1), step)]
policy = ("Refunds are given within 7 days of delivery if the item is unused. "
"The customer must send a photo of the item on WhatsApp. "
"Delivery charges are not refunded.")
for i, c in enumerate(chunk_words(policy), 1):
print(i, c)1 Refunds are given within 7 days of delivery if the item is 2 if the item is unused. The customer must send a photo of 3 send a photo of the item on WhatsApp. Delivery charges are not 4 Delivery charges are not refunded.
Each chunk repeats the last 4 words of the one before. In real documents it is often better to split by headings or paragraphs first, because they already group related sentences.
Steps 3 and 4: A Tiny Keyword Retriever
To see retrieval clearly, we start without embeddings. This retriever scores each chunk by how many important words it shares with the question:
Example
import re
handbook = """Fees: The Python course fee is paid monthly at the office or by bank transfer.
Timings: Morning batch runs 10 am to 12 pm. Evening batch runs 6 pm to 8 pm.
Location: The institute is in Orangi Town, Karachi, near the main road.
Certificates: Students get a certificate after passing the final project.
Online: Every course is also offered online on Zoom with recorded classes."""
# Step 2: split the document into chunks (here, one line = one chunk)
chunks = [line.strip() for line in handbook.splitlines() if line.strip()]
STOP = {"the", "is", "a", "an", "at", "of", "do", "i", "to", "in", "on", "and", "or", "by", "what", "when", "get", "after", "are", "also", "with", "every"}
def words(text):
return {w for w in re.findall(r"[a-z]+", text.lower()) if w not in STOP}
# Step 4: score each chunk by how many question words it shares
def retrieve(question, chunks, k=2):
q = words(question)
scored = [(len(q & words(c)), c) for c in chunks]
scored.sort(key=lambda x: x[0], reverse=True)
return [c for score, c in scored[:k] if score > 0]
question = "What are the evening batch timings?"
for c in retrieve(question, chunks):
print("-", c)- Timings: Morning batch runs 10 am to 12 pm. Evening batch runs 6 pm to 8 pm.
Only one chunk shared words with the question, so only one came back. The STOP set removes common words like "the" and "what" that would match everything.
Step 5: Building the Grounded Prompt
Now put the chunks into the prompt. Number them, so the model can say which source it used. Add this to the code above (replace its last three lines):
Example
def build_messages(question, context_chunks):
context = "\n".join(f"[{i + 1}] {c}" for i, c in enumerate(context_chunks))
system = ("Answer using ONLY the context below. Mention the source number like [1]. "
"If the answer is not in the context, say you do not know.\n\n"
f"Context:\n{context}")
return [{"role": "system", "content": system},
{"role": "user", "content": question}]
question = "Can I study online and do I get a certificate?"
for m in build_messages(question, retrieve(question, chunks)):
print(f"--- {m['role']} ---")
print(m["content"])--- system --- Answer using ONLY the context below. Mention the source number like [1]. If the answer is not in the context, say you do not know. Context: [1] Certificates: Students get a certificate after passing the final project. [2] Online: Every course is also offered online on Zoom with recorded classes. --- user --- Can I study online and do I get a certificate?
These messages are exactly what you send to the chat completions API.
The Real Program: Embeddings and an API Call
Keyword search fails when the user uses different words ("How much do I pay?" has no word "fee"). Real RAG uses embeddings from Embeddings and Vectors. This full program needs your API settings plus an AI_EMBEDDING_MODEL variable:
Example
# rag.py - pip install openai numpy python-dotenv
import os
import numpy as np
from dotenv import load_dotenv
from openai import OpenAI
load_dotenv()
client = OpenAI(base_url=os.environ["AI_BASE_URL"], api_key=os.environ["AI_API_KEY"])
chunks = [
"Fees: The Python course fee is paid monthly at the office or by bank transfer.",
"Timings: Morning batch runs 10 am to 12 pm. Evening batch runs 6 pm to 8 pm.",
"Location: The institute is in Orangi Town, Karachi, near the main road.",
"Certificates: Students get a certificate after passing the final project.",
"Online: Every course is also offered online on Zoom with recorded classes.",
]
def embed(texts):
result = client.embeddings.create(model=os.environ["AI_EMBEDDING_MODEL"], input=texts)
vectors = np.array([item.embedding for item in result.data])
return vectors / np.linalg.norm(vectors, axis=1, keepdims=True)
chunk_vectors = embed(chunks) # in a real app, do this once and save it
def retrieve(question, k=2):
q = embed([question])[0]
scores = chunk_vectors @ q # cosine similarity with every chunk
best = np.argsort(scores)[::-1][:k]
return [chunks[i] for i in best]
def answer(question):
context = "\n".join(f"[{i + 1}] {c}" for i, c in enumerate(retrieve(question)))
response = client.chat.completions.create(
model=os.environ["AI_MODEL"],
temperature=0,
messages=[
{"role": "system", "content":
"Answer using ONLY the context below. Mention the source number like [1]. "
"If the answer is not in the context, say you do not know.\n\nContext:\n" + context},
{"role": "user", "content": question},
],
)
return response.choices[0].message.content
print(answer("How much do I have to pay, and how?"))Run it with python rag.py. The model should answer from chunk [1] about monthly fees, even though the question never says "fee".
When RAG Gives Bad Answers
Most RAG problems are retrieval problems, not model problems. Check these first:
- Wrong chunks found: print the retrieved chunks for every test question
- Chunks too big or too small: try splitting by headings
- Old documents: re-index when documents change
- Mixed topics in one chunk: keep one topic per chunk
- Combine keyword and embedding search (called hybrid search) for names, codes and numbers
Always print what was retrieved while you build. If the right fact is not in the context, no prompt can make the answer correct.
Exercise
Exercise
In RAG, when does the retrieval step happen?