What Is RAG and Why Does It Matter?

Large language models have a fundamental limitation: their knowledge is frozen at training time. Ask GPT-5 about your company's internal documentation, yesterday's meeting notes, or a product released last week, and you'll get hallucinations or "I don't know."

RAG solves this by giving the agent a retrieval step before generation. Instead of relying solely on parametric memory (what the model learned during training), the agent actively fetches relevant documents from an external knowledge base and uses them as context for its response.

The result: answers that are factually grounded in your actual data, not the model's best guess.

The RAG Pipeline: 5 Stages

Every RAG system follows the same five stages, whether you're building with LangChain, LlamaIndex, or from scratch:

1. Ingestion — Load your documents (PDFs, web pages, databases, Notion pages, etc.)
2. Chunking — Split documents into manageable pieces
3. Embedding — Convert chunks into vector representations
4. Indexing — Store vectors in a vector database
5. Retrieval + Generation — At query time, retrieve relevant chunks and pass them to the LLM

Step 1: Choose Your Stack

Before writing a single line of code, pick your components. Here are the recommended defaults for 2026:

• Orchestration: LangChain or LlamaIndex (both excellent, LangChain has more ecosystem coverage, LlamaIndex has better built-in RAG abstractions)
• Embedding model: OpenAI text-embedding-3-small (best price/performance) or a local model via Ollama
• Vector store: Chroma (local, zero config) → Pinecone or Weaviate (production cloud)
• LLM: GPT-4o, Claude Sonnet, or Llama via Groq for cost savings

Step 2: Ingest Your Documents

LangChain has document loaders for almost every format. Here's a minimal example loading a directory of PDFs:

from langchain_community.document_loaders import PyPDFDirectoryLoader

loader = PyPDFDirectoryLoader("./docs/")
documents = loader.load()
print(f"Loaded {len(documents)} pages")

For web content, use WebBaseLoader. For Notion, there's a dedicated NotionDBLoader. LangChain covers 100+ source types.

Step 3: Chunk Strategically

This is where most tutorials cut corners — and where most RAG systems fail. The goal: chunks that are semantically coherent and fit within the LLM's useful attention range (roughly 200–800 tokens).

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,
    chunk_overlap=50,  # overlap preserves context across chunk boundaries
    separators=["\n\n", "\n", ".", " "]
)
chunks = splitter.split_documents(documents)
print(f"Created {len(chunks)} chunks")

Chunking mistakes to avoid:

• Chunks too large (>1000 tokens) — dilutes relevance during retrieval
• Zero overlap — loses context at boundaries
• Splitting in the middle of code blocks or tables — breaks semantic coherence

Step 4: Embed and Index

Now convert chunks to vectors and store them. Using Chroma for local development:

from langchain_openai import OpenAIEmbeddings
from langchain_chroma import Chroma

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./chroma_db"
)
print("Index built and persisted.")

For production, swap Chroma with Pinecone or Weaviate — the API is nearly identical thanks to LangChain's abstraction layer.

Step 5: Build the RAG Agent

Now wire up the retriever to an agent. Using LangChain's modern LCEL (LangChain Expression Language):

from langchain_openai import ChatOpenAI
from langchain.chains import create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain_core.prompts import ChatPromptTemplate

llm = ChatOpenAI(model="gpt-4o", temperature=0)
retriever = vectorstore.as_retriever(search_kwargs={"k": 5})

system_prompt = (
    "You are a helpful assistant. Use the following retrieved context "
    "to answer the question. If the context doesn't contain the answer, "
    "say 'I don't have information about that in my knowledge base.'\n\n"
    "Context:\n{context}"
)
prompt = ChatPromptTemplate.from_messages([
    ("system", system_prompt),
    ("human", "{input}"),
])

question_answer_chain = create_stuff_documents_chain(llm, prompt)
rag_chain = create_retrieval_chain(retriever, question_answer_chain)

response = rag_chain.invoke({"input": "What is our Q1 revenue target?"})
print(response["answer"])

Step 6: Upgrade to an Agentic RAG

Basic RAG retrieves once and generates. An agentic RAG can decide when to retrieve, what to retrieve, and can re-retrieve if the first pass wasn't sufficient. Here's how to turn your retriever into an agent tool:

from langchain.tools.retriever import create_retriever_tool
from langchain.agents import create_tool_calling_agent, AgentExecutor

retriever_tool = create_retriever_tool(
    retriever,
    name="search_knowledge_base",
    description="Search the company knowledge base for relevant information. Use this for any question about internal policies, products, or documentation."
)

tools = [retriever_tool]
agent = create_tool_calling_agent(llm, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

result = agent_executor.invoke({"input": "Compare our Q1 and Q2 targets"})
print(result["output"])

The agent now decides whether to call the retriever (and how many times) based on the query complexity. For multi-hop questions requiring several lookups, this pattern dramatically outperforms naive RAG.

Advanced Techniques Worth Knowing

Hybrid Search

Combine dense (embedding) search with sparse (keyword/BM25) search. Dense search captures semantic meaning; sparse search catches exact term matches. Most production RAG systems use both. Pinecone and Weaviate support hybrid search natively.

Re-ranking

After retrieval, use a cross-encoder re-ranker (e.g., Cohere's Rerank API or a local BGE re-ranker) to reorder chunks by actual relevance to the query. This significantly improves answer quality for the same retrieval cost.

Metadata Filtering

Add metadata to your chunks (document type, date, author, department) and filter before retrieval. This is 10x more precise than semantic search alone for structured corpora.

Query Transformation

Have the LLM rewrite or expand the user's query before retrieval. Vague queries like "what was that thing about the budget?" become "Q3 2026 budget allocation and approval process." LangChain's MultiQueryRetriever does this automatically.

Evaluation: How to Know If It's Working

Don't skip evaluation. A RAG system that feels good in demos can fail badly on real queries. Use these metrics:

• Context Precision — Are the retrieved chunks actually relevant?
• Context Recall — Did we retrieve all the relevant chunks?
• Answer Faithfulness — Does the generated answer stay grounded in the retrieved context (no hallucination)?
• Answer Relevance — Does the answer actually address the question?

Tools like Ragas, LangSmith, and Langfuse automate these evaluations against a labeled test set. All three are indexed in AgDex.

Production Checklist

• ✅ Chunking strategy validated against your specific document types
• ✅ Embedding model chosen and costs estimated at scale
• ✅ Vector store with backup and index versioning
• ✅ Retrieval evaluation (precision + recall baselines)
• ✅ Re-ranking for queries where precision matters
• ✅ Observability with LangSmith or Langfuse (trace every retrieval and generation)
• ✅ Refresh pipeline for re-indexing updated documents

Tools Referenced in This Guide

All tools mentioned are indexed in the AgDex directory: LangChain, LlamaIndex, Chroma, Pinecone, Weaviate, Ragas, LangSmith, Langfuse, Groq.