Build a RAG Pipeline with LangChain and ChromaDB

Large language models are powerful. But they only know what they were trained on. They cannot see your private documents, your notes, or your company wiki.

Retrieval-Augmented Generation, or RAG, fixes this. It fetches relevant text from your own data and feeds it to the model as context.

In this guide, you will build a full RAG pipeline with LangChain and ChromaDB. We cover loading, splitting, embedding, storing, retrieving, and generating answers.

What Is a RAG Pipeline?

A RAG pipeline has two phases. The first is indexing. You load documents, split them into chunks, turn each chunk into a vector, and store the vectors.

The second phase is retrieval and generation. A user asks a question. The system embeds the question, finds similar chunks, and passes them to the model.

This keeps answers grounded in your data. It also reduces hallucinations, because the model reads real context before replying.

Why LangChain and ChromaDB?

LangChain gives you ready-made components. Loaders, text splitters, embedders, vector store wrappers, and prompt templates all plug together.

ChromaDB is a lightweight vector database. It runs in memory or on disk. You can start with zero infrastructure and scale later.

Together, they form a clean stack. You write little glue code and focus on your data and prompts.

Step 1: Install the Dependencies

Start with a fresh virtual environment. Install LangChain, the OpenAI integration, and ChromaDB.

pip install langchain langchain-openai langchain-chroma chromadb python-dotenv

Store your API key in a .env file. Never hardcode secrets in source files.

OPENAI_API_KEY=sk-your-key-here

If you plan to call tools or structured outputs later, it helps to understand OpenAI function calling in Python. That knowledge pairs well with RAG agents.

Step 2: Load Your Documents

LangChain ships with many document loaders. We will use a simple text loader here. You can swap in PDF, HTML, or database loaders later.

from dotenv import load_dotenv
from langchain_community.document_loaders import TextLoader

load_dotenv()  # load OPENAI_API_KEY from .env

# Load a plain text file as a list of Document objects
loader = TextLoader("data/handbook.txt", encoding="utf-8")
documents = loader.load()

print(len(documents))
print(documents[0].page_content[:120])

1
Welcome to the company handbook. This document explains our

Each Document holds page content plus metadata. Metadata is useful for filtering later, such as by source file or section.

Step 3: Split Text into Chunks

Models and embeddings work best on small, focused pieces of text. Splitting is therefore a core step.

The RecursiveCharacterTextSplitter tries paragraph breaks first, then sentences, then characters. It keeps chunks readable.

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=500,      # max characters per chunk
    chunk_overlap=50,    # overlap keeps context across borders
)

chunks = splitter.split_documents(documents)
print(f"Created {len(chunks)} chunks")

Created 42 chunks

Chunk size is a trade-off. Small chunks retrieve precisely but lose context. Large chunks keep context but dilute similarity scores.

Start with 300 to 800 characters. Measure retrieval quality with real questions, then tune.

If you want to inspect chunk quality at the token level, see how tokenization with NLTK in Python works. Token counts often differ from character counts.

Step 4: Create Embeddings

Embeddings convert text into vectors. Similar meanings end up close together in vector space.

We will use OpenAI embeddings. The class OpenAIEmbeddings wraps the API call.

from langchain_openai import OpenAIEmbeddings

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

# Quick sanity check: embed one short string
vector = embeddings.embed_query("What is the refund policy?")
print(len(vector))

1536

The vector length depends on the model. Keep the same model for indexing and querying. Mixing models breaks similarity search.

Step 5: Store Vectors in ChromaDB

Now we persist the chunks and their vectors. Chroma.from_documents does both in one call.

from langchain_chroma import Chroma

vector_store = Chroma.from_documents(
    documents=chunks,
    embedding=embeddings,
    persist_directory="./chroma_db",  # saved to disk
    collection_name="handbook",
)

print(vector_store._collection.count())

42

On later runs, you do not need to re-index. Load the store with Chroma and the same embedding function.

vector_store = Chroma(
    persist_directory="./chroma_db",
    embedding_function=embeddings,
    collection_name="handbook",
)

Step 6: Retrieve Relevant Chunks

Retrieval is a similarity search. The method similarity_search returns the top matching documents.

results = vector_store.similarity_search("How do I request a refund?", k=3)

for doc in results:
    print(doc.metadata.get("source"), "->", doc.page_content[:80])

data/handbook.txt -> Refunds are processed within 5 business days...
data/handbook.txt -> To request a refund, email support with your order...
data/handbook.txt -> Our return window is 30 days from delivery...

You can also use as_retriever to wrap the store in a retriever object. That object plugs directly into chains.

retriever = vector_store.as_retriever(search_kwargs={"k": 3})

For better recall, try MMR search. It balances relevance with diversity and avoids near-duplicate chunks.

Step 7: Generate Answers with a Prompt

Now we connect retrieval to the model. A prompt template tells the model to answer only from context.

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

prompt = ChatPromptTemplate.from_template(
    """Answer the question using only the context below.
If the answer is not in the context, say you do not know.

Context:
{context}

Question: {question}
"""
)

def format_docs(docs):
    return "\n\n".join(d.page_content for d in docs)

rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

answer = rag_chain.invoke("How long do refunds take?")
print(answer)

Refunds are processed within 5 business days after approval.

That is a working RAG pipeline. Load, split, embed, store, retrieve, and generate. Each step is replaceable.

When you move beyond prototypes, you will likely add external tools and data sources. For example, you might pull records from a database or stream events from a broker.

If your context comes from databases, learn how to query an Oracle database from Python and feed those rows into your pipeline.

Best Practices and Common Pitfalls

Keep chunk metadata rich. Store source, page, and section. You can filter on it later and show citations to users.

Evaluate retrieval separately. Log the retrieved chunks for test questions. If wrong chunks appear, fix splitting or embeddings before touching the prompt.

Instruct the model to stay grounded. Tell it to say "I do not know" when context is missing. This cuts hallucinations sharply.

Watch costs. Embedding large corpora costs money. Batch your calls and cache vectors. Re-index only changed documents.

Version your collections. Use a new collection name when you change the embedding model. Old vectors are not comparable to new ones.

Conclusion

You now have a complete RAG pipeline built with LangChain and ChromaDB. It loads documents, splits them, embeds them, stores vectors, retrieves matches, and generates grounded answers.

The pattern is simple and modular. Swap the loader for PDFs. Swap ChromaDB for another vector store. Swap the model for a local one.

Start small with one document and a few test questions. Measure retrieval quality first. Then improve prompts and chunking. A clean pipeline beats a complex one every time.