Chunking, embeddings, pgvector vs Qdrant vs Pinecone, hybrid search, re-ranking, citations, evals and permissions: how RAG works and how it fails.
Retrieval-augmented generation (RAG) gives a language model access to your own documents at question time: you split content into chunks, embed them, store them in a searchable index, retrieve the best matches for each query, and pass those to the model with instructions to answer from them and cite them. It is not the same as "memory", although the two are often confused. This guide explains each stage, the real choices at each step, and the failure modes we see when RAG systems reach production.
RAG versus agent memory
A model has no persistent knowledge of your business. RAG addresses the knowledge problem: what does our handbook, product catalog or ticket history say? Memory addresses the state problem: what did this user tell the agent last week, and what has it already done? The two overlap technically, since memory is often implemented by writing summaries or facts into a store and retrieving them later, but the requirements differ.
| RAG (knowledge) | Agent memory (state) | |
|---|---|---|
| Source | Documents, wikis, tickets, databases | Conversations, user preferences, task history |
| Written by | Ingestion pipeline, controlled | The agent or user, at runtime |
| Main risk | Stale or wrong content, missing retrieval | Wrong or outdated facts being saved, privacy |
| Scope | Shared, permissioned by document | Per user or per session |
Frameworks such as LangChain document short-term memory (conversation state within a thread) separately from retrieval, which is a useful distinction to copy.
Step 1: Ingestion and chunking
Retrieval quality is decided largely before any model is called. Extract clean text (tables and PDFs are the usual trouble), keep metadata (source, title, section, date, owner, access group), then chunk.
- Chunk by structure, not by a fixed character count alone. Split on headings and paragraphs, then cap size. A chunk should make sense by itself.
- Add overlap sparingly. A small overlap protects sentences cut at boundaries; a large one bloats the index and returns near-duplicates.
- Prepend context. Including the document title and section heading in each chunk's text helps both embeddings and readers of the citation.
- Tune on your data. There is no universally correct chunk size. Test a few and measure retrieval, not intuition.
Step 2: Embeddings
An embedding model converts text into a vector so that similar meanings land near each other. Choose a model that supports your languages (English, Urdu and Roman Urdu behave differently), keep the same model for indexing and querying, and record the model version, since changing it means re-embedding everything. Larger is not always better: latency, cost and dimension count matter in production.
Step 3: Vector stores
| Option | Good fit | Trade-offs |
|---|---|---|
| pgvector (Postgres) | You already run Postgres and want vectors next to relational data, transactions and existing access control | You manage indexing (HNSW or IVFFlat) and tuning yourself; very large or very high-throughput workloads need care |
| Qdrant | Dedicated engine with rich payload filtering and support for hybrid retrieval; self-hosted or managed | Another system to operate and keep in sync |
| Pinecone | Fully managed service, minimal operations, namespaces for tenant separation | Vendor dependency and ongoing cost; data lives outside your database |
For most small and mid-sized business projects we start with pgvector because it keeps the stack simple, and move to a dedicated store only when scale or filtering needs justify it. Check each product's current documentation before deciding, as features change quickly.
Step 4: Hybrid search and re-ranking
Pure vector search is good at meaning and weak at exact tokens: product codes, invoice numbers, names, error messages. Keyword search (BM25 or Postgres full-text search) is the reverse. Hybrid search runs both and merges the results, commonly with reciprocal rank fusion. It is one of the highest-value upgrades you can make.
Re-ranking then takes the top 20 to 50 candidates and scores each against the query with a cross-encoder or a reranking API, keeping only the best handful for the prompt. Retrieval is cheap and approximate; re-ranking is slower but sharper. Add it when relevant passages appear in the candidates but not at the top.
Step 5: Generation with citations
- Instruct the model to answer only from the supplied context and to say so when the answer is not there.
- Label each chunk with an ID and require the answer to cite those IDs, then render them as links to the source page or document section.
- Validate citations in code: reject any ID that was not in the retrieved set.
- Let users see the sources. Trust comes from checkability.
Step 6: Evaluation
Without evaluation you are guessing. Build a small golden set of real questions with the expected source passage and answer, and measure two things separately.
- Retrieval: did the right chunk appear in the top k? (recall at k, and rank of the first relevant hit)
- Generation: is the answer faithful to the retrieved text, complete, and properly cited? Use human review first, then an LLM judge calibrated against those human labels.
Rerun the set whenever you change chunking, embeddings, prompts or the model. Tracing tools such as LangSmith help inspect what was retrieved for a failing question.
Freshness and permissions
Freshness. Decide how content changes reach the index: incremental sync keyed on document hash or modified date, deletions handled explicitly, and a stored last-indexed timestamp. Old versions of a policy quietly outranking the new one is a classic failure.
Permissions. Never rely on the prompt to hide restricted content. Store an access group or ACL on every chunk and filter at retrieval time using the requesting user's identity, so restricted text never reaches the model. Multi-tenant systems should separate tenants by namespace, collection or row-level security.
Common failure modes
- Answer exists but is not retrieved: poor chunking, weak embeddings for the language, missing keyword search.
- Retrieved but ignored: too many chunks, relevant passage buried mid-context, no re-ranking.
- Confident wrong answer: no "I don't know" path, stale content, contradicting documents.
- Tables and scanned PDFs lost: extraction dropped the structure.
- Prompt injection in documents: retrieved text containing instructions. Treat retrieved content as data, not commands.
- Silent drift: embedding model or content changes with no regression tests.
Frequently asked questions
Is RAG better than fine-tuning?
For factual, changing company knowledge, RAG is usually the better first choice: it updates instantly, supports citations and respects permissions. Fine-tuning suits style, format and narrow behaviours.
Do I need a dedicated vector database?
Not always. pgvector inside Postgres is enough for many business applications. Consider Qdrant or Pinecone when scale, filtering complexity or operations preferences call for it.
What chunk size should I use?
Start with structure-aware chunks of a few paragraphs, then test alternatives against your own evaluation set.
How do I stop the AI making things up?
Ground answers in retrieved text, require citations, allow "not found" answers, and measure faithfulness in your evaluation set.
Building a RAG assistant?
We design and build retrieval systems and AI agents with evaluation, permissions and citations built in. See our services and portfolio, or contact us to discuss your data.
