June 6, 2026 · 8 min read

Generative AI Data Architecture: Building the Enterprise RAG Pipeline

Designing an enterprise RAG pipeline: ingestion and chunking, hybrid dense + BM25 retrieval with RRF, cross-encoder reranking, tenant isolation and evaluation.

By Husain Abbas, Enterprise Architect & former CTO

The Problem

Enterprises want LLMs to answer questions from their own contracts, runbooks, tickets and policies. There are three obvious ways to get private knowledge into a model, and two of them break down at enterprise scale:

Fine-tuning is good for teaching format, tone and domain vocabulary. It is a poor way to store facts that change weekly. It can’t cite sources, and it can’t enforce access control: once a document has shaped the weights, every user of the model can potentially get at it.
Stuffing everything into a long context window works for a handful of documents. It doesn’t work for a document estate measured in millions of pages. Cost and latency grow with input size, and models use long contexts unevenly. Liu et al. (2023), “Lost in the Middle” showed accuracy dropping when the relevant passage sits in the middle of a long input rather than near the start or end.
Retrieval-Augmented Generation (RAG), introduced by Lewis et al. (2020), keeps knowledge in an external index and retrieves only the passages relevant to each query. Knowledge can be updated without retraining, every answer can cite its sources, and access control can be applied at retrieval time, per user.

RAG is the default architecture for enterprise knowledge assistants. Building a simple demo is easy. The hard part is retrieval quality, permissions and keeping the index fresh in production.

The Architectural Solution

A production RAG system is a data pipeline first and an LLM application second. It has two paths.

Offline: ingestion and indexing

Parse with layout awareness. PDFs, slides and spreadsheets need parsers that keep headings, lists and table structure. Plain text extraction flattens tables into meaningless token sequences.
Chunk on structure, not just size. Split on headings and paragraphs, and use a size cap only as a fallback. Store the document title, section path, source URL, version and access-control attributes with every chunk.
Add context to each chunk. A chunk that says “revenue grew 3% over the previous quarter” is not retrievable if it doesn’t name the company or period. Anthropic’s Contextual Retrieval work prepends a short, generated description of where the chunk sits in its document before embedding and BM25 indexing. In their benchmarks, contextual embeddings plus contextual BM25 cut the top-20 retrieval failure rate by 49%, and adding reranking cut it by 67% (from 5.7% to 1.9%). Those numbers are averaged over their own test sets, so treat them as directional and measure on your corpus.
Index two representations. Store a dense embedding (semantic similarity) and a sparse lexical vector (BM25) for each chunk, plus payload indexes on tenant and ACL fields.

Online: retrieval and generation

Hybrid retrieval. Run dense and BM25 queries in parallel, both constrained by the user’s tenant and permissions. Dense search finds paraphrases. BM25 finds exact tokens such as part numbers, error codes and employee IDs, which embeddings often blur.
Fuse with Reciprocal Rank Fusion (RRF). Cormack et al. (2009) showed that combining result lists by rank (score = Σ 1/(k + rank), with k = 60) reliably beats the individual rankers. It also avoids having to normalise incompatible score scales.
Rerank. A cross-encoder reads the query and each candidate together and scores relevance directly. It is more accurate than comparing embeddings but too slow to run over the whole corpus, so apply it to the top 20–50 fused candidates only.
Generate with citations. Pass the top few chunks, with source identifiers, and instruct the model to answer only from them and cite them. Return the citations to the user.

Architecture Diagram

Enterprise RAG pipeline: documents are parsed, chunked and enriched with metadata, then indexed as dense and BM25 vectors; at query time tenant-filtered hybrid retrieval is fused with RRF, reranked by a cross-encoder and passed to the LLM with citations

Core Implementation

The retriever below uses Qdrant’s hybrid Query API with two prefetches (dense and BM25) fused by RRF, then reranks with a Sentence Transformers cross-encoder. It assumes a collection with a 1,024-dimension named vector dense (the size of bge-large-en-v1.5), a sparse vector bm25 configured with the IDF modifier, and a tenant payload index on tenant_id.

import asyncio
import logging
from dataclasses import dataclass, field

from fastembed import SparseTextEmbedding
from qdrant_client import AsyncQdrantClient, models
from sentence_transformers import CrossEncoder, SentenceTransformer

logger = logging.getLogger(__name__)

# BGE v1.5 recommends this instruction for short-query -> passage retrieval (queries only).
BGE_QUERY_PREFIX = "Represent this sentence for searching relevant passages: "


@dataclass
class RetrievedChunk:
    id: str
    text: str
    metadata: dict = field(default_factory=dict)
    score: float = 0.0


class RetrievalError(RuntimeError):
    pass


class HybridRetriever:
    def __init__(self, client: AsyncQdrantClient, collection: str):
        self.client = client
        self.collection = collection
        self.dense = SentenceTransformer("BAAI/bge-large-en-v1.5")
        self.sparse = SparseTextEmbedding(model_name="Qdrant/bm25")
        self.reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")

    async def retrieve(
        self, query: str, tenant_id: str, candidates: int = 40, top_k: int = 5
    ) -> list[RetrievedChunk]:
        # Model inference is CPU/GPU-bound: keep it off the event loop.
        dense_vec = await asyncio.to_thread(
            self.dense.encode, BGE_QUERY_PREFIX + query, normalize_embeddings=True
        )
        sparse_vec = next(iter(self.sparse.query_embed(query)))

        # Tenant isolation is enforced inside the database query, in every branch.
        tenant = models.Filter(must=[
            models.FieldCondition(key="tenant_id", match=models.MatchValue(value=tenant_id))
        ])
        try:
            response = await self.client.query_points(
                collection_name=self.collection,
                prefetch=[
                    models.Prefetch(query=dense_vec.tolist(), using="dense",
                                    filter=tenant, limit=candidates),
                    models.Prefetch(
                        query=models.SparseVector(indices=sparse_vec.indices.tolist(),
                                                  values=sparse_vec.values.tolist()),
                        using="bm25", filter=tenant, limit=candidates),
                ],
                query=models.FusionQuery(fusion=models.Fusion.RRF),
                query_filter=tenant,
                limit=candidates,
                with_payload=True,
            )
        except Exception as exc:  # network, auth or schema errors from the vector store
            logger.exception("hybrid search failed", extra={"tenant_id": tenant_id})
            raise RetrievalError("vector search unavailable") from exc

        chunks = [
            RetrievedChunk(id=str(p.id), text=p.payload.get("text", ""), metadata=p.payload)
            for p in response.points if p.payload
        ]
        if not chunks:
            return []  # caller should answer "not found", not let the LLM guess

        scores = await asyncio.to_thread(self.reranker.predict, [(query, c.text) for c in chunks])
        for chunk, score in zip(chunks, scores):
            chunk.score = float(score)
        return sorted(chunks, key=lambda c: c.score, reverse=True)[:top_k]

Notes for production use:

Don’t log raw queries unless your retention policy allows it. Queries often contain PII and confidential context.
The filter appears on every prefetch and on the outer query. That’s redundant by design: a refactor that drops one filter shouldn’t cause a cross-tenant leak.
Swap components freely. The same shape works with pgvector plus PostgreSQL full-text search, OpenSearch hybrid queries, or a hosted reranker. The architecture matters more than the specific product.

Edge Cases & Limitations

Global questions defeat top-k retrieval. “What are the main risk themes across all 2025 audit reports?” can’t be answered from five chunks. Answering it means summarising across the whole corpus. Microsoft’s GraphRAG targets this kind of query-focused summarisation by building an entity graph and community summaries at index time. The trade-off is a much more expensive indexing step that is harder to keep current.
Tables and numbers. Chunking that ignores structure separates values from their row and column headers. Extract tables as Markdown or HTML, keep each table (or a header-plus-row group) as one chunk, and for analytical questions route to SQL over the source system instead of RAG.
Embedding model upgrades mean a full re-index. Vectors from different models are not comparable, so moving to a new embedding model means re-embedding everything. Plan it as a blue/green index: build the new collection alongside the old one, compare retrieval metrics on a fixed evaluation set, then switch traffic with an alias.
Permission drift. Applying ACLs at query time only works if the index’s ACL metadata is current. When a user loses access to a SharePoint folder, how long until retrieval reflects it? Drive ACL updates and deletions from change events (the same change-data-capture techniques described in zero-downtime migration patterns) and set an explicit freshness target.
Reranking latency and cost. Cross-encoder cost grows linearly with the number of candidates. Reranking 40 candidates with a small MiniLM model is cheap on a GPU and noticeable on a CPU, while large LLM-based rerankers can dominate end-to-end latency. Benchmark on your own hardware before choosing a candidate count.
Retrieved text is untrusted input. Documents can contain instructions (“ignore previous instructions and…”). If the RAG system can also call tools, apply the controls in enterprise guardrails for agentic AI.
When not to use RAG: when the corpus fits comfortably in the context window and changes rarely, or when the questions are really structured queries (counts, sums, joins) that belong in SQL or an API call.

Key Strategic Takeaways

Treat RAG as data engineering. Parsing, chunking, metadata and freshness determine answer quality more than the choice of LLM.
Use hybrid retrieval. Dense vectors handle paraphrase and BM25 handles exact identifiers. Fuse them with RRF instead of hand-tuned score weights.
Rerank a small candidate set. Cross-encoders improve precision, but measure the gain against the latency they add.
Enforce access in the query. Tenant and ACL filters belong in the vector store’s query predicates, backed by change-driven metadata updates, never in post-retrieval Python.
Measure retrieval separately. Track recall@k on a labelled question set so you can tell retrieval failures from generation failures.