Building Production RAG: Hybrid Vector Search & Reranking Architecture

Building a production-ready Retrieval-Augmented Generation (RAG) assistant involves much more than embedding text into a vector database and querying top-k results. Naive semantic vector search frequently fails on exact keyword matches, SKU numbers, or technical terminology.

In this architecture guide, we break down the exact hybrid search pipeline we engineer at dyta.ai to achieve sub-400ms query latency and 99.2% retrieval accuracy.

1. Why Pure Semantic Vector Search Fails

Dense vector embeddings (e.g., OpenAI text-embedding-3-large or Cohere embed-v3) map semantic meaning into high-dimensional vector spaces. While excellent for general context matching, pure dense search suffers from two critical flaws:

1. Exact Identifier Blindness: Queries containing specific part numbers (e.g., Part-9942-X), API methods (get_user_account()), or proper nouns often match unrelated documents that share high semantic proximity.
2. Short Query Noise: Queries with only 1–3 words generate diffuse embedding vectors that retrieve low-relevance chunks.

2. The 4-Stage Production RAG Pipeline

To solve semantic drift and exact-match failures, we implement a 4-stage hybrid retrieval architecture:

[ User Query ]
      │
      ├──> 1. PII Redaction & Sanitization Filter
      │
      ├──> 2. Dual Retriever Pipeline
      │      ├── Dense Vector Search (Qdrant HNSW)
      │      └── Sparse Keyword Search (BM25 / SPLADE)
      │
      ├──> 3. Reciprocal Rank Fusion (RRF) & Cross-Encoder Reranker
      │
      └──> 4. LLM Context Injection & Stream Response (< 400ms)

Stage 1: PII Redaction & Sanitization

Before any query payload reaches an external LLM or vector database, incoming text passes through automated PII filters (masking emails, social security numbers, and proprietary tokens).

Stage 2: Dual Hybrid Retrieval (Dense + Sparse)

We execute parallel searches in Qdrant:

• Dense HNSW Vector Search: Retrieves 25 chunks based on cosine similarity.
• Sparse BM25 Keyword Search: Retrieves 25 chunks based on exact token matches.

Stage 3: Reciprocal Rank Fusion (RRF) & Reranking

Results from both retrievers are merged using Reciprocal Rank Fusion (RRF). The top 30 merged candidates pass through a lightweight Cross-Encoder Reranker model (e.g., bge-reranker-large), selecting the top 5 highest-relevance contexts.

Stage 4: Streaming LLM Generation

The top 5 reranked context chunks are injected into the system prompt with strict zero-hallucination boundaries. Response generation streams back to the client UI with query latencies consistently below 400 milliseconds.

Key Security Takeaways

In production environments, vector databases must enforce isolated tenant namespaces and role-based access control (RBAC) so users never retrieve document chunks above their security clearance.