RAG Ingestion, Hybrid Retrieval, Reranking and Citations

The retrieval chain

1. Ingest: preserve document ID, version, title, text and access metadata.
2. Chunk: follow paragraphs or headings and retain character offsets.
3. Index: build lexical and semantic representations.
4. Retrieve: produce independent ranked candidate lists.
5. Fuse and rerank: combine complementary evidence, then score a small shortlist more carefully.
6. Cite: reconstruct the exact excerpt from the immutable source version.

Fixed-size chunks are a baseline. A policy clause should not be split simply because a token counter reaches an arbitrary number. Evaluate chunk size and overlap on real questions, and track index size and latency beside recall.

Run an end-to-end local retriever

Save as hybrid_rag_demo.py. To keep the lab runnable on an ordinary computer, latent semantic analysis provides the dense-style ranking. A production system may replace it with reviewed embeddings while preserving the same provenance and evaluation contracts.

python -m pip install scikit-learn numpy
python hybrid_rag_demo.py

# hybrid_rag_demo.py
from dataclasses import dataclass
import re
import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.decomposition import TruncatedSVD
from sklearn.preprocessing import normalize

@dataclass(frozen=True)
class Document:
    doc_id: str
    version: str
    title: str
    text: str

@dataclass(frozen=True)
class Chunk:
    chunk_id: str
    doc_id: str
    version: str
    start: int
    end: int
    text: str

def chunk_document(doc):
    chunks = []
    for number, match in enumerate(re.finditer(r"[^\n]+", doc.text), 1):
        chunks.append(Chunk(f"{doc.doc_id}#{number}", doc.doc_id,
                            doc.version, match.start(), match.end(),
                            match.group().strip()))
    return chunks

def ranked(scores, chunks):
    return [chunks[i].chunk_id for i in np.argsort(scores)[::-1]]

def rrf(rankings, k=60):
    scores = {}
    for ranking in rankings:
        for position, chunk_id in enumerate(ranking, 1):
            scores[chunk_id] = scores.get(chunk_id, 0.0) + 1 / (k + position)
    return scores

docs = [
    Document("refund", "2026-08-01", "Refund policy",
             "Customers may request a refund within 14 days.\n"
             "Digital downloads are reviewed when delivery fails."),
    Document("support", "2026-08-03", "Support guide",
             "Open a support ticket with the order number.\n"
             "A specialist reviews delivery failures within two working days."),
]
chunks = [c for doc in docs for c in chunk_document(doc)]
texts = [c.text for c in chunks]
query = "What is the refund request deadline?"

lex = TfidfVectorizer(ngram_range=(1, 2), stop_words="english")
lex_matrix = lex.fit_transform(texts)
lex_scores = (lex_matrix @ lex.transform([query]).T).toarray().ravel()

semantic_vectorizer = TfidfVectorizer(stop_words="english")
all_matrix = semantic_vectorizer.fit_transform(texts + [query])
dimensions = min(2, all_matrix.shape[1] - 1)
svd = TruncatedSVD(n_components=dimensions, random_state=7)
latent = normalize(svd.fit_transform(all_matrix))
semantic_scores = latent[:-1] @ latent[-1]

fused = rrf([ranked(lex_scores, chunks), ranked(semantic_scores, chunks)])
by_id = {c.chunk_id: c for c in chunks}
candidate_ids = sorted(fused, key=fused.get, reverse=True)[:3]

# Transparent reranker: reward query-term overlap plus fused rank score.
terms = {t for t in re.findall(r"[a-z]+", query.lower()) if len(t) > 3}
def rerank_score(chunk_id):
    words = set(re.findall(r"[a-z]+", by_id[chunk_id].text.lower()))
    return fused[chunk_id] + 0.02 * len(terms & words)

winner = max(candidate_ids, key=rerank_score)
chunk = by_id[winner]
source = next(d for d in docs if d.doc_id == chunk.doc_id)
excerpt = source.text[chunk.start:chunk.end]
assert excerpt == chunk.text
print("winner", winner)
print("citation", source.title, source.version, chunk.start, chunk.end)
print("excerpt", excerpt)

Expected result

The refund-policy chunk should win and the citation should reproduce “Customers may request a refund within 14 days.” Exact scores are implementation details. The important assertions are stable identity, stored version and byte-for-byte excerpt reconstruction.

What to replace in production

Swap latent semantic analysis for a domain-tested embedding model, and replace the transparent overlap reranker with a reviewed cross-encoder if it improves a fixed benchmark. Keep lexical retrieval for exact names, codes and uncommon terms. Filter access before scoring candidates, not after answer generation.

Measure recall at k for retrieval, mean reciprocal rank when one passage should lead, citation precision, citation completeness and answer faithfulness. An answer layer should abstain when evidence is absent or contradictory.

Keep ingestion reproducible

Store the original file hash, parser version, detected language and chunking configuration with the index build. If a PDF parser changes its reading order, the same offsets may point to different text even when the filename is unchanged. Rebuild into a new index version and keep the previous version until citations are verified. Deduplicate exact files before chunking, but preserve legitimate repeated clauses when their document context changes the meaning.

Reranking should operate on a bounded candidate set. Log stable IDs and scores for evaluation, not entire private documents. When no candidate clears the reviewed threshold, return no evidence rather than forcing the highest-scoring weak match into the answer.

Inspect one query trace

For every benchmark failure, preserve the parsed query, lexical ranking, semantic ranking, fused shortlist, reranker score and final citation. This trace shows whether the defect began during ingestion, retrieval or answer composition. Remove private text before sharing the trace outside its authorized environment.

Debug each stage separately

Knowledge check

Primary references

• Original RAG paper
• PostgreSQL full-text search controls
• pgvector documentation
• OWASP Top 10 for LLM Applications