VectorLoom: A Privacy-First RAG Assistant for PDFs

VectorLoom answers questions about your PDFs without sending a single page to the cloud. It runs the full retrieval pipeline and the language model on your own machine, whether you feed it a research paper, a textbook chapter, or an API reference. This post covers what it does, the hardest call I made building it, and the numbers I actually measured while auditing it.

What It Does and Who It Is For

VectorLoom is a question-answering system for PDF documents. You upload a file through a small Streamlit interface, the backend indexes it, and then you ask questions in plain English. Answers stream back token by token, with the source passages shown alongside so you can check where each claim came from.

It is built for people who need to read a specific document closely rather than search the open web: a student going through a textbook, a developer working through an API reference, or someone reading a research paper who wants the methodology section summarized without reading the whole thing.

The distinguishing constraint is that nothing leaves the machine. The embedding model, the vector index, and the language model all run locally. That matters for anyone reading something they cannot upload to a third-party API — a draft, a contract, an unpublished paper.

The Problem That Made Me Build It

Two separate problems pushed me toward this design. The first was privacy: most RAG tools assume you're fine sending your document to someone else's server. That assumption doesn't hold for a lot of real documents people actually want to ask questions about.

The second was quality. A naive RAG pipeline — fixed-size chunks, one embedding model, top-k cosine search — tends to produce shallow answers. A 1000-character chunk boundary can land in the middle of a methodology description, and the model answers from half of it. A single dense retriever also misses exact-term queries: if someone asks about a specific function name or a specific statistic, pure semantic search sometimes ranks a vaguely-related paragraph above the one containing the literal term.

Architecture and How It Works

The pipeline has five stages: extract, detect, chunk, index, and answer.

On upload, pypdf extracts raw text from the document. A lightweight classifier looks at the first few thousand characters for markers like DOI:, Abstract, Chapter, or fenced code blocks, and labels the document as a research paper, a textbook, technical documentation, or general text.

Chunking is where the document type actually gets used. Each type gets a parent chunk size tuned to its structure — 1500 characters for research papers split along section headers, 1200 for textbooks split along chapter markers, 800 for technical docs with code blocks kept intact. Every parent is then split again into smaller 250-character child chunks. The children get embedded and searched; the parents get handed to the language model as context. Searching on the child gives precise matches, but answering from the parent gives the model enough surrounding text to reason properly.

# config.yaml — retrieval section
retrieval:
  top_k: 24 # candidates pulled from each retriever, per query
  final_k: 8 # parents handed to the LLM after reranking
  rerank_candidates: 16 # fused candidates the cross-encoder actually scores
  reranker_model: 'cross-encoder/ms-marco-MiniLM-L-6-v2'

Retrieval itself is hybrid. A dense search over FAISS (using sentence-transformers/all-MiniLM-L6-v2, 384 dimensions) finds semantically similar children. A sparse BM25 search finds children sharing literal terms with the query. The two ranked lists are merged with Reciprocal Rank Fusion, and the merged candidates are rescored by a cross-encoder that reads the query and each passage together before the final eight are picked.

Generation runs on Qwen/Qwen2.5-1.5B-Instruct, streamed with HuggingFace's TextIteratorStreamer so the FastAPI backend can push tokens to the frontend over Server-Sent Events as they're produced, instead of waiting for the full answer.

The Hardest Technical Decision

The hardest call was whether hybrid retrieval was worth the added complexity over a single dense vector search, and if so, how to combine two retrievers whose scores live on completely different scales.

Cosine similarity from the embedding model and a BM25 score are not comparable numbers — one is bounded and the other isn't, and there's no principled way to average them directly. The two options were to calibrate and weight them, or to fuse by rank instead of by score.

# Reciprocal Rank Fusion — rank-based, no score calibration needed
k_rrf = 60
for rank, chunk_id in enumerate(dense_results):
    fused[chunk_id] = fused.get(chunk_id, 0.0) + 1.0 / (k_rrf + rank + 1)
for rank, chunk_id in enumerate(sparse_results):
    fused[chunk_id] = fused.get(chunk_id, 0.0) + 1.0 / (k_rrf + rank + 1)

I went with Reciprocal Rank Fusion. It only cares about where each candidate ranked in its own list, not its raw score, so there's nothing to calibrate between a cosine similarity and a BM25 score. The tradeoff is an extra retrieval pass and a cross-encoder rerank step on every query, which adds latency. I capped the reranker's input at 16 fused candidates specifically to keep that cost bounded, since a cross-encoder scores each query-passage pair individually and its cost scales linearly with how many pairs you hand it.

What I Measured

Some real numbers from running this system rather than estimating it.

The test suite — 13 tests covering config loading, chunking, indexing, deduplication, and retrieval — runs in 100.47 seconds on CPU, most of that spent loading the embedding and cross-encoder models once per test session.

On a small three-chunk test index, a query about study methodology produced a cross-encoder score of 3.91 for the correct passage against -9.74 and -10.86 for the two unrelated ones. That gap is the reranker doing real work — RRF alone had already put the right passage first, but the score gap shows how confidently the cross-encoder distinguishes a genuine match from noise.

End-to-end latency is dominated by generation, not retrieval: embedding a query and searching FAISS together take on the order of 50 to 60 milliseconds, while generating a full answer on CPU takes several seconds regardless of how fast retrieval was.

What I Would Do Differently

I would normalize embeddings for cosine similarity from the very first version. FAISS's IndexFlatL2 measures Euclidean distance, and MiniLM embeddings are trained for cosine similarity — those only agree when the vectors are unit-normalized. I didn't do that at first, which meant the ranking was subtly off in a way that's easy to miss in casual testing and only shows up as slightly-worse-than-expected retrieval quality.

I would also pick a language model with a large context window before writing the retrieval pipeline, not after. An early version defaulted to a model with a 2,048-token context, and once the retrieval pipeline was pulling in eight parent chunks, the prompt silently overflowed that window. Switching the default model earlier would have avoided debugging what looked like a retrieval bug but was actually a context limit.

Finally, I would add per-document metadata to each chunk from day one. Right now, uploading a second PDF adds its chunks to the same index without recording which document they came from, which makes multi-document use awkward later.

Conclusion

VectorLoom is a working answer to a narrow problem: read this specific document, locally, and let me ask it real questions without sacrificing retrieval quality for the sake of privacy. Hybrid retrieval and parent-child chunking were the two decisions that mattered most for answer quality; running everything on CPU is the main constraint left to work around.

The version running today is usable for single-document question answering. The obvious next step is GPU-backed inference to cut generation latency, and metadata tracking to make multi-document use as solid as the single-document case already is.