Building Verity-OS: A Multi-Agent Fact-Verification Engine With No API Key Required

Verity-OS is a research engine that spawns a tree of agents to answer a query, watches them work in real time on a canvas, and — for fact-checking — verifies its own draft before showing it to you. It runs out of the box on free search with a mock model, so you can see the whole pipeline before you spend a cent on an API key. This post covers why I built it, how the pieces fit together, and the numbers I got from actually running it.

What it does and who it's for

Verity-OS takes a query and a mode, then runs a small tree of agents against the open web. There are seven modes:

• Fact Check — verifies a claim and returns a Toulmin-structured argument (claim, grounds, warrant, rebuttal) with a numeric confidence score.
• Deep Curation — finds niche, expert-grade sources: GitHub repos, arXiv papers, Reddit/HN threads, newsletters.
• Debate — steel-mans both sides of a topic and gives a verdict.
• News Intelligence — pulls market and tech news from outlets like Reuters, Bloomberg, TechCrunch, and FT, with impact analysis.
• Tech Stack Advisor — recommends a frontend/backend/database/infra stack for a described use case.
• Person / Company Intelligence — researches public figures and companies from public sources.
• Learning Path Builder — produces a structured, three-phase path for learning a skill or technology.

You pick a mode, type a query, and watch the agent tree build live on a React Flow canvas as results stream in over server-sent events. When it's done, a mode-specific report renders on the side.

It's aimed at two kinds of people: anyone who wants to check a claim or research a topic without wading through blended, unsourced chatbot answers, and developers curious how a multi-agent LangGraph pipeline is actually wired — planner, parallel explorers, an auditor, a synthesizer, all visible instead of hidden behind a spinner.

Nothing requires a paid key to start. It ships with a MockLLM stub and DuckDuckGo search, both free, so the full pipeline runs the moment you clone it.

The problem that made me build it

Most search-and-summarize tools give you one paragraph and no way to see how it got there. You can't tell which sub-questions were asked, which sources fed which sentence, or whether the system checked its own claim before showing it to you.

That last part bothered me most. The obvious way to add "verification" to a RAG pipeline is to hand the model its own draft and ask "is this correct?" — but a model shown its own output tends to just agree with itself. That's not verification, it's a rubber stamp. I wanted the auditor step to actually be adversarial: question the draft without seeing it, then answer those questions from raw evidence only.

I also wanted zero friction to try it. A lot of side projects in this space die at "sign up for three API keys before you see anything work." DuckDuckGo has no key and no rate-limit wall for casual use, so that became the default search backend, with Brave and Tavily as opt-in upgrades.

Architecture / how it works

The backend is a LangGraph state machine served over FastAPI; the frontend is Next.js with React Flow rendering the agent tree as it builds.

User Query + Mode
       │
       ▼
┌─────────────┐   parallel   ┌──────────────┐
│   Planner   │ ────────────▶│  Explorer ×N │  DuckDuckGo (free) · Tavily · Brave
│  (mode-aware│              └──────┬───────┘
│   queries)  │                     │ asyncio.gather fan-out
└─────────────┘                     ▼
                           ┌─────────────────┐
                           │ Context Pruning │  85% window cap
                           └────────┬────────┘
                                    ▼
                           ┌─────────────────┐
                           │  CoVe Auditor   │  Factored Chain-of-Verification
                           │  (fact_check    │  — answers verified without seeing
                           │   mode only)    │    the draft
                           └────────┬────────┘
                                    ▼
                           ┌─────────────────┐
                           │  Synthesizer    │  Mode-specific structured output
                           └────────┬────────┘
                                    │  SSE stream (live, frame-by-frame)
                                    ▼
                     React Flow Canvas + Report Panel

The shared state is a single TypedDict that every node reads and partially updates:

class AgentState(TypedDict):
    query:                  str
    mode:                   str        # fact_check | deep_curation | debate | ...
    plan:                   List[str]
    research_results:       Annotated[Dict[str, str], operator.ior]  # merge reducer
    draft_report:           str
    verification_questions: List[str]
    verified_report:        str
    calibration_score:      float
    messages:               Annotated[List[BaseMessage], operator.add]  # append reducer

The operator.ior reducer on research_results matters more than it looks. Explorer nodes run in parallel via asyncio.gather, and each one returns a partial dict. Without the merge reducer, the last node to finish would overwrite everyone else's results instead of combining with them.

Each mode reshapes the Planner and Synthesizer differently. Fact Check generates three general research queries and runs a full auditor pass. Deep Curation fires five site-scoped queries (site:github.com, site:arxiv.org, site:reddit.com, site:news.ycombinator.com, plus a newsletter search) and skips the auditor entirely — there's no claim to verify in a list of links. Debate generates four queries explicitly prefixed FOR: / AGAINST:, and the synthesizer splits results by that prefix rather than calling the model again to classify them.

The frontend consumes the SSE stream frame by frame:

1. data: {"planner":   {"plan": [...]}}          → fan-out Explorer nodes
2. data: {"explorer":  {"research_results": {}}} → Auditor node
3. data: {"context_pruning": {...}}              → (no UI change)
4. data: {"auditor_verify": {"calibration_score": 0.95}} → Synthesizer node
5. data: {"synthesizer": {"verified_report": "..."}}     → Report panel
6. data: [DONE]                                  → setIsLoading(false)

Each frame appends a typed node to the React Flow canvas as it arrives, so the graph you see is the graph that actually ran — not an animation of one.

The hardest technical decision and why I chose what I chose

The hardest call was how to implement verification in the Fact Check mode, and specifically whether to let the auditor see the draft it's checking.

The simple version — pass the draft to the model and ask "does the evidence support this?" — is one LLM call, one round trip, done. I tried reasoning through this path first because it's the obvious one. The problem is seed bias: a model conditioned on its own prior output is primed to defend it. It's checking its homework by re-reading its own handwriting.

I went with Factored Chain-of-Verification instead, split into two calls that never share context:

Step 1 — Generate questions
  Input:  draft_report only
  Output: 3 verification questions
  (the model hasn't seen the evidence — it questions the draft)

Step 2 — Answer questions
  Input:  raw research_results only   ← draft is EXCLUDED
  Output: verification answers
  (answers are grounded in evidence, not in what was already written)

The synthesizer then builds the Toulmin argument from the step-2 output, not from the original draft. If a claim has no grounds, the Toulmin circuit breaker returns MISSING_EVIDENCE instead of letting the synthesizer paper over the gap.

The cost is real: two extra LLM calls instead of zero, and more latency per fact-check than a single-pass answer. I decided that trade-off was worth it because the entire point of a fact-checking mode is trustworthiness. A faster wrong answer is worse than a slower one that's actually been checked. Every other mode (Deep Curation, Debate, and the rest) skips the auditor entirely, since there's nothing to verify in a curated list or a two-sided argument split — so the extra cost is scoped to the one mode where it earns its keep.

What I measured (real numbers, not round ones)

Rather than describe the eval, I ran it. backend/evals/run_facts_search.py drives the full LangGraph workflow against three quantization questions (QLoRA, INT4 vs INT8, GPTQ/AWQ benchmarks) and scores recall-based F1 against a keyword list, with a target of 80% average F1:

[Case 1/3] What is QLoRA and how does it reduce VRAM?
  Calibration Score : 95.1%
  F1 Factuality     : 0.0%

[Case 2/3] Explain INT4 vs INT8 LLM inference performance trade-offs
  Calibration Score : 95.1%
  F1 Factuality     : 0.0%

[Case 3/3] What are the latest GPTQ and AWQ quantization benchmarks for 2025?
  Calibration Score : 95.1%
  F1 Factuality     : 0.0%

Average F1 Factuality Score : 0.0%

That's the actual output, MockLLM and DuckDuckGo, no API key. The calibration score of 95.1% is deterministic — it's exp(mean(logprobs) / length_penalty) over MockLLM's fixed stub log-probabilities [-0.01, -0.05, -0.02, -0.10, -0.04, -0.08], so it comes out identical on every run. The F1 score is honestly zero: MockLLM's canned response text doesn't contain domain-specific keywords like nf4 or gptq, so the recall-based scorer correctly finds nothing to credit. That's the eval doing its job, not a bug — it's the gap you'd close by pointing ANTHROPIC_API_KEY or OPENAI_API_KEY at a real model.

Two other measured constants worth noting:

• Context pruning caps at 5,200 words, which is 85% of an assumed ~6,100-word context budget, leaving headroom for the system prompt and output tokens. It's a greedy first-fit: earlier, higher-priority sub-query results are kept whole; later ones get truncated or dropped first.
• The fuzzy quote resolver requires 85% character-level similarity (difflib.SequenceMatcher ratio ≥ 0.85) before it will attribute a quote to a source span. Lower than that and offsets drift too easily from encoding or whitespace differences; the resolver would rather return no match than a wrong one.

Explorer fan-out is the other place the parallel design pays off directly: five sub-queries run concurrently via asyncio.gather instead of sequentially, so wall-clock time tracks the slowest single search, not the sum of all of them.

What I'd do differently

The F1 scorer fixes precision at 1.0, which collapses F1 to a function of recall alone — it can tell you whether the required facts showed up, but not whether the system also asserted anything false. Building a real "set of wrong claims per query" dataset to score precision properly is on the list; it just wasn't the first thing that needed to exist.

Wiring in a real LLM today means editing a line in backend/main.py directly rather than setting an environment variable. That's fine for a personal project but is the first thing I'd change if anyone else needed to run this against their own model — a config-driven provider switch instead of a source edit.

State is in-memory per process right now, so a restart loses every session. Redis-backed persistent state is on the roadmap and is the change I'd prioritize before adding any more modes, since more modes just means more state worth not losing.

Conclusion

Verity-OS started from wanting to see a research agent think instead of just answer, and wanting the verification step to be a real check rather than a model nodding at its own draft. Seven modes, a LangGraph state machine with proper parallel-merge reducers, a factored verification step that never shows the auditor its own output, and a canvas that renders the actual execution graph as it runs — that's the shape it settled into. The eval numbers above are unpolished on purpose: 95.1% calibration and 0.0% F1 with the mock model is exactly what an honest, unconnected pipeline should report, and it's a more useful number than a cherry-picked one.