The method

Each module was documented as a standalone theory file: what it does, why it exists, the algorithms, the math, and tagged research connections. That file became the intake prompt for four independent web LLMs — Kimi, DeepSeek, Claude, ChatGPT — each asked the same four targeted research questions per module.

When all four converge on the same conclusion independently, the signal is strong. When they diverge, the disagreement is the finding.

We distilled each session into a single research file with consensus findings, comparison matrices, canonical references, and actionable decisions. Twelve modules, twelve research files, zero code changes. Theory first.

What we got wrong

The code worked. The names we gave to what the code was doing were imprecise. All four LLMs independently corrected the same three labels:

None of these changed a line of code. They changed how we reason about the code — which determines what we build next.

The Delphi boundary

Sweep's triage splits into two API calls with a hard boundary between them. Call 1: the lead reviews the artifact cold, producing its own findings. No reviewer output visible. Call 2: the lead classifies all findings — its own and the reviewers' — into adopt, dismiss, or duplicate.

We originally framed this as System 2 (slow deliberation) followed by System 1 (fast classification). The research says the better framework is the Delphi method (Dalkey & Helmer, 1963): independent expert judgment in round one, controlled exposure to peer evidence in round two.

The hard API boundary isn't a design preference — it's a necessity. Research from ICLR 2025 confirms that current LLMs fundamentally lack architectural instruction/data separation. A single prompt saying “ignore Section B until completing Section A” does not produce genuine compartmentalization. Attention leaks.

Simple debiasing techniques — Chain-of-Thought, reflection, “ignore the anchor” prompting — are insufficient. The only effective mitigation is structural separation. Jury deliberation research confirms: jurors who form independent opinions before hearing others maintain more stable positions and are less susceptible to majority dominance.

Condorcet, not multi-start

Sweep sends code to diverse models in parallel — different architectures, different training data, different blind spots. We originally called this multi-start optimization. The research points to a more precise framework: Condorcet's Jury Theorem (1785) plus ensemble diversity theory (Breiman, 2001).

Condorcet: if N independent voters each have probability p > 0.5 of being correct, majority correctness approaches 1 as N grows. Breiman (Random Forests): ensemble error is bounded by ρ̄(1−s²)/s² where ρ̄ is mean correlation and s is individual strength.

Three caveats the math doesn't advertise:

Independence is the critical assumption. LLMs sharing training data create correlated failures. The effective number of independent reviewers may be smaller than N.

Detection probability varies by defect type. A subtle race condition with p = 0.05 across all models: (0.95)⁵ ≈ 0.77 — still 77% miss probability.

The defect space is open-ended. No ensemble can guarantee exhaustion of an unbounded space. Convergence means “below detection threshold,” not “no defects remain.”

The strongest counterargument came from 2025 research: Self-MoA (ensembling one model's outputs via resampling) outperformed Mixed-MoA (different models) by 6.6% on AlpacaEval. But production data tells a different story — every individual frontier model failed where the ensemble succeeded. The finding that one reviewer contributed 0% unique signal validated Krogh-Vedelsby: a model that always agrees with others adds no value to the ensemble.

Viola-Jones in the normalization layer

Raw LLM output is unpredictable. The normalization pipeline uses a two-tier cascade classifier — the same pattern Viola and Jones used for real-time face detection in 2001. Cheap filter first, expensive model second.

Research confirmed a nuance: structured output (JSON schema enforcement) can degrade reasoning quality. Tam et al. (EMNLP 2024) proved format restrictions hurt reasoning tasks but improve classification tasks. Finding extraction is classification-adjacent — structured output is net positive for the use case.

The highest-value improvement identified: per-finding fallthrough. Parse each finding independently through regex. Only LLM-normalize individual failures instead of the entire response. All four LLMs recommended this independently.

Parse at the boundary, trust internally

The type system uses Python TypedDicts with zero runtime validation. The research identified one principle that changes where validation belongs: “Parse, Don't Validate” (Alexis King, 2019). Parsers return typed evidence of validity. Validators return booleans.

Two trust boundaries exist: LLM output entering the normalizer, and persisted JSON loaded from disk. Everything between is trusted. Adding schema validation at these two entry points catches malformed data before it propagates silently — the LLM equivalent of parameterized SQL queries.

LLM output (untrusted)
    ↓ validate against schema
Finding (trusted dict)
    ↓ internal pipeline — no validation
Classification (trusted dict)
    ↓ persist to disk

The research also identified a Python type system upgrade: PEP 655 (Required/NotRequired) replaces blanket total=False with per-field optionality. Some fields are always present; others arrive later as data moves through the pipeline — a practical form of what type theory calls row polymorphism (Wand, 1987; Rémy, 1989).

Shadow paging, not write-ahead logging

Every state mutation writes to a temp file in the same directory, then atomically swaps via os.replace(). We called this “simplified WAL.” The research corrected us: this is the shadow paging pattern — write the new version to a fresh location and atomically swap the pointer. WAL implies redo/undo logging with replay capability.

The critical gap all four LLMs identified: rename atomicity ≠ rename durability. The pattern is process-crash-safe (SIGKILL, exception, OOM-kill) on local filesystems. It is not power-loss-safe without fsync() before the rename. For a developer CLI tool, the risk is acceptable. For production, it isn't.

The strongest infrastructure improvement identified: flock() over PID-based lockfiles. Kernel-managed, auto-cleanup on process death, no stale detection needed, no PID recycling risk. Eliminates the entire retry-with-cleanup codepath. Same complexity.

Convergence as a stopping heuristic

The pipeline iterates until adopted = 0 for two consecutive passes. We called this fixed-point iteration. The research says it's closer to a stopping heuristic than a formal fixed point — the operator (reviewers + triage) is stochastic, nonstationary, and path-dependent.

The closest analogy from the literature: fuzzing campaign saturation. Green Fuzzing (ISSTA 2023) stops campaigns when coverage of predicted-vulnerable functions stalls, achieving 17–40% time savings with under 6% miss rate. The optimal criterion is not “zero findings” but “marginal cost exceeds expected value of the next discovery.”

The Wald sequential analysis framework (1947) formalizes this: stop when the evidence is strong enough for a decision. At $0.014 per pass, the extra pass is nearly free — the conservative heuristic is defensible. But severity-weighted convergence would be more principled: a zero-P0/P1 pass is more indicative than a zero-P3 pass.

Three behavior modes, validated

Sweep documents three reviewer behavior modes. The research validated all three against 2024–2025 literature:

Mode 3 is the only acceptable behavior. It's enforced structurally — stateless prompts, no prior findings, fresh context per pass — not through instructions. “Please don't anchor on these findings” does not work. The bias operates below the level of conscious prompt-following.

Persist the state, not the conversation.

Prompt injection as a probabilistic boundary

Every untrusted input is wrapped in nonce-tagged security fences. Random 6-character hex nonces prevent tag forgery. Both opening and closing tags are escaped inside the content. The model is explicitly told: this is data, not instructions.

The research confirmed: delimiter-based defenses are probabilistic, not guaranteed. LLMs lack architectural instruction/data separation (ICLR 2025). The defense frontier has moved to instruction hierarchy training (OpenAI, 2024) and dual-LLM architectures (Google CaMeL, 2025).

For Sweep's threat model — reviewing your own source code, not adversarial user input — the prompt-level defense is sufficient. The deterministic parser acts as an accidental second layer: any injection that breaks the output format fails parsing. Defense in depth, discovered after the fact.

FrugalGPT — the architecture we accidentally discovered

Stanford published FrugalGPT in 2023 (Chen, Zaharia & Zou): up to 98% cost reduction while matching GPT-4 quality through three strategies — prompt adaptation, LLM approximation, and model cascading. Route easy tasks to cheap models, escalate only when needed.

Sweep independently converged on the same pattern. Cheap reviewers (DeepSeek, Gemini Flash-Lite, GPT-5 Nano) handle the broad sweep. An expensive lead (Grok 4 Fast) handles the hard part — classification under ambiguity. The normalization layer adds another cascade: deterministic regex handles 80% for zero tokens, LLM fallback handles the rest.

RouteLLM (Berkeley, 2024) extended this: 85% cost reduction at 95% quality using learned routing on preference data. De Koninck et al. (ICML 2025) showed even cascading between similar-sized models delivers savings — contradicting the assumption that you need a small→large hierarchy.

The pricing trajectory makes this increasingly absurd. Epoch AI data: GPT-4-class capabilities cost $60/M tokens in 2023, $0.75/M in 2026. A 98% reduction in two years. LLM inference costs are declining faster than Moore's Law during the microprocessor revolution — roughly 10x per year.

This is why we deferred batch mode. The engineering cost of a second code path exceeds the savings at $0.014/pass. Prompt caching (52% savings, already implemented) and model routing are higher-ROI levers that apply to every call, not just batch-eligible ones.

Alarm fatigue and the P1 threshold

Severity miscalibration erodes trust faster than misclassification. The research here is unambiguous, and it changed how we think about Sweep's output quality.

In medical settings, 72–99% of clinical alarms are false. Clinicians disable or ignore 23–44% of alarms as a result. In static analysis, high false positive rates are the number one reason teams abandon tools. The pattern is universal: when the signal is noisy, humans learn to ignore it.

Greptile's production data crystallized this. Users tolerated 2% of factually incorrect comments. What drove them away was 79% of comments being “nits” — technically correct but not valuable. The problem wasn't accuracy. It was calibration.

The asymmetry matters: systematic bias (always calling P1) is more trust-destroying than random error, because developers learn the signal is unreliable and mentally downgrade all future P1s — even after calibration improves. Trust recovery after calibration failure is much slower than trust establishment.

All four LLMs converged on the same recommendation:

This is why the post-triage verification gate exists. It's not just a quality check — it's a trust-preservation mechanism. Every false P1 that reaches the developer is a withdrawal from a finite trust account.

Counting what you can't see

The hardest question in automated review: how many bugs did we miss? Sweep's eval harness measures pipeline quality — normalization accuracy, classification correctness. But it doesn't measure review quality: did we find real bugs? How many are still hiding?

The breakthrough came from ecology. The Lincoln-Peterson estimator (1930) was designed to count fish in a lake: tag a sample, release them, catch another sample, count how many are tagged. The overlap between samples estimates the total population.

Briand et al. (2006) adapted this to software defects: independent inspectors as “capture sessions,” shared findings as “recaptures.” The math is simple:

If Claude finds 15 issues, Gemini finds 12, and 8 overlap:

N̂ = (15 × 12) / 8 = 22.5

Found:    19 unique (15 + 12 − 8)
Estimated: 22.5 total
Implied:   ~3.5 missed by both

The critical assumption is independence — the same assumption Condorcet requires and Sweep's stateless architecture enforces. No known-issues injection, no prior findings visible, Mode 3 behavior. The design that makes the ensemble work is the same design that makes the estimator valid.

Dawid & Skene (1979) extended this with EM-based error rate estimation: learn each reviewer's confusion matrix without ground truth. Applied to Sweep, this would reveal which reviewers systematically miss which categories — security-focused models missing performance issues, systems-focused models missing UX bugs.

The data already exists. Every triage pass records which reviewer found which finding and whether it was adopted. Thirty lines of stdlib Python. Zero new API calls. Coverage confidence per run, for free.

What we deliberately didn't build

Research validates not building things as strongly as it validates building them. Six proposals were killed by cross-validated consensus. Two were killed unanimously — 6/6 reviewers in the meta-review agreed.

The dedup module is an interesting case. It's wired in but effectively dormant — the 0.85 Jaccard threshold catches zero pairs in production because reviewers use completely different vocabulary for the same bug. All meaningful dedup happens during triage, where the lead has semantic understanding. Research (He et al., 2024, SoftDedup) argues near-duplicates contain valuable signal about reviewer agreement — the lead may actually benefit from seeing them. The module stays as cheap insurance. When it fires, something interesting happened.

The strongest counterargument

“Rethinking Mixture-of-Agents” (2025) tested whether model diversity actually matters. Self-MoA — ensembling outputs from only the single best model via temperature-varied resampling — outperformed Mixed-MoA (diverse models) by 6.6% on AlpacaEval 2.0.

If true for code review, Sweep's entire multi-model architecture is unnecessary overhead. You'd just run Claude five times with different temperatures.

Production data disagrees. Across Sweep's runs, every individual frontier model failed where the ensemble succeeded. DeepSeek found systems bugs the others missed. Gemini caught web framework issues no one else flagged. Grok surfaced edge cases in auth flows that Claude and GPT both overlooked.

The resolution may be domain-specific. AlpacaEval measures conversational quality — a domain where a single strong model resampled at different temperatures explores the response space adequately. Code review is different: architectural diversity in training data correlates with different blind spots. A model trained heavily on systems code sees race conditions a web-focused model misses, and vice versa.

The question isn't settled. But the practical test is simple: measure pairwise finding overlap between reviewer models. If two reviewers always find the same things, one is redundant. If they rarely overlap, both contribute genuine independent signal. LLM-TOPLA (EMNLP 2024) shows diversity is measurable and optimizable — not just a heuristic.

The fix rate problem

Google's most important finding on static analysis (Sadowski et al., CACM 2018): 84% of bugs filed from analysis tool output were never fixed. Integration into code review proved more effective. Compiler integration was most effective of all.

Compiler errors (block build)       → highest fix rate
Code review integration (inline PR)  → high fix rate
Dashboard / bug tracker              → findings get lost

Every successful LLM review tool — Qodo PR-Agent, CodeRabbit, AI Review — has converged on direct PR comments, not SARIF dashboards. The delivery mechanism matters more than the format. Inline comments with reasoning transparency build trust. Dashboard findings without explanation erode it.

Developer trust in AI is actively declining: from 43% (2024) to 33% (2025). AI-assisted completion is 55.8% faster in controlled experiments but 19% slower in real-world studies due to verification overhead — the “productivity paradox.”

Sweep currently exports SARIF for IDE/CI integration. The research says PR comment integration is the higher-impact path for developer-facing experience. SARIF remains valuable as a secondary channel for CI/CD gates, compliance, and aggregation. But the primary integration — the one that determines whether findings get fixed — needs to be inline, conversational, and transparent about its reasoning.

What the research says to build next

Nine improvements. Five are under 100 lines. Two are config changes. One requires a new module. All are research-backed by at least three independent LLMs citing the same literature.

Theory before iteration

Most tools iterate endlessly — run more passes, add more features, fix more edge cases. We stopped and asked: what does the literature say about what we already built?

The answer surprised us. The architecture is sound — Delphi boundary, ensemble diversity, cascade normalization, stateless reviewer protocol — all validated by independent research stretching from Condorcet (1785) to Self-MoA (2025). But the labels were wrong. The evaluation gap was solvable with ecology's species-counting math. Seven improvements were hiding in 30-line patches. And the most important lesson wasn't technical at all: severity calibration matters more than detection accuracy, because trust erodes faster than it builds.

Forty-eight research sessions. Twelve modules. Six terminology corrections. Nine actionable improvements. Six killed proposals. One codebase that didn't change a line of code — but whose next evolution is now grounded in three centuries of converging theory.

The same problem Sweep was built to solve.