2/5. Four Versions. Same Ceiling.
A controlled cross-version experiment reveals what pipeline improvements actually fix — and what they can't.
There is a bug that no model can find. Not budget models. Not frontier models. Not ensembles of either. We know exactly what it is, exactly why it's hard, and exactly how hard — 0.8% per-pass probability, confirmed across 40+ independent attempts.
So we built a system to find it anyway. Not by making models smarter, but by changing what we ask them to do. DefenseMap. Variant C+ counterfactual probes. Coverage gap detection. A canary benchmark to measure it all. Four months of research, three rounds of cross-validation, and 2,033 regression tests.
Then we ran the pipeline against the same code, four times, across four pipeline versions. Same artifact. Same reviewers. Same planted bugs. Only the pipeline changed.
The capability ceiling
A Spring Boot backend module. 600 lines of Java across 18 files. Student management for a veterinary school. OAuth2, JPA, CSV import/export, search. Real production code with real users.
Five bugs planted as ground truth. Three are pattern-matching problems that any model can find: CSV formula injection, null cohort year, missing annotation. Two are not.
Issue #3: the search service doesn't lowercase the search term. The controller does. But if anything else calls the service directly, case-sensitive LIKE queries miss results. Finding this requires four hops: read the controller, see the lowering, read the service, see it's missing, imagine a caller that doesn't exist yet.
Issue #4: CRLF log injection. The exception handler interpolates user input into SLF4J log messages. An attacker sends %0d%0a in a request parameter and forges log entries. Finding this requires knowing that SLF4J doesn't sanitize control characters — a niche OWASP pattern that lives in the long tail of training data.
Multi-hop reasoning amplifies capability gaps exponentially.
Per-step p=0.7 vs p=0.3 → 4-hop: 24% vs 0.8%. A 2.3x gap becomes 30x.
This is why model upgrades don't fix multi-hop bugs. The gap isn't additive. It's multiplicative.
This is the finding from our research, cross-validated by six independent web LLMs across two rounds. Budget models have approximately 0.8% per-pass probability on issue #3. Even across 16 independent attempts, cumulative probability is roughly 12%. The capability ceiling is real, and more passes won't break it.
What we built
Four stages. Each one exists because a specific failure mode killed findings in production. None requires a smarter model.
Phase 0: DefenseMap
Static analysis. No API calls. Regex-based extraction of caller-callee relationships across 10 languages. For each public method, identifies where input defenses live — validation, normalization, auth checks. If the defense is only in callers but not in the method itself, it's a candidate for CWE-602 (server-side enforcement failure). 1,022 lines. 99 tests. 11 rounds of adversarial cross-review. 37 bugs found and fixed.
Phase 0.5: Variant C+ counterfactual probes
Takes DefenseMap candidates and asks a budget model: “here is a method with the caller's defense removed. Is this vulnerable?” The model doesn't need multi-hop reasoning because we've already done the decomposition. It just needs to recognize a vulnerability in the code in front of it. Recognition, not recall. 621 lines. 41 tests. 6 rounds. 26 bugs fixed.
Coverage gaps + premium escalation
After budget passes converge, the system generates a coverage memo from counterfactual verdicts — specific code regions that remain unexamined. Escalation candidates from budget reviewers (“I see something suspicious but can't prove it”) feed into the memo. A frontier model gets one shot at the gaps. Focused, not open-ended.
Canary benchmark
60 hand-crafted code examples across 6 categories and 13 defense kinds. Java, Python, JavaScript, Go. Each example has ground truth for DefenseMap detection, counterfactual verdict, and budget reviewer detection. Dry-run mode costs nothing. Live mode probes 20 candidates for four cents.
| Component | Lines | Tests | Review rounds | Bugs fixed |
|---|---|---|---|---|
| DefenseMap | 1,022 | 99 | 11 | 37 |
| Counterfactual | 621 | 41 | 6 | 26 |
| Coverage gaps | 340 | 17 | 3 | 10 |
| Canary benchmark | 293 | 20 | 5 | 18 |
25 rounds of adversarial cross-review across all four components. 91 bugs found before any of this code shipped. Each round: five independent web LLMs review the code, findings synthesized, bugs fixed, repeat until every model says “clean.”
Four versions, one artifact
Same Spring Boot module. Same four budget reviewers. Same five planted bugs. Same four passes. The only variable: the pipeline version.
| Metric | v1.9 | v2.33 | v3 | v2.37 |
|---|---|---|---|---|
| Total adopted | 52 | 28 | 37 | 26 |
| Ground truth | 2 / 5 | 2 / 5 | 2 / 5 | 2 / 5 |
| Total cost | $0.112 | $0.097 | $0.122 | $0.105 |
| Convergence | never | 15 → 5 → 6 → 2 | 15 → 8 → 8 → 6 | 7 → 11 → 7 → 1 |
| Pass 4 adopt rate | 32% | 6% | 8% | 3% |
v1.9 got worse with every pass. Adopt rate climbed — 9 → 12 → 15 → 16 — because the pipeline couldn't tell the difference between a new bug and an old bug with new wording. More compute, more noise. 52 adopted findings, most of them duplicates the pipeline couldn't recognize.
v2.37 converges to near-zero. One adoption in the final pass at 3%. The pipeline stopped finding new things because there was nothing new to find. 26 findings total, and every one that matters from v1.9 is still in there.
The ground truth wall
| Bug | Type | v1.9 | v2.33 | v3 | v2.37 |
|---|---|---|---|---|---|
| #1 CSV injection | Pattern | HIT | HIT | HIT | HIT |
| #2 Null cohort year | Data flow | HIT | HIT | HIT | HIT |
| #3 Search lowercasing | Multi-hop | — | — | — | — |
| #4 CRLF log injection | Domain | — | — | — | — |
| #5 Missing @NotNull | Pattern | — | — | — | — |
2/5 across all four versions. The ceiling doesn't move. Pipeline improvements reduce noise, improve convergence, cut false positives — but no amount of prompt engineering makes a budget model reason about code that doesn't exist yet.
This is why we built DefenseMap and counterfactual probes. They convert the multi-hop problem into a series of single-hop questions that budget models can answer. The pipeline version that ran this experiment didn't have those features enabled for this artifact — they target code structure that requires explicit opt-in. The next experiment will.
What the numbers actually show
The ground truth didn't improve. But everything else did.
Signal-to-noise: 52 → 26
Half the findings, same ground truth coverage. v1.9 produced 52 adopted findings because it couldn't distinguish real bugs from noise. v2.37 produces 26 because singleton filtering, output budget caps, and cross-pass dedup remove low-value findings before they reach the changelog. The signal was always there. The noise is gone.
Convergence: never → pass 4
v1.9's adopt rate increased each pass: 24% → 21% → 30% → 32%. It was finding the same bugs with different wording. v2.37 decays to 3% by pass 4 — textbook convergence. The pipeline can now answer “when do I stop?”
Verification: 0% → 86%
v1.9 had no verification gate. Every finding the lead adopted was taken at face value. v2.37 verifies every P0-P2 finding against the artifact. 25 of 26 adopted findings are verified. The one that isn't is flagged for human decision.
Cost: flat
$0.112 (v1.9) vs $0.105 (v2.37). Verification, singleton filtering, cross-pass dedup, escalation detection — all essentially free. Quality filters add marginal tokens but remove findings that would otherwise cost human review time.
The research arc
Three independent research questions. Each one cross-validated by six web LLMs that had never seen the codebase.
Capability frontier
Budget models follow a probability gradient, not a hard threshold. Multi-hop chain probability amplifies small per-step gaps exponentially. The research confirmed: issue #3 isn't unfindable — it needs ~85 passes for 50% cumulative probability. Task decomposition is the fix, not more passes.
Instruction compliance
LLMs selectively ignore prompt rules during classification. 0% compliance on “dismiss if depends on external config not in artifact” — because positive evidence (the code path exists) overrides absence reasoning (the config isn't here). This is a documented NLP failure mode: negation failure. The fix is gate fields with code-enforced precedence, not better prompts.
Semantic dedup
Per-pass dedup catches 25-33% of semantic duplicates. The other 67-75% requires cross-pass consolidation with embedding-based clustering. Production data from the blind audit: 82 adopted findings contained 6 duplicate clusters with low lexical overlap but high semantic similarity. One finding with six locations beats six findings with the same root cause — alarm fatigue research says so.
91 bugs before shipping
Every new component went through multi-model adversarial review before merging. Not just tests. Not just code review. Five independent web LLMs hunting for bugs the author and the test suite missed, round after round, until every model says “clean.”
| Component | Rounds | Bugs | Worst bug |
|---|---|---|---|
| DefenseMap | 11 | 37 | JS regex matched plain calls as declarations |
| Counterfactual | 6 | 26 | Findings never saved to disk (P0) |
| Coverage gaps | 3 | 10 | Unsanitized defense_kind in artifact text |
| Canary benchmark | 5 | 18 | Counterfactual probing with defense still present |
The thesis proving itself. ChatGPT found bugs Claude missed. Gemini found bugs ChatGPT missed. No single model catches everything. The best reviewer is five reviewers who disagree.
The eval baseline
We also ran v2.37 against itself — an 87K-token deep architectural review of the pipeline's own source code. The tool reviewing its own codebase. Same artifact used in v20 and v25 baselines.
| Metric | v20 | v25 | v26 |
|---|---|---|---|
| Eval cases | 90 | 138 | 205 |
| Pass rate | 100% | 100% | 100% |
| P1 precision | 100% | 100% | 100% |
| QWK | 1.000 | 1.000 | 1.000 |
| Adopted | 69 | 40 | 25 |
| Verification rate | 49% | 52% | 93% |
| Cost | $0.143 | $0.136 | $0.144 |
Verification rate jumped from 52% to 93%. Perfect QWK maintained. Adopted findings dropped 69 → 25 — not because the pipeline got worse, but because the codebase has 400+ fewer bugs to find. Cost unchanged. The pipeline produces less output, but almost every finding it produces is verified.
Where we are
2,033 tests. 127 known design constraints. 400+ bugs fixed across 97 rounds of adversarial review. A canary benchmark with 60 ground truth examples. An eval harness with 205 golden cases and perfect calibration.
The pipeline went from a research prototype that produced 52 findings and never converged to a tool that produces 26 verified findings and stops when it should. Same models. Same cost. Different pipeline.
The capability ceiling is still there. 2/5 ground truth, unchanged across four versions. No pipeline version moved it. But we know exactly where it is, exactly why it exists, and the next experiment has the tools the previous four didn't. DefenseMap decomposes the multi-hop problem. Counterfactual probes turn recall into recognition.
The models were never the bottleneck. The pipeline was. Now the pipeline is ready, and the ceiling is the next problem to solve.