Same Code. Same Models. 15 Versions Later.
A controlled re-run of our blind audit reveals what pipeline improvements actually matter.
Five days ago, we ran our multi-model review pipeline against a production Spring Boot module. It found 82 findings. It took hours to triage. More than half were wrong.
We ranked #1 in raw detection — catching bugs no other model found. And dead last in practical quality. The output was unusable.
So we fixed the pipeline. Not the findings — the machine that produces them. Then we ran it again. Same artifact. Same reviewers. Same lead model. The only difference: fifteen versions of pipeline improvements.
82 → 28 findings. $0.20 → $0.10. 3.5/5 → 4/5 ground truth hits.
The experiment
The setup is identical to the original blind audit. A Spring Boot backend module — ~600 lines of Java across 18 files. Student management for a veterinary school. OAuth2, JPA, CSV import/export, search. Real production code with real users.
A separate CC session planted five hidden bugs as ground truth. Issue #1 (CSV injection) through Issue #5 (missing @NotNull). Two of those bugs — search term lowercasing and CRLF log injection — had never been found by any budget model at any price point across eleven passes.
Four budget reviewers. Four passes. One premium frontier-model pass. Total cost: ten cents.
Before and after
| Metric | v1.9 (7 passes) | v2.33 (4+premium) | Delta |
|---|---|---|---|
| Total adopted | 82 | 28 | -66% |
| Ground truth | 3.5 / 5 | 4 / 5 | +0.5 |
| Total cost | $0.20 | $0.10 | -50% |
| Convergence | never | 15 → 5 → 6 → 2 | clean decay |
| Duplicates | ~34 | 10 | -71% |
| Triageable? | no | yes | — |
Less output. Better detection. Half the cost. The pipeline went from “raw detection king but untriageable” to “strong detection and actionable output.”
The bug that broke the ceiling
Issue #4: CRLF log injection. An attacker sends %0d%0a in a request parameter. The exception handler interpolates it into an SLF4J log message. The attacker forges log entries.
No budget model had ever found this. Not in seven passes. Not in eleven. It requires knowing that SLF4J doesn't sanitize control characters, that CRLF injection is a specific OWASP category, and that the attack path runs from user input through exception handling to log output.
In v2.29, we added a security cheat sheet to the reviewer prompt. Six lines. It lists rare-but-critical vulnerability patterns matched to the detected tech stack. For Java: Log injection / CRLF: untrusted text written to logs.
Pass 4, Grok found it. Verified by the post-triage gate. Rated P2.
Recall is hard. Recognition is easy.
The cheat sheet doesn't teach the model. It primes the pattern in working memory.
This is the difference between asking a model “find all security bugs” and asking “look for this specific pattern in this code.” Budget models have the knowledge. They lack the retrieval. Six lines of prompt engineering turned a zero-percent detection rate into a confirmed find.
The one that got away
Issue #3: the search service doesn't lowercase the search term. The controller does. But if anything else calls the service directly, case-sensitive LIKE queries miss results.
Finding this requires imagining a caller that doesn't exist yet. Four hops: read the controller, see the lowering, read the service, see it's missing, imagine a future caller, conclude the defense is in the wrong place. Per our research, budget models have ~0.8% per-pass probability on this. Even across 16 independent attempts, cumulative probability is ~12%.
The premium pass (GPT-4o) didn't find it either. It flagged SQL injection in the same method — right neighborhood, wrong bug. The escalation system correctly directed premium attention toward the search service, but the model diagnosed the wrong vulnerability.
This is the capability ceiling. Not a pipeline problem. Not a prompt problem. A reasoning depth problem that no model at any price point reliably solves. The research says so. The data confirms it.
What actually moved the needle
Fifteen versions of pipeline improvements between the two runs. Here is what mattered, ranked by impact:
Output budget + singleton filter
Cap adopted findings per pass. Filter unverified singletons. This alone took 82 → ~35. The biggest quality-of-life change. The pipeline had signal buried in noise — the budget models were finding real bugs, but 50 false positives made them invisible.
Security cheat sheet
Six lines of prompt. One ground truth hit. The highest impact-per-token feature in the entire pipeline. Transforms recall into recognition for niche attack patterns.
Cross-pass dedup
The original run oscillated: 29 → 7 → 14 → 5 → 3 → 11 → 13. Passes 6 and 7 re-discovered old bugs. Cross-pass dedup killed the oscillation. The decay is now monotonic: 15 → 5 → 6 → 2.
Escalation hybrid
Budget reviewers can now say “I see something suspicious but can't prove it.” Three escalation candidates fed into the premium pass coverage memo, directing the frontier model toward MeController auth and CSV seeder concurrency. The system captures intuition that would otherwise vanish.
The hidden insight
Gemini Flash-Lite went from 16% signal to 39% signal. Same model. Same prompts. Same artifact. The model didn't get better. The pipeline got better at extracting what was already there.
This is the thesis playing out in data. Budget models have signal. They find real bugs. They also produce enormous amounts of noise. The original pipeline couldn't separate the two. The new pipeline can.
The models were never the bottleneck. The pipeline was.
When Gemini-lite fires 25 findings and 4 are real, the question is not “why is Gemini-lite so noisy?” The question is “why is the triage pipeline adopting the other 21?” Fix the pipeline, and the model you thought was your weakest reviewer becomes your most improved.
Convergence
The original run never converged. Seven passes. Bayesian P(zero next pass) stuck at 0.1%. The pipeline kept re-discovering the same bugs with different wording.
The v2.33 run shows clean decay: 15 → 5 → 6 → 2. Chapman capture-recapture estimates 95% coverage after four passes. The Bayesian gate reached 3% — still below the 80% threshold, but trending in the right direction. Two more passes would likely cross it.
The pipeline can now answer the question “when do I stop?” Not perfectly. But better than “never.”
What this means
Five days. Fifteen versions. One controlled experiment. The pipeline went from a research prototype that produced unusable output to a tool that finds 80% of planted bugs at ten cents and produces a reviewable report.
The one bug it can't find requires reasoning about code that doesn't exist yet. That's not a pipeline problem. That's a frontier of AI capability. Every model, at every price point, misses it. Including the premium ones.
Everything else — the pattern matching, the data flow tracing, the security recognition, the severity calibration — budget models can do it. They just need a pipeline that knows how to listen.
Budget models have signal. Build a pipeline that extracts it.