Conviso AST in the OWASP Benchmark: 97.1% precision with 94% fewer false positives

We ran Conviso AST against the 2,740 test cases of OWASP Benchmark v1.2. It identified 1,118 real vulnerabilities and raised only 33 false alerts, 94% fewer than Semgrep OSS, which found the same vulnerabilities under the same conditions.

This article covers the numbers, how we measured them, where Conviso AST still falls short, and how AI-driven false positive analysis decides what reaches your team.

The SAST problem was never finding issues. It was trusting what gets found

Rule-based SAST scanners cover the entire codebase and are deterministic, but they don't understand how the application behaves. The result is volume: in many alerts, attacker-controlled data never reaches the vulnerable point. When most of the queue is noise, developers learn to ignore the scanner, and the tool stops doing its job.

AI models used on their own flip the problem. They reason well about the snippet in front of them, but they don't sweep the repository systematically, and their output varies from run to run. The benchmarks vendors published in 2026 show this pattern: high precision, low recall.

Conviso AST combines both approaches. The scanner guarantees coverage; the AI reviews every alert before it becomes work for someone. To find out whether that combination holds up, we tested it on a public dataset with a public answer key, and we're publishing the full method.

The results: 97 out of every 100 alerts are real vulnerabilities

Across the 2,740 cases, Conviso AST reached 97.1% precision and 79.0% recall, with an F1 score of 87.1%. For whoever handles triage, the difference is concrete: false alerts to review dropped from 552 (Semgrep OSS) to 33 on the same dataset.

Of the 2,258 alerts the scanner sent to the AI for review, no real vulnerability was dismissed, and 94.3% of the false alerts were removed. In 7 of the benchmark's 11 categories, detection exceeded 90%.

The false positive rate alone doesn't tell you whether a tool finds vulnerabilities: a tool that reports nothing has a rate of zero. That's why we always publish all four metrics together.

How we measured

We chose a public dataset with a public answer key so anyone can reproduce the test. No one at Conviso labeled any cases.

• Test set: OWASP Benchmark v1.2, an open project used as a reference by SAST vendors. It contains 2,740 executable Java servlets: 1,415 with a real vulnerability and 1,325 that look vulnerable but aren't exploitable. The cases span 11 categories, each tied to a CWE.
• Answer key: the expectedresults-1.2.csv file published by OWASP, which states for each case whether it's vulnerable and which CWE applies.
• Preparation: we removed comments, class names, and routes that give away the answer (for example, /sqli-00/), without changing a single line of code. This keeps the AI from leaning on hints planted in the benchmark itself.
• Conviso AST: conviso-ast scanner 4.0.7, with 1,154 rules, plus AI false positive analysis (Claude Opus 5.5), which receives the full file for each alert.
• Semgrep OSS: version 1.146.0 with the p/default ruleset, run by us on the same cases.
• Scoring: OWASP BenchmarkUtils criteria. A case counts as identified when the tool reports the category's CWE in the case's file. We ran 5 execution rounds in October 2026.

With the answer key, each case falls into one of four groups:

From these four numbers:

• Precision = TP / (TP + FP) = 1,118 / 1,151 = 97.1%
• Recall = TP / (TP + FN) = 1,118 / 1,415 = 79.0%
• F1 = 2·TP / (2·TP + FP + FN) = 2,236 / 2,566 = 87.1%
• False positive rate = FP / (FP + TN) = 33 / 1,325 = 2.5%

Comparison: same recall, 94% less noise

We ran Semgrep OSS ourselves on the same cases, under the same conditions. Both tools reached the same recall, 79.0%. The difference is what comes with it: 552 false alerts from Semgrep OSS, 33 from Conviso AST.

Cycode published an OWASP Benchmark result, but only the false positive rate, with no recall. Without recall, there's no way to know how many real vulnerabilities were missed.

What other vendors published in 2026

Several vendors released AI SAST benchmarks this year. Each used its own codebase and its own scoring criteria, so the table below shows where the market stands rather than ranking Conviso AST against them. The figures were copied from the original publications on 10/14/2026 and were not re-run.

Wiz only published the success rate of Atlas on CyberGym (90.9% across 1,507 vulnerabilities), which measures a different task: reproducing already known vulnerabilities. That's why it's left out of the table.

The recurring pattern is the trade-off between the two metrics: tools with high precision tend to have low recall, and vice versa. The datasets differ in language, size, and scoring criteria, so direct comparisons only hold within the same dataset.

Results by category

Seven of the eleven categories exceed 90% detection, including SQL injection, command injection, and path traversal. The 33 false alerts are concentrated in four injection categories; the other seven finished with none.

* Conviso AST identifies all 130 weak cryptography cases, but reports them as CWE-326 (inadequate encryption strength). The OWASP Benchmark's strict rule only accepts CWE-327, so these cases are left out of the main total. Accepting CWEs equivalent to the category's, recall rises to 88.2% and F1 to 92.6%. The reference score in this article follows the strict rule.

In practice: why the AI dismissed a command injection alert

Every dismissed alert comes with the AI's reasoning, so the team can audit the decision instead of trusting it blindly. OWASP Benchmark case 00090 shows how this works.

The scanner saw an HTTP parameter coming from a cookie and a call to Runtime.exec() in the same file. By code pattern, it's a legitimate alert. The AI followed the data flow and reached a different conclusion:

The cookie value (param) is never assigned to bar. The condition (7*42)-86 > 200 is always true (208 > 200), so bar is always the fixed constant. The command comes only from a fixed echo helper or is empty. No user-controlled input reaches Runtime.exec(), so there is no command injection.

A rule-based scanner doesn't evaluate that a condition is always true, nor track which variable receives user data. This is exactly the kind of reasoning that, done by hand, eats up the team's triage time.

How it works: scanner coverage, AI judgment

Conviso AST splits the work into two stages, each doing what it does best. The scanner is responsible for recall; the AI protects precision.

1. The scanner sweeps the entire codebase. conviso-ast, with 1,154 rules, analyzes every supported file and raises alerts based on known patterns. This stage is deterministic and doesn't depend on the AI choosing what to look at.
2. The AI receives the full file for each alert. Not just the flagged line, but the context around it.
3. The AI checks whether attacker-controlled data reaches the vulnerable point. It follows assignments, conditions, and calls between the input and the sink, as in case 00090.
4. Only confirmed alerts reach Conviso Platform. Dismissed ones are logged with their reasoning, available for review.

Because the AI works on what the scanner found, it doesn't reduce coverage: in the benchmark, none of the real vulnerabilities evaluated were dismissed. The entire gain is in what no longer reaches the team.

Where there's still room to improve

A benchmark is only useful if it also shows where the tool falls short. Three points stand out in the data:

• Recall depends on the scanner. The AI filters, but it can't find what the scanner didn't flag. The 297 missed vulnerabilities come from that stage, especially trust boundary violation (51.8%) and weak hashing (69.0%), which are where the rules will evolve next.
• The OWASP Benchmark is Java and synthetic. These are servlets built to test tools, not real applications with multiple services. It's good for measuring precision and recall against a reliable answer key, but it doesn't cover business logic flaws such as IDOR and access control.
• The AI isn't deterministic. That's why we ran it 5 times and published the consolidated result instead of picking the best round.

Any new result we publish will follow the same rule as this one: public dataset, public answer key, and open method.

Sources

1. Endor Labs. AI SAST benchmark, 06/30/2026.
2. Semgrep. 3.5x more true positives, 06/12/2026.
3. Semgrep. Mythos IDOR benchmark, 08/25/2026.
4. Checkmarx Zero. Proof, not promises, 06/11/2026.
5. Checkmarx. Your scanner's accuracy claims are only half the story, 06/16/2026.
6. Corgea. Corgea vs Aikido, 07/02/2026.
7. Corgea. Corgea vs Snyk, 07/02/2026.
8. DeepSource. Benchmarks.
9. Tal et al. Snyk VulnBench JS 1.0, arXiv:2606.15762, 2026.
10. Aikido. Aikido vs Claude Security (Mythos), 08/24/2026.
11. Cycode. Next-generation SAST, 2025, updated in 2026.
12. Block Engineering. benchmrk, 04/20/2026.
13. OWASP. OWASP Benchmark.
14. Wiz. Atlas: Wiz's AI vulnerability researcher, 07/27/2026.