Share this
Vulnerability Discovery with AI and Conventional Tools
by Reflare Research Team on Nov 17, 2026, 10:11:57 AM
Anthropic recently reported that its Project Glasswing partners had uncovered at least 129,000 verified software vulnerabilities. Palo Alto Networks reported 14,090 confirmed findings across 3,915 open-source projects in a two-month campaign. If model-assisted systems can find flaws at this scale, what role remains for conventional security testing tools?
.jpg?width=1200&height=800&name=Vulnerability%20Discovery%20with%20AI%20and%20Conventional%20Tools%20(1200).jpg)
Found one. Now someone has to fix it.
Palo Alto’s NOVA system combines multiple models with specialised analysis tools and independent validation in clean environments. Anthropic’s total comes from partial partner reports with differing triage practices. Neither report compares those results with a similarly resourced programme of conventional security testing, so the totals cannot tell a team which existing tools it could safely retire.
To decide what to keep in a testing programme, we need to distinguish the jobs those tools do. Static application security testing, or SAST, examines code without running the target application. Dynamic application security testing, or DAST, commonly refers to testing a running web application through its external interfaces. Fuzzing repeatedly executes a target with generated or mutated inputs, looking for failures. The examples below also consider where a large language model, or LLM, could help, and how a discovery can become a lasting check.
The methods overlap, because fuzzing is also dynamic testing. Web and API fuzzing can form part of DAST, while a fuzzer aimed at a library function tests execution inside a process. The separate fuzzing section examines how input generation, coverage feedback and failure detection work together.
What the finding mix shows
Palo Alto assigned about 4% of its findings to memory and calculation issues, rising to 8% when resource-management and denial-of-service issues were included. The remaining categories included access-control failures, path traversal, code injection, server-side request forgery (SSRF) and prototype pollution. That is a broader range of failures than a crash-only campaign is designed to recognise.
Those percentages do not measure what existing tools missed. Path traversal and code injection can be investigated with static data flow queries or targeted runtime tests. Stateful fuzzing can explore workflows, and an application-specific assertion can detect an unauthorised action that causes no crash. A vulnerability’s category does not determine which technique can find it.
SAST follows the input through the code
SAST ranges from matching syntax to tracing how data moves through a program. CodeQL’s data flow analysis can follow values between functions and through object properties, while taint tracking follows attacker influence through transformations. A query can therefore investigate whether input from a request can reach a sensitive operation.
Palo Alto also describes NOVA’s discovery as largely static analysis, tracing external input towards dangerous operations. Its model-based review and CodeQL’s program analysis are different ways of investigating those relationships.
Consider a Python service that looks up customers in SQLite. The HTTP handler passes a query-string value to the following function without first constraining it to an integer. The connection is supplied by the application, and the function inserts the value directly into SQL.
def find_customer(db, customer_id): sql = f"SELECT id, name FROM customers WHERE id = {customer_id}" return db.execute(sql).fetchall()
The source is the request parameter, the sensitive operation is db.execute, and the formatted string carries attacker influence between them. A value such as 1 OR 1=1 changes the condition so that it selects every row. A suitable taint analysis can flag the path without sending that request to a live service. It still needs to establish where the argument comes from and whether an earlier check constrains it.
For SQLite, binding the parameter keeps the supplied value out of the SQL syntax.
def find_customer(db, customer_id): sql = "SELECT id, name FROM customers WHERE id = ?" return db.execute(sql, (customer_id,)).fetchall()
Python’s SQLite interface passes the value separately from the statement, so the same string no longer becomes an extra SQL condition. That addresses this injection route, while permission to read a particular customer remains a separate question.
Static analysis can inspect code that a running test never reaches. Once the failure pattern is understood, a query can search for related cases across a codebase and run again as the software changes. That is useful when several handlers share an internal database wrapper.
The limits come from the program model and the question encoded in the query. Consider an application that passes requests through an internal query builder before reaching the database. If the analyser does not understand that wrapper, it may miss the connection. Modelling every unknown function as passing through everything can create the opposite problem, producing paths that are irrelevant or impossible in practice.
DAST tests what the application actually permits
A suspicious code path may be blocked by authentication middleware or deployment settings. DAST exercises the running application to find out what can actually be reached under the conditions tested.
An active scanner such as ZAP applies known attacks to selected targets. That remains useful for finding familiar weaknesses through real requests. Its coverage still depends on reaching the right routes with the necessary accounts and application state. A scan that spends its time at the login page has little to say about a privileged workflow behind it.
Some failures require an understanding of the application’s rules. Suppose a service requires a request to be approved by someone other than its creator. A useful dynamic test creates a request and attempts approval using the same identity, then checks the stored result. In the following Python test, creator and auditor are authenticated HTTP clients connected to an isolated test instance. The fixture gives the auditor permission to read the request, and the API contract specifies a 403 response for forbidden approval.
created = creator.post("/requests", json={"amount": 100}) assert created.status_code == 201 request_id = created.json()["id"] response = creator.post(f"/requests/{request_id}/approve") assert response.status_code == 403 stored = auditor.get(f"/requests/{request_id}") assert stored.status_code == 200 assert stored.json()["state"] == "pending"
The final assertion matters because a rejection response alone does not establish that nothing changed. An implementation could update the request before discovering the permission error. This test assumes synchronous processing, so an asynchronous service would also need it to wait for the operation to settle.
An integration test like this checks authorisation through the running API. Generic DAST scanners do not automatically know the application’s approval policy. As OWASP’s business logic guidance explains, making that rule explicit allows it to be tested automatically.
Fuzzing searches beyond the first example
A coverage-guided fuzzer such as libFuzzer mutates inputs and retains those that reach new code, using them as the basis for further variations. It can explore combinations that would be tedious to construct manually, provided the test setup gets those inputs to the relevant behaviour.
Consider a small binary format whose first byte declares how many payload bytes follow. This deliberately flawed C parser calculates a checksum using that length without checking whether the supplied buffer contains enough bytes. The final function is the entry point through which libFuzzer supplies each candidate input.
#include <stddef.h> #include <stdint.h> static uint8_t checksum(const uint8_t *data, size_t size) { if (size == 0) return 0; uint8_t sum = 0; for (size_t i = 1; i <= data[0]; ++i) sum ^= data[i]; return sum; } int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) { volatile uint8_t result = checksum(data, size); (void)result; return 0; }
The one-byte input 0x01 claims a payload byte that does not exist. Reading data[1] therefore crosses the buffer boundary. In a build instrumented with AddressSanitizer, that invalid access can produce a report even if the program would otherwise continue running. Coverage feedback guides the fuzzer’s search, while the sanitizer detects the memory error. The volatile result keeps the checksum observable so the compiler cannot simply discard the calculation as unused.
For this format, the missing condition is data[0] <= size - 1, checked after establishing that size is non-zero. A static analyser might also identify the missing bounds check. Fuzzing adds a concrete triggering input and can keep searching neighbouring lengths and payloads after the repair. The input demonstrates a memory error in this parser, while its effect in a product would still depend on how the parser is reached and used.
The small example puts the missing check directly in reach of the generated input. Complex formats may need enough structure to pass initial parsing, and stateful services may require several successful operations before a later request becomes interesting. More processing power cannot compensate for a campaign that repeatedly fails before reaching the code under investigation.
Google reported 26 vulnerabilities found using AI-generated or enhanced fuzz targets in 2024, including CVE-2024-9143 in OpenSSL. The team said that, as far as it could tell, the OpenSSL flaw would not have been discoverable with the existing human-written targets. It also reported a cJSON bug found by an AI-generated target in a function that already had a human-written harness. The model helped expose behaviour to testing, and fuzzing supplied the executions that revealed the defect.
Reaching more code is only part of the problem. The test also needs a way to recognise failure, often called an oracle. A runtime memory detector can report an out-of-bounds access, while an assertion can check that a rejected operation leaves protected state unchanged. A crash-only test has no reason to complain when an unauthorised request succeeds cleanly. Changing that requires a different check, even if the input generation stays the same.
Fuzzing also extends beyond malformed files and memory corruption. Microsoft’s RESTler generates sequences of API requests, using dependencies between operations to reach deeper service states. A resource may have to be created before it can be modified or deleted. Security questions involving the order of operations can therefore be tested systematically, provided the test knows which outcomes are forbidden.
An LLM-assisted investigation
NOVA’s reported workflow runs discovery, proof-of-concept construction and validation autonomously until final human review. A team can also use models for individual parts of an investigation. In the example below, the model’s proposed job is to identify a security assumption and help construct a test for it.
Anthropic’s property-based testing research describes an agent that inferred properties from Python code and documentation, wrote tests in Hypothesis and ran them. It found bugs in popular Python packages, showing one way a model can propose what a test should check. The study covered software bugs generally, rather than security vulnerabilities alone.
As an illustration, consider a service that checks permission when a user submits work, then places the operation on a queue. A separate worker executes it later. If the policy requires permission to remain valid at execution time, revoking the user’s role between submission and execution should prevent the protected action. A worker that relies entirely on the earlier decision may violate that rule, even though the submission endpoint authenticated the user correctly.
Some systems intentionally authorise work at submission, so the investigation has to establish which contract applies before treating delayed execution as a vulnerability.
An LLM could trace the submission handler, queue message and worker to identify where permission is checked. To test whether the worker relies on a stale decision, the following export regression keeps it paused until explicitly invoked. Access is revoked through the normal application service before the worker runs.
def test_revoked_user_cannot_export(app, worker, output_store): user, dataset = app.create_authorised_export_fixture() job = app.submit_export(user, dataset) app.revoke_access(user, dataset) worker.run(job.id) assert app.job_status(job.id) == "denied" assert not output_store.exists(job.id)
The test preserves a valid submission and changes the user’s permission before execution. It also needs a positive case in which an authorised user’s export completes, otherwise a worker that rejects every job could appear secure. The researcher must check that the setup preserves normal application behaviour and that the asserted policy is correct.
The investigation could then use static analysis to locate other consumers of the stored permission decision, dynamic testing to exercise the sequence through public interfaces, and stateful fuzzing to explore related orderings and retries.
The model can also be wrong about any of those steps. It may overlook a downstream permission check, assume that an attacker controls an internal field, or write a test that bypasses the normal application setup. A plausible explanation needs to survive examination of its prerequisites. Agreement from another model may help challenge the argument, but does not establish that those prerequisites hold.
In the property-based testing study, 56% of 50 manually selected reports from an initial set of 984 were judged valid bugs, and 32% were both valid and worth reporting. After the researchers introduced ranking, their top-scoring reports had validity and reportability rates of 86% and 81%. Generating reports was only the start of the work needed to select useful findings.
Keep the result after the research ends
A useful finding identifies a security failure under conditions the maintainer can understand and verify. That may come from a reproducible test or sufficiently strong code and configuration evidence. A crash in an artificial setup does not by itself establish a reachable attack, just as a static warning does not establish that the suspected path is feasible.
The fix should leave behind more than a report. A new regression can protect the business rule, a static query can search for the same mistake elsewhere, and a retained fuzz input can keep exercising the failure. These are ways to turn an expensive investigation into checks that the development process can continue to run.
Anthropic has acknowledged that verifying, prioritising and fixing findings remains difficult, despite the number of vulnerabilities uncovered. Increasing the supply of candidates is useful only if teams can establish which matter and carry the repairs through.
A security team has to judge the combination against its own software. A previously missed failure that can be reproduced, repaired and checked again is a better reason to change the testing programme than the size of another organisation’s discovery total.
Share this
- October 2026 (1)
- September 2026 (1)
- August 2026 (1)
- June 2026 (1)
- May 2026 (1)
- April 2026 (1)
- March 2026 (1)
- February 2026 (1)
- January 2026 (1)
- December 2025 (1)
- November 2025 (1)
- October 2025 (1)
- September 2025 (1)
- August 2025 (1)
- July 2025 (1)
- June 2025 (1)
- May 2025 (1)
- April 2025 (1)
- March 2025 (1)
- February 2025 (1)
- January 2025 (1)
- December 2024 (1)
- November 2024 (1)
- October 2024 (1)
- September 2024 (1)
- August 2024 (1)
- July 2024 (1)
- June 2024 (1)
- April 2024 (2)
- February 2024 (1)
- January 2024 (1)
- December 2023 (1)
- November 2023 (1)
- October 2023 (1)
- September 2023 (1)
- August 2023 (1)
- July 2023 (1)
- June 2023 (2)
- May 2023 (2)
- April 2023 (3)
- March 2023 (4)
- February 2023 (3)
- January 2023 (5)
- December 2022 (1)
- November 2022 (2)
- October 2022 (1)
- September 2022 (11)
- August 2022 (5)
- July 2022 (1)
- May 2022 (3)
- April 2022 (1)
- February 2022 (4)
- January 2022 (3)
- December 2021 (2)
- November 2021 (3)
- October 2021 (2)
- September 2021 (1)
- August 2021 (1)
- June 2021 (1)
- May 2021 (14)
- February 2021 (1)
- October 2020 (1)
- September 2020 (1)
- July 2020 (1)
- June 2020 (1)
- May 2020 (1)
- April 2020 (2)
- March 2020 (1)
- February 2020 (1)
- January 2020 (3)
- December 2019 (1)
- November 2019 (2)
- October 2019 (3)
- September 2019 (5)
- August 2019 (2)
- July 2019 (3)
- June 2019 (3)
- May 2019 (2)
- April 2019 (3)
- March 2019 (2)
- February 2019 (3)
- January 2019 (1)
- December 2018 (3)
- November 2018 (5)
- October 2018 (4)
- September 2018 (3)
- August 2018 (3)
- July 2018 (4)
- June 2018 (4)
- May 2018 (2)
- April 2018 (4)
- March 2018 (5)
- February 2018 (3)
- January 2018 (3)
- December 2017 (2)
- November 2017 (4)
- October 2017 (3)
- September 2017 (5)
- August 2017 (3)
- July 2017 (3)
- June 2017 (4)
- May 2017 (4)
- April 2017 (2)
- March 2017 (4)
- February 2017 (2)
- January 2017 (1)
- December 2016 (1)
- November 2016 (4)
- October 2016 (2)
- September 2016 (4)
- August 2016 (5)
- July 2016 (3)
- June 2016 (5)
- May 2016 (3)
- April 2016 (4)
- March 2016 (5)
- February 2016 (4)


