Cloudflare's Security Audit Skill: The Gate Is the Point

I gave a model one line: "find the security vulnerabilities in this code". It came back with one finding. SQL injection, severity critical, app.py line 13. I fed that answer to the validator inside Cloudflare's new security audit skill, and it was refused.

Then I wrote a second answer about the same bug, with a source trace, a reproduction and a fix. That one passed. Same bug, two answers, one accepted. That gap is this whole article.

The repo is cloudflare/security-audit-skill, and it is number one on GitHub's weekly trending list. It gained 15,072 stars in seven days and had 47,213 when I read the GitHub API on 26 October 2026. People are sharing it as a free AI pentester. I installed it and tested what it actually enforces. The real story is a JSON schema, not a scanner.

Why "run an AI audit" is so tempting

Most of what I run for clients is software nobody has audited: a Flask admin panel written over a weekend, or a forked dashboard with a few local patches. On a homelab or a small client VPS, that is normal.

Nobody pays for a pentest on a box like that, so asking a coding agent for bugs feels like free coverage. The catch is what comes back: a tidy list of plausible findings, mostly marked critical, and no way to tell which are real without doing the work yourself.

What the skill actually is

Cloudflare describes it as "a coding-agent skill for multi-phase security audits with independently verified, machine-readable findings". It seeded their vulnerability discovery harness. Their post on building the harness says they started with a roughly 450-line security-audit skill run on a single repository.

The published version runs six phases:

• Reconnaissance
• Coverage-led hunting
• Candidate validation
• Structured output
• Independent record verification
• Target-neutral reporting

Here is what surprised me. The repo is 20 files, 314,670 bytes and 24,768 words of instruction text. The only executable code is two zero-dependency Node validators plus their two test files, 3,037 lines in total.

So nothing in it scans your code. It is a specification that tells your agent how to hunt and how to write up findings, plus two programs that check the write-up. Both pass their own tests:

$ node validate-findings.test.cjs
ℹ tests 34
ℹ pass 34
ℹ fail 0
$ node validate-coverage-ledger.test.cjs
ℹ tests 31
ℹ pass 31
ℹ fail 0

Installing it, and one trip-up with the agent id

I installed it on our own server with Node 26.7.0 and npm 11.19.0. My first attempt used -a claude, which failed with ■ Invalid agents: claude. The correct id is claude-code. Here is the version that works:

$ npx skills add https://github.com/cloudflare/security-audit-skill --skill security-audit -a claude-code -y
◇  Found 1 skill
●  Selected 1 skill: security-audit
│  security-audit  Safe   0 alerts   Med Risk
◇  Installed 1 skill
│  ✓ security-audit (copied)
│    → .claude/skills/security-audit
└  Done!  Review skills before use; they run with full agent permissions.

The files land in .claude/skills/security-audit. The scan row has three columns, Gen, Socket and Snyk, reading Safe, 0 alerts and Med Risk. I did not get the detail behind the Snyk rating, so I won't guess at it.

Also, the CLI documents --metadata <json> Attach valid JSON to the install telemetry event, so the installer sends a telemetry event. The skill itself only reads local source. On a client box with strict egress rules, the installer is the step to think about.

The gate is the whole point

Every finding must end up as a record with one of three verdicts:

• confirmed: a complete source trace plus a bounded, observed result.
• needs_validation: names the exact unresolved fact, and has no severity field.
• rejected: a disproved candidate.

To see how strict this is, I wrote records by hand and ran them through the repo's own validator. The target was a 13-line Flask app. First I proved the bug against a two-row SQLite table:

$ python3 repro.py
exact name  'alice'          -> (1, '[email protected]')
payload     x' OR '1'='1     -> (1, '[email protected]')
missing name 'zoe'           -> None

The crafted value returns alice's row, while a name that does not exist returns nothing. That is the injection in one line of output.

Test 1: the lazy record

I wrote it the way agents usually answer. Title, description, severity "critical", empty trace, empty evidence. The validator reported 10 errors:

$ node validate-findings.cjs fake_confirmed.json
ERROR: $[0]: must match exactly one schema in oneOf; matched 0
ERROR: $[0]: missing required field "conditions"
ERROR: $[0]: missing required field "execution"
ERROR: $[0]: missing required field "remediation"
ERROR: $[0]: missing required field "confidence"
ERROR: $[0].severity: expected object, got string
ERROR: $[0].trace: must have at least 1 item(s), got 0
ERROR: $[0].evidence: must have at least 1 item(s), got 0
ERROR: $[0]: confirmed finding requires a visible execution observed_result
ERROR: $[0]: confirmed finding requires visible remediation
FAIL: 10 validation error(s)

Severity must be an object, not a word. A confirmed finding must show what ran, what was observed and how to fix it.

Test 2: a model's natural answer

I gave deepseek-flash the same file at temperature 0 with a naive prompt: "find the security vulnerabilities, return JSON with title, severity, description, file, line". It spent 802 reasoning tokens and returned one finding, SQL Injection, critical, app.py line 13. It found the correct bug, and was still refused:

$ node validate-findings.cjs model_findings.json
ERROR: $[0]: must match exactly one schema in oneOf; matched 0
FAIL: 1 validation error(s)

The bug was right and the shape was wrong: a line number is not evidence.

Test 3: the record that passes

Then I built it properly. A trace from app.py line 10, where request.args.get reads the input, to line 13, where db.execute runs it. Evidence at line 13, a condition, an execution block with the payload, a remediation with the fixed line, and severity as likelihood times impact.

$ node validate-findings.cjs good_confirmed.json
PASS: 1 findings valid

The rule most people will skip

Without an OS-enforced sandbox, the skill must not execute target code, and the lead stays needs_validation. Sandbox means no external network, a sanitized allowlisted environment, a read-only target, writes only to scratch, and explicit CPU, memory, disk and wall-clock limits. Probing deployed endpoints, external services, production identities or shared infrastructure is forbidden outright.

So I tested the blocked case. Same bug, verdict needs_validation, and I added a severity anyway, the way an eager agent would:

$ node validate-findings.cjs unproven_with_severity.json
ERROR: $[0]: must match exactly one schema in oneOf; matched 0
ERROR: $[0]: unexpected field "severity"
ERROR: $[0]: needs_validation finding must not contain severity
FAIL: 3 validation error(s)

An unproven lead cannot carry a severity at all. For me, that refusal is the most valuable thing in the repo. It is what stops an agent handing you a wall of fake criticals it never ran.

The honest limits

Cloudflare is upfront in their own post. They write that "a single run finds only about half the bugs you'd catch across multiple runs", and the ones found are "skewed toward the simpler and less subtle". Their three walls were context exhaustion, persistence and cross-repo reasoning.

They also say findings.json is "checked mechanically for schema adherence, not correctness". Read that twice. My Test 3 passed because I filled in the right fields. A wrong trace with the right fields would pass too.

And the repo itself has not moved. MIT licence, created 18 June 2026, 2,929 forks, 97 open issues, and the last push was 14 September 2026, which is six weeks before this week's star spike. So the surge is attention, not a release: nothing in the code changed while 15,072 people starred it.

My take: as a free AI pentester, it is overhyped, because it is only as good as the agent and sandbox you bring. As a discipline for making an agent prove its claims, it is genuinely good.

How I would use it on a client box

• Sandbox first: a network-disabled container, target mounted read-only, scratch for writes, hard limits. Without that, every lead stays needs_validation, and I accept it.
• One target repo per run, as Cloudflare started.
• Run it twice and diff the findings. If one run catches about half, a second pass is cheap.
• Treat confirmed as a work queue and needs_validation as a to-do list for a human.

This is the exact container I probed on our server, using an image that was already on the box. The skill does not check any of this, so the sandbox is yours to get right:

$ docker run --rm --network none --read-only --tmpfs /scratch \
    --memory 512m --cpus 1 --pids-limit 128 \
    -v "$PWD:/target:ro" -w /target python:3-alpine sh -c '
      echo "scratch: $(touch /scratch/ok && echo writable)"
      echo "target:  $(touch /target/ok 2>&1 | head -1)"
      python3 -c "import urllib.request;urllib.request.urlopen(\"https://example.com\",timeout=5)"'
scratch: writable
target:  touch: /target/ok: Read-only file system
urllib.error.URLError: <urlopen error [Errno -3] Try again>

Three lines, three controls: scratch writes, a read-only target, a dead network. Errno -3 is a name resolution failure, which is what "no network" looks like from inside a container. If the target needs its own build dependencies, bake them into an image first, because fetching them at run time is the one thing this setup will not allow.

One thing before you point it at anything: an audit agent needs read access to the source and a way to run code inside the sandbox. If your agent already has more access than that, fix that first. My earlier post on cutting an AI agent's credentials is the place to start.

Frequently asked questions

Is this a free AI pentester?

No. It is instructions for your coding agent plus two validators. Hunting quality depends on your agent.

Does the skill send my code anywhere?

The skill only reads local source. The installer has a telemetry event, and your agent has its own data path, so check both.

Can I run it without a sandbox?

Yes, but nothing should execute. Every lead stays needs_validation with no severity.

Does a PASS mean the bug is real?

No. It means schema adherence, not correctness. A human still reads the trace.

The bottom line

I drafted this with AI assistance and used an AI coding agent to run the install, the validator tests and the four records on our own server. Every number comes from those runs, the GitHub API or Cloudflare's post.

Don't install this expecting a scanner. Install it for the gate. The most useful thing it did in my tests was say no to a confident "critical" with nothing behind it, and that is exactly what an unaudited client box needs.