Cloudflare's Security Audit Skill Has 9,300 Stars and 14 Commits, So We Tried to Feed It a Fake Finding

A repository with 9,300 stars and 14 commits is unusual enough to look at twice. cloudflare/security-audit-skill went up on 18 June and has averaged roughly a hundred stars a day since, on a codebase of 2,345 lines across 22 files, most of which are Markdown.

It is a coding-agent skill: a package of prompts that turns whatever agent you already run into a security auditor working through six phases. That description makes it sound like nothing, and for most of its bulk it is. The part that is not nothing is a 461-line JSON schema and two validators that enforce it, and those are the reason we spent an afternoon here.

What is actually in it

The split matters more than the total:

• 1,778 lines of Markdown across 15 files. SKILL.md plus one hunting-class file per target type: memory safety, client side, cloud and deployment, supply chain, RPC and messaging, AI and LLM, and so on.
• 1,645 lines of CommonJS in two validators, validate-findings.cjs and validate-coverage-ledger.cjs.
• 1,392 lines of tests for those two validators, which is 85% as much test code as product code.
• 461 lines of JSON schema defining three verdicts: confirmed, needs_validation, and rejected.

There is no package.json anywhere in the repo and no dependencies. The validators run on a bare Node install and use the built-in test runner. We cloned it and ran both suites:

$ node validate-findings.test.cjs
# tests 34
# pass 34
# fail 0
node validate-findings.test.cjs  0.84s user 0.14s system

$ node validate-coverage-ledger.test.cjs
# tests 31
# pass 31
# fail 0
node validate-coverage-ledger.test.cjs  0.65s user 0.12s system

65 tests, all green, both suites finishing in under a second on a laptop. Zero setup. That is a nicer first five minutes than most security tooling gives you.

We tried to slip a fake finding past it

The claim in the repo description is "independently verified, machine-readable findings". The failure mode it is built against is obvious to anyone who has asked an LLM to review code: you get back four confident paragraphs about a vulnerability that does not exist, with no file, no line, and no way to check.

So we wrote exactly that. One finding, plausible, about our own codebase, with a title and a description and a severity and nothing else:

{
  "verdict": "confirmed",
  "title": "Admin app has no authentication",
  "description": "The admin application does not implement
    authentication, which could allow unauthorized access.",
  "severity": "high"
}

Twelve errors, exit code 1. The useful ones were not the missing-field complaints but these two:

ERROR: confirmed finding requires a visible execution observed_result
ERROR: confirmed finding requires visible remediation

That is the whole design in one line. To file something as confirmed you must supply a trace with file and line numbers for entrypoint and sink, evidence, an attacker perspective, the payload, the instructions, and what you observed when you ran it. If you cannot produce an observed result, the only verdict available to you is needs_validation, which demands an explicit blocker and a validation plan instead and is not allowed to carry a severity at all.

We then wrote a properly formed finding about our own admin app, one where the honest observed result was "connection refused, because the server binds 127.0.0.1". It passed. So the gate is not just rejecting everything.

What we measured

What we handed it	Result	Verdict
Vague finding, no trace or evidence	12 errors, exit 1	Rejected
Fully evidenced finding, honest null result	PASS, exit 0	Accepted
Invalid enum carrying ANSI and bidi bytes	Echoed as \u001b, zero raw control bytes in output	Escaped
Bidi override inside an accepted title	PASS, exit 0, passed through unchanged	Not filtered
Both test suites on a bare Node install	65 of 65 pass, under 1s each	Green

That fourth row deserves a sentence of fairness. The validator's escaping is designed to protect its own diagnostics, so a hostile repo cannot inject terminal escapes into an error message printed on the auditor's machine, and we confirmed it does that: we piped the output through od and found zero raw 0x1b bytes. It is not a content filter on values it accepts. Whether that matters depends entirely on what renders REPORT.md afterwards, which is outside this repo's scope. Worth knowing before you cat a generated report.

What it does not do

• It finds nothing by itself. There is no analysis engine here. Without an agent driving it, this repo is 22 files that do nothing.
• It will not run the risky part for you. SKILL.md requires an OS-enforced sandbox with no network, read-only target, and explicit CPU and memory limits before any target code executes. If you cannot provide that, the instruction is to report the missing sandbox as a blocker rather than run anything. One Hacker News commenter, 9el, asked why that requirement was there at all, which suggests most people will skip it.
• It is not deterministic. Two runs will not produce the same findings, which makes it awkward as a merge gate.
• It does not replace a static analyzer. Different job. Pattern scanners are cheap, repeatable, and terrible at authorization logic. This is the inverse on all three.

Who made it, and the bus factor

Three contributors and 14 commits total. GitHub attributes 11 of 13 mapped commits to a single account, literally-dan. That is a bus factor of one on a repo with 9,300 stars, 500 forks and 17 open issues.

Normally that would be the warning in this section. Here it is close to irrelevant, because the entire artifact is 143 KB of text under MIT and you can vendor the whole thing into your own repo in one command. If Cloudflare loses interest tomorrow, your copy keeps working. The README is candid that this is the single-repo starting point that seeded a larger internal harness, not the harness itself.

A prompt you can fork in ten seconds has a bus factor problem that does not matter.

One more field note from the thread, because it is the kind of thing that costs you an hour otherwise: wslh reports that skills which explicitly frame the task as security research sometimes trigger refusals from frontier models, and works around it by splitting bug-class skills from the security framing. We did not reproduce that, so treat it as one practitioner's experience rather than a finding.

Verdict

Adopt today if you already run an agent on your codebase and you want its security output to be falsifiable. Install it, and even if you never run a full audit, steal the schema. The confirmed/needs_validation split is the useful idea and it costs nothing to adopt.

Wait if you were hoping to replace Semgrep or CodeQL in CI. Non-deterministic output and a per-run token bill are the wrong shape for a merge gate, and nothing in this repo pretends otherwise.

What would change our mind: a published token cost for a reference repo. The absence of one is conspicuous when the loudest reaction in the thread is people counting theirs.

Your turn

If you have pointed an agent at your own codebase for a security pass, we want the number people keep asking for and nobody publishes: how many tokens did one run cost you, and how many of the findings survived a second look? That ratio is the only thing that decides whether this category is real.

We did something similar to another agent skill recently, running its own rules back against our published work to see whether it held up: sepia Hit 2,618 Stars in 18 Days, So We Ran Its Ban List Against Our Own Corpus. Same question, different target.