All Posts

security

Security Scanning on Free Tools, Now That Finding Bugs Is Cheap

PHP2Shell was found with AI and attacked before its patch shipped. Why that is the new normal, how we built SAST, dependency, secret, IaC and image scanning from free tools, and what a green security check still does not tell us.

On 21 October the PHP project shipped fixes for two flaws, soon nicknamed PHP2Shell, that let anyone run code on a web server running PHP without logging in. Every release from 5.4 to 8.5 was affected, under PHP-FPM, mod_php and CGI alike. Two research teams had found them independently in August, and both used AI models to do it. Amazon saw an attempt to exploit one of them four days before the fix was published. Government sites in the Netherlands, France and Finland went offline as a precaution. Nearly two weeks after the release, Censys still counted 29 million of 46 million reachable servers on a vulnerable PHP, while new exploits kept appearing, in Kaspersky’s words, “in batches — courtesy of AI”.

We don’t run PHP, and a memory bug in a language runtime is nothing exotic. What has changed is how cheap such bugs are to find, and how short the gap is between a fix going public and someone using it. A published fix is a description of the bug, and reading a diff is exactly what these models are good at.

We had run that experiment on ourselves months earlier. On 12 April we ran two AI reviews of our own repository overnight, report only: three Claude agents working in parallel, and Codex on its own. The code base was about 94,000 lines of Go and 23,000 of TypeScript, plus the Kubernetes manifests. The first review came back with 36 findings, four of them rated critical:

  • access and refresh tokens in localStorage, where any script on the page could read them;
  • a JWT signing secret with a default value in the config struct;
  • a Redis with no password, holding sessions and OTP codes;
  • no Kubernetes NetworkPolicy, so any pod could reach that Redis.

All four are fixed. Six days later, a review of a plan, a withdrawal feature for organisers that did not exist as code yet, found that an admin demoted after submitting a request could still be paid out, that the password re-check could be replayed, and that two clicks on a slow connection could queue two withdrawals against the same funds.

Those reviews cost us one evening of compute, and nothing stops anyone else from buying the same thing and pointing it at our code, or at code shaped like it. We also write most of our code with AI agents, so more of it lands each week than any of us reads line by line. The cost of finding a bug has dropped for attackers and defenders at the same time, and the side that gains is the one that looks first.

This post is about the cheap, boring half of our answer: scanners that run on every merge, built from free tools, and what it took before I trusted their output. The half that no scanner covers, reviewing logic, is in section 8, followed by what we still haven’t done.

The Map

Kind What it looks for Tool When it runs
SAST Dangerous patterns in our own code: injection, unsafe exec, weak cookies Semgrep CE 1.176.1, p/default rules vendored (397 rules) Every merge, weekly
Dependencies Known CVEs in Go modules and npm lockfiles Trivy, govulncheck, Dependabot Merge, weekly, release; Dependabot continuously
Secrets Tokens and keys in the tree, the image, and every commit GitHub holds Trivy’s secret scanner; Betterleaks 1.10.0 for git history Pull request, merge, weekly, release
IaC Kubernetes manifests, Dockerfiles Trivy misconfig Merge, weekly
Container image OS packages and Go binaries in what we ship Trivy image Merge, release, weekly
DAST The running application, from outside Nothing yet —
Logic review Authorization, races, money flows AI red-team reviews Per plan, by hand

The repository is private, and on a private repository GitHub’s CodeQL scanning and secret push protection are paid add-ons. Everything in this table is free. That constraint shaped the map more than anything else did.

1. Pick Tools With Canaries, Not With Blog Posts

Scanner comparisons are easy to find and hard to apply to your own code. So before choosing, we ran a bake-off on 12 September: Trivy, Grype, OSV-Scanner, govulncheck, gitleaks, and Semgrep with two rule sets, three repetitions each on a hosted runner. Before each scan we planted things a correct scanner had to find:

Planted Result
npm minimist@1.2.5 (CVE-2021-44906) Trivy, Grype and OSV-Scanner all found it
A Go module version with a known advisory, called Every dependency scanner found it
The same module, imported but never called govulncheck correctly stayed quiet
A Pod with privileged: true Trivy found it (KSV-0017)
Three ghp_ tokens: a plain file, a _test.go, a .md Trivy’s defaults: 1 of 3. gitleaks: 3 of 3
exec.Command("sh", "-c", r.URL.Query().Get("x")) The smaller Semgrep rule set missed it in 3 of 3

Three lessons from that table changed the configuration.

Defaults skip the places secrets actually end up. Trivy’s built-in allow-rules ignore anything that looks like a test, an example, Markdown or a locale directory. In our repository that was 115,000 lines of Go tests, every planning document and every .env.example. One config file fixes it, and the bake-off measured both sides: 3 of 3 canaries found, still 0 real secrets in the tree.

# .trivy/secret.yaml
disable-allow-rules:
  - tests
  - examples
  - markdown
  - locale-dir

A faster rule set is not a cheaper one. The smaller Semgrep set ran 23% faster than p/default and missed the most obvious command injection you can write in Go, three times in a row. It was out.

Scan with the toolchain production ships. govulncheck checks the standard library of whichever Go runs it. Run under the version pinned in go.mod, it reported 28 advisories in code we call, 17 of them in the standard library. Run under the Go version our production builder actually used, eight patches newer, it reported 11 and 0. The workflow now resolves the newest patch of go.mod’s minor version, which is what the golang:<minor>-alpine builder floats to:

minor=$(awk '/^go /{split($2, v, "."); print v[1] "." v[2]; exit}' api/go.mod)
echo "go-version=${minor}.x" >> "$GITHUB_OUTPUT"   # setup-go, check-latest: true

What we did not adopt says as much. Grype reported more critical Go findings than Trivy, but every extra one was the same advisory with a different severity label. OSV-Scanner found three middleware advisories that Trivy misses, because they are scoped to a subpackage; govulncheck reports them too, and none is in code we call. The rule written before the first run was that a challenger had to show more verified, fixable HIGH or CRITICAL findings. None did, so we kept fewer tools: Trivy for breadth, govulncheck for reachability, Semgrep for our own code. Secrets got a second, harder look two months later (section 6).

The chosen set costs about 70 seconds of tool time. Trivy’s vulnerability database downloads cold in 6 seconds, so we don’t cache it.

2. Silence Is Never a Pass

A scanner that crashes, loads no rules or reads an empty database can still exit 0 and print nothing, which is exactly what a clean scan looks like. So no scanner step is allowed to fail the job. Each one records its exit code in a file, and one script, report.sh, turns all of them into a verdict. Its own test suite runs first in every job, because it is the one piece of code whose bug would make every result meaningless.

Then every run has to prove that each scanner did its job:

Evidence Red when
Canaries planted outside the workspace Any canary is missed
Trivy’s database age Older than 48 hours
Packages Trivy saw Fewer than 100 Go modules or 1,000 npm packages
govulncheck’s Go version Not the version production builds with
govulncheck’s module list Fewer than 40 modules, or missing chi, pgx or stdlib
Semgrep rules loaded Not equal to the 397 in the lock file
The image scanned Not Alpine, or no Go binary inside

The Semgrep canary is the pattern the smaller rule set missed, written to a directory of its own on every run:

func Handler(w http.ResponseWriter, r *http.Request) {
    _ = exec.Command("sh", "-c", r.URL.Query().Get("x")).Run()
}

Trivy’s secret scanner has a sabotage test on top: a manual run can plant a fresh random token in the workspace, and passes only if the scan reported it and the token appears nowhere in the summary or the artifact. “Not in the report” alone would also be true of a scan that saw nothing, and a report that prints the secret it found is a second leak.

Even the glue lies. Given a path that does not exist, yq prints its usage to stdout and exits 0, so a step converting our ignore file to JSON “succeeded” and wrote help text. The step now requires the file to exist and the output to be a JSON object.

3. The Defaults Were Hiding Our Tests

Two weeks after the bake-off, I looked at what Semgrep’s built-in ignore list excluded in our repository: 688 files, and 686 of them were tests. Since the day it was introduced, the SAST scanner had never read a line of test code.

Test code is not a safe place for a vulnerability. It runs in CI next to credentials, it shells out, it writes files, and it is where a supply-chain payload would be least looked at. Bringing it into scope:

  Files Findings
Before 1,656 118
After 2,387 271

Nothing fell out of scope in the trade. Of the new findings, exactly one was blocking: a dangerous-exec-command on exec.Command(os.Args[0]), the standard “re-run the test binary as the CLI” helper. It is waived at the line, with a comment naming the condition that would make the waiver wrong.

There is a trap in how you do this. .semgrepignore does not add to Semgrep’s defaults, it replaces them. A file containing only plans/ took a full scan from 2,080 files to 2,325, and pulled .git into scope. --exclude on the command line is additive. Whatever exclusion mechanism you use, count the files scanned before and after.

4. The Scanner Is Part of the Attack Surface

On 19 March 2026, almost every version tag of aquasecurity/trivy-action, and the tags of setup-trivy, were force-pushed to a credential stealer that read the runner’s memory. Any workflow running uses: aquasecurity/trivy-action@<tag> that day ran it, with whatever secrets its job held. A security scanner is a third-party program you run on every commit, often in a job that can also deploy.

So the scan follows a few rules:

  • No wrapper actions. A local composite action downloads one Trivy release and refuses it unless its SHA-256 matches the pin, taken from the release’s checksum file and verified with cosign when the pin is bumped. Semgrep runs from an image pinned by digest. govulncheck is a pinned module version, checked against the Go checksum database. New scanner versions wait seven days before we adopt them.
  • The scan job holds nothing worth stealing. It has contents: read and no other permission. The job that scans the deployed images needs packages: read to pull them from our registry, and logs out of the registry before any scanner starts.
  • A scanner cannot poison a build. The Go setup in the scan job has its cache turned off, so nothing a scanner writes can be restored by a later build.
permissions:
  contents: read
# ...
SEMGREP_IMAGE: semgrep/semgrep@sha256:34ab619bf1391a24bfda3f05debd0d8a6ce3093c2d5f9d39cfc00f83c1397823

The scan workflows pin every action by commit SHA. Our test and deploy workflows mostly don’t yet, and those are the jobs with the most secrets. That is the first item in section 9.

5. Where Scans Run, and What Turns Them Red

When What it scans What it can do
Every pull request Its own commits, for secrets only Go red; not a required check
Every merge to main The tree, the API image built from it, and every commit GitHub holds Go red, and alert
Weekly, Monday 03:00 The tree, every commit, and the images running right now Go red; images: report only
Every production deploy The release commit, and the exact image to ship Block the deploy

The scanners used to run on every pull request too. We moved them to merge-only to save Actions minutes, and the cost should be said plainly: a vulnerable dependency is now reported after it lands on main, not before. What tipped it was that the scan had never been a required check, so on a pull request it was a signal somebody had to choose to read. Pull requests still run the cheap guards, the ones that take seconds: the rules on our ignore files, since this week a secret scan of the pull request’s own commits (section 6), and a check that the production deploy is still wired to its gate. Every way to disable that gate is a one-line, innocent-looking edit, an if: on the deploy job or a trimmed needs:.

What turns a merge red is narrow on purpose:

  • a fixable CRITICAL vulnerability;
  • a govulncheck finding in code we actually call, with a fix available;
  • a Semgrep finding of HIGH or above;
  • any secret;
  • any missing evidence from section 2.

Everything else is recorded and does not block. An accepted risk goes into an ignore file, where each entry must carry a statement and an expiry no more than 90 days out, scoped to a package or a path, and a script enforces those rules. Both ignore files are empty today. Nothing has been formally accepted, which means everything is either fixed or still waiting.

The release gate runs between building the production image and deploying it, and rests on one rule: the tooling is trusted, the target is not. The commit being released is checked out into src/, the gate’s own commit into tools/, and every script that decides anything runs from tools/, so a release cannot ship the reporter that judges it. The digest pulled must equal the digest built. That check survives the gate’s bypass mode, and a bypass needs a written reason, which goes into the summary and the alert.

On 30 September it blocked v0.28.0 with every scanner clean:

"reasons": ["evidence:step-failed:check-gate-wiring"]
check-gate-wiring.sh: no such file: .github/workflows/security-scan.yml

A change two days earlier had taught the wiring check to read a second workflow, by a path relative to the repository root. That path exists in a normal checkout, and not in the gate’s src/ and tools/ layout. The tempting fix, skipping the rule when the file is missing, is exactly how a gate starts failing open. We passed the gate the right path instead. The blocked tag stays as the record.

The sweep of deployed images exists for CVEs published after an image shipped. On 8 October a run of it found HIGH advisories in OpenSSL and in excelize, our spreadsheet library, inside the API image running in production. Several of the fixes needed Go 1.26 or newer, and the OpenSSL one needed an apk upgrade in the runtime stage: the base image tag still shipped the vulnerable packages, and the fixed ones existed only in the package repository.

Image HIGH MEDIUM / LOW UNKNOWN
Running in production 14 50 2
After the upgrade 0 0 1 (no fix exists; govulncheck: not called)

Upgrading Go also changed the bytes our certificate renderer writes for the same drawing, which a golden test caught before release. A security bump is a behaviour change, and it goes through the same tests as any other.

That sweep reports and never goes red, deliberately. A finding in a running image was not introduced by the last merge, so failing that run would blame whoever merged last, and what it calls for is a rebuild and a release rather than a code change.

6. A Deleted Secret Is Still on GitHub

The September bake-off tested secret scanning with three ghp_ tokens. That measured whether Trivy reads test files and Markdown. It did not measure whether Trivy knows the credentials we actually hold, or whether it looks everywhere they can end up.

Trivy reads the tree at the tip of main. A token added in one commit of a pull request and removed in the next never reaches that tree, and a squash merge leaves neither commit in main’s history. GitHub keeps both in refs/pull/<n>/head, and anyone who can clone the repository can fetch them. When we counted this week, main had 1,114 commits and the repository as a whole 1,669, across 445 pull request heads.

So we ran a second bake-off the way we ran the first. Every tool read a clone that included the pull request heads, with the network off, and we planted 25 canaries, each shaped like a credential we actually hold and filled with a fresh random value, one of them existing only base64-encoded in a Kubernetes Secret. Canaries caught out of 25, then distinct false-positive values over the whole history:

Tool Canaries False positives
Betterleaks 1.10.0 23 78
detect-secrets 1.5.0 23 1,408 (tree only)
gitleaks 8.30.1 21 23
Titus 1.3.0 17 55
Kingfisher 2.18.1 14 19
TruffleHog 3.99.8 11 19
Trivy 0.74.0, our config 5 0

No tool found a real secret anywhere in the history. The scanner we already ran caught 5 of 25: it missed most of the provider tokens, the base64 value and every generic shape.

We took Betterleaks, which tied for the most canaries with a fraction of detect-secrets’ noise. But every tool’s default rules missed the same two canaries, both random strings under names we chose. A value like that has no provider prefix to match, and to an entropy rule it is one more hash. So the configuration adds one rule of our own:

# .betterleaks.toml, abridged: one name shown; the real list names every secret we hold
[[rules]]
id = "cme-known-secret-name"
regex = '''\b(JWT_SECRET)\b["']?[ \t]*[:=][ \t]*["']?([^\s"'`]{8,})'''
secretGroup = 2

An allowlist keeps references and placeholders quiet ($VAR, <...>, os.Getenv), so a manifest that names a secret without holding it does not fire. With that rule, Betterleaks caught 25 of 25.

A rule built on a list of names goes stale the day someone adds a secret that isn’t on it. So a second script reads every secret name out of our Kubernetes manifests and workflows, writes each one into a scratch directory with a fresh random value, as NAME=value and as NAME: "value", and fails unless the rule reports both. Its first run found 70 names; the hand-written list had 23. Each of the others is now either in the rule or exempted with a written reason, such as an account id that is not a secret.

Section 3 brought tests into SAST. Here we did the opposite for three rules, and measured it first. The generic rules guess from keywords and entropy, and test code is full of fake keys: with them reading tests, the whole history produced 284 findings; with tests kept out of those three rules, 97, and the same 25 canaries either way. The provider rules and our own rule still read tests, because a real key pasted into a test is the case worth catching. All 97 were false positives, each reviewed and waived by fingerprint (commit, file, rule, line), so a waiver covers one occurrence and nothing after it.

It runs in two places:

  • Every pull request, over the pull request’s own commits (base..head), not the merge result, because the merge result is exactly where a deleted token doesn’t show. It runs with the pull request’s own configuration, so a waiver added in the same pull request is something the reviewer has to read. Like the old scans, it is not a required check yet.
  • Every merge and every weekly run, over every commit of every branch, tag and pull request head: about 1,700 commits in a few seconds. A checkout fetches branches and tags only, so the job fetches refs/pull/*/head first, and the verdict wants at least 400 of them. A sweep that quietly read only main is red.

Section 2’s rules apply. Four canaries run on every scan, one of them a throwaway repository where a commit adds a token and the next one deletes it. git counts the commits and added lines on its own, so a scan of an empty range cannot pass for a clean one. The binary is installed the way Trivy is in section 4: one release, checked against a SHA-256 pin.

A finding here means the secret is already on GitHub. Deleting the line doesn’t help; rotating the credential does.

7. Green Is Not Clean

Here is this Monday’s weekly run. Verdict: pass. Findings: 518.

Kind Count Mostly
SAST (Semgrep) 307 82 actions referenced by tag; 113 cookie flags, 108 in test files
Vulnerabilities 113 Mostly npm packages, many in build and test tooling; 4 from govulncheck, none called
Misconfigurations 98 Kubernetes securityContext settings
Of all of these, HIGH 55  

Dependabot tells the same story from another side. When we turned it on in September, main had 168 open alerts, 11 of them critical. Since then 181 have closed as fixed, 13 critical and 71 high among them. Today 89 are open: none critical, 29 high, and 18 of those 29 in development dependencies. None has been dismissed, and five Dependabot pull requests are open, the oldest from 6 October.

A gate should block only what is both severe and actionable, or people learn to ignore it. But a gate calibrated like that says nothing about the backlog behind it, and “the check is green” quietly turns into “we’re fine”. Two weeks after PHP2Shell’s fix, most of the servers Censys could see were still unpatched; that is what “we’ll get to it” looks like at internet scale. The backlog needs its own owner and its own clock, and we have neither yet.

8. What No Scanner Will Find

Go back to April. None of those four criticals is a CVE, and only the default secret looks like a pattern a rule could match. The withdrawal findings were not in code at all. The classes of bug that hurt a platform like ours, a check that doesn’t hold under concurrency, a permission read at the wrong moment, a money flow that can run twice, are invisible to every scanner above.

That is where AI review earns its place. We have written 29 red-team reviews since April, and this is how we run them now:

  • Review the plan before the code for anything that touches money, authentication or tenancy. A finding in a plan costs a paragraph. The same finding after release costs an incident.
  • Give the reviewer an attacker’s brief and a stopping rule: report only, no fixes, a failure scenario for every finding.
  • Reproduce every claim before acting on it. A reviewer’s mistakes arrive in the same confident tone as its findings. One April finding filed among the criticals, missing CSRF protection, said in its own text that our bearer-token design already mitigated it.
  • Turn each confirmed finding into a test that fails without the fix, so the class of bug cannot quietly come back.

9. What’s Still Open

All of these can be done with free tools, and none is done yet:

  1. DAST. Nothing tests the running application from outside. The plan is OWASP ZAP’s baseline and API scans against staging, driven by our existing OpenAPI spec, on the nightly E2E schedule, and Schemathesis to fuzz the same API from the same spec.
  2. Secrets before push. The pull request check in section 6 finds a token once it is already on GitHub, where only rotation helps. The same binary and configuration can run as a pre-commit hook, and the pull request check can become required.
  3. Every action pinned by SHA, deploy workflows first, with Dependabot’s github-actions ecosystem to keep the pins fresh and zizmor to lint the workflows.
  4. A clock for the backlog: a fix window per severity for runtime dependencies, and real ignore entries, with expiry dates, for the things we decide to accept.
  5. Rules from our own findings. We vendor p/default and have no rules of our own. Every bug class an AI review finds once should become a Semgrep rule in this repository, so a scanner catches it the second time.
  6. The image and the cluster: a smaller runtime image, an SBOM and a keyless cosign signature for every release, verified at deploy, and the securityContext findings above fixed.

What I’d Carry Forward

  • Make every scanner prove it ran, on every run. A scan that saw nothing and a scan of a clean tree produce the same output.
  • Read what the defaults exclude. Ours kept all of our tests out of SAST and all of our Markdown out of secret scanning, silently, for weeks.
  • Scan what an attacker can fetch, not what main shows. A squash merge removes a commit from your history, not from GitHub, and no default rule knows what your own secrets are called.
  • Treat security tooling as code that holds credentials. Pin it, verify it, and give it the smallest permissions that work.
  • Block only what is severe and fixable, then give the rest an owner and a clock. A green check is a statement about the gate, not about the code.
  • Scanners find known patterns, reviews find design flaws. Both are cheap now, and both are just as cheap for whoever is looking at your code from the other side.