Security Scanning on Free Tools, Now That Finding Bugs Is Cheap
PHP2Shell was found with AI and attacked before its patch shipped. Why that is the new normal, how we built SAST, dependency, secret, IaC and image scanning from free tools, and what a green security check still does not tell us.
On 21 October the PHP project shipped fixes for two flaws, soon nicknamed PHP2Shell, that
let anyone run code on a web server running PHP without logging in. Every release from
5.4 to 8.5 was affected, under PHP-FPM, mod_php and CGI alike. Two research teams had
found them independently in August, and
both used AI models to do it. Amazon saw an
attempt to exploit one of them four days before the fix was published. Government sites
in the Netherlands, France and Finland went offline as a precaution. Nearly two weeks
after the release,
Censys still counted 29 million of 46 million reachable servers on a vulnerable PHP,
while new exploits kept appearing, in Kaspersky’s words, “in batches — courtesy of AI”.
We don’t run PHP, and a memory bug in a language runtime is nothing exotic. What has changed is how cheap such bugs are to find, and how short the gap is between a fix going public and someone using it. A published fix is a description of the bug, and reading a diff is exactly what these models are good at.
We had run that experiment on ourselves months earlier. On 12 April we ran two AI reviews of our own repository overnight, report only: three Claude agents working in parallel, and Codex on its own. The code base was about 94,000 lines of Go and 23,000 of TypeScript, plus the Kubernetes manifests. The first review came back with 36 findings, four of them rated critical:
- access and refresh tokens in
localStorage, where any script on the page could read them; - a JWT signing secret with a default value in the config struct;
- a Redis with no password, holding sessions and OTP codes;
- no Kubernetes
NetworkPolicy, so any pod could reach that Redis.
All four are fixed. Six days later, a review of a plan, a withdrawal feature for organisers that did not exist as code yet, found that an admin demoted after submitting a request could still be paid out, that the password re-check could be replayed, and that two clicks on a slow connection could queue two withdrawals against the same funds.
Those reviews cost us one evening of compute, and nothing stops anyone else from buying the same thing and pointing it at our code, or at code shaped like it. We also write most of our code with AI agents, so more of it lands each week than any of us reads line by line. The cost of finding a bug has dropped for attackers and defenders at the same time, and the side that gains is the one that looks first.
This post is about the cheap, boring half of our answer: scanners that run on every merge, built from free tools, and what it took before I trusted their output. The half that no scanner covers, reviewing logic, is in section 8, followed by what we still haven’t done.
The Map
| Kind | What it looks for | Tool | When it runs |
|---|---|---|---|
| SAST | Dangerous patterns in our own code: injection, unsafe exec, weak cookies | Semgrep CE 1.176.1, p/default rules vendored (397 rules) |
Every merge, weekly |
| Dependencies | Known CVEs in Go modules and npm lockfiles | Trivy, govulncheck, Dependabot | Merge, weekly, release; Dependabot continuously |
| Secrets | Tokens and keys in the tree, the image, and every commit GitHub holds | Trivy’s secret scanner; Betterleaks 1.10.0 for git history | Pull request, merge, weekly, release |
| IaC | Kubernetes manifests, Dockerfiles | Trivy misconfig | Merge, weekly |
| Container image | OS packages and Go binaries in what we ship | Trivy image | Merge, release, weekly |
| DAST | The running application, from outside | Nothing yet | — |
| Logic review | Authorization, races, money flows | AI red-team reviews | Per plan, by hand |
The repository is private, and on a private repository GitHub’s CodeQL scanning and secret push protection are paid add-ons. Everything in this table is free. That constraint shaped the map more than anything else did.
1. Pick Tools With Canaries, Not With Blog Posts
Scanner comparisons are easy to find and hard to apply to your own code. So before choosing, we ran a bake-off on 12 September: Trivy, Grype, OSV-Scanner, govulncheck, gitleaks, and Semgrep with two rule sets, three repetitions each on a hosted runner. Before each scan we planted things a correct scanner had to find:
| Planted | Result |
|---|---|
npm minimist@1.2.5 (CVE-2021-44906) |
Trivy, Grype and OSV-Scanner all found it |
| A Go module version with a known advisory, called | Every dependency scanner found it |
| The same module, imported but never called | govulncheck correctly stayed quiet |
A Pod with privileged: true |
Trivy found it (KSV-0017) |
Three ghp_ tokens: a plain file, a _test.go, a .md |
Trivy’s defaults: 1 of 3. gitleaks: 3 of 3 |
exec.Command("sh", "-c", r.URL.Query().Get("x")) |
The smaller Semgrep rule set missed it in 3 of 3 |
Three lessons from that table changed the configuration.
Defaults skip the places secrets actually end up. Trivy’s built-in allow-rules ignore
anything that looks like a test, an example, Markdown or a locale directory. In our
repository that was 115,000 lines of Go tests, every planning document and every
.env.example. One config file fixes it, and the bake-off measured both sides: 3 of 3
canaries found, still 0 real secrets in the tree.
# .trivy/secret.yaml
disable-allow-rules:
- tests
- examples
- markdown
- locale-dir
A faster rule set is not a cheaper one. The smaller Semgrep set ran 23% faster than
p/default and missed the most obvious command injection you can write in Go, three times
in a row. It was out.
Scan with the toolchain production ships. govulncheck checks the standard library of
whichever Go runs it. Run under the version pinned in go.mod, it reported 28 advisories
in code we call, 17 of them in the standard library. Run under the Go version our
production builder actually used, eight patches newer, it reported 11 and 0. The workflow
now resolves the newest patch of go.mod’s minor version, which is what the
golang:<minor>-alpine builder floats to:
minor=$(awk '/^go /{split($2, v, "."); print v[1] "." v[2]; exit}' api/go.mod)
echo "go-version=${minor}.x" >> "$GITHUB_OUTPUT" # setup-go, check-latest: true
What we did not adopt says as much. Grype reported more critical Go findings than Trivy, but every extra one was the same advisory with a different severity label. OSV-Scanner found three middleware advisories that Trivy misses, because they are scoped to a subpackage; govulncheck reports them too, and none is in code we call. The rule written before the first run was that a challenger had to show more verified, fixable HIGH or CRITICAL findings. None did, so we kept fewer tools: Trivy for breadth, govulncheck for reachability, Semgrep for our own code. Secrets got a second, harder look two months later (section 6).
The chosen set costs about 70 seconds of tool time. Trivy’s vulnerability database downloads cold in 6 seconds, so we don’t cache it.
2. Silence Is Never a Pass
A scanner that crashes, loads no rules or reads an empty database can still exit 0 and
print nothing, which is exactly what a clean scan looks like. So no scanner step is
allowed to fail the job. Each one records its exit code in a file, and one script,
report.sh, turns all of them into a verdict. Its own test suite runs first in every job,
because it is the one piece of code whose bug would make every result meaningless.
Then every run has to prove that each scanner did its job:
| Evidence | Red when |
|---|---|
| Canaries planted outside the workspace | Any canary is missed |
| Trivy’s database age | Older than 48 hours |
| Packages Trivy saw | Fewer than 100 Go modules or 1,000 npm packages |
| govulncheck’s Go version | Not the version production builds with |
| govulncheck’s module list | Fewer than 40 modules, or missing chi, pgx or stdlib |
| Semgrep rules loaded | Not equal to the 397 in the lock file |
| The image scanned | Not Alpine, or no Go binary inside |
The Semgrep canary is the pattern the smaller rule set missed, written to a directory of its own on every run:
func Handler(w http.ResponseWriter, r *http.Request) {
_ = exec.Command("sh", "-c", r.URL.Query().Get("x")).Run()
}
Trivy’s secret scanner has a sabotage test on top: a manual run can plant a fresh random token in the workspace, and passes only if the scan reported it and the token appears nowhere in the summary or the artifact. “Not in the report” alone would also be true of a scan that saw nothing, and a report that prints the secret it found is a second leak.
Even the glue lies. Given a path that does not exist, yq prints its usage to stdout and
exits 0, so a step converting our ignore file to JSON “succeeded” and wrote help text. The
step now requires the file to exist and the output to be a JSON object.
3. The Defaults Were Hiding Our Tests
Two weeks after the bake-off, I looked at what Semgrep’s built-in ignore list excluded in our repository: 688 files, and 686 of them were tests. Since the day it was introduced, the SAST scanner had never read a line of test code.
Test code is not a safe place for a vulnerability. It runs in CI next to credentials, it shells out, it writes files, and it is where a supply-chain payload would be least looked at. Bringing it into scope:
| Files | Findings | |
|---|---|---|
| Before | 1,656 | 118 |
| After | 2,387 | 271 |
Nothing fell out of scope in the trade. Of the new findings, exactly one was blocking: a
dangerous-exec-command on exec.Command(os.Args[0]), the standard “re-run the test
binary as the CLI” helper. It is waived at the line, with a comment naming the condition
that would make the waiver wrong.
There is a trap in how you do this. .semgrepignore does not add to Semgrep’s defaults, it
replaces them. A file containing only plans/ took a full scan from 2,080 files to
2,325, and pulled .git into scope. --exclude on the command line is additive. Whatever
exclusion mechanism you use, count the files scanned before and after.
4. The Scanner Is Part of the Attack Surface
On 19 March 2026, almost every version tag of aquasecurity/trivy-action, and the tags of
setup-trivy, were force-pushed to a credential stealer
that read the runner’s memory. Any workflow running uses: aquasecurity/trivy-action@<tag>
that day ran it, with whatever secrets its job held. A security scanner is a third-party
program you run on every commit, often in a job that can also deploy.
So the scan follows a few rules:
- No wrapper actions. A local composite action downloads one Trivy release and refuses it unless its SHA-256 matches the pin, taken from the release’s checksum file and verified with cosign when the pin is bumped. Semgrep runs from an image pinned by digest. govulncheck is a pinned module version, checked against the Go checksum database. New scanner versions wait seven days before we adopt them.
- The scan job holds nothing worth stealing. It has
contents: readand no other permission. The job that scans the deployed images needspackages: readto pull them from our registry, and logs out of the registry before any scanner starts. - A scanner cannot poison a build. The Go setup in the scan job has its cache turned off, so nothing a scanner writes can be restored by a later build.
permissions:
contents: read
# ...
SEMGREP_IMAGE: semgrep/semgrep@sha256:34ab619bf1391a24bfda3f05debd0d8a6ce3093c2d5f9d39cfc00f83c1397823
The scan workflows pin every action by commit SHA. Our test and deploy workflows mostly don’t yet, and those are the jobs with the most secrets. That is the first item in section 9.
5. Where Scans Run, and What Turns Them Red
| When | What it scans | What it can do |
|---|---|---|
| Every pull request | Its own commits, for secrets only | Go red; not a required check |
Every merge to main |
The tree, the API image built from it, and every commit GitHub holds | Go red, and alert |
| Weekly, Monday 03:00 | The tree, every commit, and the images running right now | Go red; images: report only |
| Every production deploy | The release commit, and the exact image to ship | Block the deploy |
The scanners used to run on every pull request too. We moved them to merge-only to save
Actions minutes, and the cost should be said plainly: a vulnerable dependency is now
reported after it lands on main, not before. What tipped it was that the scan had never
been a required check, so on a pull request it was a signal somebody had to choose to
read. Pull requests still run the cheap guards, the ones that take seconds: the rules on
our ignore files, since this week a secret scan of the pull request’s own commits
(section 6), and a check that the production deploy is still wired to its gate. Every way to disable that gate is a one-line, innocent-looking edit, an
if: on the deploy job or a trimmed needs:.
What turns a merge red is narrow on purpose:
- a fixable CRITICAL vulnerability;
- a govulncheck finding in code we actually call, with a fix available;
- a Semgrep finding of HIGH or above;
- any secret;
- any missing evidence from section 2.
Everything else is recorded and does not block. An accepted risk goes into an ignore file, where each entry must carry a statement and an expiry no more than 90 days out, scoped to a package or a path, and a script enforces those rules. Both ignore files are empty today. Nothing has been formally accepted, which means everything is either fixed or still waiting.
The release gate runs between building the production image and deploying it, and
rests on one rule: the tooling is trusted, the target is not. The commit being released is
checked out into src/, the gate’s own commit into tools/, and every script that
decides anything runs from tools/, so a release cannot ship the reporter that judges it.
The digest pulled must equal the digest built. That check survives the gate’s bypass
mode, and a bypass needs a written reason, which goes into the summary and the alert.
On 30 September it blocked v0.28.0 with every scanner clean:
"reasons": ["evidence:step-failed:check-gate-wiring"]
check-gate-wiring.sh: no such file: .github/workflows/security-scan.yml
A change two days earlier had taught the wiring check to read a second workflow, by a
path relative to the repository root. That path exists in a normal checkout, and not in
the gate’s src/ and tools/ layout. The tempting fix, skipping the rule when the file is
missing, is exactly how a gate starts failing open. We passed the gate the right path
instead. The blocked tag stays as the record.
The sweep of deployed images exists for CVEs published after an image shipped. On
8 October a run of it found HIGH advisories in OpenSSL and in excelize, our spreadsheet
library, inside the API image running in production. Several of the fixes needed Go 1.26
or newer, and the OpenSSL one needed an
apk upgrade in the runtime stage: the base image tag still shipped the vulnerable
packages, and the fixed ones existed only in the package repository.
| Image | HIGH | MEDIUM / LOW | UNKNOWN |
|---|---|---|---|
| Running in production | 14 | 50 | 2 |
| After the upgrade | 0 | 0 | 1 (no fix exists; govulncheck: not called) |
Upgrading Go also changed the bytes our certificate renderer writes for the same drawing, which a golden test caught before release. A security bump is a behaviour change, and it goes through the same tests as any other.
That sweep reports and never goes red, deliberately. A finding in a running image was not introduced by the last merge, so failing that run would blame whoever merged last, and what it calls for is a rebuild and a release rather than a code change.
6. A Deleted Secret Is Still on GitHub
The September bake-off tested secret scanning with three ghp_ tokens. That measured
whether Trivy reads test files and Markdown. It did not measure whether Trivy knows the
credentials we actually hold, or whether it looks everywhere they can end up.
Trivy reads the tree at the tip of main. A token added in one commit of a pull request
and removed in the next never reaches that tree, and a squash merge leaves neither commit
in main’s history. GitHub keeps both in refs/pull/<n>/head, and anyone who can clone
the repository can fetch them. When we counted this week, main had 1,114 commits and
the repository as a whole 1,669, across 445 pull request heads.
So we ran a second bake-off the way we ran the first. Every tool read a clone that included the pull request heads, with the network off, and we planted 25 canaries, each shaped like a credential we actually hold and filled with a fresh random value, one of them existing only base64-encoded in a Kubernetes Secret. Canaries caught out of 25, then distinct false-positive values over the whole history:
| Tool | Canaries | False positives |
|---|---|---|
| Betterleaks 1.10.0 | 23 | 78 |
| detect-secrets 1.5.0 | 23 | 1,408 (tree only) |
| gitleaks 8.30.1 | 21 | 23 |
| Titus 1.3.0 | 17 | 55 |
| Kingfisher 2.18.1 | 14 | 19 |
| TruffleHog 3.99.8 | 11 | 19 |
| Trivy 0.74.0, our config | 5 | 0 |
No tool found a real secret anywhere in the history. The scanner we already ran caught 5 of 25: it missed most of the provider tokens, the base64 value and every generic shape.
We took Betterleaks, which tied for the most canaries with a fraction of detect-secrets’ noise. But every tool’s default rules missed the same two canaries, both random strings under names we chose. A value like that has no provider prefix to match, and to an entropy rule it is one more hash. So the configuration adds one rule of our own:
# .betterleaks.toml, abridged: one name shown; the real list names every secret we hold
[[rules]]
id = "cme-known-secret-name"
regex = '''\b(JWT_SECRET)\b["']?[ \t]*[:=][ \t]*["']?([^\s"'`]{8,})'''
secretGroup = 2
An allowlist keeps references and placeholders quiet ($VAR, <...>, os.Getenv), so
a manifest that names a secret without holding it does not fire. With
that rule, Betterleaks caught 25 of 25.
A rule built on a list of names goes stale the day someone adds a secret that isn’t on
it. So a second script reads every secret name out of our Kubernetes manifests and
workflows, writes each one into a scratch directory with a fresh random value, as
NAME=value and as NAME: "value", and fails unless the rule reports both. Its first run
found 70 names; the hand-written list had 23. Each of the others is now either in the
rule or exempted with a written reason, such as an account id that is not a secret.
Section 3 brought tests into SAST. Here we did the opposite for three rules, and measured it first. The generic rules guess from keywords and entropy, and test code is full of fake keys: with them reading tests, the whole history produced 284 findings; with tests kept out of those three rules, 97, and the same 25 canaries either way. The provider rules and our own rule still read tests, because a real key pasted into a test is the case worth catching. All 97 were false positives, each reviewed and waived by fingerprint (commit, file, rule, line), so a waiver covers one occurrence and nothing after it.
It runs in two places:
- Every pull request, over the pull request’s own commits (
base..head), not the merge result, because the merge result is exactly where a deleted token doesn’t show. It runs with the pull request’s own configuration, so a waiver added in the same pull request is something the reviewer has to read. Like the old scans, it is not a required check yet. - Every merge and every weekly run, over every commit of every branch, tag and pull
request head: about 1,700 commits in a few seconds. A checkout fetches branches and tags
only, so the job fetches
refs/pull/*/headfirst, and the verdict wants at least 400 of them. A sweep that quietly read onlymainis red.
Section 2’s rules apply. Four canaries run on every scan, one of them a throwaway repository where a commit adds a token and the next one deletes it. git counts the commits and added lines on its own, so a scan of an empty range cannot pass for a clean one. The binary is installed the way Trivy is in section 4: one release, checked against a SHA-256 pin.
A finding here means the secret is already on GitHub. Deleting the line doesn’t help; rotating the credential does.
7. Green Is Not Clean
Here is this Monday’s weekly run. Verdict: pass. Findings: 518.
| Kind | Count | Mostly |
|---|---|---|
| SAST (Semgrep) | 307 | 82 actions referenced by tag; 113 cookie flags, 108 in test files |
| Vulnerabilities | 113 | Mostly npm packages, many in build and test tooling; 4 from govulncheck, none called |
| Misconfigurations | 98 | Kubernetes securityContext settings |
| Of all of these, HIGH | 55 |
Dependabot tells the same story from another side. When we turned it on in September,
main had 168 open alerts, 11 of them critical. Since then 181 have closed as fixed,
13 critical and 71 high among them. Today 89 are open: none critical, 29 high, and 18 of
those 29 in development dependencies. None has been dismissed, and five Dependabot pull
requests are open, the oldest from 6 October.
A gate should block only what is both severe and actionable, or people learn to ignore it. But a gate calibrated like that says nothing about the backlog behind it, and “the check is green” quietly turns into “we’re fine”. Two weeks after PHP2Shell’s fix, most of the servers Censys could see were still unpatched; that is what “we’ll get to it” looks like at internet scale. The backlog needs its own owner and its own clock, and we have neither yet.
8. What No Scanner Will Find
Go back to April. None of those four criticals is a CVE, and only the default secret looks like a pattern a rule could match. The withdrawal findings were not in code at all. The classes of bug that hurt a platform like ours, a check that doesn’t hold under concurrency, a permission read at the wrong moment, a money flow that can run twice, are invisible to every scanner above.
That is where AI review earns its place. We have written 29 red-team reviews since April, and this is how we run them now:
- Review the plan before the code for anything that touches money, authentication or tenancy. A finding in a plan costs a paragraph. The same finding after release costs an incident.
- Give the reviewer an attacker’s brief and a stopping rule: report only, no fixes, a failure scenario for every finding.
- Reproduce every claim before acting on it. A reviewer’s mistakes arrive in the same confident tone as its findings. One April finding filed among the criticals, missing CSRF protection, said in its own text that our bearer-token design already mitigated it.
- Turn each confirmed finding into a test that fails without the fix, so the class of bug cannot quietly come back.
9. What’s Still Open
All of these can be done with free tools, and none is done yet:
- DAST. Nothing tests the running application from outside. The plan is OWASP ZAP’s baseline and API scans against staging, driven by our existing OpenAPI spec, on the nightly E2E schedule, and Schemathesis to fuzz the same API from the same spec.
- Secrets before push. The pull request check in section 6 finds a token once it is already on GitHub, where only rotation helps. The same binary and configuration can run as a pre-commit hook, and the pull request check can become required.
- Every action pinned by SHA, deploy workflows first, with Dependabot’s
github-actionsecosystem to keep the pins fresh and zizmor to lint the workflows. - A clock for the backlog: a fix window per severity for runtime dependencies, and real ignore entries, with expiry dates, for the things we decide to accept.
- Rules from our own findings. We vendor
p/defaultand have no rules of our own. Every bug class an AI review finds once should become a Semgrep rule in this repository, so a scanner catches it the second time. - The image and the cluster: a smaller runtime image, an SBOM and a keyless cosign
signature for every release, verified at deploy, and the
securityContextfindings above fixed.
What I’d Carry Forward
- Make every scanner prove it ran, on every run. A scan that saw nothing and a scan of a clean tree produce the same output.
- Read what the defaults exclude. Ours kept all of our tests out of SAST and all of our Markdown out of secret scanning, silently, for weeks.
- Scan what an attacker can fetch, not what
mainshows. A squash merge removes a commit from your history, not from GitHub, and no default rule knows what your own secrets are called. - Treat security tooling as code that holds credentials. Pin it, verify it, and give it the smallest permissions that work.
- Block only what is severe and fixable, then give the rest an owner and a clock. A green check is a statement about the gate, not about the code.
- Scanners find known patterns, reviews find design flaws. Both are cheap now, and both are just as cheap for whoever is looking at your code from the other side.