What Actually Made Our CI Faster, and What I Threw Away
PR checks went from 8.5 minutes to 3–5. Most of the gain came from one frozen Go cache. A couple of my other ideas cost more than they saved.
Our monorepo (Go API, Next.js app, a Cloudflare Worker, some deploy scripts) runs one GitHub Actions workflow on every pull request. In early September it took a median of 516 seconds, and it was 80% of our Actions bill: 3,515 runner-minutes in 30 days, more than our plan includes.
Two days and seven pull requests later:
Data Table
| Stage | Runs | Median wall time (s) | Billed minutes per run |
|---|---|---|---|
| Before | 22 | 516 | 28 |
| Caches + shards | 8 | 219 | 27 |
| Selection: all groups | 6 | 296 | 27.5 |
| Selection: one tree | 7 | 190 | 13 |
Short version: a frozen cache was most of it, parallel jobs fixed the wait but not the bill, and the bill needed a different fix. The details, including what didn’t work:
1. Draw the Timeline First
A run ends when its slowest job ends. So before changing anything I pulled job and step timings from the Actions API for one typical run and drew them.
Data Table
| Run | Job | Duration (s) |
|---|---|---|
| Before | Go build + test | 526 |
| Before | Web build + test | 434 |
| Before | API integration | 298 |
| Before | API migrations | 70 |
| Before | API runtime grants | 53 |
| Before | Storage scripts | 38 |
| Before | Cloudflare Worker | 28 |
| Before | Migration guards | 17 |
| After | Go test (race) | 216 |
| After | Go build + vet | 170 |
| After | API integration | 157 |
| After | Web test 1/3 | 155 |
| After | Web test 3/3 | 124 |
| After | Web test 2/3 | 112 |
| After | API migrations | 101 |
| After | Web build | 86 |
| After | Migration guards | 46 |
| After | Storage scripts | 38 |
| After | Cloudflare Worker | 35 |
| After | API runtime grants | 27 |
The before panel shows three problems:
go buildandgo vetran in series before the race suite, but couldn’t catch anything it wouldn’t.go testcompiles every package anyway, andvetwascontinue-on-error. They now run in their own job.- The web tests were one 345-second block. They’re now three vitest shards, with
next buildin a separate job. - Orange everywhere: most of the Go jobs’ time was compiling. That turned out to be a cache bug.
2. The Cache That Never Updated
Every Go job’s log said the same thing:
Cache hit occurred on the primary key setup-go-Linux-x64-ubuntu24-go-1.25.14-b0adb3d…, not saving cache.
actions/setup-go keys its cache on the Go version and go.sum, with no date and no
restore-keys. Whichever job finishes first after a dependency change writes the entry,
and nothing can overwrite it. Ours was a 17-second migration check that compiles one
package. So on every run the race job downloaded 95 modules and compiled everything from
scratch.
We turned setup-go’s cache off and keep our own build cache: one entry per build variant,
rolled daily, saved only from main. Pull request caches can’t be shared, and our 10 GB
cache budget was already 8.5 GB full.
key: go-build-${{ runner.os }}-${{ inputs.variant }}-${{ hashFiles('api/go.sum') }}-${{ steps.stamp.outputs.day }}
restore-keys: |
go-build-${{ runner.os }}-${{ inputs.variant }}-${{ hashFiles('api/go.sum') }}-
go-build-${{ runner.os }}-${{ inputs.variant }}-
Two traps in that key, now documented next to it:
- Two jobs share a variant only if they compile the same flags and the same packages.
The fastest job still wins the save. Let the migration check share the
plainvariant and you’ve rebuilt the setup-go bug yourself. restore-keysmatch by prefix.go-build-Linux-race-also matches a variant calledrace-rest. The guard for that first went out with a PR I closed (section 5). It’s back now as the action’s first step: a variant name outside[a-z0-9_]fails the job before any cache is touched.
3. One Run Is an Anecdote
The first run of the branch took 370 seconds, and I nearly wrote “516 → 370” in the PR. The next two took 403 and 413.
Data Table
| Measurement | Samples (s) |
|---|---|
| go test -race step | 235, 301, 369, 382 |
| Whole run | 370, 403, 413 |
What made a later single run believable was the mechanism showing up in the step timings. When the build cache first hit, the tag-gated integration suites went from 268s to 109s, which is their compile phase disappearing. The restore step took 5–6 seconds instead of the 1-second no-op it logged on every miss.
4. The Fix I Was Proudest of Saved Four Seconds
In the PR description I’d called the module cache the root cause. Then I measured it cold and warm:
Data Table
| Job | No cache (s) | Warm cache (s) |
|---|---|---|
| Migration guards | 14 | 11 |
| API migrations | 15 | 11 |
| Go build + vet | 18 | 13 |
| Go test (race) | 20 | 17 |
| API integration | 19 | 18 |
GitHub’s runners reach the Go module proxy about as fast as they reach the cache service,
so I deleted the module cache. The migration check got faster without it: 63 seconds
cold, 53 warm, 46 with no module cache at all. setup-go keeps cache: false, with a
comment explaining why, because turning it back on is exactly the tidy-up someone would
do in good faith.
5. Sharding That Would Have Cost More
Next target: the race suite’s two heaviest packages, internal/handler (~145s) and
internal/service (~112s). I split their 1,594 test functions across two runners,
797 each, and checked from go test -json that every test ran exactly once. It worked.
I closed it without merging.
Data Table
| Stage | Runs | Go test job last | API integration last | Web last |
|---|---|---|---|---|
| Before | 22 | 21 | 0 | 1 |
| Caches and shards | 15 | 11 | 4 | 0 |
| Job selection, full runs | 10 | 6 | 4 | 0 |
By then the race job and API integration were trading places as the slowest job. Halving one moves the finish line only as far as the other, and every shard repeats checkout, toolchain setup and compilation. More runner-minutes, same wait.
Three things I learned writing it anyway:
go test -run ''runs every test, not none.- Anchor the pattern (
^(TestA|TestB)$), orTestAalso matchesTestAB. - Sort test names with
LC_ALL=C. glibc’sen_UScollation ignores underscores, so two machines can split the same list differently.
6. What t.Parallel() Found Instead
The actual long pole was the Postgres-backed integration suite, which runs serially.
Turning on t.Parallel() found two bugs before it found any speed:
- Fixture cleanup deadlocked. Two cleanups walking overlapping foreign keys each held
a row the other needed. Postgres aborted one with
40P01, and its rows leaked into the next run. The fix retries the whole cleanup transaction, bounded and with jitter. - The skip check read the wrong line. CI fails on unexpected skips, and it found each
skip’s reason with
grep -B1 '^--- SKIP'. With interleaved output,t.Cleanupprints between a skip message and its--- SKIPline, so an unexpected skip could pass. It now attributes lines using the=== RUN,=== CONTand=== NAMEheaders.
Parallel mode is still off. Test phone numbers are deterministic and collide, and a few revenue tests assert on global totals. That’s the next piece of work.
7. The Bill Needed a Different Fix
At about 3.5 minutes per PR the wait was fine, but a run still billed ~27 minutes. GitHub rounds each job up to a minute, shards add jobs, and every change still ran every job.
Plain path filters weren’t safe. Our Go tests read the Kubernetes manifests and the workflow files. Our web tests read the API’s permission list. A skipped check looks exactly like a passing one. So the selector fails closed:
Replaying 157 old runs through the selector predicted 27% fewer billed minutes. The 22 runs since came in at 29%: 432 minutes against 605. A web-only PR now bills about 13 minutes instead of 27.
The cost is about 25 seconds on full runs, because every job waits for the selector (~17s) and the final “Test complete” job (~8s). Full runs are also slower than that explains (296s vs 219s median). I haven’t found the rest yet, and six samples aren’t enough to guess.
What I’d Do First Next Time
- Draw the timeline before touching anything, and redraw it after every change. The slowest job moves.
- Trust step timings that match a mechanism. Don’t trust one wall-clock number.
- Measure a cache cold and warm before crediting it.
- Test anything that decides not to run tests. Every one of ours had a quiet failure
mode:
vitest listignoring--shard, a shard that could own zero tests and still pass, a skip check reading the wrong line, a selector that could emitnullas a path.