What Actually Made Our CI Faster, and What I Threw Away

Our monorepo has a Go API, a Next.js frontend, a Cloudflare Worker and a handful of deploy scripts, and one GitHub Actions workflow tests all of it. In early September a pull request waited a median of 516 seconds for that workflow. Nobody had complained about it. People opened another tab, lost the thread of what they were doing, and came back later. That kind of cost never shows up anywhere.

The bill did show it. Over the 30 days to September 18 the test workflow used 3,515 runner-minutes, 80% of the repository’s Actions usage and more than our plan includes for the whole month. So there were two problems, time and money. It took me a couple of wrong turns to see that they have different fixes.

This is what we changed, in the order we changed it, including the parts that didn’t work.

Start With the Job That Sets the Clock

A workflow run finishes when its slowest job finishes. Queueing took about 15 seconds, and cancel-in-progress was already cancelling superseded runs, so neither was the problem. One job finished last in 21 of the 22 pull request runs I looked at:

Job	Median	Where the time went (one representative run)
Go build + test	514s	`go build` 113s, `go vet` 32s, then `go test -race` 369s, ~146s of it compiling
Web build + test	414s	install 30s, one 345s vitest run, `next build` 46s
API integration	316s	126s of compiling before the first test line

Laid out like that, three things stand out. A lot of the time is compilation, not tests. go build and go vet sit in front of the race suite. And the web tests run as one block.

The Cache That Stopped Updating

The compile time was the odd part, because actions/setup-go caches both Go’s module cache and its build cache. Every Go job’s log had the explanation:

Cache hit occurred on the primary key setup-go-Linux-x64-ubuntu24-go-1.25.14-b0adb3d…, not saving cache.

setup-go keys its cache on the Go version and go.sum, nothing else. There’s no date or run number in the key and no restore-keys, so once an entry exists for a given go.sum, nothing writes to it again. Whichever job finishes first after a dependency change creates the entry. Every job after that restores it and skips the save.

Our fastest Go job is a 16-second check of migration file numbering that compiles one package. It won that race every time, and the entry it left behind was 41 MB. The race job downloaded 95 modules and compiled everything from scratch on every run. The integration job downloaded 124.

I turned setup-go’s cache off (cache: false) and replaced it with two of our own:

a module cache, still keyed on go.sum, with every Go job running go mod download first so the entry is complete no matter which job writes it;
a build cache, which can’t be keyed on go.sum because our own source changes invalidate it. It rolls daily, keeps one entry per build variant (race, tag-gated, plain), and is saved only from main.

Saving only from main is about space as much as correctness. A pull request’s cache entries can only be read by that pull request, and our 10 GB cache budget was already 8.5 GB full, mostly npm and BuildKit layers. A new 170 MB entry per pull request would have evicted entries other jobs relied on. We also added a small workflow that deletes a pull request’s caches when it closes. It can’t act retroactively, so I deleted the backlog by hand: 15 entries, 2.7 GB.

Stop Running Things in Series That Don’t Gate Anything

go build and go vet ran before go test -race. But go test compiles every package in the module, including the ones without tests, so a compile error that go build would catch fails the test step anyway. The two don’t share build artifacts either, because race and non-race builds live in separate parts of the build cache. And vet was marked continue-on-error, so it couldn’t fail the run at all. That’s 145 seconds of the slowest job spent on two steps that couldn’t catch anything the race suite wouldn’t. Both still run, in their own job next to the race suite.

The web suite became a three-way vitest shard, and next build moved to its own job, since it doesn’t depend on the tests passing.

Sharding has a failure mode I care about more than speed: a test file that ends up in no shard never runs, and nothing turns red. So before trusting it I ran each shard and compared the result with an unsharded run. Unsharded, it was 214 files and 2,199 tests. The shards had 72, 71 and 71 files, and no file appeared in more than one.

My first version of that check compared the union of vitest list --filesOnly --shard=i/3 against the full file list, and it passed. It turns out vitest list ignores --shard and prints all 214 files for every shard, so the union was complete by construction. Asserting that the shards were disjoint is what caught it.

I also didn’t want the speed-up to come from quietly dropping a step. A small script diffed every run: step between main and the branch, keyed on job and command. Before believing its green, I broke the workflow three ways on purpose (removed go vet, shrank the matrix, deleted the build job) and made sure it went red each time.

The First Numbers Were Optimistic

The first run of the branch took 370 seconds. It was tempting to put “516s to 370s” in the pull request and move on. The second run took 413 seconds, and a third took 403.

Hosted runners vary more than most people expect. The same go test -race ./... step took 369, 301, 382 and 235 seconds across four runs with the same cache state. Anything smaller than about 40 seconds from a single run isn’t a result. At that point the fair summary was 516 seconds down to about 390.

It also made one job worse. API integration took 366, 387 and 399 seconds, against 245 to 354 before. Every sample was above the old maximum, so this wasn’t noise. The job had lost whatever the frozen setup-go entry happened to contain, and its replacement, the build cache, is saved only from main, so it didn’t exist yet.

The Fix I Was Proudest of Saved Four Seconds

In the pull request description I’d called the module cache the root cause. After merging I measured it properly: each job’s module step cold (a cache miss, so a plain go mod download) and warm (a restore).

Job	Cold	Warm
Migration guards	14s	11s
API migrations	15s	11s
Go build + vet	18s	13s
Go test (race)	20s	17s
API integration	19s	18s

Three to five seconds per job, for a 709 MB entry, 7% of our 10 GB cache budget. GitHub’s runners reach the Go module proxy about as fast as they reach the cache service. The module cache had been broken, but fixing it wasn’t worth much. The missing build cache had been the expensive part all along.

So I removed it. The migration guards job, the one that should have benefited most, took 63 seconds with a cold module cache, 53 with a warm one, and 46 with no module cache at all. Restoring 709 MB was slower than downloading the handful of modules it needed.

setup-go still has cache: false, with a comment above it explaining why. Turning that flag back on is the kind of tidy-up someone does in good faith, and it would bring the original bug straight back.

The Build Cache, Once It Existed

The first pull request to restore a build cache entry finished in 220 seconds. That’s one sample, but the step timings matched the mechanism, which is what made it believable. The tag-gated integration suites dropped from 268 to 109 seconds, which is roughly their compile phase disappearing. And the cache restore step took 5 to 6 seconds, instead of the 1-second no-op it had logged on every earlier miss.

go build and go vet were then the last Go job still compiling from cold, because a plain build shares nothing with the race or tag-gated variants. A third variant brought that job from about 170 seconds to about a minute.

Two rules came out of this. Both are written into the cache action’s comments, because both are easy to break.

First, two jobs can share a variant only if they compile with the same flags and the same scope. On main, every job holding a variant races to save it, and actions/cache skips the save once the key exists, so the fastest job wins. If the migration guards job shared the plain variant, it would freeze a near-empty cache for the job that builds everything. That’s the setup-go bug again, in our own code.

Second, restore-keys match by prefix. Our fallback key for the race variant is go-build-Linux-race-, which would also match a variant named race-rest. I wrote a check that rejects variant names outside [a-z0-9_], but it lived in the pull request I describe next, and I closed that one. None of our current variant names contain a dash, so the trap isn’t live. It’s on my list.

Sharding That Would Have Cost More

With the caches warm, most of the race suite’s time was two packages: internal/handler at about 145 seconds and internal/service at about 112. The race detector is nearly all of that. Without -race the same packages take 17.6 and 5.2 seconds. They do image downscaling and bcrypt, which is the memory-heavy kind of work the detector makes expensive.

Top-level Go tests in a package run one after another unless they call t.Parallel(), so splitting a package’s tests across two runners nearly halves its time. I wrote that split: 1,594 test functions, 797 per shard, with each shard checking the go test -json output to confirm that exactly the tests assigned to it had reported a result. There were a few sharp edges:

go test -run '' doesn’t run nothing. It runs everything. A shard with no tests assigned in a package must not invoke that package at all.
The -run pattern has to be anchored, or TestA also selects TestAB.
The assignment sorts with LC_ALL=C. glibc’s en_US collation ignores underscores, so two machines would otherwise shard the same tree differently.

It worked, and I closed it without merging.

By then the race suite wasn’t reliably the slowest job. It and API integration traded places from run to run: the race suite finished last in 11 of 15 runs, integration in the other 4. Halving one of them only moves the finish line as far as the other. And every extra shard repeats checkout, toolchain setup and compilation, so it would have added runner-minutes to every pull request without making any of them finish sooner. I’d done the work before re-checking which job was last. That check takes a minute, and I now do it after every change.

Trying to Parallelize the Integration Suite

The real long pole was internal/integration, a Postgres-backed suite that runs serially. Turning on t.Parallel() didn’t make it faster yet, but it did find two bugs that had only been waiting for concurrency to show up.

The first was a deadlock in fixture cleanup. Each test’s cleanup deletes its rows by walking foreign keys in one transaction, taking row locks table by table. Two cleanups over overlapping data would each hold a row the other needed next. Postgres would abort one with 40P01, and the loser’s fixtures leaked into the next run. The fix retries the whole purge on deadlock and serialization failures, a bounded number of times with jitter. That’s only safe because the purge is a single transaction, so an abort rolls back every decision it made. The test for it reproduces a real Postgres deadlock rather than a mocked error. Postgres chose the purge as the victim in 26 of 26 attempts, and the test treats an attempt where it chooses the other side as inconclusive rather than as a pass.

The second was in our skip accounting. CI fails when an integration test skips for a reason that isn’t on an allowlist, and it found each skip’s reason with grep -B1 '^--- SKIP'. With parallel tests the output interleaves, and t.Cleanup runs right between a test’s skip message and its --- SKIP line. A neighbouring test printing an allowed phrase in that gap made an unexpected skip pass. The check now attributes each line to a test using the === RUN, === CONT, === NAME and === PAUSE headers that go test -v prints.

Parallel mode is still off. Two things block it, and both are in the tests rather than the infrastructure. Test users get phone numbers from a deterministic helper, so parallel tests collide on them. And a few revenue summary tests assert on totals that other tests’ rows change. That’s the next piece of work if wall time matters again.

Cutting the Bill: Run Only What a Change Can Affect

After all that, a pull request took about three and a half minutes, and the bill hadn’t moved. A run still cost about 27 billed minutes, the same as before, because GitHub rounds every job up to a whole minute and the shards had added jobs. And every change still ran everything, so a pull request that only touched web/ ran six Go and database jobs.

The obvious fix is path filters: run the Go jobs when api/** changes. In this repository that’s unsafe, because tests read across trees. Some Go tests read the Kubernetes manifests under deploy/, the workflow files and the deployment guide, to check that they agree with the code. Some web tests read the API’s permission list in api/internal/authz/policy_actions.go. An allowlist would skip those tests silently, and a skipped check looks exactly like a check that passed.

So the selector is a routing table that fails closed. A first job maps the changed paths to four groups (Go, web, the Cloudflare Worker, storage scripts), and a group is skipped only when every changed path is known not to reach it. An unknown path, an empty file list, a path outside the repository, or a diff that GitHub truncated at 300 files all run everything. There are 15 cross-tree reads today, and a separate check fails when a test starts reading a path the routing table doesn’t account for.

Before merging, I replayed all 157 historical runs through the selector script itself. It would have skipped 1,091 billed minutes and spent 157 on the selector job, a net saving of 27%. In the 22 runs since it merged, the real figure is 432 billed minutes against 605 if every run had been a full one, about 29%. A web-only pull request now runs six jobs, bills about 13 minutes and finishes in about 190 seconds.

It isn’t free. Every job now waits for the selector, which takes about 17 seconds, and a final aggregation job adds about 8 more. Full runs since the change have a median of 296 seconds over six runs, against 219 over eight before it. About 25 seconds of that gap is the gating. I haven’t pinned down the rest, and I’m not going to guess from six samples.

That final job, “Test complete”, is the only required status check on main. Branch protection matches required checks by job name, and most of our jobs are now skipped on purpose, so requiring each one would block every pull request that skipped it. “Test complete” fails if any job the selector marked as needed failed, was cancelled, or was skipped. That last case matters, because needs: on its own treats a skipped job as fine.

Where It Ended Up

Pull request runs	Runs	Median wall time	Billed minutes per run
Before any of this	22	516s	28
Caches and shards	8	219s	27
Job selection, every group runs	6	296s	27.5
Job selection, one part of the repo changed	7	190s	13

If I did this again, I’d start with the table at the top of this post and redo it after every change, because the job that sets the clock moves. I’d measure any cache I’d “fixed” both warm and cold before crediting it. I wouldn’t believe a single run. And I’d treat anything that decides not to run a test as something to test in its own right. Every piece of this that skips work had a way to fail quietly: a file listing that ignored --shard, a shard that could own zero tests and still report OK, a skip check that read the wrong line, a selector that could print null as a file path. None of those turned up as a red check. They turned up because something compared what actually ran against what was supposed to, or because I broke the check on purpose to see whether it noticed.