All Posts

ci

What Actually Made Our CI Faster, and What I Threw Away

PR checks went from 8.5 minutes to 3–5. Most of the gain came from one frozen Go cache. A couple of my other ideas cost more than they saved.

Our monorepo (Go API, Next.js app, a Cloudflare Worker, some deploy scripts) runs one GitHub Actions workflow on every pull request. In early September it took a median of 516 seconds, and it was 80% of our Actions bill: 3,515 runner-minutes in 30 days, more than our plan includes.

Two days and seven pull requests later:

Median wall timeBefore: 516s; Caches + shards: 219s; Selection: all groups: 296s; Selection: one tree: 190sMedian wall time0s200s400s600sBefore22 runsBefore: 516s median wall time (22 runs)516sCaches + shards8 runsCaches + shards: 219s median wall time (8 runs)219sSelection: all groups6 runsSelection: all groups: 296s median wall time (6 runs)296sSelection: one tree7 runsSelection: one tree: 190s median wall time (7 runs)190sBilled runner-minutes per runBefore: 28; Caches + shards: 27; Selection: all groups: 27.5; Selection: one tree: 13Billed runner-minutes per run0102030Before22 runsBefore: 28 billed minutes per run (22 runs)28Caches + shards8 runsCaches + shards: 27 billed minutes per run (8 runs)27Selection: all groups6 runsSelection: all groups: 27.5 billed minutes per run (6 runs)27.5Selection: one tree7 runsSelection: one tree: 13 billed minutes per run (7 runs)13
Caches and parallel jobs cut the wait but not the bill. Running only the jobs a change can reach cut the bill, and costs about 25 seconds when every group runs.
Data Table
StageRunsMedian wall time (s)Billed minutes per run
Before2251628
Caches + shards821927
Selection: all groups629627.5
Selection: one tree719013

Short version: a frozen cache was most of it, parallel jobs fixed the wait but not the bill, and the bill needed a different fix. The details, including what didn’t work:

1. Draw the Timeline First

A run ends when its slowest job ends. So before changing anything I pulled job and step timings from the Actions API for one typical run and drew them.

Job timelines of one CI run before and one afterBefore, the Go job ran build, vet and a race test step in series and finished at 526 seconds; about 146 seconds of the test step and 126 seconds of the integration suites were compilation. After, build and vet run in their own job, web tests run as three shards, the build cache removes most compilation, and the run finishes at 216 seconds.SetupBuild, vet or compileTestsBefore: one run, 526s0s100s200s300s400s500sGo build + testGo build + test: 526s in totalGo build + test: 113s of build, vet or compileGo build + test: 32s of build, vet or compileGo build + test: 146s of build, vet or compileGo build + test: 223s of testsWeb build + testWeb build + test: 434s in totalWeb build + test: 345s of testsWeb build + test: 46s of build, vet or compileAPI integrationAPI integration: 298s in totalAPI integration: 126s of build, vet or compileAPI integration: 142s of testsAPI migrationsAPI migrations: 70s in totalAPI migrations: 42s of testsAPI runtime grantsAPI runtime grants: 53s in totalAPI runtime grants: 15s of testsStorage scriptsStorage scripts: 38s in totalStorage scripts: 27s of testsCloudflare WorkerCloudflare Worker: 28s in totalCloudflare Worker: 9s of testsMigration guardsMigration guards: 17s in totalMigration guards: 5s of testsrun ends at 526sAfter caches and shards: one run, 216s0s100s200s300s400s500sGo test (race)Go test (race): 216s in totalGo test (race): 204s of testsGo build + vetGo build + vet: 170s in totalGo build + vet: 132s of build, vet or compileGo build + vet: 32s of build, vet or compileAPI integrationAPI integration: 157s in totalAPI integration: 109s of testsWeb test 1/3Web test 1/3: 155s in totalWeb test 1/3: 115s of testsWeb test 3/3Web test 3/3: 124s in totalWeb test 3/3: 89s of testsWeb test 2/3Web test 2/3: 112s in totalWeb test 2/3: 76s of testsAPI migrationsAPI migrations: 101s in totalAPI migrations: 77s of testsWeb buildWeb build: 86s in totalWeb build: 51s of build, vet or compileMigration guardsMigration guards: 46s in totalMigration guards: 36s of testsStorage scriptsStorage scripts: 38s in totalStorage scripts: 27s of testsCloudflare WorkerCloudflare Worker: 35s in totalCloudflare Worker: 11s of testsAPI runtime grantsAPI runtime grants: 27s in totalAPI runtime grants: 6s of testsrun ends at 216s
The same workflow before and after. Orange is time spent building or compiling: most of it was a cache that never refreshed. Runs 35042913056 and 35127376056.
Data Table
RunJobDuration (s)
BeforeGo build + test526
BeforeWeb build + test434
BeforeAPI integration298
BeforeAPI migrations70
BeforeAPI runtime grants53
BeforeStorage scripts38
BeforeCloudflare Worker28
BeforeMigration guards17
AfterGo test (race)216
AfterGo build + vet170
AfterAPI integration157
AfterWeb test 1/3155
AfterWeb test 3/3124
AfterWeb test 2/3112
AfterAPI migrations101
AfterWeb build86
AfterMigration guards46
AfterStorage scripts38
AfterCloudflare Worker35
AfterAPI runtime grants27

The before panel shows three problems:

  • go build and go vet ran in series before the race suite, but couldn’t catch anything it wouldn’t. go test compiles every package anyway, and vet was continue-on-error. They now run in their own job.
  • The web tests were one 345-second block. They’re now three vitest shards, with next build in a separate job.
  • Orange everywhere: most of the Go jobs’ time was compiling. That turned out to be a cache bug.

2. The Cache That Never Updated

How the setup-go cache frozesetup-go keys its cache on the Go version and go.sum only. After a dependency change, the migration guards job finishes first, after 17 seconds, and saves a 41 MB entry holding one package's modules. The other jobs cannot overwrite the key. Every later run restores that entry as a hit, so the Go test job downloads 95 modules and the integration job 124, and both compile everything from scratch.The run after go.sum changes: every job misses, then races to saveMigration guardsMigration guards: 17sSaves the cache entryfinishes first and saves the entry: 41 MB, what one package neededAPI migrationsAPI migrations: 70sKey already taken: not savedkey already taken: nothing savedAPI integrationAPI integration: 298sKey already taken: not savedGo build + testGo build + test: 526sKey already taken: not savedEvery run after that, until go.sum changes againRestore the 41 MB entrysame key, so it counts as a hitGo test (race)95 modules, cold compileAPI integration124 modules, cold compile
setup-go’s key has no date and no fallback, so the fastest job decides what every later run gets. Ours was a 17-second check that compiles one package.

Every Go job’s log said the same thing:

Cache hit occurred on the primary key setup-go-Linux-x64-ubuntu24-go-1.25.14-b0adb3d…, not saving cache.

actions/setup-go keys its cache on the Go version and go.sum, with no date and no restore-keys. Whichever job finishes first after a dependency change writes the entry, and nothing can overwrite it. Ours was a 17-second migration check that compiles one package. So on every run the race job downloaded 95 modules and compiled everything from scratch.

We turned setup-go’s cache off and keep our own build cache: one entry per build variant, rolled daily, saved only from main. Pull request caches can’t be shared, and our 10 GB cache budget was already 8.5 GB full.

key: go-build-${{ runner.os }}-${{ inputs.variant }}-${{ hashFiles('api/go.sum') }}-${{ steps.stamp.outputs.day }}
restore-keys: |
  go-build-${{ runner.os }}-${{ inputs.variant }}-${{ hashFiles('api/go.sum') }}-
  go-build-${{ runner.os }}-${{ inputs.variant }}-

Two traps in that key, now documented next to it:

  • Two jobs share a variant only if they compile the same flags and the same packages. The fastest job still wins the save. Let the migration check share the plain variant and you’ve rebuilt the setup-go bug yourself.
  • restore-keys match by prefix. go-build-Linux-race- also matches a variant called race-rest. The guard for that first went out with a PR I closed (section 5). It’s back now as the action’s first step: a variant name outside [a-z0-9_] fails the job before any cache is touched.

3. One Run Is an Anecdote

The first run of the branch took 370 seconds, and I nearly wrote “516 → 370” in the PR. The next two took 403 and 413.

Run-to-run variance on GitHub-hosted runnersThe same go test -race step took 235, 301, 369 and 382 seconds on four runs, 147 seconds apart. Three whole runs with a cold build cache took 370, 403 and 413 seconds.200s250s300s350s400s450sgo test -race step4 runs, same cache statego test -race step: 235sgo test -race step: 301sgo test -race step: 369sgo test -race step: 382s235s382s147s apartWhole run3 runs, cold build cacheWhole run: 370sWhole run: 403sWhole run: 413s370s413s43s apart
Nothing changed between these samples except the runner. A single run cannot show a difference smaller than about 40 seconds.
Data Table
MeasurementSamples (s)
go test -race step235, 301, 369, 382
Whole run370, 403, 413

What made a later single run believable was the mechanism showing up in the step timings. When the build cache first hit, the tag-gated integration suites went from 268s to 109s, which is their compile phase disappearing. The restore step took 5–6 seconds instead of the 1-second no-op it logged on every miss.

4. The Fix I Was Proudest of Saved Four Seconds

In the PR description I’d called the module cache the root cause. Then I measured it cold and warm:

Module download time per job, with and without the Go module cacheWithout the cache the module step took 14 to 20 seconds per job; with a 709 MB cache restore it took 11 to 18 seconds. The saving was 1 to 5 seconds per job.No cache: go mod downloadWarm: 709 MB restored0s5s10s15s20s25sMigration guardsMigration guards: 14s without a cacheMigration guards: 11s with the cache−3sAPI migrationsAPI migrations: 15s without a cacheAPI migrations: 11s with the cache−4sGo build + vetGo build + vet: 18s without a cacheGo build + vet: 13s with the cache−5sGo test (race)Go test (race): 20s without a cacheGo test (race): 17s with the cache−3sAPI integrationAPI integration: 19s without a cacheAPI integration: 18s with the cache−1s
The module cache saved one to five seconds a job and took 7% of the 10 GB cache budget. It is gone now.
Data Table
JobNo cache (s)Warm cache (s)
Migration guards1411
API migrations1511
Go build + vet1813
Go test (race)2017
API integration1918

GitHub’s runners reach the Go module proxy about as fast as they reach the cache service, so I deleted the module cache. The migration check got faster without it: 63 seconds cold, 53 warm, 46 with no module cache at all. setup-go keeps cache: false, with a comment explaining why, because turning it back on is exactly the tidy-up someone would do in good faith.

5. Sharding That Would Have Cost More

Next target: the race suite’s two heaviest packages, internal/handler (~145s) and internal/service (~112s). I split their 1,594 test functions across two runners, 797 each, and checked from go test -json that every test ran exactly once. It worked. I closed it without merging.

Which job finished last in each runBefore, the Go test job finished last in 21 of 22 runs. After caches and shards it finished last in 11 of 15, with API integration last in the other 4. With job selection, 6 of 10 and 4 of 10.Go test jobAPI integrationWeb build + testBefore22 runsBefore: Go test job finished last in 21 of 22 runs21Before: Web build + test finished last in 1 of 22 runsCaches and shards15 runsCaches and shards: Go test job finished last in 11 of 15 runs11Caches and shards: API integration finished last in 4 of 15 runs4Job selection, full10 runsJob selection, full: Go test job finished last in 6 of 10 runs6Job selection, full: API integration finished last in 4 of 10 runs4
Once two jobs trade places as the slowest, speeding up either one barely moves the finish line.
Data Table
StageRunsGo test job lastAPI integration lastWeb last
Before222101
Caches and shards151140
Job selection, full runs10640

By then the race job and API integration were trading places as the slowest job. Halving one moves the finish line only as far as the other, and every shard repeats checkout, toolchain setup and compilation. More runner-minutes, same wait.

Three things I learned writing it anyway:

  • go test -run '' runs every test, not none.
  • Anchor the pattern (^(TestA|TestB)$), or TestA also matches TestAB.
  • Sort test names with LC_ALL=C. glibc’s en_US collation ignores underscores, so two machines can split the same list differently.

6. What t.Parallel() Found Instead

The actual long pole was the Postgres-backed integration suite, which runs serially. Turning on t.Parallel() found two bugs before it found any speed:

  • Fixture cleanup deadlocked. Two cleanups walking overlapping foreign keys each held a row the other needed. Postgres aborted one with 40P01, and its rows leaked into the next run. The fix retries the whole cleanup transaction, bounded and with jitter.
  • The skip check read the wrong line. CI fails on unexpected skips, and it found each skip’s reason with grep -B1 '^--- SKIP'. With interleaved output, t.Cleanup prints between a skip message and its --- SKIP line, so an unexpected skip could pass. It now attributes lines using the === RUN, === CONT and === NAME headers.

Parallel mode is still off. Test phone numbers are deterministic and collide, and a few revenue tests assert on global totals. That’s the next piece of work.

7. The Bill Needed a Different Fix

At about 3.5 minutes per PR the wait was fine, but a run still billed ~27 minutes. GitHub rounds each job up to a minute, shards add jobs, and every change still ran every job.

Plain path filters weren’t safe. Our Go tests read the Kubernetes manifests and the workflow files. Our web tests read the API’s permission list. A skipped check looks exactly like a passing one. So the selector fails closed:

How the test workflow picks jobsA first job reads the changed files and maps them to four groups: Go and database, web, the Cloudflare Worker and storage scripts. A group runs unless every changed path is known not to reach it. An unknown path, an empty list, a path outside the repository or a diff truncated at 300 files runs every group. A final Test complete job is the only required check.Changed filesfrom the GitHub APISelect jobsrouting table, ~17sGo and database5 jobs on a pull requestWebbuild + 3 test shardsCloudflare Worker1 jobStorage scripts1 jobTest completerequired checkRuns every group when a path is unknown, the list is empty,a path leaves the repo, or the diff hit the 300-file cap.
The selector fails closed. Branch protection requires only “Test complete”, which fails if a job the selector needed was skipped, cancelled or failed.

Replaying 157 old runs through the selector predicted 27% fewer billed minutes. The 22 runs since came in at 29%: 432 minutes against 605. A web-only PR now bills about 13 minutes instead of 27.

The cost is about 25 seconds on full runs, because every job waits for the selector (~17s) and the final “Test complete” job (~8s). Full runs are also slower than that explains (296s vs 219s median). I haven’t found the rest yet, and six samples aren’t enough to guess.

What I’d Do First Next Time

  • Draw the timeline before touching anything, and redraw it after every change. The slowest job moves.
  • Trust step timings that match a mechanism. Don’t trust one wall-clock number.
  • Measure a cache cold and warm before crediting it.
  • Test anything that decides not to run tests. Every one of ours had a quiet failure mode: vitest list ignoring --shard, a shard that could own zero tests and still pass, a skip check reading the wrong line, a selector that could emit null as a path.