One Spot VM per CI Job, and Everything It Took to Get There
How our GitHub Actions jobs moved from hosted runners to a Linode host, back, to one always-on GCP Spot VM, and finally to a fresh Spot VM for every job in three pools sized from measurements, with the failures behind each move.
Every CI job in our main repository now runs on its own virtual machine. A small service on Cloud Run creates a Google Cloud Spot VM when a job is queued, sized from what that job was measured to use: 1, 2 or 4 vCPUs. The VM runs that one job, and it is deleted when the job ends. Nothing a job does survives into the next one, and nothing on the VM can call a Google API.
It took three weeks and five different homes for our CI to end up there:
This post covers the last two steps on GCP: why we built them, how they work, and the problems we hit in production, including the ones I caused. It assumes you know what a GitHub Actions runner is and have seen a webhook before.
1. The Constraints
cmevietnam/cme is a private monorepo: a Go API, a Next.js app, a Cloudflare Worker and
some deploy scripts. A full run of its Test workflow is 13 jobs plus a selector job, and
a few of them need things a small VM does not have:
| Job | What makes it awkward |
|---|---|
| Go test (race) | -race needs cgo; peak memory ~3 GB |
| API integration, migrations, grants | A postgres service container on the fixed host port 5432 |
| Storage scripts | Starts PostgreSQL containers from inside shell tests |
| Security scan (post-merge) | Three scanners and a docker build: peaked at 3,832 MB |
Deploy jobs are not on this list on purpose. They hold the production cluster’s credentials and stay on GitHub’s runners whatever happens to the rest.
The money side: on a private repository every hosted minute counts against the plan. Over 1–4 October we used 1,514 billed minutes for 605 jobs. Projected over a month that is about 11,500 minutes, or roughly $51 past the 3,000 the plan includes.
2. What the Linode Attempt Taught Us
We had already self-hosted once, in September, on Linode. It ended after a week, and the reason shaped everything that came after.
The final setup was one 4 GB host running two runner instances. Three jobs publish PostgreSQL on port 5432. A label kept them from running at the same time, but nothing stopped them running one after another on the same instance:
07:03:12-07:10:30 cme-runner-1-1 API integration failed at its test step
07:10:47-07:11:09 cme-runner-1-1 API runtime grants Bind for 0.0.0.0:5432 failed
07:11:30-07:11:49 cme-runner-1-1 API migrations same
The failed job’s container was still holding 5432 seventy-two seconds after the job ended. The host did have a cleanup timer for exactly this, but it waited for a moment with no job running, and with two runner instances that moment almost never came. We added concurrency, and the concurrency stopped the cleanup that fixed its own failure mode.
The lesson I carried over: shared state between jobs on one host is the real problem. Memory and CPU are easy to measure. Leftover containers, ports, caches and files are not, and every job adds more of them.
3. Step One on GCP: One Always-On Spot VM
The first GCP version deliberately changed as little as possible: one VM, one runner, one job at a time.
Why GCP. For roughly the same money, GCP gave twice the memory:
| Option (2 vCPU, Singapore) | RAM | $/month | Notes |
|---|---|---|---|
GCP n2d-standard-2 Spot |
8 GB | ~$25 | VM, 50 GB disk and IP, all in |
Linode g6-standard-2 |
4 GB | $24 | Never preempted |
AWS c6a.large / m7i-flex.large Spot |
4 / 8 GB | ~$35 | Disk and IPv4 billed separately |
8 GB meant the security scan, which had come within 78 MB of crashing the 4 GB Linode host, could run here too. The cost is preemption: Google gives a Spot VM 30 seconds of notice and publishes no preemption rate.
What changed from Linode:
- A separate GCP project (
cme-ci-runners). A CI job on a self-hosted runner is root on the VM, because the runner user is in thedockergroup. In the same project as our production buckets, one over-broad IAM grant would hand CI jobs our users’ files. - The GitHub App key moved to Secret Manager. On Linode it was pushed over SSH after every rebuild, behind a Touch ID prompt. A Spot VM has to come back from a preemption on its own, so the VM read the key into memory each time it minted a token.
- Preemption stopped the VM instead of deleting it, keeping its caches, and a Cloud
Scheduler job called
instances.startevery five minutes. - A reset script ran as root before every job, removing every container, network and anonymous volume. With one runner there is always a gap between jobs, so the port clash could not come back.
It worked. Every job passed. And here is what one run looked like:
Data Table
| Run | Job | Runner | Waited (s) | Ran (s) |
|---|---|---|---|---|
| always-on VM | Select jobs | GitHub-hosted | 3 | 15 |
| always-on VM | API migrations | self-hosted | 281 | 82 |
| always-on VM | Go build + vet | self-hosted | 380 | 96 |
| always-on VM | API runtime grants | self-hosted | 493 | 31 |
| always-on VM | Migration guards | self-hosted | 541 | 17 |
| always-on VM | Web build | self-hosted | 574 | 108 |
| always-on VM | cdn Worker | self-hosted | 701 | 70 |
| always-on VM | Go test (race) | self-hosted | 788 | 443 |
| always-on VM | Web test 1/3 | self-hosted | 1248 | 216 |
| always-on VM | Web test 2/3 | self-hosted | 1482 | 207 |
| always-on VM | Storage scripts | self-hosted | 1707 | 315 |
| always-on VM | Web test 3/3 | self-hosted | 2039 | 221 |
| always-on VM | API integration | self-hosted | 2279 | 378 |
| always-on VM | Test complete | self-hosted | 17 | 14 |
Forty-five minutes for a pull request is not a CI system, it is a queue. The obvious fix, more runners on the same VM, was exactly what had ended the Linode pool.
4. Step Two: One VM per Job
If two jobs must never share a host, give every job its own host. On a cloud with per-second billing this is not as expensive as it sounds: a VM that exists for six minutes costs six minutes.
There are three moving parts:
- The image. Ubuntu 24.04 with the GitHub runner, Docker, Go, Node, the module cache and the jobs’ container images already installed. A workflow rebuilds it every week, and boot-tests it before any VM can use it.
- The manager. A Go service on Cloud Run, standard library only, about 1,400 lines plus 76 tests. It scales to zero and runs as a single instance.
- The reconciler. The same service, called by Cloud Scheduler every minute.
Here is the life of one job:
Just-in-time runner configs
The piece that makes this safe is GitHub’s
just-in-time runner configuration.
The manager calls POST /orgs/{org}/actions/runners/generate-jitconfig and gets back an
opaque blob that registers exactly one runner, once, in one runner group, with fixed
labels. That blob goes into the VM’s metadata, and the VM starts the runner with it:
cfg=$(curl -sf -H 'Metadata-Flavor: Google' \
http://metadata.google.internal/computeMetadata/v1/instance/attributes/jit-config)
sudo -u runner -H ./run.sh --jitconfig "$cfg"
# ...the job runs...
systemctl poweroff
The VM never sees the App key or a registration token. A job that turns hostile can, at worst, use the VM it is already on, and that VM is deleted when the job ends.
The VM list is the state
The manager keeps no database. Every decision starts by listing the VMs in the project and the runners registered on GitHub, and works out what to do from there. That makes it trivially restartable, and it makes one rule load-bearing:
A listing that fails is never read as “no VMs”.
That rule is in the code because the rehearsal broke it. A hand-written launch loop standing in for the manager got one failed listing, concluded the pool was empty, and launched VMs past its cap. The only thing that stopped it was the CPU quota.
The reconciler exists because GitHub does not retry a failed webhook delivery. If the
manager is cold-starting, crashes, or a deploy is in progress when a queued event
arrives, that job would wait until its timeout. Every minute the reconciler:
- deletes VMs that are powered off, preempted, or older than any job’s timeout;
- removes offline runners whose VM no longer exists;
- counts the jobs queued for more than 30 seconds, subtracts the VMs that are booting or listening, and creates VMs for the difference, up to the cap.
The webhook is the fast path, the reconciler is the correct path. Either one alone would eventually run every job.
Routing jobs to it
The repository decides where its jobs go with one variable, RUNNER_TIER. The workflow
turns it into a runs-on label list:
runner_docker: ${{ (vars.RUNNER_TIER == 'gcp-eph'
&& (github.event_name != 'pull_request'
|| github.event.pull_request.head.repo.full_name == github.repository)
&& !endsWith(github.actor, '[bot]'))
&& '["self-hosted","cme-ci-eph"]' || '["ubuntu-latest"]' }}
The middle of that expression is a security control, not configuration. Pull requests
from forks and [bot] actors never reach our VMs, because a job there is root on the
machine. Two more outputs, runner_heavy and runner_light, repeat the same condition
for the bigger and smaller VMs of section 8. Rolling back to hosted runners is one
command: gh variable delete RUNNER_TIER.
5. Baking an Image That Is Not Slower Than GitHub’s
The first images ran our Go tests much slower than GitHub’s own runners did. Each fix below came from a measurement, not a guess:
Data Table
| Measurement | Runner | Note | Seconds |
|---|---|---|---|
| Compile before the first test, Go race step | Hosted ubuntu-24.04 | gcc 13.3, glibc 2.39 | 85 |
| Compile before the first test, Go race step | Our VM, Ubuntu 24.04 | gcc 13.3, glibc 2.39 | 129 |
| Compile before the first test, Go race step | Our VM, Ubuntu 26.04 | gcc 15.2, glibc 2.43 | 251 |
| Whole Go race step, by workspace path | /home/runner/work | the hosted path | 578 |
| Whole Go race step, by workspace path | /opt/actions-runner/_work | the runner’s default | 734 |
| Download Go modules, once per job | Hosted runner | 13 | |
| Download Go modules, once per job | Our VM, Singapore | 33 |
- Ubuntu 24.04, not 26.04. Go’s build cache keys include the C toolchain, because
-raceneeds cgo. A cache written on 24.04 (gcc 13.3, glibc 2.39) almost completely missed on 26.04 (gcc 15.2, glibc 2.43). Using the same release as GitHub’subuntu-24.04lets the two share cache entries. - The hosted runner’s paths. Go also keys compiled packages by source directory. With
the workspace at the runner’s default
/opt/actions-runner/_work, the Go race step took 734 s; at/home/runner/work, where GitHub’s runners work, 578 s. We use the hosted path. - Go and modules preinstalled.
setup-gootherwise downloads Go for every job (10–11 s), and ~114 modules took 31–33 s from Singapore against 13 s on hosted. - Node and the jobs’ container images preinstalled.
postgres:16-alpine, the migration tool, the pinned Semgrep and Playwright images are pulled at bake time, 5.6 GB of the 20 GB disk. Measured afterwards:setup-gowent from ~10 s to ~1 s,setup-nodefrom ~30 s to ~22 s, and starting a service container from ~16 s to ~9 s.
The Go build cache key in cme also gained an OS segment (cme #335), so runners on different Ubuntu
releases stop restoring, and overwriting, each other’s entries. One subtle case: the
always-on VM stayed on 26.04 until it was retired, on purpose. Its GOCACHE pointed
somewhere else, so any cache it saved from the standard path would have been nearly
empty. actions/cache never overwrites an existing key, so if it had shared the
ubuntu24 key it could have won the race for the day’s entry and left every
ephemeral VM a cold cache.
Three bugs in the image pipeline are worth describing, because all three looked like success:
Go was preinstalled, and every job downloaded it anyway. The image put Go exactly
where setup-go looks, and the runner’s systemd unit set RUNNER_TOOL_CACHE to point
there. But the boot script starts the runner with sudo -u runner, and sudo drops the
environment. The variable never reached a job, the tool cache defaulted to
<work>/_tool, and every job spent ~12 s downloading a Go that was already on disk.
Nothing failed. The runner now reads the variable from the .env file in its own
directory, which it loads at start.
A systemd ordering cycle that deleted the runner. The runner’s unit said
After=cloud-final.service. But cloud-final is ordered after multi-user.target,
which wants the runner unit. systemd resolved the cycle by quietly dropping the runner’s
start job. The VM booted and the runner never started. The bake script now
rejects that line, and refuses to publish an image until a VM booted from it prints a
marker (CME_JIT_NO_CONFIG) on the serial console, proving systemd started the unit.
A boot check that could never pass. The first version of that check was
serial-output | grep -q CME_JIT_NO_CONFIG under set -o pipefail. grep -q exits
as soon as it finds a match. The command still writing into the pipe then dies of
SIGPIPE, and pipefail reports that as failure, so the check reported no match even when
the marker was there. The check now reads the console into a variable first. It was then
verified both ways: green on a good image, red with its own message on an image without
the marker.
Keeping the image fresh
An image with a module cache and pinned container images goes stale as the repository
moves on. A workflow in cme now rebuilds it every Monday at 04:00 (UTC+7), when CI is
idle, on a GitHub-hosted runner:
- No key anywhere. The workflow gets GCP credentials through Workload Identity
Federation, and the provider issues a token to exactly one workflow file on
cme’smainbranch. That identity can manage VMs namedcme-bake*and images namedcme-runner-*, and nothing else: not the runner pools, not the manager, no secret. - No SSH either. The builder VM installs everything through cloud-init, cleans cloud-init’s state and powers itself off. The bake waits for a marker on the serial console and for the VM to stop.
- Nothing enters the image family unchecked. The bake creates a candidate image
outside the family, boots a VM from it, and only after the boot check passes copies
it into family
cme-runner. The manager boots the family’s newest image, so a new image needs no deploy, and rolling back is deprecating the bad image. The family keeps the newest three.
That pipeline had its own share of surprises. The marker shows up on the console about
40 s after boot, but once the VM has stopped, about 40 s later, the console reads
empty. A wait that looked for “marker present and VM stopped” in the same poll never saw
both, and timed out on a bake that had worked. The marker is now remembered when it is
first seen. The same wait also tripped over
serial | grep -q under pipefail a second time, in new code. And the first scheduled
bake got all the way through building, then failed creating the image: images create
--labels needs compute.images.setLabels, which the name-scoped role did not have.
6. Security: What Changed
The boundary is the same as before. The runner group admits one repository, and the
runs-on guard keeps forks and bots off. What changed is what a hostile job inside that
boundary could reach:
| Always-on VM | One VM per job | |
|---|---|---|
| GitHub App key | Readable by every job, via the VM’s identity | Only in the manager; the VM has no identity |
| What a job can register | Any runner the App can, for an hour | Nothing: the JIT config is spent |
| Persistence | Survives into the next job | Deleted with the VM |
The manager itself can only touch its own VMs. Its role to create and delete instances is granted with an IAM condition on the resource name, so a bug in the manager cannot delete anything else in the project:
condition {
title = "only-cme-eph"
# One startsWith() for the instances and one for the disks of each pool's prefix:
# cme-eph-, cme-ephx-, cme-ephl-.
expression = join(" || ", flatten([for p in local.pool_prefixes : [
"resource.name.startsWith(\"${local.zone_path}/instances/${p}\")",
"resource.name.startsWith(\"${local.zone_path}/disks/${p}\")",
]]))
}
Listing instances, using the image and the subnet have no per-name resource to put a condition on, so they sit in a second role without one. We confirmed the condition with Policy Troubleshooter rather than trusting the expression.
The GitHub App’s permissions needed the same scrutiny. While a job could read the key,
the App’s permissions were every job’s permissions. It still held
Administration: write on cme, left over from an earlier setup, which would have let
any job rewrite branch protection. That was removed. The per-job design needed one new
permission, Actions: read, to receive workflow_job events. It was only acceptable
because the same change took the key away from every job.
7. Rolling It Out
We ran two rehearsals before writing the manager. A shell loop stood in for it,
re-running the Test workflow on one commit (c3d11d72) so every variant ran the same
code:
| Attempt | Runners | Test workflow |
|---|---|---|
| 1 | The always-on VM | 45.1 min |
| 2 | One VM per job, first image | 23.3 min |
| 3 | One VM per job, hosted-like paths | 17.2 min |
Then the manager went live next to the always-on VM, registering runners under a new
label, cme-ci-eph. Setting RUNNER_TIER=gcp-eph moved the jobs; deleting it would
have moved them back. The plan called for a week side by side. After one clean live
run (14 of 14 jobs passed, nothing left behind) we retired the always-on VM instead. That
was a judgement call, and it means we have not yet seen a preemption or a full month’s
bill on the new setup.
8. What Broke in Production
Quota, not money, set the size of the pool
The project’s N2D_CPUS quota in the region is 16, which is eight 2-vCPU VMs. We
asked for 32 and were denied. The IN_USE_ADDRESSES quota was raised from 8 to 16.
So the cap was 8, until it was split into three pools (below).
The first live run hit the cap one VM early: every 8th insert was accepted, then failed
in its operation with QUOTA_EXCEEDED. The always-on VM was still running and held 2 of
the 16 vCPUs. The cap went to 7 until that VM was retired, then back to 8.
The rehearsals had found a subtler version: a VM being deleted still holds its quota.
For about a minute it sits in STOPPING with its CPUs and IP, and creates fail. In one
rehearsal, not counting those VMs cost 12 QUOTA_EXCEEDED errors. So the cap
counts every VM that holds quota, and a powered-off one does not, because Compute Engine
charges CPU quota for running VMs only.
Freed slots were refilled minutes late
In the first live run, the five jobs beyond the cap waited up to 728 seconds for a VM,
far longer than any job took. The manager’s log showed why: a job’s completed webhook
arrives while its VM is still up, the delete then holds the VM in STOPPING for a minute,
and only a later reconcile could use the slot. Three things made it worse:
- the reconciler ran every 2 minutes;
- it counted a finished VM, one in
STOPPINGor one still up after GitHub had removed its runner, as idle, so each one cancelled out a waiting job; - it counted powered-off VMs toward the cap.
The fix was to define idle precisely. A VM counts as idle only while it is booting or listening and its runner is still registered. The JIT config registers the runner before the VM exists, and GitHub removes an ephemeral runner when its job ends, so a registered runner on a live VM means a job is still coming:
func willTakeAJob(v VM, r Runner, registered bool, listed map[string]bool) bool {
switch v.Status {
case "PROVISIONING", "STAGING", "RUNNING":
return registered && (!r.Busy || !listed[r.Name])
}
return false
}
The reconciler now runs every minute, which is Cloud Scheduler’s shortest interval. Re-running the same commit after the fix:
Data Table
| Run | Job | Runner | Waited (s) | Ran (s) |
|---|---|---|---|---|
| first live run | Select jobs | GitHub-hosted | 2 | 21 |
| first live run | API integration | self-hosted | 39 | 571 |
| first live run | Web test 2/3 | self-hosted | 42 | 213 |
| first live run | cdn Worker | self-hosted | 45 | 86 |
| first live run | Go test (race) | self-hosted | 47 | 744 |
| first live run | Go build + vet | self-hosted | 48 | 239 |
| first live run | API runtime grants | self-hosted | 56 | 42 |
| first live run | Storage scripts | self-hosted | 248 | 236 |
| first live run | API migrations | self-hosted | 250 | 137 |
| first live run | Web test 3/3 | self-hosted | 367 | 215 |
| first live run | Web test 1/3 | self-hosted | 488 | 205 |
| first live run | Migration guards | self-hosted | 493 | 61 |
| first live run | Web build | self-hosted | 728 | 124 |
| first live run | Test complete | self-hosted | 47 | 14 |
| after the fix | Select jobs | GitHub-hosted | 3 | 17 |
| after the fix | Storage scripts | self-hosted | 47 | 242 |
| after the fix | Go test (race) | self-hosted | 47 | 400 |
| after the fix | Go build + vet | self-hosted | 52 | 60 |
| after the fix | Web build | self-hosted | 52 | 126 |
| after the fix | API runtime grants | self-hosted | 53 | 43 |
| after the fix | Migration guards | self-hosted | 58 | 60 |
| after the fix | Web test 1/3 | self-hosted | 59 | 212 |
| after the fix | Web test 2/3 | self-hosted | 63 | 200 |
| after the fix | cdn Worker | self-hosted | 214 | 83 |
| after the fix | Web test 3/3 | self-hosted | 214 | 209 |
| after the fix | API migrations | self-hosted | 273 | 133 |
| after the fix | API integration | self-hosted | 333 | 335 |
| after the fix | Test complete | self-hosted | 43 | 16 |
| three pools | Select jobs | GitHub-hosted | 3 | 20 |
| three pools | API runtime grants | GitHub-hosted | 2 | 24 |
| three pools | Go build + vet · std | self-hosted | 19 | 48 |
| three pools | Web build · std | self-hosted | 44 | 115 |
| three pools | Storage scripts · std | self-hosted | 44 | 229 |
| three pools | cdn Worker · light | self-hosted | 48 | 91 |
| three pools | Go test (race) · heavy | self-hosted | 50 | 250 |
| three pools | Migration guards · light | self-hosted | 52 | 72 |
| three pools | API integration · heavy | self-hosted | 53 | 292 |
| three pools | API migrations · light | self-hosted | 54 | 158 |
| three pools | Web test · heavy | self-hosted | 54 | 233 |
| three pools | Test complete | GitHub-hosted | 3 | 8 |
| hosted | Select jobs | GitHub-hosted | 3 | 19 |
| hosted | API runtime grants | GitHub-hosted | 2 | 23 |
| hosted | Migration guards | GitHub-hosted | 2 | 35 |
| hosted | Go build + vet | GitHub-hosted | 2 | 51 |
| hosted | Web build | GitHub-hosted | 2 | 90 |
| hosted | API migrations | GitHub-hosted | 2 | 106 |
| hosted | Web test 1/3 | GitHub-hosted | 2 | 141 |
| hosted | Web test 2/3 | GitHub-hosted | 2 | 154 |
| hosted | Web test 3/3 | GitHub-hosted | 2 | 178 |
| hosted | Storage scripts | GitHub-hosted | 2 | 221 |
| hosted | API integration | GitHub-hosted | 2 | 282 |
| hosted | Go test (race) | GitHub-hosted | 2 | 385 |
| hosted | cdn Worker | GitHub-hosted | 10 | 58 |
| hosted | Test complete | GitHub-hosted | 2 | 6 |
Same commit, fdd12c17 |
First live run | After the fix |
|---|---|---|
| Longest wait for a VM | 728 s | 333 s |
| Total wait, all 13 jobs | 2,898 s | 1,508 s |
| Whole run | 936 s | 749 s |
That is one run on each side, and the second run also had a warmer Go cache, which shortens the jobs (Go race: 744 s → 400 s) but not the waits. The waits are the part the fix explains.
Why not refill the slot from the completed webhook itself? Because the slot is not
free yet when it arrives, GitHub expects the webhook to be answered within ten seconds,
and a Cloud Run service with CPU only during requests cannot wait a minute after
answering.
One VM too many: two listings are not one snapshot
The re-run above left one VM idle afterwards. Its launch was a race between two GitHub API calls:
17:25:03 cme-eph-65d42610ee takes "Test complete"
17:25:05 reconcile: the runner list says that runner is busy,
the job list still says "Test complete" is queued
→ a queued job with no idle VM coming for it → launch another
The runner list and the job list are separate API calls, so they are not one snapshot:
a runner can read busy while the job it took still reads queued. A VM that the reconciler launches picks up its job about 50
seconds later, close to the next one-minute tick, so this is not a rare race. The
reconciler now reads every job of the active runs, not only the queued ones. A busy
runner that no listed job names must be on a job still listed as queued, so its VM counts
against that job. That is the listed map in the function above.
Thirteen jobs, eight VMs: sizing pools from readings
Even with slots refilled quickly, the cap of 8 meant five jobs always queued behind short ones. In one run API integration waited 373 s for a VM. Three changes followed, and one experiment was dropped:
- Sharding was dropped. I split the Go race job in two (cme #346) and closed it after one run: the binding constraint was VM concurrency, not job length, and a fourteenth job made the queue longer.
- Sub-minute jobs went back to hosted runners (cme #351). A VM takes ~45–50 s to
boot. For
Test complete(hosted p50 7 s) andAPI runtime grants(28 s), that wait was longer than the job, and each took a slot from a long job. The rule is “hosted maximum under 60 s”; two jobs with a 61 s and 63 s maximum stayed. - The three web test shards became one job on a 4-vCPU VM (cme #350). vitest 4 runs
CPUs − 1workers by default, so each 2-vCPU shard had been running one worker.--maxWorkers=100%and one bigger VM replaced three boots and three slots.
The fourth change was more pools, each on a different machine family, because each family has its own CPU quota. To choose machine types from evidence rather than guesses, the VM now prints its peak memory, peak disk and a CPU busy share on the serial console, and the manager reads the console before it deletes the VM:
JOB_RESOURCES job=<id> name="<job name>" vm=<vm> machine=<type> mem_peak_mb=<n> disk_peak_mb=<n> cpu_busy_pct=<p> cores=<c>
A reading the console does not hold is logged as unknown, never as zero, and an
unreadable console never keeps a VM alive. A day of those lines decided the pools:
| Pool | Machine | vCPU / GB | Cap | Jobs | Spot $/h |
|---|---|---|---|---|---|
| heavy | n2-highcpu-4 | 4 / 4 | 4 | Go race (~2.3 cores), Web test (~3.2), API integration | ~0.068 |
| standard | n2d-standard-2 | 2 / 8 | 6 | Go build + vet, Web build, Storage scripts | ~0.025 |
| light | n1-standard-1 | 1 / 3.75 | 4 | API migrations, Migration guards, cdn Worker | ~0.014 |
Two of those rows were corrected by the readings themselves. The heavy pool started on
n2-standard-4 with 16 GB; its jobs peaked at 2.57 GB, so it moved to n2-highcpu-4
for about 26% less per hour. And API integration looked like a light-pool job (~0.7 of 4
cores, 1.6 GB) until it ran there: 374 s instead of 289–295 s, and it is the job that
sets the run’s wall clock. It went back to the heavy pool after one run. The other three
light jobs got 17–42 s slower and still finish well before it.
The quota that binds now is not a family’s but CPUS_ALL_REGIONS: 32 vCPUs shared by
every family. 6 × 2 + 4 × 4 + 4 × 1 is exactly 32, and a Terraform precondition fails
the plan if the caps ever ask for more. That left nothing for the weekly bake’s builder,
so the bake runs when CI is idle and waits out QUOTA_EXCEEDED (up to 30 minutes)
instead of failing.
The first preemption
It came on 8 October, and it hit the shortest job there is. The VM running
Test complete was preempted at 18:14:01 UTC, twelve seconds after the job started.
What followed was worse than a failed job:
18:13:49 Test complete starts on cme-eph-16d0bb0d5b
18:14:01 Spot preemption; the VM is gone
GitHub still shows the runner as busy, and refuses to delete it:
HTTP 422, "currently running a job"
every reconcile: delete runner → 422 → return with the error
18:23:50 GitHub fails the job; the runner can be removed again
For nine minutes, every reconcile round ended at step 2 with that error, before the step that launches VMs for queued jobs. The standard pool could neither rescue a queued job nor refill a freed slot. The other pools kept working, because reconcile already ran each pool on its own. The fix is to collect delete failures, finish the round, and report them at the end. A failed listing still stops the round before anything is touched: those are two different kinds of error, and only one of them makes the rest of the round unsafe.
Smaller ones, each a few minutes of confusion
- Cloud Run reserves paths ending in
z. The deploy script’s smoke test got Google’s own HTML 404 for/healthzwhile/webhookand/reconcileanswered correctly. The request never reached our handler. The known issues page recommends avoiding every path that ends inz. It is/healthnow, with a test that/healthzis not a route. - Cloud Build said
PERMISSION_DENIEDto a project owner right after its API was enabled. The same command a minute later succeeded. The deploy script retries it. - A pinned App installation id answered 404. The App had been uninstalled when the
Linode pool was retired, and a reinstall issues a new id. The manager now looks it up
on each token mint with
GET /orgs/{org}/installation. terraform applywould have deleted the manager. The manager’s image is a Terraform variable whose default is empty, which means “no manager”. A bare apply would have planned its destruction. A wrapper script now reads the live image from the service and passes it in, and there is nomake applytarget at all.- The resource log named the wrong machine.
JOB_RESOURCESfirst took the machine type from the pool’s configuration, so three VMs created asn2-standard-4just before the heavy pool changed type were logged asn2-highcpu-4. It now reads the type from the instance itself. - A stopped VM has an empty console. The VM used to power off 6 s after its job,
and the
completedwebhook could arrive later than that, finding nothing to read. It now waits 30 s. - An IAM binding vanished with its VM. On the always-on design, the Scheduler’s
permission to start the VM was bound to the instance. Rebuilding the VM deleted the
binding, but Terraform still had it in state and planned nothing. We caught it in the
plan for that rebuild and fixed it with
replace_triggered_byon the instance id.
9. Where It Stands
Wall time of the Test workflow, for every run in which every job ran:
Data Table
| Where | Wall time (s) | Run IDs |
|---|---|---|
| GitHub-hosted | 270, 414, 417, 569 | 36920574158, 37059314856, 37392619174, 37197430332 |
| Always-on Spot VM | 2706, 2764 | 37469981570 (attempt 1), 37548628028 |
| One VM per job, rehearsal | 1400, 1030 | 37469981570 (attempts 2 and 3) |
| One VM per job, live | 936, 749, 806, 598 | 37556576150 (attempts 1 and 2), 37662879165, 37666229300 |
| Three sized pools, live | 379 | 37829083245 |
And the estimated monthly cost:
Data Table
| Option | Basis | Dollars per month |
|---|---|---|
| GitHub-hosted | projected, past the free minutes | 51 |
| Linode, 1 host | list price, 2 runners | 24 |
| GCP, always-on Spot VM | estimate | 26 |
| GCP, one Spot VM per job | estimate | 9–10 |
Read those two charts together, honestly. With one pool, GitHub’s hosted runners were clearly faster: 4.5 to 9.5 minutes against our 10 to 15.6. The job timelines show why: every job waited about 50 seconds for its VM to boot, and with 13 jobs and a cap of 8, the last five also waited for a slot.
The three pools removed the queue, and the jobs that set the wall clock now get 4 vCPUs. The latest full run took 379 s, inside the hosted range and close to the ~363 s median of hosted runs in the days before (cme #349). That is one run, and earlier in this project single runs were off by more than that. I will call it parity once a week of runs agrees. The boot wait has not gone anywhere: most self-hosted jobs still start 44–54 s after they are queued, and that is now most of the difference from hosted.
What we got in exchange is a CI bill that should drop from a projected ~$51 a month to roughly the ~$9–10 estimated for one pool (the three pools have not been re-estimated), no shared state between jobs, and a runner fleet in which no job holds a credential.
What is still open:
- The boot wait. ~50 s per job is the largest remaining cost in wall time. A small pool of VMs booted ahead of demand would hide it, at the price of paying for idle VMs. Not built.
- Preemption. We have seen one now. The job on that VM fails and has to be re-run by hand, and GitHub takes about ten minutes to notice. Nothing retries it automatically.
- Cost. No month has been billed yet, and the estimate predates the pools.
- Cloud NAT instead of a public IPv4 per VM. Not priced. NAT bills processed data, and these VMs download a lot.
What I’d Tell Someone Starting This
- Measure where the time goes before choosing a design. The 45-minute timeline made the case for one VM per job in one picture. The job timelines then showed exactly how much of the remaining gap was boot and how much was queueing, which is what pointed at more pools rather than sharding.
- Make the reconciler the correct path, and the webhook only the fast one. Webhooks get lost and GitHub does not retry them. A design that needs every delivery to arrive will eventually strand a job.
- Never treat a failed listing as an empty one, and never let one failed item stop the round. The first would launch VMs past every limit. The second, after the first preemption, froze a pool for nine minutes. They look like opposite rules; both come from asking what the rest of the round can still safely do.
- Size machines from readings of the job, not from the job’s name. API integration looked like a one-core job and was the slowest one on a one-core VM. The heavy pool looked like it needed 16 GB and peaked at 2.57.
- Define every state your capacity math depends on, against the provider’s actual behaviour. “Idle”, “holding quota” and “done” each turned out to mean something different from what I first wrote, and each wrong guess showed up as a queue or a quota error.
- Treat everything a CI job can read as something every contributor can read. That question decided where the App key lives, which permissions it holds, and why the runner VMs have no Google identity at all.