All Posts

ci

One Spot VM per CI Job, and Everything It Took to Get There

How our GitHub Actions jobs moved from hosted runners to a Linode host, back, to one always-on GCP Spot VM, and finally to a fresh Spot VM for every job in three pools sized from measurements, with the failures behind each move.

Every CI job in our main repository now runs on its own virtual machine. A small service on Cloud Run creates a Google Cloud Spot VM when a job is queued, sized from what that job was measured to use: 1, 2 or 4 vCPUs. The VM runs that one job, and it is deleted when the job ends. Nothing a job does survives into the next one, and nothing on the VM can call a Google API.

It took three weeks and five different homes for our CI to end up there:

Where the CI jobs of cmevietnam/cme ran, from mid-September to OctoberGitHub-hosted runners until 20 September, when the free minutes ran out. A Linode host with two runners from 20 to 26 September, retired after a port clash between jobs. GitHub-hosted again until 6 October, projected at about 51 dollars a month over the free minutes. One always-on GCP Spot VM for a day, which ran the 13 jobs one after another in about 45 minutes. Since 7 October, one GCP Spot VM per job, in three pools sized per job since 8 October; the latest full run took 6.3 minutes.Where it runs nowTried, then leftGitHub-hosteduntil 20 Sep3,061 of 3,000 free minutes usedLinode, self-hosted20–26 Sep1 host, 2 runners: port clashGitHub-hosted again26 Sep – 6 Octprojected ~$51/month overGCP: one always-on Spot VM6–7 Oct13 jobs in series: ~45 minGCP: one Spot VM per jobsince 7 Oct, pools since 8 Oct6.3 min on the latest full run
Three weeks, five homes. Each move was forced by the one before it: minutes, then a shared host, then a bill, then a queue.

This post covers the last two steps on GCP: why we built them, how they work, and the problems we hit in production, including the ones I caused. It assumes you know what a GitHub Actions runner is and have seen a webhook before.

1. The Constraints

cmevietnam/cme is a private monorepo: a Go API, a Next.js app, a Cloudflare Worker and some deploy scripts. A full run of its Test workflow is 13 jobs plus a selector job, and a few of them need things a small VM does not have:

Job What makes it awkward
Go test (race) -race needs cgo; peak memory ~3 GB
API integration, migrations, grants A postgres service container on the fixed host port 5432
Storage scripts Starts PostgreSQL containers from inside shell tests
Security scan (post-merge) Three scanners and a docker build: peaked at 3,832 MB

Deploy jobs are not on this list on purpose. They hold the production cluster’s credentials and stay on GitHub’s runners whatever happens to the rest.

The money side: on a private repository every hosted minute counts against the plan. Over 1–4 October we used 1,514 billed minutes for 605 jobs. Projected over a month that is about 11,500 minutes, or roughly $51 past the 3,000 the plan includes.

2. What the Linode Attempt Taught Us

We had already self-hosted once, in September, on Linode. It ended after a week, and the reason shaped everything that came after.

The final setup was one 4 GB host running two runner instances. Three jobs publish PostgreSQL on port 5432. A label kept them from running at the same time, but nothing stopped them running one after another on the same instance:

07:03:12-07:10:30  cme-runner-1-1  API integration      failed at its test step
07:10:47-07:11:09  cme-runner-1-1  API runtime grants   Bind for 0.0.0.0:5432 failed
07:11:30-07:11:49  cme-runner-1-1  API migrations       same

The failed job’s container was still holding 5432 seventy-two seconds after the job ended. The host did have a cleanup timer for exactly this, but it waited for a moment with no job running, and with two runner instances that moment almost never came. We added concurrency, and the concurrency stopped the cleanup that fixed its own failure mode.

The lesson I carried over: shared state between jobs on one host is the real problem. Memory and CPU are easy to measure. Leftover containers, ports, caches and files are not, and every job adds more of them.

3. Step One on GCP: One Always-On Spot VM

The first GCP version deliberately changed as little as possible: one VM, one runner, one job at a time.

Why GCP. For roughly the same money, GCP gave twice the memory:

Option (2 vCPU, Singapore) RAM $/month Notes
GCP n2d-standard-2 Spot 8 GB ~$25 VM, 50 GB disk and IP, all in
Linode g6-standard-2 4 GB $24 Never preempted
AWS c6a.large / m7i-flex.large Spot 4 / 8 GB ~$35 Disk and IPv4 billed separately

8 GB meant the security scan, which had come within 78 MB of crashing the 4 GB Linode host, could run here too. The cost is preemption: Google gives a Spot VM 30 seconds of notice and publishes no preemption rate.

What changed from Linode:

  • A separate GCP project (cme-ci-runners). A CI job on a self-hosted runner is root on the VM, because the runner user is in the docker group. In the same project as our production buckets, one over-broad IAM grant would hand CI jobs our users’ files.
  • The GitHub App key moved to Secret Manager. On Linode it was pushed over SSH after every rebuild, behind a Touch ID prompt. A Spot VM has to come back from a preemption on its own, so the VM read the key into memory each time it minted a token.
  • Preemption stopped the VM instead of deleting it, keeping its caches, and a Cloud Scheduler job called instances.start every five minutes.
  • A reset script ran as root before every job, removing every container, network and anonymous volume. With one runner there is always a gap between jobs, so the port clash could not come back.

It worked. Every job passed. And here is what one run looked like:

Job timeline of one Test run on the always-on VMEvery job is queued at 18 seconds. The single runner takes them one at a time, so the jobs form a staircase: API integration, the last of them, waits 2,279 seconds before it starts, and the run ends at 2,706 seconds, 2,198 of them running jobs on the VM.Waiting for a runnerRunning on our VMRunning on GitHub’s runnerOne always-on VM: run 37469981570, 2,706s0s500s1000s1500s2000s2500sSelect jobsSelect jobs: waited 3s for a runnerSelect jobs: ran 15sAPI migrationsAPI migrations: waited 281s for a runnerAPI migrations: ran 82sGo build + vetGo build + vet: waited 380s for a runnerGo build + vet: ran 96sAPI runtime grantsAPI runtime grants: waited 493s for a runnerAPI runtime grants: ran 31sMigration guardsMigration guards: waited 541s for a runnerMigration guards: ran 17sWeb buildWeb build: waited 574s for a runnerWeb build: ran 108scdn Workercdn Worker: waited 701s for a runnercdn Worker: ran 70sGo test (race)Go test (race): waited 788s for a runnerGo test (race): ran 443sWeb test 1/3Web test 1/3: waited 1248s for a runnerWeb test 1/3: ran 216sWeb test 2/3Web test 2/3: waited 1482s for a runnerWeb test 2/3: ran 207sStorage scriptsStorage scripts: waited 1707s for a runnerStorage scripts: ran 315sWeb test 3/3Web test 3/3: waited 2039s for a runnerWeb test 3/3: ran 221sAPI integrationAPI integration: waited 2279s for a runnerAPI integration: ran 378sTest completeTest complete: waited 17s for a runnerTest complete: ran 14sdone at 2706s
Thirteen jobs, one runner. The 2 vCPU / 8 GB VM had room for every job; it did not have room for two at once without bringing back the port clash that ended the Linode pool.
Data Table
RunJobRunnerWaited (s)Ran (s)
always-on VMSelect jobsGitHub-hosted315
always-on VMAPI migrationsself-hosted28182
always-on VMGo build + vetself-hosted38096
always-on VMAPI runtime grantsself-hosted49331
always-on VMMigration guardsself-hosted54117
always-on VMWeb buildself-hosted574108
always-on VMcdn Workerself-hosted70170
always-on VMGo test (race)self-hosted788443
always-on VMWeb test 1/3self-hosted1248216
always-on VMWeb test 2/3self-hosted1482207
always-on VMStorage scriptsself-hosted1707315
always-on VMWeb test 3/3self-hosted2039221
always-on VMAPI integrationself-hosted2279378
always-on VMTest completeself-hosted1714

Forty-five minutes for a pull request is not a CI system, it is a queue. The obvious fix, more runners on the same VM, was exactly what had ended the Linode pool.

4. Step Two: One VM per Job

If two jobs must never share a host, give every job its own host. On a cloud with per-second billing this is not as expensive as it sounds: a VM that exists for six minutes costs six minutes.

The moving parts of one Spot VM per jobGitHub sends a workflow_job webhook to a manager on Cloud Run, which checks its HMAC signature. The manager asks GitHub for a just-in-time runner config and creates a Spot VM, in one of three pools, from a weekly baked image in Compute Engine; its role can create and delete only the pools' own VMs. The VM has no service account; it connects out to GitHub over HTTPS and takes one job. Cloud Scheduler calls the manager's reconcile endpoint every minute. The GitHub App key lives in Secret Manager and only the manager reads it.GitHubActions job queuerunner group cme-ci:cmevietnam/cme onlySecret Managerthe GitHub App keyManager on Cloud Run/webhook: HMAC-checked/reconcile: OIDC tokenscales to zero, 1 instanceCloud Schedulerevery minuteCompute Engineproject cme-ci-runnerscme-eph…Spot, 1, 2 or 4 vCPUno service accountcme-eph…Spot, 1, 2 or 4 vCPUno service accountcme-eph…Spot, 1, 2 or 4 vCPUno service account3 pools, up to 14 VMs,one job eachImage cme-runnerrebuilt weekly, andboot-checked firstworkflow_jobJIT configcreate, deletecme-eph* only/reconcileread by the manager onlythe runner connects out over HTTPS and takes one job
Nothing on a runner VM can talk to Google Cloud, and nothing on it outlives its job. The only long-lived credential, the GitHub App key, never leaves the manager.

There are three moving parts:

  • The image. Ubuntu 24.04 with the GitHub runner, Docker, Go, Node, the module cache and the jobs’ container images already installed. A workflow rebuilds it every week, and boot-tests it before any VM can use it.
  • The manager. A Go service on Cloud Run, standard library only, about 1,400 lines plus 76 tests. It scales to zero and runs as a single instance.
  • The reconciler. The same service, called by Cloud Scheduler every minute.

Here is the life of one job:

One job, from queued to deleted1: GitHub sends a queued workflow_job webhook to the manager. 2: The manager asks GitHub for a just-in-time runner config, which registers runner cme-eph-x. 3: The manager creates a Spot VM with the config in its metadata. 4: The VM boots the baked image and is listening about 50 seconds after the create call. 5: The runner connects to GitHub and takes the job. 6: The job runs. 7: GitHub sends a completed webhook naming the runner. 8: The VM powers itself off. 9: The manager deletes the VM, which stays in STOPPING, holding quota, for about a minute. Every minute a reconcile pass repairs whatever the webhooks missed.GitHubManagerCompute EngineRunner VM1. workflow_job: queued2. JIT config for cme-eph-x3. insert VM, config in metadata4. boot the baked imagelistening ~50s after 35. run.sh --jitconfig: takes the jobThe job runs6. the job runs7. workflow_job: completed8. powers itself off9. delete cme-eph-xSTOPPING ~1 min, holds quotaEvery minute, Cloud Scheduler calls /reconcile: delete stopped or expired VMs and runners with no VM,then create VMs for jobs queued more than 30 s that no booting or listening VM is coming for.
Steps 7 and 9 are an optimisation, not a dependency: a VM that powers off with no webhook is deleted by the next reconcile, so a lost delivery costs a minute of disk, not hours of CPU.

Just-in-time runner configs

The piece that makes this safe is GitHub’s just-in-time runner configuration. The manager calls POST /orgs/{org}/actions/runners/generate-jitconfig and gets back an opaque blob that registers exactly one runner, once, in one runner group, with fixed labels. That blob goes into the VM’s metadata, and the VM starts the runner with it:

cfg=$(curl -sf -H 'Metadata-Flavor: Google' \
  http://metadata.google.internal/computeMetadata/v1/instance/attributes/jit-config)
sudo -u runner -H ./run.sh --jitconfig "$cfg"
# ...the job runs...
systemctl poweroff

The VM never sees the App key or a registration token. A job that turns hostile can, at worst, use the VM it is already on, and that VM is deleted when the job ends.

The VM list is the state

The manager keeps no database. Every decision starts by listing the VMs in the project and the runners registered on GitHub, and works out what to do from there. That makes it trivially restartable, and it makes one rule load-bearing:

A listing that fails is never read as “no VMs”.

That rule is in the code because the rehearsal broke it. A hand-written launch loop standing in for the manager got one failed listing, concluded the pool was empty, and launched VMs past its cap. The only thing that stopped it was the CPU quota.

The reconciler exists because GitHub does not retry a failed webhook delivery. If the manager is cold-starting, crashes, or a deploy is in progress when a queued event arrives, that job would wait until its timeout. Every minute the reconciler:

  1. deletes VMs that are powered off, preempted, or older than any job’s timeout;
  2. removes offline runners whose VM no longer exists;
  3. counts the jobs queued for more than 30 seconds, subtracts the VMs that are booting or listening, and creates VMs for the difference, up to the cap.

The webhook is the fast path, the reconciler is the correct path. Either one alone would eventually run every job.

Routing jobs to it

The repository decides where its jobs go with one variable, RUNNER_TIER. The workflow turns it into a runs-on label list:

runner_docker: ${{ (vars.RUNNER_TIER == 'gcp-eph'
  && (github.event_name != 'pull_request'
  || github.event.pull_request.head.repo.full_name == github.repository)
  && !endsWith(github.actor, '[bot]'))
  && '["self-hosted","cme-ci-eph"]' || '["ubuntu-latest"]' }}

The middle of that expression is a security control, not configuration. Pull requests from forks and [bot] actors never reach our VMs, because a job there is root on the machine. Two more outputs, runner_heavy and runner_light, repeat the same condition for the bigger and smaller VMs of section 8. Rolling back to hosted runners is one command: gh variable delete RUNNER_TIER.

5. Baking an Image That Is Not Slower Than GitHub’s

The first images ran our Go tests much slower than GitHub’s own runners did. Each fix below came from a measurement, not a guess:

Three measured fixes in the runner imageCompiling before the first Go race test took 85 seconds on hosted Ubuntu 24.04, 129 seconds on our Ubuntu 24.04 VM and 251 seconds on our Ubuntu 26.04 VM, whose newer C toolchain missed the shared cache. The whole Go race step took 578 seconds with the workspace at /home/runner/work and 734 seconds at /opt/actions-runner/_work. Downloading Go modules took 13 seconds on a hosted runner and 31 to 33 seconds on our VM in Singapore.Compile before the first test, Go race step0s100s200s300sHosted ubuntu-24.04gcc 13.3, glibc 2.39Hosted ubuntu-24.04: 85s85sOur VM, Ubuntu 24.04gcc 13.3, glibc 2.39Our VM, Ubuntu 24.04: 129s129sOur VM, Ubuntu 26.04gcc 15.2, glibc 2.43Our VM, Ubuntu 26.04: 251s251sWhole Go race step, by workspace path0s200s400s600s800s/home/runner/workthe hosted path/home/runner/work: 578s578s/opt/actions-runner/_workthe runner’s default/opt/actions-runner/_work: 734s734sDownload Go modules, once per job0s10s20s30s40sHosted runnerHosted runner: 13s13sOur VM, SingaporeOur VM, Singapore: 33s31–33s
Orange is what the first image did. Each one is now baked out: Ubuntu 24.04, the hosted runner’s paths, and Go plus the module cache installed in the image. The race step on 26.04 was measured before the move to 24.04.
Data Table
MeasurementRunnerNoteSeconds
Compile before the first test, Go race stepHosted ubuntu-24.04gcc 13.3, glibc 2.3985
Compile before the first test, Go race stepOur VM, Ubuntu 24.04gcc 13.3, glibc 2.39129
Compile before the first test, Go race stepOur VM, Ubuntu 26.04gcc 15.2, glibc 2.43251
Whole Go race step, by workspace path/home/runner/workthe hosted path578
Whole Go race step, by workspace path/opt/actions-runner/_workthe runner’s default734
Download Go modules, once per jobHosted runner13
Download Go modules, once per jobOur VM, Singapore33
  • Ubuntu 24.04, not 26.04. Go’s build cache keys include the C toolchain, because -race needs cgo. A cache written on 24.04 (gcc 13.3, glibc 2.39) almost completely missed on 26.04 (gcc 15.2, glibc 2.43). Using the same release as GitHub’s ubuntu-24.04 lets the two share cache entries.
  • The hosted runner’s paths. Go also keys compiled packages by source directory. With the workspace at the runner’s default /opt/actions-runner/_work, the Go race step took 734 s; at /home/runner/work, where GitHub’s runners work, 578 s. We use the hosted path.
  • Go and modules preinstalled. setup-go otherwise downloads Go for every job (10–11 s), and ~114 modules took 31–33 s from Singapore against 13 s on hosted.
  • Node and the jobs’ container images preinstalled. postgres:16-alpine, the migration tool, the pinned Semgrep and Playwright images are pulled at bake time, 5.6 GB of the 20 GB disk. Measured afterwards: setup-go went from ~10 s to ~1 s, setup-node from ~30 s to ~22 s, and starting a service container from ~16 s to ~9 s.

The Go build cache key in cme also gained an OS segment (cme #335), so runners on different Ubuntu releases stop restoring, and overwriting, each other’s entries. One subtle case: the always-on VM stayed on 26.04 until it was retired, on purpose. Its GOCACHE pointed somewhere else, so any cache it saved from the standard path would have been nearly empty. actions/cache never overwrites an existing key, so if it had shared the ubuntu24 key it could have won the race for the day’s entry and left every ephemeral VM a cold cache.

Three bugs in the image pipeline are worth describing, because all three looked like success:

Go was preinstalled, and every job downloaded it anyway. The image put Go exactly where setup-go looks, and the runner’s systemd unit set RUNNER_TOOL_CACHE to point there. But the boot script starts the runner with sudo -u runner, and sudo drops the environment. The variable never reached a job, the tool cache defaulted to <work>/_tool, and every job spent ~12 s downloading a Go that was already on disk. Nothing failed. The runner now reads the variable from the .env file in its own directory, which it loads at start.

A systemd ordering cycle that deleted the runner. The runner’s unit said After=cloud-final.service. But cloud-final is ordered after multi-user.target, which wants the runner unit. systemd resolved the cycle by quietly dropping the runner’s start job. The VM booted and the runner never started. The bake script now rejects that line, and refuses to publish an image until a VM booted from it prints a marker (CME_JIT_NO_CONFIG) on the serial console, proving systemd started the unit.

A boot check that could never pass. The first version of that check was serial-output | grep -q CME_JIT_NO_CONFIG under set -o pipefail. grep -q exits as soon as it finds a match. The command still writing into the pipe then dies of SIGPIPE, and pipefail reports that as failure, so the check reported no match even when the marker was there. The check now reads the console into a variable first. It was then verified both ways: green on a good image, red with its own message on an image without the marker.

Keeping the image fresh

An image with a module cache and pinned container images goes stale as the repository moves on. A workflow in cme now rebuilds it every Monday at 04:00 (UTC+7), when CI is idle, on a GitHub-hosted runner:

  1. No key anywhere. The workflow gets GCP credentials through Workload Identity Federation, and the provider issues a token to exactly one workflow file on cme’s main branch. That identity can manage VMs named cme-bake* and images named cme-runner-*, and nothing else: not the runner pools, not the manager, no secret.
  2. No SSH either. The builder VM installs everything through cloud-init, cleans cloud-init’s state and powers itself off. The bake waits for a marker on the serial console and for the VM to stop.
  3. Nothing enters the image family unchecked. The bake creates a candidate image outside the family, boots a VM from it, and only after the boot check passes copies it into family cme-runner. The manager boots the family’s newest image, so a new image needs no deploy, and rolling back is deprecating the bad image. The family keeps the newest three.

That pipeline had its own share of surprises. The marker shows up on the console about 40 s after boot, but once the VM has stopped, about 40 s later, the console reads empty. A wait that looked for “marker present and VM stopped” in the same poll never saw both, and timed out on a bake that had worked. The marker is now remembered when it is first seen. The same wait also tripped over serial | grep -q under pipefail a second time, in new code. And the first scheduled bake got all the way through building, then failed creating the image: images create --labels needs compute.images.setLabels, which the name-scoped role did not have.

6. Security: What Changed

The boundary is the same as before. The runner group admits one repository, and the runs-on guard keeps forks and bots off. What changed is what a hostile job inside that boundary could reach:

  Always-on VM One VM per job
GitHub App key Readable by every job, via the VM’s identity Only in the manager; the VM has no identity
What a job can register Any runner the App can, for an hour Nothing: the JIT config is spent
Persistence Survives into the next job Deleted with the VM

The manager itself can only touch its own VMs. Its role to create and delete instances is granted with an IAM condition on the resource name, so a bug in the manager cannot delete anything else in the project:

condition {
  title = "only-cme-eph"
  # One startsWith() for the instances and one for the disks of each pool's prefix:
  # cme-eph-, cme-ephx-, cme-ephl-.
  expression = join(" || ", flatten([for p in local.pool_prefixes : [
    "resource.name.startsWith(\"${local.zone_path}/instances/${p}\")",
    "resource.name.startsWith(\"${local.zone_path}/disks/${p}\")",
  ]]))
}

Listing instances, using the image and the subnet have no per-name resource to put a condition on, so they sit in a second role without one. We confirmed the condition with Policy Troubleshooter rather than trusting the expression.

The GitHub App’s permissions needed the same scrutiny. While a job could read the key, the App’s permissions were every job’s permissions. It still held Administration: write on cme, left over from an earlier setup, which would have let any job rewrite branch protection. That was removed. The per-job design needed one new permission, Actions: read, to receive workflow_job events. It was only acceptable because the same change took the key away from every job.

7. Rolling It Out

We ran two rehearsals before writing the manager. A shell loop stood in for it, re-running the Test workflow on one commit (c3d11d72) so every variant ran the same code:

Attempt Runners Test workflow
1 The always-on VM 45.1 min
2 One VM per job, first image 23.3 min
3 One VM per job, hosted-like paths 17.2 min

Then the manager went live next to the always-on VM, registering runners under a new label, cme-ci-eph. Setting RUNNER_TIER=gcp-eph moved the jobs; deleting it would have moved them back. The plan called for a week side by side. After one clean live run (14 of 14 jobs passed, nothing left behind) we retired the always-on VM instead. That was a judgement call, and it means we have not yet seen a preemption or a full month’s bill on the new setup.

8. What Broke in Production

Quota, not money, set the size of the pool

The project’s N2D_CPUS quota in the region is 16, which is eight 2-vCPU VMs. We asked for 32 and were denied. The IN_USE_ADDRESSES quota was raised from 8 to 16. So the cap was 8, until it was split into three pools (below).

The first live run hit the cap one VM early: every 8th insert was accepted, then failed in its operation with QUOTA_EXCEEDED. The always-on VM was still running and held 2 of the 16 vCPUs. The cap went to 7 until that VM was retired, then back to 8.

The rehearsals had found a subtler version: a VM being deleted still holds its quota. For about a minute it sits in STOPPING with its CPUs and IP, and creates fail. In one rehearsal, not counting those VMs cost 12 QUOTA_EXCEEDED errors. So the cap counts every VM that holds quota, and a powered-off one does not, because Compute Engine charges CPU quota for running VMs only.

Freed slots were refilled minutes late

In the first live run, the five jobs beyond the cap waited up to 728 seconds for a VM, far longer than any job took. The manager’s log showed why: a job’s completed webhook arrives while its VM is still up, the delete then holds the VM in STOPPING for a minute, and only a later reconcile could use the slot. Three things made it worse:

  • the reconciler ran every 2 minutes;
  • it counted a finished VM, one in STOPPING or one still up after GitHub had removed its runner, as idle, so each one cancelled out a waiting job;
  • it counted powered-off VMs toward the cap.

The fix was to define idle precisely. A VM counts as idle only while it is booting or listening and its runner is still registered. The JIT config registers the runner before the VM exists, and GitHub removes an ephemeral runner when its job ends, so a registered runner on a live VM means a job is still coming:

func willTakeAJob(v VM, r Runner, registered bool, listed map[string]bool) bool {
	switch v.Status {
	case "PROVISIONING", "STAGING", "RUNNING":
		return registered && (!r.Busy || !listed[r.Name])
	}
	return false
}

The reconciler now runs every minute, which is Cloud Scheduler’s shortest interval. Re-running the same commit after the fix:

Job timelines of two per-job runs and one GitHub-hosted runIn the first live run with one VM per job, the first jobs start about 45 seconds after they are queued, the time a VM takes to boot, and jobs beyond the cap wait up to 728 seconds; the run takes 936 seconds. After the refill fix, on the same commit, the longest wait is 333 seconds and the run takes 749 seconds. With three sized pools and the sub-minute jobs on hosted runners, every job starts within 54 seconds and the run takes 379 seconds. On GitHub-hosted runners every job starts within 10 seconds and the run takes 417 seconds.Waiting for a runnerRunning on our VMRunning on GitHub’s runnerOne VM per job, first live run: 936s0s200s400s600s800sSelect jobsSelect jobs: waited 2s for a runnerSelect jobs: ran 21sAPI integrationAPI integration: waited 39s for a runnerAPI integration: ran 571sWeb test 2/3Web test 2/3: waited 42s for a runnerWeb test 2/3: ran 213scdn Workercdn Worker: waited 45s for a runnercdn Worker: ran 86sGo test (race)Go test (race): waited 47s for a runnerGo test (race): ran 744sGo build + vetGo build + vet: waited 48s for a runnerGo build + vet: ran 239sAPI runtime grantsAPI runtime grants: waited 56s for a runnerAPI runtime grants: ran 42sStorage scriptsStorage scripts: waited 248s for a runnerStorage scripts: ran 236sAPI migrationsAPI migrations: waited 250s for a runnerAPI migrations: ran 137sWeb test 3/3Web test 3/3: waited 367s for a runnerWeb test 3/3: ran 215sWeb test 1/3Web test 1/3: waited 488s for a runnerWeb test 1/3: ran 205sMigration guardsMigration guards: waited 493s for a runnerMigration guards: ran 61sWeb buildWeb build: waited 728s for a runnerWeb build: ran 124sTest completeTest complete: waited 47s for a runnerTest complete: ran 14sdone at 936sSame commit, after the refill fix: 749s0s200s400s600s800sSelect jobsSelect jobs: waited 3s for a runnerSelect jobs: ran 17sStorage scriptsStorage scripts: waited 47s for a runnerStorage scripts: ran 242sGo test (race)Go test (race): waited 47s for a runnerGo test (race): ran 400sGo build + vetGo build + vet: waited 52s for a runnerGo build + vet: ran 60sWeb buildWeb build: waited 52s for a runnerWeb build: ran 126sAPI runtime grantsAPI runtime grants: waited 53s for a runnerAPI runtime grants: ran 43sMigration guardsMigration guards: waited 58s for a runnerMigration guards: ran 60sWeb test 1/3Web test 1/3: waited 59s for a runnerWeb test 1/3: ran 212sWeb test 2/3Web test 2/3: waited 63s for a runnerWeb test 2/3: ran 200scdn Workercdn Worker: waited 214s for a runnercdn Worker: ran 83sWeb test 3/3Web test 3/3: waited 214s for a runnerWeb test 3/3: ran 209sAPI migrationsAPI migrations: waited 273s for a runnerAPI migrations: ran 133sAPI integrationAPI integration: waited 333s for a runnerAPI integration: ran 335sTest completeTest complete: waited 43s for a runnerTest complete: ran 16sdone at 749sThree sized pools, 12 jobs, run 37829083245: 379s0s200s400s600s800sSelect jobsSelect jobs: waited 3s for a runnerSelect jobs: ran 20sAPI runtime grantsAPI runtime grants: waited 2s for a runnerAPI runtime grants: ran 24sGo build + vet · stdGo build + vet · std: waited 19s for a runnerGo build + vet · std: ran 48sWeb build · stdWeb build · std: waited 44s for a runnerWeb build · std: ran 115sStorage scripts · stdStorage scripts · std: waited 44s for a runnerStorage scripts · std: ran 229scdn Worker · lightcdn Worker · light: waited 48s for a runnercdn Worker · light: ran 91sGo test (race) · heavyGo test (race) · heavy: waited 50s for a runnerGo test (race) · heavy: ran 250sMigration guards · lightMigration guards · light: waited 52s for a runnerMigration guards · light: ran 72sAPI integration · heavyAPI integration · heavy: waited 53s for a runnerAPI integration · heavy: ran 292sAPI migrations · lightAPI migrations · light: waited 54s for a runnerAPI migrations · light: ran 158sWeb test · heavyWeb test · heavy: waited 54s for a runnerWeb test · heavy: ran 233sTest completeTest complete: waited 3s for a runnerTest complete: ran 8sdone at 379sGitHub-hosted, 14 jobs, for reference: 417s0s200s400s600s800sSelect jobsSelect jobs: waited 3s for a runnerSelect jobs: ran 19sAPI runtime grantsAPI runtime grants: waited 2s for a runnerAPI runtime grants: ran 23sMigration guardsMigration guards: waited 2s for a runnerMigration guards: ran 35sGo build + vetGo build + vet: waited 2s for a runnerGo build + vet: ran 51sWeb buildWeb build: waited 2s for a runnerWeb build: ran 90sAPI migrationsAPI migrations: waited 2s for a runnerAPI migrations: ran 106sWeb test 1/3Web test 1/3: waited 2s for a runnerWeb test 1/3: ran 141sWeb test 2/3Web test 2/3: waited 2s for a runnerWeb test 2/3: ran 154sWeb test 3/3Web test 3/3: waited 2s for a runnerWeb test 3/3: ran 178sStorage scriptsStorage scripts: waited 2s for a runnerStorage scripts: ran 221sAPI integrationAPI integration: waited 2s for a runnerAPI integration: ran 282sGo test (race)Go test (race): waited 2s for a runnerGo test (race): ran 385scdn Workercdn Worker: waited 10s for a runnercdn Worker: ran 58sTest completeTest complete: waited 2s for a runnerTest complete: ran 6sdone at 417s
The dashed boxes are the cost of our own runners: about 50 seconds of boot for every job, and, with one pool, a queue for the jobs beyond its cap. With three pools nothing queues. The first run also had a cold Go cache, after the cache key changed.
Data Table
RunJobRunnerWaited (s)Ran (s)
first live runSelect jobsGitHub-hosted221
first live runAPI integrationself-hosted39571
first live runWeb test 2/3self-hosted42213
first live runcdn Workerself-hosted4586
first live runGo test (race)self-hosted47744
first live runGo build + vetself-hosted48239
first live runAPI runtime grantsself-hosted5642
first live runStorage scriptsself-hosted248236
first live runAPI migrationsself-hosted250137
first live runWeb test 3/3self-hosted367215
first live runWeb test 1/3self-hosted488205
first live runMigration guardsself-hosted49361
first live runWeb buildself-hosted728124
first live runTest completeself-hosted4714
after the fixSelect jobsGitHub-hosted317
after the fixStorage scriptsself-hosted47242
after the fixGo test (race)self-hosted47400
after the fixGo build + vetself-hosted5260
after the fixWeb buildself-hosted52126
after the fixAPI runtime grantsself-hosted5343
after the fixMigration guardsself-hosted5860
after the fixWeb test 1/3self-hosted59212
after the fixWeb test 2/3self-hosted63200
after the fixcdn Workerself-hosted21483
after the fixWeb test 3/3self-hosted214209
after the fixAPI migrationsself-hosted273133
after the fixAPI integrationself-hosted333335
after the fixTest completeself-hosted4316
three poolsSelect jobsGitHub-hosted320
three poolsAPI runtime grantsGitHub-hosted224
three poolsGo build + vet · stdself-hosted1948
three poolsWeb build · stdself-hosted44115
three poolsStorage scripts · stdself-hosted44229
three poolscdn Worker · lightself-hosted4891
three poolsGo test (race) · heavyself-hosted50250
three poolsMigration guards · lightself-hosted5272
three poolsAPI integration · heavyself-hosted53292
three poolsAPI migrations · lightself-hosted54158
three poolsWeb test · heavyself-hosted54233
three poolsTest completeGitHub-hosted38
hostedSelect jobsGitHub-hosted319
hostedAPI runtime grantsGitHub-hosted223
hostedMigration guardsGitHub-hosted235
hostedGo build + vetGitHub-hosted251
hostedWeb buildGitHub-hosted290
hostedAPI migrationsGitHub-hosted2106
hostedWeb test 1/3GitHub-hosted2141
hostedWeb test 2/3GitHub-hosted2154
hostedWeb test 3/3GitHub-hosted2178
hostedStorage scriptsGitHub-hosted2221
hostedAPI integrationGitHub-hosted2282
hostedGo test (race)GitHub-hosted2385
hostedcdn WorkerGitHub-hosted1058
hostedTest completeGitHub-hosted26
Same commit, fdd12c17 First live run After the fix
Longest wait for a VM 728 s 333 s
Total wait, all 13 jobs 2,898 s 1,508 s
Whole run 936 s 749 s

That is one run on each side, and the second run also had a warmer Go cache, which shortens the jobs (Go race: 744 s → 400 s) but not the waits. The waits are the part the fix explains.

Why not refill the slot from the completed webhook itself? Because the slot is not free yet when it arrives, GitHub expects the webhook to be answered within ten seconds, and a Cloud Run service with CPU only during requests cannot wait a minute after answering.

One VM too many: two listings are not one snapshot

The re-run above left one VM idle afterwards. Its launch was a race between two GitHub API calls:

17:25:03  cme-eph-65d42610ee takes "Test complete"
17:25:05  reconcile: the runner list says that runner is busy,
          the job list still says "Test complete" is queued
          → a queued job with no idle VM coming for it → launch another

The runner list and the job list are separate API calls, so they are not one snapshot: a runner can read busy while the job it took still reads queued. A VM that the reconciler launches picks up its job about 50 seconds later, close to the next one-minute tick, so this is not a rare race. The reconciler now reads every job of the active runs, not only the queued ones. A busy runner that no listed job names must be on a job still listed as queued, so its VM counts against that job. That is the listed map in the function above.

Thirteen jobs, eight VMs: sizing pools from readings

Even with slots refilled quickly, the cap of 8 meant five jobs always queued behind short ones. In one run API integration waited 373 s for a VM. Three changes followed, and one experiment was dropped:

  • Sharding was dropped. I split the Go race job in two (cme #346) and closed it after one run: the binding constraint was VM concurrency, not job length, and a fourteenth job made the queue longer.
  • Sub-minute jobs went back to hosted runners (cme #351). A VM takes ~45–50 s to boot. For Test complete (hosted p50 7 s) and API runtime grants (28 s), that wait was longer than the job, and each took a slot from a long job. The rule is “hosted maximum under 60 s”; two jobs with a 61 s and 63 s maximum stayed.
  • The three web test shards became one job on a 4-vCPU VM (cme #350). vitest 4 runs CPUs − 1 workers by default, so each 2-vCPU shard had been running one worker. --maxWorkers=100% and one bigger VM replaced three boots and three slots.

The fourth change was more pools, each on a different machine family, because each family has its own CPU quota. To choose machine types from evidence rather than guesses, the VM now prints its peak memory, peak disk and a CPU busy share on the serial console, and the manager reads the console before it deletes the VM:

JOB_RESOURCES job=<id> name="<job name>" vm=<vm> machine=<type> mem_peak_mb=<n> disk_peak_mb=<n> cpu_busy_pct=<p> cores=<c>

A reading the console does not hold is logged as unknown, never as zero, and an unreadable console never keeps a VM alive. A day of those lines decided the pools:

Pool Machine vCPU / GB Cap Jobs Spot $/h
heavy n2-highcpu-4 4 / 4 4 Go race (~2.3 cores), Web test (~3.2), API integration ~0.068
standard n2d-standard-2 2 / 8 6 Go build + vet, Web build, Storage scripts ~0.025
light n1-standard-1 1 / 3.75 4 API migrations, Migration guards, cdn Worker ~0.014

Two of those rows were corrected by the readings themselves. The heavy pool started on n2-standard-4 with 16 GB; its jobs peaked at 2.57 GB, so it moved to n2-highcpu-4 for about 26% less per hour. And API integration looked like a light-pool job (~0.7 of 4 cores, 1.6 GB) until it ran there: 374 s instead of 289–295 s, and it is the job that sets the run’s wall clock. It went back to the heavy pool after one run. The other three light jobs got 17–42 s slower and still finish well before it.

The quota that binds now is not a family’s but CPUS_ALL_REGIONS: 32 vCPUs shared by every family. 6 × 2 + 4 × 4 + 4 × 1 is exactly 32, and a Terraform precondition fails the plan if the caps ever ask for more. That left nothing for the weekly bake’s builder, so the bake runs when CI is idle and waits out QUOTA_EXCEEDED (up to 30 minutes) instead of failing.

The first preemption

It came on 8 October, and it hit the shortest job there is. The VM running Test complete was preempted at 18:14:01 UTC, twelve seconds after the job started. What followed was worse than a failed job:

18:13:49  Test complete starts on cme-eph-16d0bb0d5b
18:14:01  Spot preemption; the VM is gone
          GitHub still shows the runner as busy, and refuses to delete it:
          HTTP 422, "currently running a job"
          every reconcile: delete runner → 422 → return with the error
18:23:50  GitHub fails the job; the runner can be removed again

For nine minutes, every reconcile round ended at step 2 with that error, before the step that launches VMs for queued jobs. The standard pool could neither rescue a queued job nor refill a freed slot. The other pools kept working, because reconcile already ran each pool on its own. The fix is to collect delete failures, finish the round, and report them at the end. A failed listing still stops the round before anything is touched: those are two different kinds of error, and only one of them makes the rest of the round unsafe.

Smaller ones, each a few minutes of confusion

  • Cloud Run reserves paths ending in z. The deploy script’s smoke test got Google’s own HTML 404 for /healthz while /webhook and /reconcile answered correctly. The request never reached our handler. The known issues page recommends avoiding every path that ends in z. It is /health now, with a test that /healthz is not a route.
  • Cloud Build said PERMISSION_DENIED to a project owner right after its API was enabled. The same command a minute later succeeded. The deploy script retries it.
  • A pinned App installation id answered 404. The App had been uninstalled when the Linode pool was retired, and a reinstall issues a new id. The manager now looks it up on each token mint with GET /orgs/{org}/installation.
  • terraform apply would have deleted the manager. The manager’s image is a Terraform variable whose default is empty, which means “no manager”. A bare apply would have planned its destruction. A wrapper script now reads the live image from the service and passes it in, and there is no make apply target at all.
  • The resource log named the wrong machine. JOB_RESOURCES first took the machine type from the pool’s configuration, so three VMs created as n2-standard-4 just before the heavy pool changed type were logged as n2-highcpu-4. It now reads the type from the instance itself.
  • A stopped VM has an empty console. The VM used to power off 6 s after its job, and the completed webhook could arrive later than that, finding nothing to read. It now waits 30 s.
  • An IAM binding vanished with its VM. On the always-on design, the Scheduler’s permission to start the VM was bound to the instance. Rebuilding the VM deleted the binding, but Terraform still had it in state and planned nothing. We caught it in the plan for that rebuild and fixed it with replace_triggered_by on the instance id.

9. Where It Stands

Wall time of the Test workflow, for every run in which every job ran:

Wall time of the Test workflow by where its jobs ranGitHub-hosted runners took 4.5 to 9.5 minutes over four runs. The always-on Spot VM took 45.1 and 46.1 minutes. Two rehearsals of one VM per job took 23.3 and 17.2 minutes, four live runs with one pool took 10.0 to 15.6 minutes, and the one full run with three sized pools so far took 6.3 minutes.0 min10 min20 min30 min40 min50 minGitHub-hosted4 runs, 1–6 OctGitHub-hosted: 270s (4.5 min)GitHub-hosted: 414s (6.9 min)GitHub-hosted: 417s (7.0 min)GitHub-hosted: 569s (9.5 min)4.5–9.5 minAlways-on Spot VM2 runsAlways-on Spot VM: 2706s (45.1 min)Always-on Spot VM: 2764s (46.1 min)45.1–46.1 minOne VM per job, rehearsal2 runs, hand-run loopOne VM per job, rehearsal: 1400s (23.3 min)One VM per job, rehearsal: 1030s (17.2 min)17.2–23.3 minOne VM per job, live4 runs, 7–8 Oct, 1 poolOne VM per job, live: 936s (15.6 min)One VM per job, live: 749s (12.5 min)One VM per job, live: 806s (13.4 min)One VM per job, live: 598s (10.0 min)10.0–15.6 minThree sized pools, live1 run, 8 OctThree sized pools, live: 379s (6.3 min)6.3 min
Only runs in which every job ran, timed from the first job queued to the last job done. The workflow had 14 jobs until the web shards merged into one on 8 October, and has 12 since. The last row is a single run: inside the hosted range, not yet a measured median.
Data Table
WhereWall time (s)Run IDs
GitHub-hosted270, 414, 417, 56936920574158, 37059314856, 37392619174, 37197430332
Always-on Spot VM2706, 276437469981570 (attempt 1), 37548628028
One VM per job, rehearsal1400, 103037469981570 (attempts 2 and 3)
One VM per job, live936, 749, 806, 59837556576150 (attempts 1 and 2), 37662879165, 37666229300
Three sized pools, live37937829083245

And the estimated monthly cost:

Monthly cost of each way of running CIGitHub-hosted runners were projected at about 51 dollars a month past the free minutes. One Linode host cost 24 dollars. One always-on GCP Spot VM is estimated at about 26 dollars, and one Spot VM per job at about 9 to 10 dollars.0102030405060GitHub-hostedprojected, past the free minutesGitHub-hosted: about $51 a month~$51Linode, 1 hostlist price, 2 runnersLinode, 1 host: about $24 a month~$24GCP, always-on Spot VMestimateGCP, always-on Spot VM: about $26 a month~$26GCP, one Spot VM per jobestimateGCP, one Spot VM per job: about $9–10 a month~$9–10US dollars per month for cmevietnam/cme’s CI
The per-job estimate was made for one pool of 2-vCPU VMs, from the measured load: 4,600 jobs and 9,100 job-minutes a month, plus 50 seconds of boot each. The three pools change the mix and have not been re-estimated. The GCP figures are estimates until a full month is billed.
Data Table
OptionBasisDollars per month
GitHub-hostedprojected, past the free minutes51
Linode, 1 hostlist price, 2 runners24
GCP, always-on Spot VMestimate26
GCP, one Spot VM per jobestimate9–10

Read those two charts together, honestly. With one pool, GitHub’s hosted runners were clearly faster: 4.5 to 9.5 minutes against our 10 to 15.6. The job timelines show why: every job waited about 50 seconds for its VM to boot, and with 13 jobs and a cap of 8, the last five also waited for a slot.

The three pools removed the queue, and the jobs that set the wall clock now get 4 vCPUs. The latest full run took 379 s, inside the hosted range and close to the ~363 s median of hosted runs in the days before (cme #349). That is one run, and earlier in this project single runs were off by more than that. I will call it parity once a week of runs agrees. The boot wait has not gone anywhere: most self-hosted jobs still start 44–54 s after they are queued, and that is now most of the difference from hosted.

What we got in exchange is a CI bill that should drop from a projected ~$51 a month to roughly the ~$9–10 estimated for one pool (the three pools have not been re-estimated), no shared state between jobs, and a runner fleet in which no job holds a credential.

What is still open:

  • The boot wait. ~50 s per job is the largest remaining cost in wall time. A small pool of VMs booted ahead of demand would hide it, at the price of paying for idle VMs. Not built.
  • Preemption. We have seen one now. The job on that VM fails and has to be re-run by hand, and GitHub takes about ten minutes to notice. Nothing retries it automatically.
  • Cost. No month has been billed yet, and the estimate predates the pools.
  • Cloud NAT instead of a public IPv4 per VM. Not priced. NAT bills processed data, and these VMs download a lot.

What I’d Tell Someone Starting This

  • Measure where the time goes before choosing a design. The 45-minute timeline made the case for one VM per job in one picture. The job timelines then showed exactly how much of the remaining gap was boot and how much was queueing, which is what pointed at more pools rather than sharding.
  • Make the reconciler the correct path, and the webhook only the fast one. Webhooks get lost and GitHub does not retry them. A design that needs every delivery to arrive will eventually strand a job.
  • Never treat a failed listing as an empty one, and never let one failed item stop the round. The first would launch VMs past every limit. The second, after the first preemption, froze a pool for nine minutes. They look like opposite rules; both come from asking what the rest of the round can still safely do.
  • Size machines from readings of the job, not from the job’s name. API integration looked like a one-core job and was the slowest one on a one-core VM. The heavy pool looked like it needed 16 GB and peaked at 2.57.
  • Define every state your capacity math depends on, against the provider’s actual behaviour. “Idle”, “holding quota” and “done” each turned out to mean something different from what I first wrote, and each wrong guess showed up as a queue or a quota error.
  • Treat everything a CI job can read as something every contributor can read. That question decided where the App key lives, which permissions it holds, and why the runner VMs have no Google identity at all.