Leaving Supabase Storage for GCS, With an Outage in the Middle
Why we chose Supabase, why 5 GB of egress a month stopped being enough, and what it took to move 439 objects and 255 database rows to Google Cloud Storage without a user-visible change.
On 19 September at 05:15 every production image went dark at once. Not a 404:
HTTP/2 402
Service for this project is restricted due to the following violations:
exceed_cached_egress_quota. The project owner must upgrade their plan or
remove spend caps to restore service.
We were a week into replacing Supabase Storage with Google Cloud Storage when its bandwidth ran out. Part of that bandwidth was mine — the inventory script I wrote for the migration downloaded the whole production bucket to hash it, twice in one day.
1. Why Supabase, and Why We Left
Supabase gave a small team Postgres and object storage in an afternoon, for nothing. That was the right call and it is not the interesting part. The interesting part is the arithmetic that eventually ended it:
free plan 5 GB cached egress per month
production bucket 557 MB
─────────────────────────────────────────────
≈ 9 full reads of everything we own, per month
— shared between visitors, link previews, and
any tool you point at the bucket
next tier up 250 GB / $25 a month
$25 is not a lot of money. It also does not fix anything structural: by September the database had already moved to self-hosted Postgres on Linode, and what was left of Supabase was one job with four pieces of coupling.
| Left behind | Why it mattered |
|---|---|
| One service key | Same credential over both environments’ buckets |
| A MIME and size gate in a dashboard | Not in the repo, not in review; silently 500’d uploads on 20 July |
*.supabase.co in the database |
More than a dozen columns, growing weekly |
| Egress metered per read | Browsers fetch images directly, so the meter runs where we cannot see it |
2. The Outage
| During the restriction | |
|---|---|
| Public images and documents | Dead — 402 from the storage host |
| Private certificates, IP attachments | Dead — the API proxies them from Supabase |
| New uploads on production | Dead — production still wrote to Supabase |
| Staging | Dead — same Supabase project |
| Web, API, login, all data | Fine — Postgres is self-hosted |
Two things were worse than the outage:
- No way out through the front door. The S3-compatible endpoint returned 402 too, so nothing could be copied out while the restriction held. The copy to GCS had not run yet — the only copy of every production media object was inside a project that would not serve it.
- Nothing noticed. API logs were clean, because browsers fetch images without touching our API. The post-deploy smoke test covers the API and the web app, so it could not reach the failure either. It was silent until a person looked at a page.
3. The Script That Helped Cause It
Phase 0 takes an inventory of every bucket so the copy can be proved later. The first version hashed like this:
rclone hashsum SHA256 --download "$REMOTE:$bucket" # 623 MB, every run
The listing already carries a usable hash for most objects, and returns it without putting a single object byte on the wire. Only a handful still have to be downloaded:
rclone lsjson -R --hash --metadata "$REMOTE:$bucket"
Data Table
| Bucket | Objects | Downloaded to hash, first version | Downloaded to hash, today |
|---|---|---|---|
| cme production public | 439 | 623 MB (every object) | 130 MB (12 objects) |
| cme staging public | 341 | ~12 MB (every object) | 0 MB (no object needs it) |
Two bugs came out of that change, both worse than the thing being optimised.
NR == FNR stays true when the first file is empty — and an empty hash file had
just become the common case. awk reads no records from the first file, so the second
file’s first record is both NR 1 and FNR 1, and every record after it keeps the two
counters equal:
NR == FNR { h[$2] = $1; next } # swallowed the whole listing instead
FILENAME == hashmap { h[$2] = $1; next } # the fix: select on FILENAME
The snapshot came out empty under an OK verdict.
The cheap pass silently weakened a real guard. The cross-check that had caught an earlier bug worked because the hash pass walked the whole bucket. It no longer does:
# An INDEPENDENT count of the bucket, metadata only. This is what catches a listing
# that under-reports — the missing -R bug, where lsjson returned one directory entry
# and the run called a 285-object bucket empty.
rclone size --json "$REMOTE:$bucket"
Removing -R from a copy of the script turns 19 cases red. Neither bug broke an
existing test; both opened a new silent path.
4. What It Looks Like Now
| Supabase | GCS | |
|---|---|---|
| Public URL in the DB | https://<ref>.supabase.co/storage/v1/object/public/<b>/<key> |
https://cdn.<root>/<key> |
| Private ref in the DB | a full Supabase URL | private://<key> — no host, bucket from config |
| App identity | one key, both environments | per env: objectCreator + objectViewer public, objectUser private |
| CDN identity | — | per env: one HMAC key, objectViewer on one bucket |
| Public bucket | public | not public — public access prevention enforced |
| Cache-Control | max-age=3600 (default) |
set on upload: public public, max-age=3600, private private, no-store |
No delete permission on the public bucket means the API cannot overwrite an object
there, because replacing one in GCS needs create + delete. Writes carry a
precondition on top of that. Neither bucket is public, for two reasons: allUsers with
objectViewer includes storage.objects.list, so anyone could enumerate every avatar;
and a public bucket lets a caller bypass Cloudflare and bill us GCS egress per request —
we had just learned what an unmetered read path costs.
Config refuses to start a pod in a state that cannot work:
case c.NormalizeLegacyWrites && c.WriteBackend != StorageBackendGCS:
return fmt.Errorf("STORAGE_NORMALIZE_LEGACY_WRITES requires STORAGE_WRITE_BACKEND=%q", StorageBackendGCS)
case c.PublicBucket == c.PrivateBucket:
return errors.New("GCS_PUBLIC_BUCKET and GCS_PRIVATE_BUCKET must differ: private certificates would become public")
case os.Getenv("STORAGE_EMULATOR_HOST") != "":
return fmt.Errorf("STORAGE_EMULATOR_HOST must not be set in env=%s: the SDK would send every request there", c.Env)
Every stored value goes through one resolver that turns it into a (tier, key) pair by
anchored prefix, before any I/O, so a poisoned column cannot address an arbitrary
host or path:
const (
RefPublicCDN RefKind = iota + 1 // https://cdn.<root>/<key>
RefPrivate // private://<key>
RefLegacyPublic // <supabase>/storage/v1/object/public/<bucket>/<key>
RefLegacyPrivate // <supabase>/storage/v1/object/<certBucket>/<key>
)
Three Worker details that were not obvious:
- A cacheable
HEADis converted to aGETat origin by Cloudflare, so the Worker signs and fetches aGETand drops the body. Signing aHEADupstream signs a request that never arrives. - Cache API entries live per data centre and cannot be purged globally by URL, so every
stored entry carries a
Cache-Tagand the incident procedure is a purge by tag. - The client
Cache-Controlis decided by status, not by matching the stored string, because the zone rewrites that string on the way out:
export function toClientCacheControl(
status: number,
storedCacheControl: string | null,
) {
if (status === 200 || status === 206) return CLIENT_CACHE_CONTROL_OK;
if (status === 404) return CLIENT_CACHE_CONTROL_NOT_FOUND;
return storedCacheControl;
}
Live, today:
$ curl -sI https://cdn.cmevietnam.org.vn/uploads/_canary/canary.txt
HTTP/2 200
cache-control: public, max-age=3600
access-control-allow-origin: *
x-content-type-options: nosniff
x-cme-cdn: 1
5. The Order Is the Design, So a Machine Enforces It
Two arrows are not preferences:
- Reads must understand both forms before anything is written in the new one.
Production ran a pre-migration image for ten days — pods from 9 September, code merged
on the 12th — and that image cannot parse
https://cdn…orprivate://at all. - The write normaliser must go on before the URL rewrite. An editor tab opened before
the rewrite’s
COMMITstill holds the old page; saving it writes a legacy URL into a column that has just been converted, with nothing left to repoint it.
Neither lives in a runbook. The image carries a format label, the namespace declares the minimum it accepts, and the deploy refuses a mismatch before migrations and before the apply:
readonly LABEL_KEY=org.cmevietnam.storage-format
readonly CONFIGMAP=cme-storage-format
storage-format-guard: OK image=2 required=2 exit 0
storage-format-guard: REFUSED image=1 required=2 exit 1
storage-preflight: PASS
And the rewrite script reads the flag off every running pod rather than trusting a manifest:
kc get pods -l app=cme-api --field-selector=status.phase=Running -o \
'jsonpath={range .items[*]}{.metadata.name}{" "}{.spec.containers[0].env[?(@.name=="STORAGE_WRITE_BACKEND")].value}{" "}{.spec.containers[0].env[?(@.name=="STORAGE_NORMALIZE_LEGACY_WRITES")].value}{"\n"}{end}'
A --reverse is exempt: demanding the migration’s own preconditions would refuse to undo
at exactly the moment undoing is what is needed.
6. Where It Landed
rclone check --download 0 differences found
storage-rewrite: OK env=prod rows=255 columns=12
storage-urls: OK env=prod public=273 private=4 verified=277/277
db-rewritable-matches=0
| Production | Staging | |
|---|---|---|
| Objects copied and content-verified | 435 public + 4 private | 359, plus 43 brought over from production |
| Metadata asserted | 100% of both buckets | 100% of both buckets |
| Rows rewritten | 255 across 12 columns | 46 across 7 columns |
| References re-fetched after COMMIT | 277 / 277 | 47 / 47 |
Rows still matching supabase.co |
23, on purpose | 0 |
The 23 are audit_logs.details — the recorded diff of a news edit that no surface
renders. Rewriting them would make the audit record claim an edit that never happened.
Nothing is irreversible yet. Every changed row sits in a manifest keyed by a run id,
--reverse applies only where the current value still matches what was written and
reports a conflict rather than overwriting a later edit, and not one Supabase object has
been deleted. Emails have been sent carrying those links, and bookmarks and Open Graph
caches still hold them, so the 90-day compatibility window has not started.
Left to do: delete the Supabase code path once the normaliser has been on for seven days with no legacy reads logged, then decommission.
7. What I’d Carry Forward
- Tooling that reads a metered resource is part of the bill. “It is just an inventory” was 623 MB a run.
- The busiest path had no monitoring, because it did not go through our code. That is a reason to watch it, not a reason to skip it.
- A green suite is not evidence when the change opens a new path. Neither inventory bug broke an existing test; both were found by running the thing against a real bucket.
- If an order is a safety property, make a machine enforce it — a label, a ConfigMap, a preflight Job, a refusal in a script. Not a line in a runbook.