All Posts

storage

Leaving Supabase Storage for GCS, With an Outage in the Middle

Why we chose Supabase, why 5 GB of egress a month stopped being enough, and what it took to move 439 objects and 255 database rows to Google Cloud Storage without a user-visible change.

On 19 September at 05:15 every production image went dark at once. Not a 404:

HTTP/2 402
Service for this project is restricted due to the following violations:
exceed_cached_egress_quota. The project owner must upgrade their plan or
remove spend caps to restore service.

We were a week into replacing Supabase Storage with Google Cloud Storage when its bandwidth ran out. Part of that bandwidth was mine — the inventory script I wrote for the migration downloaded the whole production bucket to hash it, twice in one day.

1. Why Supabase, and Why We Left

Supabase gave a small team Postgres and object storage in an afternoon, for nothing. That was the right call and it is not the interesting part. The interesting part is the arithmetic that eventually ended it:

free plan           5 GB  cached egress per month
production bucket   557 MB
                    ─────────────────────────────────────────────
                    ≈ 9 full reads of everything we own, per month
                      — shared between visitors, link previews, and
                        any tool you point at the bucket
next tier up        250 GB / $25 a month

$25 is not a lot of money. It also does not fix anything structural: by September the database had already moved to self-hosted Postgres on Linode, and what was left of Supabase was one job with four pieces of coupling.

Left behind Why it mattered
One service key Same credential over both environments’ buckets
A MIME and size gate in a dashboard Not in the repo, not in review; silently 500’d uploads on 20 July
*.supabase.co in the database More than a dozen columns, growing weekly
Egress metered per read Browsers fetch images directly, so the meter runs where we cannot see it

2. The Outage

  During the restriction
Public images and documents Dead — 402 from the storage host
Private certificates, IP attachments Dead — the API proxies them from Supabase
New uploads on production Dead — production still wrote to Supabase
Staging Dead — same Supabase project
Web, API, login, all data Fine — Postgres is self-hosted

Two things were worse than the outage:

  • No way out through the front door. The S3-compatible endpoint returned 402 too, so nothing could be copied out while the restriction held. The copy to GCS had not run yet — the only copy of every production media object was inside a project that would not serve it.
  • Nothing noticed. API logs were clean, because browsers fetch images without touching our API. The post-deploy smoke test covers the API and the web app, so it could not reach the failure either. It was silent until a person looked at a page.

3. The Script That Helped Cause It

Phase 0 takes an inventory of every bucket so the copy can be proved later. The first version hashed like this:

rclone hashsum SHA256 --download "$REMOTE:$bucket"   # 623 MB, every run

The listing already carries a usable hash for most objects, and returns it without putting a single object byte on the wire. Only a handful still have to be downloaded:

rclone lsjson -R --hash --metadata "$REMOTE:$bucket"
Bytes leaving the production bucket for one inventory runHashing the inventory by streaming every object cost 623 MB per run. Taking the hash the listing already carries, and streaming only the 12 objects that have none, costs 130 MB.Hashed by downloadingHashed from listing metadata0100200300400500600700Every object streamedthe first versionEvery object streamed: 623 MB per run623 MBListing hash, stream the rest12 objects without oneListing hash, stream the rest: 130 MB per run130 MBMegabytes transferred out of the bucket, per run
Measured on 19 September, when the bucket held 439 objects. The free plan allows 5 GB of cached egress a month, and the first version ran twice in one day — against a bucket that was 557 MB on 13 September.
Data Table
BucketObjectsDownloaded to hash, first versionDownloaded to hash, today
cme production public439623 MB (every object)130 MB (12 objects)
cme staging public341~12 MB (every object)0 MB (no object needs it)

Two bugs came out of that change, both worse than the thing being optimised.

NR == FNR stays true when the first file is empty — and an empty hash file had just become the common case. awk reads no records from the first file, so the second file’s first record is both NR 1 and FNR 1, and every record after it keeps the two counters equal:

NR == FNR   { h[$2] = $1; next }              # swallowed the whole listing instead
FILENAME == hashmap { h[$2] = $1; next }      # the fix: select on FILENAME

The snapshot came out empty under an OK verdict.

The cheap pass silently weakened a real guard. The cross-check that had caught an earlier bug worked because the hash pass walked the whole bucket. It no longer does:

# An INDEPENDENT count of the bucket, metadata only. This is what catches a listing
# that under-reports — the missing -R bug, where lsjson returned one directory entry
# and the run called a 285-object bucket empty.
rclone size --json "$REMOTE:$bucket"

Removing -R from a copy of the script turns 19 cases red. Neither bug broke an existing test; both opened a new silent path.

4. What It Looks Like Now

How an object is read and written after the move to Google Cloud StorageA public read goes from the browser to cdn.cmevietnam.org.vn, where a Cloudflare Worker guards the path, answers from the Workers Cache API, and otherwise makes one HMAC-signed GET to the cme-prod-public bucket, which has public access prevention on. A private read goes to the API, which parses the stored private://<key> reference before any I/O and proxies the bytes from cme-prod-private. A write is stored under uploads/<id>/<uuidv7>.<ext> with a does-not-exist precondition, and the database stores either an https://cdn.cmevietnam.org.vn/<key> URL or a private://<key> reference, never a bucket host.Public readBrowseran <img> on a pagecdn.cmevietnam.org.vnWorker: path guard, Cache API,then one HMAC-signed GETcme-prod-publicpublic access prevention on,no allUsers bindingPrivate readBrowsera certificate PDFAPIRefResolver parses private://<key>before any I/O, then proxiescme-prod-privateunreachable from the internet,even with the keyWriteUpload handlersniffs the typeGCS putuploads/<id>/<uuidv7>.<ext>If DoesNotExist, Cache-Control setThe DB stores a refhttps://cdn.<root>/<key>, orprivate://<key> — never a hostNo bucket is public. The Worker holds a read-only HMAC key scoped to onebucket in one environment; no identity spans both environments.
Every stored value is parsed into a tier and a key before any I/O, so a poisoned column cannot address an arbitrary host or path. Staging is the same picture with its own buckets and key.
  Supabase GCS
Public URL in the DB https://<ref>.supabase.co/storage/v1/object/public/<b>/<key> https://cdn.<root>/<key>
Private ref in the DB a full Supabase URL private://<key> — no host, bucket from config
App identity one key, both environments per env: objectCreator + objectViewer public, objectUser private
CDN identity — per env: one HMAC key, objectViewer on one bucket
Public bucket public not public — public access prevention enforced
Cache-Control max-age=3600 (default) set on upload: public public, max-age=3600, private private, no-store

No delete permission on the public bucket means the API cannot overwrite an object there, because replacing one in GCS needs create + delete. Writes carry a precondition on top of that. Neither bucket is public, for two reasons: allUsers with objectViewer includes storage.objects.list, so anyone could enumerate every avatar; and a public bucket lets a caller bypass Cloudflare and bill us GCS egress per request — we had just learned what an unmetered read path costs.

Config refuses to start a pod in a state that cannot work:

case c.NormalizeLegacyWrites && c.WriteBackend != StorageBackendGCS:
    return fmt.Errorf("STORAGE_NORMALIZE_LEGACY_WRITES requires STORAGE_WRITE_BACKEND=%q", StorageBackendGCS)
case c.PublicBucket == c.PrivateBucket:
    return errors.New("GCS_PUBLIC_BUCKET and GCS_PRIVATE_BUCKET must differ: private certificates would become public")
case os.Getenv("STORAGE_EMULATOR_HOST") != "":
    return fmt.Errorf("STORAGE_EMULATOR_HOST must not be set in env=%s: the SDK would send every request there", c.Env)

Every stored value goes through one resolver that turns it into a (tier, key) pair by anchored prefix, before any I/O, so a poisoned column cannot address an arbitrary host or path:

const (
    RefPublicCDN    RefKind = iota + 1 // https://cdn.<root>/<key>
    RefPrivate                         // private://<key>
    RefLegacyPublic                    // <supabase>/storage/v1/object/public/<bucket>/<key>
    RefLegacyPrivate                   // <supabase>/storage/v1/object/<certBucket>/<key>
)

Three Worker details that were not obvious:

  • A cacheable HEAD is converted to a GET at origin by Cloudflare, so the Worker signs and fetches a GET and drops the body. Signing a HEAD upstream signs a request that never arrives.
  • Cache API entries live per data centre and cannot be purged globally by URL, so every stored entry carries a Cache-Tag and the incident procedure is a purge by tag.
  • The client Cache-Control is decided by status, not by matching the stored string, because the zone rewrites that string on the way out:
export function toClientCacheControl(
  status: number,
  storedCacheControl: string | null,
) {
  if (status === 200 || status === 206) return CLIENT_CACHE_CONTROL_OK;
  if (status === 404) return CLIENT_CACHE_CONTROL_NOT_FOUND;
  return storedCacheControl;
}

Live, today:

$ curl -sI https://cdn.cmevietnam.org.vn/uploads/_canary/canary.txt
HTTP/2 200
cache-control: public, max-age=3600
access-control-allow-origin: *
x-content-type-options: nosniff
x-cme-cdn: 1

5. The Order Is the Design, So a Machine Enforces It

The seven steps of the migration, and where it standsExpand, flip writes, copy and verify, normalize writes, and rewrite URLs are done. Contract, which deletes the Supabase code path, and decommission, which ends with the project deleted, are still ahead. Each step is reversible on its own, and none needs an image rollback.DoneStill ahead1. ExpandNew code, flags offPR 1a, 100 files2. Flip writesNew objects go to GCSstaging, then v0.26.03. Copy and verify439 objects copied0 differences found4. Normalizer onA stale tab cannotwrite a legacy URL5. Rewrite URLs255 rows, 12 columns277/277 refs resolve6. ContractDelete the Supabasecode path7. DecommissionRead-only 90 days,then the project goes
The order is not a preference. Normalizing writes has to precede the URL rewrite, or an editor tab opened beforehand writes a legacy URL into a column that has just been converted.

Two arrows are not preferences:

  • Reads must understand both forms before anything is written in the new one. Production ran a pre-migration image for ten days — pods from 9 September, code merged on the 12th — and that image cannot parse https://cdn… or private:// at all.
  • The write normaliser must go on before the URL rewrite. An editor tab opened before the rewrite’s COMMIT still holds the old page; saving it writes a legacy URL into a column that has just been converted, with nothing left to repoint it.

Neither lives in a runbook. The image carries a format label, the namespace declares the minimum it accepts, and the deploy refuses a mismatch before migrations and before the apply:

readonly LABEL_KEY=org.cmevietnam.storage-format
readonly CONFIGMAP=cme-storage-format

storage-format-guard: OK image=2 required=2        exit 0
storage-format-guard: REFUSED image=1 required=2   exit 1
storage-preflight:    PASS

And the rewrite script reads the flag off every running pod rather than trusting a manifest:

kc get pods -l app=cme-api --field-selector=status.phase=Running -o \
  'jsonpath={range .items[*]}{.metadata.name}{" "}{.spec.containers[0].env[?(@.name=="STORAGE_WRITE_BACKEND")].value}{" "}{.spec.containers[0].env[?(@.name=="STORAGE_NORMALIZE_LEGACY_WRITES")].value}{"\n"}{end}'

A --reverse is exempt: demanding the migration’s own preconditions would refuse to undo at exactly the moment undoing is what is needed.

6. Where It Landed

rclone check --download    0 differences found
storage-rewrite: OK env=prod rows=255 columns=12
storage-urls:    OK env=prod public=273 private=4 verified=277/277
db-rewritable-matches=0
  Production Staging
Objects copied and content-verified 435 public + 4 private 359, plus 43 brought over from production
Metadata asserted 100% of both buckets 100% of both buckets
Rows rewritten 255 across 12 columns 46 across 7 columns
References re-fetched after COMMIT 277 / 277 47 / 47
Rows still matching supabase.co 23, on purpose 0

The 23 are audit_logs.details — the recorded diff of a news edit that no surface renders. Rewriting them would make the audit record claim an edit that never happened.

Nothing is irreversible yet. Every changed row sits in a manifest keyed by a run id, --reverse applies only where the current value still matches what was written and reports a conflict rather than overwriting a later edit, and not one Supabase object has been deleted. Emails have been sent carrying those links, and bookmarks and Open Graph caches still hold them, so the 90-day compatibility window has not started.

Left to do: delete the Supabase code path once the normaliser has been on for seven days with no legacy reads logged, then decommission.

7. What I’d Carry Forward

  • Tooling that reads a metered resource is part of the bill. “It is just an inventory” was 623 MB a run.
  • The busiest path had no monitoring, because it did not go through our code. That is a reason to watch it, not a reason to skip it.
  • A green suite is not evidence when the change opens a new path. Neither inventory bug broke an existing test; both were found by running the thing against a real bucket.
  • If an order is a safety property, make a machine enforce it — a label, a ConfigMap, a preflight Job, a refusal in a script. Not a line in a runbook.