Cutover
Production moved from Render to the self-hosted DigitalOcean droplet on 2026-10-12. Render stays suspended (not deleted) until the 7-day rollback window ends, around 2026-10-20.
Cutover
How api.legendary-arena.com and its PostgreSQL moved from Render to the droplet, what went wrong, and the exact rollback.
Summary
- When: 2026-10-12, about 19:00–19:30 UTC (midday PDT). This was moved up from the planned 11 PM window because production was idle: 0 active matches in the previous 15 minutes and 4 matches created in 24 hours.
- Downtime: 10 m 10 s, as measured by UptimeRobot at its 5-minute check granularity.
- Data: a read-only final dump of Render’s database, restored with
51-restore.ps1. All 41 tables and 6,425 rows were identical, andschema_migrationswas 45. - Engine record: DECISIONS D-24707.
Before the window
- Production secrets installed on the droplet by key name (Secrets and Environment
).
LEGENDS_PUBLISHER_ENABLED=falseuntil the restore. - Every caller pinned to the onrender hostname was moved to
api.(PLAN §9 step 0). The Stripe webhook was the one that needed it. - The droplet was redeployed to the same
maincommit Render was serving. - Monitoring was live: healthchecks.io
la-droplet-monitorand UptimeRobot onapi./health. - The production
backup.envwas prepared on the operator’s desktop (the bucket, the pCloud token, three ping URLs), but not installed until after the restore.
The steps, as run
- Rollback rehearsal on
api-next. Switch it to a DNS-only CNAME →legendary-arena-server.onrender.com, confirm it resolves to Render, then switch it back to the proxied A record. Each change showed in public DNS within seconds. - Suspend Render (Render → service → Settings → Suspend). Writes freeze, and
api.and the onrender host answer503. - Final dump (operator desktop, read-only):
PGOPTIONS='-c default_transaction_read_only=on',pg_dump -Fc --no-owner --no-acl. Record per-table counts from the source. - Copy and restore (operator):Compare
scp -i $HOME/.ssh/id_ed25519 <dump> operator@<droplet-ip>:/tmp/ ssh -i $HOME/.ssh/id_ed25519 operator@<droplet-ip> 'sudo systemctl stop la-server && cd /opt/la-lab/infra && sudo RESTORE_DUMP=/tmp/<dump> RESTORE_TARGET_DB=la RESTORE_ALLOW_LIVE_TARGET=true pwsh -NoProfile -File scripts/51-restore.ps1; sudo systemctl start la-server'51’s row counts with the source, table by table. Shred the dump on the droplet. - Move the legends publisher.
LEGENDS_PUBLISHER_ENABLED=trueon the droplet (then restartla-server), andfalsein Render’s environment. Only the box with the live database publishes. - Cloudflare. Add
api.legendary-arena.comto the Configuration RuleOrigin TLS strict — api hosts(an Or condition). Change theapirecord from the DNS-only CNAME to an A record → the droplet, Proxied. - Remove Render’s custom domain (Render → service → Settings → Custom Domains → delete
api.legendary-arena.com). This step wasn’t in the plan; see below. - Verify through Cloudflare.
/health200,/health/legends-publisherok, Socket.IO handshake 200, WebSocket upgrade101, the droplet IP refuses 443, andjournalctl -u la-serveris clean. Then play a signed-in match and confirm it lands in the droplet’s database. - Backups. Install the production
backup.env, sync50-backup.ps1, and run it once. Confirm the object exists in R2legendary-arena-db-backupsand in pCloud (byte-identical), and thatla-droplet-backupandla-droplet-pcloudare green. Thengh workflow disable db-backup.ymlin the engine repo. - Delete the local dump and run
pnpm check:domainsin the engine repo (11 ok, 0 failed).
What went wrong: Render’s custom domain
After step 6, public DNS returned Cloudflare’s proxy addresses, but api. still served Render’s “Service Suspended” page. That held even when the request was pinned to a Cloudflare edge IP, and the response carried x-render-routing: suspend-by-user.
Render runs on Cloudflare too. When api. was added as a custom domain in Render, Render registered it as a custom hostname in its own Cloudflare for SaaS account, and that registration takes precedence over our zone’s proxied record. Deleting the custom domain in Render’s dashboard released it, and the next request reached the droplet.
A local DNS cache can also show the old answer for up to the record’s TTL. Clear it (Clear-DnsClientCache, or chrome://net-internals/#dns) before deciding something is wrong.
Rollback (until ~2026-10-20)
- Cloudflare: change the
apirecord back to a CNAME →legendary-arena-server.onrender.com, DNS only (grey). - Render: Resume the web service. Then Custom Domains → add
api.legendary-arena.comand wait for Render to verify it and issue the certificate (a few minutes). - The publisher: set
truein Render’s environment andfalseon the droplet. - Verify
api./healthreaches Render.
Anything written on the droplet after the flip isn’t in Render’s database. A rollback after real traffic needs that data copied back: a dump from the droplet restored into Render. Decide that deliberately; don’t improvise it.
After the window (Phase 7)
Delete the Render web service and database, the rehearsal bucket, the api-next record and its rule entry, and the engine’s db-backup.yml. Then regenerate the engine’s architecture inventory. Deploys stay manual (40-app-deploy.ps1) until the deploy-on-push workflow lands.