Legendary Arena Lab

Cutover

ops

Production moved from Render to the self-hosted DigitalOcean droplet on 2026-10-12. Render stays suspended (not deleted) until the 7-day rollback window ends, around 2026-10-20.

Cutover

How api.legendary-arena.com and its PostgreSQL moved from Render to the droplet, what went wrong, and the exact rollback.

Summary

  • When: 2026-10-12, about 19:00–19:30 UTC (midday PDT). This was moved up from the planned 11 PM window because production was idle: 0 active matches in the previous 15 minutes and 4 matches created in 24 hours.
  • Downtime: 10 m 10 s, as measured by UptimeRobot at its 5-minute check granularity.
  • Data: a read-only final dump of Render’s database, restored with 51-restore.ps1. All 41 tables and 6,425 rows were identical, and schema_migrations was 45.
  • Engine record: DECISIONS D-24707.

Before the window

  • Production secrets installed on the droplet by key name (Secrets and Environment ). LEGENDS_PUBLISHER_ENABLED=false until the restore.
  • Every caller pinned to the onrender hostname was moved to api. (PLAN §9 step 0). The Stripe webhook was the one that needed it.
  • The droplet was redeployed to the same main commit Render was serving.
  • Monitoring was live: healthchecks.io la-droplet-monitor and UptimeRobot on api./health.
  • The production backup.env was prepared on the operator’s desktop (the bucket, the pCloud token, three ping URLs), but not installed until after the restore.

The steps, as run

  1. Rollback rehearsal on api-next. Switch it to a DNS-only CNAME → legendary-arena-server.onrender.com, confirm it resolves to Render, then switch it back to the proxied A record. Each change showed in public DNS within seconds.
  2. Suspend Render (Render → service → Settings → Suspend). Writes freeze, and api. and the onrender host answer 503.
  3. Final dump (operator desktop, read-only): PGOPTIONS='-c default_transaction_read_only=on', pg_dump -Fc --no-owner --no-acl. Record per-table counts from the source.
  4. Copy and restore (operator):
    scp -i $HOME/.ssh/id_ed25519 <dump> operator@<droplet-ip>:/tmp/
    ssh -i $HOME/.ssh/id_ed25519 operator@<droplet-ip> 'sudo systemctl stop la-server && cd /opt/la-lab/infra && sudo RESTORE_DUMP=/tmp/<dump> RESTORE_TARGET_DB=la RESTORE_ALLOW_LIVE_TARGET=true pwsh -NoProfile -File scripts/51-restore.ps1; sudo systemctl start la-server'
    
    Compare 51’s row counts with the source, table by table. Shred the dump on the droplet.
  5. Move the legends publisher. LEGENDS_PUBLISHER_ENABLED=true on the droplet (then restart la-server), and false in Render’s environment. Only the box with the live database publishes.
  6. Cloudflare. Add api.legendary-arena.com to the Configuration Rule Origin TLS strict — api hosts (an Or condition). Change the api record from the DNS-only CNAME to an A record → the droplet, Proxied.
  7. Remove Render’s custom domain (Render → service → Settings → Custom Domains → delete api.legendary-arena.com). This step wasn’t in the plan; see below.
  8. Verify through Cloudflare. /health 200, /health/legends-publisher ok, Socket.IO handshake 200, WebSocket upgrade 101, the droplet IP refuses 443, and journalctl -u la-server is clean. Then play a signed-in match and confirm it lands in the droplet’s database.
  9. Backups. Install the production backup.env, sync 50-backup.ps1, and run it once. Confirm the object exists in R2 legendary-arena-db-backups and in pCloud (byte-identical), and that la-droplet-backup and la-droplet-pcloud are green. Then gh workflow disable db-backup.yml in the engine repo.
  10. Delete the local dump and run pnpm check:domains in the engine repo (11 ok, 0 failed).

What went wrong: Render’s custom domain

After step 6, public DNS returned Cloudflare’s proxy addresses, but api. still served Render’s “Service Suspended” page. That held even when the request was pinned to a Cloudflare edge IP, and the response carried x-render-routing: suspend-by-user.

Render runs on Cloudflare too. When api. was added as a custom domain in Render, Render registered it as a custom hostname in its own Cloudflare for SaaS account, and that registration takes precedence over our zone’s proxied record. Deleting the custom domain in Render’s dashboard released it, and the next request reached the droplet.

A local DNS cache can also show the old answer for up to the record’s TTL. Clear it (Clear-DnsClientCache, or chrome://net-internals/#dns) before deciding something is wrong.

Rollback (until ~2026-10-20)

  1. Cloudflare: change the api record back to a CNAME → legendary-arena-server.onrender.com, DNS only (grey).
  2. Render: Resume the web service. Then Custom Domains → add api.legendary-arena.com and wait for Render to verify it and issue the certificate (a few minutes).
  3. The publisher: set true in Render’s environment and false on the droplet.
  4. Verify api./health reaches Render.

Anything written on the droplet after the flip isn’t in Render’s database. A rollback after real traffic needs that data copied back: a dump from the droplet restored into Render. Decide that deliberately; don’t improvise it.

After the window (Phase 7)

Delete the Render web service and database, the rehearsal bucket, the api-next record and its rule entry, and the engine’s db-backup.yml. Then regenerate the engine’s architecture inventory. Deploys stay manual (40-app-deploy.ps1) until the deploy-on-push workflow lands.