Deploy Server
Deploy Server
Migration in progress.
api.legendary-arena.comand its PostgreSQL are moving off Render onto a self-hosted DigitalOcean Ubuntu droplet fronted by Cloudflare, perdocs/PLAN.md. Until decommission, Render stays warm as the rollback target.
Summary
Clones the engine repo, builds the server and its workspace dependencies with the pinned toolchain, runs migrations, then installs and starts the la-server systemd unit and waits until /health answers 200. This is the step that turns a provisioned droplet into a running apps/server on 127.0.0.1:3000.
Mechanics
The owning artifact infra/scripts/40-app-deploy.ps1
requires root and stops on the first failing command. Stages:
Inputs. Requires the operator user,
/opt/node/bin/node(from Node Runtime ), and/etc/la/.envwith at leastDATABASE_URLandPORT. It then sets the env file toroot:root, mode600: systemd readsEnvironmentFileas root before dropping to the service user, so no other account needs to read the secrets.Repository. Under
APP_ROOT(default/opt/la-app, owned byoperator), clonesREPO_URL(defaulthttps://github.com/barefootbetters/legendary-arena.git, the public engine repo) once intorepo/with no working tree, then fetches.DEPLOY_REF(defaultmain) resolves to one commit: a 40-hex ref is used as is, a branch name meansorigin/<branch>.Release directory. The commit is checked out as a git worktree at
releases/<40-hex sha>, next to the releasela-serveris running. Nothing the running server reads changes. A release whose build finished carries a.la-release-builtmarker and is reused as is (a redeploy, a roll forward). A directory without the marker is a failed build; it is deleted and rebuilt.Build, inside the release directory, as the operator:
pnpm install --frozen-lockfile # fails on lockfile drift — no silent fallback pnpm --filter @legendary-arena/server... build # the server and its workspace deps onlyRender built the whole monorepo; the front-ends deploy to Cloudflare Pages, so building them here only costs time.
Migrations (default on,
RUN_MIGRATIONS=true, as on Render). Runs the new release’snode scripts/migrate.mjsas a transient systemd unit withEnvironmentFile=/etc/la/.env, so migrations see exactly the environment the server will. A failed build or migration stops here, withcurrentand the live commit unchanged.Unit. Copies
infra/systemd/la-server.serviceinto place,daemon-reload,enable. Also copiesinfra/systemd/la-server-standby.service, which is never enabled.Switch, restart through a standby, health gates. Both units run from
WorkingDirectory=/opt/la-app/current, a symlink. The script repoints it at the new release (an atomic rename) and then restarts. systemd resolves the link when a process starts, so a running server keeps its own release. Whilela-serverstops and starts, nothing listens on:3000, and every request through Nginx would fail (“Failed to fetch” in the browser). So the script first startsla-server-standby: the new release on127.0.0.1:3001. Once the standby’s/healthanswers 200, it restartsla-server. Nginx’sla_serverupstream lists:3001as abackup, so requests the restarting primary refuses go to the standby instead. Whenla-server’s/healthanswers 200, the script stops the standby, and its websocket clients reconnect tola-server. Each gate polls for up toHEALTH_TIMEOUT_SECONDS(120) and prints the last 60 journal lines of the unit that failed.- Standby never healthy: the script stops it, points
currentback at the old release, and fails.la-serverwas not touched and keeps serving the old release. la-servernever healthy on the new release: the script pointscurrentback, restartsla-serveron the old release, waits for its/health, stops the standby, and fails. If the old release does not come back either, it fails and leaves the standby serving the new one.- No standby when
/etc/nginx/conf.d/la-upstream.confis missing (run20-nginx.ps1) orla-serveris not running yet (a first deploy): a plain restart, with the same switch back on failure.
- Standby never healthy: the script stops it, points
Record. Only after the health gate passes: the release that was live becomes
previous(the rollback target), the commit is written to/var/lib/la-deploy/live-commit, whichla-deploy statusreports, and every directory underreleases/other thancurrentandpreviousis deleted.
Layout:
/opt/la-app/repo/ engine clone, no working tree; releases are its worktrees
/opt/la-app/releases/<sha>/ one built commit each; current + previous are kept
/opt/la-app/current -> releases/… what la-server and la-server-standby run
/opt/la-app/previous -> … the release that was live before; -Rollback target
/var/lib/la-deploy/live-commit the commit that passed the health gate last
The unit runs /opt/node/bin/node --import ./apps/server/node_modules/tsx/dist/loader.mjs apps/server/src/index.mjs, the same start command as render.yaml. Node runs directly (no pnpm wrapper) so systemd’s SIGTERM reaches the server’s own graceful-shutdown handler. Restart=on-failure, After/Wants postgresql.service.
Environment overrides: OPERATOR_USER, APP_ROOT (the units hard-code /opt/la-app/current), LEGACY_APP_DIR, LA_LIVE_COMMIT_FILE, REPO_URL, DEPLOY_REF, RUN_MIGRATIONS, LA_ENV_FILE, SYSTEMD_SRC, STANDBY_SYSTEMD_SRC, HEALTH_TIMEOUT_SECONDS.
Interactions
- Node Runtime
— provides
/opt/node(Node 24.18.0 + pnpm); run it first. - Secrets and Env
—
/etc/la/.env, built frominfra/.env.example. - PostgreSQL Setup
— the local database
DATABASE_URLpoints at. - Backup and Restore — at rehearsal and cutover, restore the data before the first deploy so migrations run against it.
- Deploy Pipeline — the CI workflow that re-runs this flow on push.
Edge Cases
- Lockfile drift fails the deploy. The old fallback to a plain
pnpm installcould drift dependencies silently; now the deploy stops and you fix the lockfile in the engine repo. PORTmust be 3000. Nginx proxies to127.0.0.1:3000, while the server’s own default is 8000..env.examplesets it. The standby’s3001is fixed in its unit, in40-app-deploy.ps1, and in the upstream20-nginx.ps1writes.- Two servers run for a few seconds. During the handoff both processes share the database. The bot-ally lease handoff (engine WP-437) and the boardgame.io Postgres store were built for Render’s overlapping rolling deploys, which behaved the same way. A client that moved to the standby reconnects to
la-serverwhen the standby stops. - Existing non-git
/opt/la-app/repo. The script refuses to delete a directory it did not create; move it aside first. - The first release deploy on a droplet that deployed in place. Until 2026-10, every deploy checked out and rebuilt
/opt/legendary-arenaunder the running server. The first release deploy clonesrepo/borrowing that checkout’s git objects, pointscurrentat the checkout before installing the new units (so a crash-restart still runs what is live), and leaves it asprevious: the first rollback goes back to it. Nothing writes to it any more and nothing deletes it. Oncepreviouspoints intoreleases/, remove it by hand:sudo rm -rf /opt/legendary-arena. - Installing this change on the droplet.
/opt/la-labis a copy of the lab files, not a git checkout, so it is installed from a staged archive, as lab #39 was. Run both scripts as root under the gate’s lock.install-on-droplet.shchecks for 2 GB free on/. It then copies40-app-deploy.ps1,50-backup.ps1,la-deploy.ps1and both units into/opt/la-lab, keeping.bak-YYYYMMDDcopies, and re-runs41-deploy-user.ps1with the CI key already pinned fordeployer. That re-run is needed because/usr/local/sbin/la-deployis an installed copy. The script deploys nothing, and/etc/systemd/systemkeeps the old units until the first release deploy installs the new ones. That first release deploy happens on the next engine push, or right away with the separatemigrate-on-droplet.sh, which redeploys the live commit as the first release. - A rollback does not reverse migrations. The older release runs against the newer schema, as with any hand rollback. Before rolling back past a release that added a migration, check that the older code tolerates it.
- Disk. Two built releases stay on disk. pnpm hard-links
node_modulesfrom the shared store, so the second costs the checkout (about 135 MB) and the build output. - First boot on an empty database runs every migration; on a restored copy, only the ones the copy is missing.
- Health endpoints:
/health(gate) and/health/legends-publisher.
Execute
# Deploy main: build a release, migrate, switch, restart, wait for health
sudo flock -w 600 /run/la-deploy.lock pwsh -File /opt/la-lab/infra/scripts/40-app-deploy.ps1
# Deploy a specific ref
sudo DEPLOY_REF='<40-hex commit or branch>' flock -w 600 /run/la-deploy.lock pwsh -File /opt/la-lab/infra/scripts/40-app-deploy.ps1
# Roll back to the release that was live before the last deploy (no rebuild).
# Running it again rolls forward.
sudo flock -w 600 /run/la-deploy.lock pwsh -File /opt/la-lab/infra/scripts/40-app-deploy.ps1 -Rollback
flock on the gate’s lock keeps a hand deploy from overlapping a CI deploy.
Verify
systemctl status la-server --no-pager
# Expected: Active: active (running)
curl -fsS http://127.0.0.1:3000/health
# Expected: HTTP 200
journalctl -u la-server -n 50 --no-pager
# Expected: a clean start, no Restart=on-failure loop
readlink /opt/la-app/current /opt/la-app/previous; cat /var/lib/la-deploy/live-commit; echo
curl -fsS http://127.0.0.1:3000/version
# Expected: current is releases/<live-commit>, and /version's gitSha is its prefix
The provision smoke workflow
first deploys engine main’s parent with the in-place 40-app-deploy.ps1 from lab #39 (the layout production had), then, on a fresh Ubuntu 24.04 runner: the first release deploy through the gate, a redeploy of the live commit, a rollback to the in-place checkout, a roll forward, a deploy that never gets healthy (it must leave the live release serving and status unchanged), a release-to-release deploy that prunes releases/, and a rollback between two releases. Each check compares the current link, la-deploy status, and the running server’s own /api/version. It checks /health through Nginx and TLS, then re-runs the script to prove the update path. Its restart-under-traffic step holds a websocket open, probes Nginx every 0.1 s, and fails if any request fails during the standby deploy. The job summary also shows a bare systemctl restart for comparison.
References
infra/scripts/40-app-deploy.ps1— owning deploy script.infra/systemd/la-server.service— the unit it installs.infra/systemd/la-server-standby.service— the standby that serves during the restart.infra/nginx/la-upstream.conf— the upstream with the standby asbackup.infra/.env.example— template for/etc/la/.env.docs/PLAN.md— migration plan and pins.content/app/deploy-server.md— original source operator page.