Legendary Arena Lab

Deploy Server

wiki

Deploy Server

Migration in progress. api.legendary-arena.com and its PostgreSQL are moving off Render onto a self-hosted DigitalOcean Ubuntu droplet fronted by Cloudflare, per docs/PLAN.md . Until decommission, Render stays warm as the rollback target.

Summary

Clones the engine repo, builds the server and its workspace dependencies with the pinned toolchain, runs migrations, then installs and starts the la-server systemd unit and waits until /health answers 200. This is the step that turns a provisioned droplet into a running apps/server on 127.0.0.1:3000.

Mechanics

The owning artifact infra/scripts/40-app-deploy.ps1 requires root and stops on the first failing command. Stages:

  1. Inputs. Requires the operator user, /opt/node/bin/node (from Node Runtime ), and /etc/la/.env with at least DATABASE_URL and PORT. It then sets the env file to root:root, mode 600: systemd reads EnvironmentFile as root before dropping to the service user, so no other account needs to read the secrets.

  2. Repository. Under APP_ROOT (default /opt/la-app, owned by operator), clones REPO_URL (default https://github.com/barefootbetters/legendary-arena.git, the public engine repo) once into repo/ with no working tree, then fetches. DEPLOY_REF (default main) resolves to one commit: a 40-hex ref is used as is, a branch name means origin/<branch>.

  3. Release directory. The commit is checked out as a git worktree at releases/<40-hex sha>, next to the release la-server is running. Nothing the running server reads changes. A release whose build finished carries a .la-release-built marker and is reused as is (a redeploy, a roll forward). A directory without the marker is a failed build; it is deleted and rebuilt.

  4. Build, inside the release directory, as the operator:

    pnpm install --frozen-lockfile                 # fails on lockfile drift — no silent fallback
    pnpm --filter @legendary-arena/server... build  # the server and its workspace deps only
    

    Render built the whole monorepo; the front-ends deploy to Cloudflare Pages, so building them here only costs time.

  5. Migrations (default on, RUN_MIGRATIONS=true, as on Render). Runs the new release’s node scripts/migrate.mjs as a transient systemd unit with EnvironmentFile=/etc/la/.env, so migrations see exactly the environment the server will. A failed build or migration stops here, with current and the live commit unchanged.

  6. Unit. Copies infra/systemd/la-server.service into place, daemon-reload, enable. Also copies infra/systemd/la-server-standby.service , which is never enabled.

  7. Switch, restart through a standby, health gates. Both units run from WorkingDirectory=/opt/la-app/current, a symlink. The script repoints it at the new release (an atomic rename) and then restarts. systemd resolves the link when a process starts, so a running server keeps its own release. While la-server stops and starts, nothing listens on :3000, and every request through Nginx would fail (“Failed to fetch” in the browser). So the script first starts la-server-standby: the new release on 127.0.0.1:3001. Once the standby’s /health answers 200, it restarts la-server. Nginx’s la_server upstream lists :3001 as a backup, so requests the restarting primary refuses go to the standby instead. When la-server’s /health answers 200, the script stops the standby, and its websocket clients reconnect to la-server. Each gate polls for up to HEALTH_TIMEOUT_SECONDS (120) and prints the last 60 journal lines of the unit that failed.

    • Standby never healthy: the script stops it, points current back at the old release, and fails. la-server was not touched and keeps serving the old release.
    • la-server never healthy on the new release: the script points current back, restarts la-server on the old release, waits for its /health, stops the standby, and fails. If the old release does not come back either, it fails and leaves the standby serving the new one.
    • No standby when /etc/nginx/conf.d/la-upstream.conf is missing (run 20-nginx.ps1 ) or la-server is not running yet (a first deploy): a plain restart, with the same switch back on failure.
  8. Record. Only after the health gate passes: the release that was live becomes previous (the rollback target), the commit is written to /var/lib/la-deploy/live-commit, which la-deploy status reports, and every directory under releases/ other than current and previous is deleted.

Layout:

/opt/la-app/repo/                  engine clone, no working tree; releases are its worktrees
/opt/la-app/releases/<sha>/        one built commit each; current + previous are kept
/opt/la-app/current -> releases/…  what la-server and la-server-standby run
/opt/la-app/previous -> …          the release that was live before; -Rollback target
/var/lib/la-deploy/live-commit     the commit that passed the health gate last

The unit runs /opt/node/bin/node --import ./apps/server/node_modules/tsx/dist/loader.mjs apps/server/src/index.mjs, the same start command as render.yaml. Node runs directly (no pnpm wrapper) so systemd’s SIGTERM reaches the server’s own graceful-shutdown handler. Restart=on-failure, After/Wants postgresql.service.

Environment overrides: OPERATOR_USER, APP_ROOT (the units hard-code /opt/la-app/current), LEGACY_APP_DIR, LA_LIVE_COMMIT_FILE, REPO_URL, DEPLOY_REF, RUN_MIGRATIONS, LA_ENV_FILE, SYSTEMD_SRC, STANDBY_SYSTEMD_SRC, HEALTH_TIMEOUT_SECONDS.

Interactions

Edge Cases

  • Lockfile drift fails the deploy. The old fallback to a plain pnpm install could drift dependencies silently; now the deploy stops and you fix the lockfile in the engine repo.
  • PORT must be 3000. Nginx proxies to 127.0.0.1:3000, while the server’s own default is 8000. .env.example sets it. The standby’s 3001 is fixed in its unit, in 40-app-deploy.ps1, and in the upstream 20-nginx.ps1 writes.
  • Two servers run for a few seconds. During the handoff both processes share the database. The bot-ally lease handoff (engine WP-437) and the boardgame.io Postgres store were built for Render’s overlapping rolling deploys, which behaved the same way. A client that moved to the standby reconnects to la-server when the standby stops.
  • Existing non-git /opt/la-app/repo. The script refuses to delete a directory it did not create; move it aside first.
  • The first release deploy on a droplet that deployed in place. Until 2026-10, every deploy checked out and rebuilt /opt/legendary-arena under the running server. The first release deploy clones repo/ borrowing that checkout’s git objects, points current at the checkout before installing the new units (so a crash-restart still runs what is live), and leaves it as previous: the first rollback goes back to it. Nothing writes to it any more and nothing deletes it. Once previous points into releases/, remove it by hand: sudo rm -rf /opt/legendary-arena.
  • Installing this change on the droplet. /opt/la-lab is a copy of the lab files, not a git checkout, so it is installed from a staged archive, as lab #39 was. Run both scripts as root under the gate’s lock. install-on-droplet.sh checks for 2 GB free on /. It then copies 40-app-deploy.ps1, 50-backup.ps1, la-deploy.ps1 and both units into /opt/la-lab, keeping .bak-YYYYMMDD copies, and re-runs 41-deploy-user.ps1 with the CI key already pinned for deployer. That re-run is needed because /usr/local/sbin/la-deploy is an installed copy. The script deploys nothing, and /etc/systemd/system keeps the old units until the first release deploy installs the new ones. That first release deploy happens on the next engine push, or right away with the separate migrate-on-droplet.sh, which redeploys the live commit as the first release.
  • A rollback does not reverse migrations. The older release runs against the newer schema, as with any hand rollback. Before rolling back past a release that added a migration, check that the older code tolerates it.
  • Disk. Two built releases stay on disk. pnpm hard-links node_modules from the shared store, so the second costs the checkout (about 135 MB) and the build output.
  • First boot on an empty database runs every migration; on a restored copy, only the ones the copy is missing.
  • Health endpoints: /health (gate) and /health/legends-publisher.

Execute

# Deploy main: build a release, migrate, switch, restart, wait for health
sudo flock -w 600 /run/la-deploy.lock pwsh -File /opt/la-lab/infra/scripts/40-app-deploy.ps1

# Deploy a specific ref
sudo DEPLOY_REF='<40-hex commit or branch>' flock -w 600 /run/la-deploy.lock pwsh -File /opt/la-lab/infra/scripts/40-app-deploy.ps1

# Roll back to the release that was live before the last deploy (no rebuild).
# Running it again rolls forward.
sudo flock -w 600 /run/la-deploy.lock pwsh -File /opt/la-lab/infra/scripts/40-app-deploy.ps1 -Rollback

flock on the gate’s lock keeps a hand deploy from overlapping a CI deploy.

Verify

systemctl status la-server --no-pager
# Expected: Active: active (running)

curl -fsS http://127.0.0.1:3000/health
# Expected: HTTP 200

journalctl -u la-server -n 50 --no-pager
# Expected: a clean start, no Restart=on-failure loop

readlink /opt/la-app/current /opt/la-app/previous; cat /var/lib/la-deploy/live-commit; echo
curl -fsS http://127.0.0.1:3000/version
# Expected: current is releases/<live-commit>, and /version's gitSha is its prefix

The provision smoke workflow first deploys engine main’s parent with the in-place 40-app-deploy.ps1 from lab #39 (the layout production had), then, on a fresh Ubuntu 24.04 runner: the first release deploy through the gate, a redeploy of the live commit, a rollback to the in-place checkout, a roll forward, a deploy that never gets healthy (it must leave the live release serving and status unchanged), a release-to-release deploy that prunes releases/, and a rollback between two releases. Each check compares the current link, la-deploy status, and the running server’s own /api/version. It checks /health through Nginx and TLS, then re-runs the script to prove the update path. Its restart-under-traffic step holds a websocket open, probes Nginx every 0.1 s, and fails if any request fails during the standby deploy. The job summary also shows a bare systemctl restart for comparison.

References