Legendary Arena Lab

Monitoring

ops

Migration in progress. api.legendary-arena.com and PostgreSQL are moving from Render to a self-hosted DigitalOcean Ubuntu host behind Cloudflare. Render remains rollback-ready until decommission phase.

Monitoring

Render emailed you when the service died. On your own box, nothing does unless you set it up. A check that has never fired an alert hasn’t been proven to work.

Summary

This page owns infra/scripts/60-monitor.ps1. Every 5 minutes (la-health.timer) it checks that:

  • la-server is active
  • GET http://127.0.0.1:<PORT>/health returns 200
  • GET http://127.0.0.1:<PORT>/health/db returns 200 — SELECT 1 against PostgreSQL (engine D-24711). /health never touches the database, so without this check a dead database left every monitor green while signed-in routes returned 500. A failure puts the route’s reason (timeout or query_failed) in the alert.
  • <PORT> is listening on loopback only (ss -tln). LISTEN_HOST=127.0.0.1 in /etc/la/.env binds Node to loopback, so :3000 stays closed even if UFW is turned off while troubleshooting; a direct client there could forge the cf-connecting-ip header the guest rate limiters trust. The alert names the exposed address.
  • the root disk is below MONITOR_DISK_ALERT_PERCENT (default 85)
  • the newest local backup in /var/backups/la is younger than MONITOR_BACKUP_MAX_AGE_HOURS (default 26)
  • la-cf-ufw-refresh.service (the weekly Cloudflare-range refresh, Cloudflare TLS ) is not in a failed state. The refresh has no ping of its own, so this check is how a failed fetch or ufw error reaches you.
  • memory use has not held at or above MONITOR_MEMORY_ALERT_PERCENT (default 70) for MONITOR_MEMORY_SUSTAIN_CHECKS runs in a row (default 3, so 15 minutes). Used memory is MemTotal − MemAvailable, so page cache doesn’t count. One high sample is a spike; three in a row is the level D-002 means by “memory holds above ~70%”.
  • no more than MONITOR_SWAP_OUT_ALERT_MB (default 8) went out to swap since the last run, unless a deploy ran in between. It watches pages written to swap (pswpout in /proc/vmstat), not “swap used”: pages a deploy pushed out stay counted as used until something touches them, so that figure can sit above zero for days after a normal deploy. A deploy counts if journalctl shows la-deploy’s Deploying <sha> line (CI) or a sudo line naming 40-app-deploy.ps1 (by hand) since the last run. 08-swap.ps1 sized the swapfile for the deploy’s pnpm -r build, so its swap-out is expected.
  • a pending reboot (/var/run/reboot-required, written when a kernel or core library update installs) is younger than MONITOR_REBOOT_MAX_AGE_DAYS (default 7). Until then it’s a warning in the ping body, not a failure (see Pending reboot ).

The memory and swap checks are the D-002 resize trigger made automatic. When either fails for real (not a one-off), resize CPU and RAM only to 4 vCPU / 8 GB.

The run-to-run counters (swap-out pages, memory-high streak, when a reboot was first seen) live in /var/lib/la-monitor/state.json. A reboot resets them, because /proc/vmstat restarts at boot.

Before checking, it waits for any running deploy to release /run/la-deploy.lock (the lock la-deploy.ps1 holds). A deploy restarts la-server behind the :3001 standby, so the public site stays up, but :3000 refuses connections for about 4 seconds. On 2026-10-21 a check landed in that gap and sent a false DOWN alert. If the lock is still held after 5 minutes, the check fails and names it.

If every check passes, it pings MONITOR_HEALTHCHECK_URL with a body like ok; memory 38% used, swap 0 MB used, plus any pending-reboot warning. If any check fails, it pings <url>/fail with the reasons in the body (the same reading in parentheses) and exits 1. healthchecks.io keeps each ping’s body in the check’s event log, so the capacity reading is visible there without SSH.

Pending reboot

Since 2026-11-13, 07-fail2ban-upgrades.ps1 has unattended-upgrades reboot the droplet by itself at 10:30 UTC (2:30 AM Pacific in winter) on a day an update needs it (Fail2ban and Unattended Upgrades ). So the normal sequence is: the upgrade run at ~06:30 UTC sets the flag, la-health shows the warning for about four hours, and the 10:30 reboot clears it.

The check below is the backstop for when that doesn’t happen. A pending reboot isn’t an outage, and paging every 5 minutes from the first day would teach you to ignore the check. So for its first MONITOR_REBOOT_MAX_AGE_DAYS days it only rides in the ping body: warning: reboot required since 2026-11-12 06:41 UTC (linux-image-6.8.0-88-generic, ...). After that the check fails, and the DOWN email names the same packages. The age counts from the first run that saw the flag, because update-notifier rewrites the file for each package that needs a reboot.

A warning that survives past 10:30 UTC means the automatic reboot didn’t run. Check sudo grep -i reboot /var/log/unattended-upgrades/unattended-upgrades.log | tail -5. To clear it by hand, in a quiet hour:

cat /var/run/reboot-required.pkgs
sudo reboot
# ~1 minute later, from the workstation:
curl -s https://api.legendary-arena.com/health/db

la-server, PostgreSQL and the la-* timers are enabled units, so they come back by themselves. The first la-health run is 2 minutes after boot.

The ping service is healthchecks.io (free Hobbyist plan, $0). It works as a dead-man’s switch: it alerts on a /fail ping and when pings stop arriving, so a droplet that’s down, unreachable or wedged also alerts.

50-backup.ps1 pings its own check (BACKUP_HEALTHCHECK_URL in backup.env) after each successful backup, so a backup that silently stops running is caught on its own schedule. It doesn’t wait for the 26-hour backup-age limit. The pCloud mirror reports to a third check (BACKUP_PCLOUD_HEALTHCHECK_URL), so a mirror problem never turns the backup check red; see Backup and Restore .

The three checks

Create all three in the healthchecks.io dashboard (Add Check). Email is the default alert channel.

CheckScheduleGracePinged by
la-droplet-monitorSimple, period 5 min10 min60-monitor.ps1 (la-health.timer)
la-droplet-backupCron 17 9 * * *, UTC2 h50-backup.ps1 (la-backup.timer): dump + R2
la-droplet-pcloudCron 17 9 * * *, UTC2 h50-backup.ps1: the pCloud mirror only (missing token or failed copy → /fail)

The backup check uses a cron schedule that matches the timer exactly, so a late or skipped backup alerts within the grace period.

Configuration

/etc/la/monitor.env is root-only (600 root:root):

MONITOR_HEALTHCHECK_URL=https://hc-ping.com/<uuid>
MONITOR_DISK_ALERT_PERCENT=85
MONITOR_BACKUP_MAX_AGE_HOURS=26

The memory, swap and reboot limits (MONITOR_MEMORY_ALERT_PERCENT=70, MONITOR_MEMORY_SUSTAIN_CHECKS=3, MONITOR_SWAP_OUT_ALERT_MB=8, MONITOR_REBOOT_MAX_AGE_DAYS=7) default in the script. Add a line only to change one.

Then add one line to /etc/la/backup.env:

BACKUP_HEALTHCHECK_URL=https://hc-ping.com/<uuid>

Anyone who has a ping URL can mark a check up or down, so treat it like a password. The operator writes both files and installs them personally, the same way as backup.env:

scp -i $HOME\.ssh\id_ed25519 "<folder>\monitor.env" "<folder>\backup.env" operator@<droplet-ip>:/tmp/
ssh -i $HOME\.ssh\id_ed25519 operator@<droplet-ip> 'sudo install -m 600 -o root -g root /tmp/monitor.env /etc/la/monitor.env; sudo install -m 600 -o root -g root /tmp/backup.env /etc/la/backup.env; rm /tmp/monitor.env /tmp/backup.env'

Remove CRLF line endings from a file saved on Windows: sudo sed -i 's/\r$//' /etc/la/monitor.env /etc/la/backup.env. Check the result by key name only, without printing any values:

sudo awk -F= '/^[A-Z]/{ print $1, (length($2) ? "set" : "EMPTY") }' /etc/la/monitor.env

Execute

sudo pwsh -NoProfile -File infra/scripts/60-monitor.ps1 -Install   # once: la-health.service + .timer
sudo systemctl start la-health.service                              # one check now
sudo journalctl -u la-health.service -n 5 --no-pager -o cat

A healthy run logs [60-monitor] All checks passed. memory 38% used, swap 0 MB used. A failed ping logs Ping to <url> failed: …. The ping URL appears in that line, so keep journal excerpts out of public places.

Prove an alert fires

sudo systemctl stop la-server
sudo systemctl start la-health.service      # exits 1; pings /fail with the reasons
# confirm the DOWN email, then:
sudo systemctl start la-server
sudo systemctl start la-health.service      # passes; the check returns to up

Only run this before cutover, or inside an announced window: stopping la-server stops the game.

Rehearsed (2026-10-12)

CheckResult
60 -InstallPass: la-health.timer registered, every 5 min
One monitor runPass: All checks passed., ping received
One backup run (la-backup.service)Pass: Backup complete., ping received
Alert proof: la-server stoppedPass: FAILED: la-server is not active; /health unreachable: Connection refused, /fail ping sent
RecoveryPass: /health 200, next run passed

Harmless noise: under systemd, PowerShell 7.6 prints a NamedPipeIPC_ServerListenerError … Operation canceled line as it exits. The unit still deactivates successfully.

Alert channel

Every check alerts through two channels that share nothing (healthchecks.io → Integrations, both assigned to all checks):

ChannelPriorityWhy
Pushover → the operator’s phoneHigh for DOWN (breaks through Do Not Disturb), Low for UP (silent)Hard to miss, and doesn’t depend on the mailbox
healthchecks.io’s own email to the operator—The record, searchable later

Why two. Until 2026-11-13 email was the only channel, and it failed the way email does: it was delivered and not seen. Around 2026-10-28 14:00 UTC a PowerShell snap refresh stopped la-health, la-backup and la-cf-ufw-refresh (see “If the check never runs at all” below). healthchecks.io marked la-droplet-monitor DOWN on time, at 14:14 UTC (5-minute period plus 10-minute grace), and the email reached the operator’s inbox a minute later. la-droplet-backup followed on 2026-10-29 at 11:17 UTC. Neither email was noticed for about 15 days. A Microsoft mailbox outage on 2026-11-13 at first made it look as if the alerts had never arrived, but their received timestamps settled it. A high-priority push to the phone is the channel meant to be seen; the email stays as the record. Emergency priority is deliberately not used: it repeats until acknowledged, which is too loud for a check that can blip (the 2026-10-21 deploy-gap false DOWN).

Keep Pushover licensed. The phone app is a one-time $4.99 per platform after a 30-day trial. The trial started 2026-11-13, so buy the license before about 2026-12-13; once the trial lapses, delivery to that phone stops without any error. To prove the path at any time: Integrations → Pushover → Test!.

PLAN §1 names Brevo as the alert path. Routing alerts through Brevo would add a second account and API key to the alert path, only to deliver the same email.

When la-droplet-monitor goes DOWN

  1. Convert the alert time to UTC. healthchecks.io shows it in your local zone (e.g. 22:55:30 -0700), but the droplet logs in UTC (05:55:30).
  2. Read the ping body. Connection refused (127.0.0.1:3000) means the server process wasn’t listening. /health/db returned 503 means PostgreSQL. a deploy has held /run/la-deploy.lock means a deploy is stuck. used for 3 runs in a row or MB went out to swap is the D-002 resize trigger: check free -m and ps aux --sort=-rss | head before resizing, in case one runaway process is the cause. reboot required since … means a kernel update has waited a week. Disk, backup-age, loopback, and la-cf-ufw-refresh failures name themselves.
  3. Check whether it’s still down: curl -s https://api.legendary-arena.com/health and sudo systemctl start la-health.service. If the next timer run passed, healthchecks.io has already marked the check up again.
  4. Line the alert up with the journals (use sudo, or operator sees only its own entries):
sudo journalctl -t la-deploy --since "<UTC time - 10 min>" --until "<UTC time + 10 min>"   # a deploy running?
sudo journalctl -u la-server --since ... --until ... | grep -E "listening|Stopped|exited"    # restart or crash?
sudo journalctl -u la-server-standby --since ... --until ...                                # was :3001 covering?
sudo grep "<yyyy/mm/dd hh:mm>" /var/log/nginx/error.log                                     # did users see errors?

A restart with no deploy around it means a crash: read the la-server lines just before the stop. If nginx has no upstream errors in that minute, users weren’t affected.

If the check never runs at all. systemctl status la-health showing status=203/EXEC means systemd can’t execute the unit’s ExecStart binary, so no check ran and no ping went out. On 2026-11-13 this turned out to have stopped la-health, la-backup and la-cf-ufw-refresh together. PowerShell is a snap on the droplet, and their units had been installed with a versioned /snap/powershell/<rev>/… path that a snap refresh deleted. The last backup was 2026-10-28. Repair it with grep -lE '/snap/(bin/pwsh|powershell/[0-9]+/)' /etc/systemd/system/*.service | xargs -r sudo sed -i -E 's#(/snap/bin/pwsh|/snap/powershell/[0-9]+/opt/powershell/pwsh)#/snap/powershell/current/opt/powershell/pwsh#', then sudo systemctl daemon-reload, and start each service once. The provisioners now install the snap’s current path, which snapd keeps pointed at the live revision. They avoid /snap/bin/pwsh: it goes through snap run, which moves the process into its own scope, so the script’s lines would leave journalctl -u <unit>.

If the check dumps core. Job for la-health.service failed because a fatal signal was delivered causing the control process to dump core, with The temporary HOME directory '/tmp/pwsh-root-…' already exists but does not have the expected user-only permission in journalctl -u la-health, means the unit has no HOME. systemd sets none for a root service, and PowerShell 7.7.0 (snap revision 456, published 2026-11-13) refuses to start without one; 7.6.9 tolerated it. The provisioners now write Environment=HOME=/root into all three units. Repair an older box with for u in la-health la-backup la-cf-ufw-refresh; do sudo grep -q '^Environment=HOME=' /etc/systemd/system/$u.service || sudo sed -i '/^Type=oneshot$/a Environment=HOME=/root' /etc/systemd/system/$u.service; done, then sudo systemctl daemon-reload.

Worked example (2026-10-21 05:55 UTC). A DOWN alert with Connection refused landed 1 s into the ~4 s restart of the 9dcde5f8 deploy. The standby served traffic, so nginx logged no errors, and the next run passed. It was a false alarm, and lab #43 closed the gap: the check now waits on the deploy lock.

What to watch

The timer catches “down”. These signs catch “unhealthy”, the slow failures that came before Render’s freezes:

  • Restart loops: systemctl show la-server -p NRestarts should stay at 0 between deploys. journalctl -u la-server --since "1 hour ago" | grep -c Started counts the starts.
  • CPU load: uptime load average should stay well under 1 per core (under 2 on this 2-vCPU box). Sustained load near or above 2 starves the socket loop and freezes clients.
  • Memory: la-health now alerts on the memory and swap halves of the D-002 trigger (see Summary), and every ping body carries the reading. free -m shows it by hand. CPU above ~50% under real traffic is the one half still watched by hand.
  • Query load: pg_stat_statements is preloaded (30-postgres.ps1), so sudo -u postgres psql -d la -c "select calls, round(total_exec_time) as ms, left(query, 80) from pg_stat_statements order by total_exec_time desc limit 10" shows where database time goes. select pg_stat_statements_reset() starts a fresh window.
  • DB connections: sudo -u postgres psql -c "select count(*) from pg_stat_activity where datname='la'".

journald basics

journalctl -u la-server -f                      # follow the app log
journalctl -u la-server --since "30 min ago"    # recent window
journalctl -u la-server -p err --since today    # errors only
journalctl -u la-health -u la-backup -n 20      # the timers' last runs
systemctl list-timers 'la-*'                    # when each timer last ran and next runs

What this does not cover

The on-box check calls 127.0.0.1 directly, so it never tests Nginx, TLS, Cloudflare or DNS. The healthchecks.io switch covers the “whole droplet is gone” case. The public path is covered by an external monitor: UptimeRobot (free plan, created 2026-10-12) checks https://api.legendary-arena.com/health/db every 5 minutes and emails on failure. It was green against Render before cutover and followed api. to the droplet with no change. It watched /health until 2026-10-13, when it moved to /health/db (engine D-24711), so it now fails on a dead database as well as on a dead server, Nginx, TLS, Cloudflare or DNS.

References

  • Plan §4.6 (monitoring.md stub), §5 (script manifest), and §9 (Cutover)
  • Backup and Restore : 50-backup.ps1 and backup.env