Monitoring
Migration in progress.
api.legendary-arena.comand PostgreSQL are moving from Render to a self-hosted DigitalOcean Ubuntu host behind Cloudflare. Render remains rollback-ready until decommission phase.
Monitoring
Render emailed you when the service died. On your own box, nothing does unless you set it up. A check that has never fired an alert hasn’t been proven to work.
Summary
This page owns infra/scripts/60-monitor.ps1. Every 5 minutes (la-health.timer) it checks that:
la-serveris activeGET http://127.0.0.1:<PORT>/healthreturns 200GET http://127.0.0.1:<PORT>/health/dbreturns 200 —SELECT 1against PostgreSQL (engine D-24711)./healthnever touches the database, so without this check a dead database left every monitor green while signed-in routes returned 500. A failure puts the route’s reason (timeoutorquery_failed) in the alert.<PORT>is listening on loopback only (ss -tln).LISTEN_HOST=127.0.0.1in/etc/la/.envbinds Node to loopback, so:3000stays closed even if UFW is turned off while troubleshooting; a direct client there could forge thecf-connecting-ipheader the guest rate limiters trust. The alert names the exposed address.- the root disk is below
MONITOR_DISK_ALERT_PERCENT(default 85) - the newest local backup in
/var/backups/lais younger thanMONITOR_BACKUP_MAX_AGE_HOURS(default 26) la-cf-ufw-refresh.service(the weekly Cloudflare-range refresh, Cloudflare TLS ) is not in a failed state. The refresh has no ping of its own, so this check is how a failed fetch or ufw error reaches you.- memory use has not held at or above
MONITOR_MEMORY_ALERT_PERCENT(default 70) forMONITOR_MEMORY_SUSTAIN_CHECKSruns in a row (default 3, so 15 minutes). Used memory isMemTotal − MemAvailable, so page cache doesn’t count. One high sample is a spike; three in a row is the level D-002 means by “memory holds above ~70%”. - no more than
MONITOR_SWAP_OUT_ALERT_MB(default 8) went out to swap since the last run, unless a deploy ran in between. It watches pages written to swap (pswpoutin/proc/vmstat), not “swap used”: pages a deploy pushed out stay counted as used until something touches them, so that figure can sit above zero for days after a normal deploy. A deploy counts ifjournalctlshowsla-deploy’sDeploying <sha>line (CI) or asudoline naming40-app-deploy.ps1(by hand) since the last run.08-swap.ps1sized the swapfile for the deploy’spnpm -r build, so its swap-out is expected. - a pending reboot (
/var/run/reboot-required, written when a kernel or core library update installs) is younger thanMONITOR_REBOOT_MAX_AGE_DAYS(default 7). Until then it’s a warning in the ping body, not a failure (see Pending reboot ).
The memory and swap checks are the D-002 resize trigger made automatic. When either fails for real (not a one-off), resize CPU and RAM only to 4 vCPU / 8 GB.
The run-to-run counters (swap-out pages, memory-high streak, when a reboot was first seen) live in /var/lib/la-monitor/state.json. A reboot resets them, because /proc/vmstat restarts at boot.
Before checking, it waits for any running deploy to release /run/la-deploy.lock (the lock la-deploy.ps1 holds). A deploy restarts la-server behind the :3001 standby, so the public site stays up, but :3000 refuses connections for about 4 seconds. On 2026-10-21 a check landed in that gap and sent a false DOWN alert. If the lock is still held after 5 minutes, the check fails and names it.
If every check passes, it pings MONITOR_HEALTHCHECK_URL with a body like ok; memory 38% used, swap 0 MB used, plus any pending-reboot warning. If any check fails, it pings <url>/fail with the reasons in the body (the same reading in parentheses) and exits 1. healthchecks.io keeps each ping’s body in the check’s event log, so the capacity reading is visible there without SSH.
Pending reboot
Since 2026-11-13, 07-fail2ban-upgrades.ps1 has unattended-upgrades reboot the droplet by itself at 10:30 UTC (2:30 AM Pacific in winter) on a day an update needs it (Fail2ban and Unattended Upgrades
). So the normal sequence is: the upgrade run at ~06:30 UTC sets the flag, la-health shows the warning for about four hours, and the 10:30 reboot clears it.
The check below is the backstop for when that doesn’t happen. A pending reboot isn’t an outage, and paging every 5 minutes from the first day would teach you to ignore the check. So for its first MONITOR_REBOOT_MAX_AGE_DAYS days it only rides in the ping body: warning: reboot required since 2026-11-12 06:41 UTC (linux-image-6.8.0-88-generic, ...). After that the check fails, and the DOWN email names the same packages. The age counts from the first run that saw the flag, because update-notifier rewrites the file for each package that needs a reboot.
A warning that survives past 10:30 UTC means the automatic reboot didn’t run. Check sudo grep -i reboot /var/log/unattended-upgrades/unattended-upgrades.log | tail -5. To clear it by hand, in a quiet hour:
cat /var/run/reboot-required.pkgs
sudo reboot
# ~1 minute later, from the workstation:
curl -s https://api.legendary-arena.com/health/db
la-server, PostgreSQL and the la-* timers are enabled units, so they come back by themselves. The first la-health run is 2 minutes after boot.
The ping service is healthchecks.io
(free Hobbyist plan, $0). It works as a dead-man’s switch: it alerts on a /fail ping and when pings stop arriving, so a droplet that’s down, unreachable or wedged also alerts.
50-backup.ps1 pings its own check (BACKUP_HEALTHCHECK_URL in backup.env) after each successful backup, so a backup that silently stops running is caught on its own schedule. It doesn’t wait for the 26-hour backup-age limit. The pCloud mirror reports to a third check (BACKUP_PCLOUD_HEALTHCHECK_URL), so a mirror problem never turns the backup check red; see Backup and Restore
.
The three checks
Create all three in the healthchecks.io dashboard (Add Check). Email is the default alert channel.
| Check | Schedule | Grace | Pinged by |
|---|---|---|---|
la-droplet-monitor | Simple, period 5 min | 10 min | 60-monitor.ps1 (la-health.timer) |
la-droplet-backup | Cron 17 9 * * *, UTC | 2 h | 50-backup.ps1 (la-backup.timer): dump + R2 |
la-droplet-pcloud | Cron 17 9 * * *, UTC | 2 h | 50-backup.ps1: the pCloud mirror only (missing token or failed copy → /fail) |
The backup check uses a cron schedule that matches the timer exactly, so a late or skipped backup alerts within the grace period.
Configuration
/etc/la/monitor.env is root-only (600 root:root):
MONITOR_HEALTHCHECK_URL=https://hc-ping.com/<uuid>
MONITOR_DISK_ALERT_PERCENT=85
MONITOR_BACKUP_MAX_AGE_HOURS=26
The memory, swap and reboot limits (MONITOR_MEMORY_ALERT_PERCENT=70, MONITOR_MEMORY_SUSTAIN_CHECKS=3, MONITOR_SWAP_OUT_ALERT_MB=8, MONITOR_REBOOT_MAX_AGE_DAYS=7) default in the script. Add a line only to change one.
Then add one line to /etc/la/backup.env:
BACKUP_HEALTHCHECK_URL=https://hc-ping.com/<uuid>
Anyone who has a ping URL can mark a check up or down, so treat it like a password. The operator writes both files and installs them personally, the same way as backup.env:
scp -i $HOME\.ssh\id_ed25519 "<folder>\monitor.env" "<folder>\backup.env" operator@<droplet-ip>:/tmp/
ssh -i $HOME\.ssh\id_ed25519 operator@<droplet-ip> 'sudo install -m 600 -o root -g root /tmp/monitor.env /etc/la/monitor.env; sudo install -m 600 -o root -g root /tmp/backup.env /etc/la/backup.env; rm /tmp/monitor.env /tmp/backup.env'
Remove CRLF line endings from a file saved on Windows: sudo sed -i 's/\r$//' /etc/la/monitor.env /etc/la/backup.env. Check the result by key name only, without printing any values:
sudo awk -F= '/^[A-Z]/{ print $1, (length($2) ? "set" : "EMPTY") }' /etc/la/monitor.env
Execute
sudo pwsh -NoProfile -File infra/scripts/60-monitor.ps1 -Install # once: la-health.service + .timer
sudo systemctl start la-health.service # one check now
sudo journalctl -u la-health.service -n 5 --no-pager -o cat
A healthy run logs [60-monitor] All checks passed. memory 38% used, swap 0 MB used. A failed ping logs Ping to <url> failed: …. The ping URL appears in that line, so keep journal excerpts out of public places.
Prove an alert fires
sudo systemctl stop la-server
sudo systemctl start la-health.service # exits 1; pings /fail with the reasons
# confirm the DOWN email, then:
sudo systemctl start la-server
sudo systemctl start la-health.service # passes; the check returns to up
Only run this before cutover, or inside an announced window: stopping la-server stops the game.
Rehearsed (2026-10-12)
| Check | Result |
|---|---|
60 -Install | Pass: la-health.timer registered, every 5 min |
| One monitor run | Pass: All checks passed., ping received |
One backup run (la-backup.service) | Pass: Backup complete., ping received |
Alert proof: la-server stopped | Pass: FAILED: la-server is not active; /health unreachable: Connection refused, /fail ping sent |
| Recovery | Pass: /health 200, next run passed |
Harmless noise: under systemd, PowerShell 7.6 prints a NamedPipeIPC_ServerListenerError … Operation canceled line as it exits. The unit still deactivates successfully.
Alert channel
Every check alerts through two channels that share nothing (healthchecks.io → Integrations, both assigned to all checks):
| Channel | Priority | Why |
|---|---|---|
| Pushover → the operator’s phone | High for DOWN (breaks through Do Not Disturb), Low for UP (silent) | Hard to miss, and doesn’t depend on the mailbox |
| healthchecks.io’s own email to the operator | — | The record, searchable later |
Why two. Until 2026-11-13 email was the only channel, and it failed the way email does: it was delivered and not seen. Around 2026-10-28 14:00 UTC a PowerShell snap refresh stopped la-health, la-backup and la-cf-ufw-refresh (see “If the check never runs at all” below). healthchecks.io marked la-droplet-monitor DOWN on time, at 14:14 UTC (5-minute period plus 10-minute grace), and the email reached the operator’s inbox a minute later. la-droplet-backup followed on 2026-10-29 at 11:17 UTC. Neither email was noticed for about 15 days. A Microsoft mailbox outage on 2026-11-13 at first made it look as if the alerts had never arrived, but their received timestamps settled it. A high-priority push to the phone is the channel meant to be seen; the email stays as the record. Emergency priority is deliberately not used: it repeats until acknowledged, which is too loud for a check that can blip (the 2026-10-21 deploy-gap false DOWN).
Keep Pushover licensed. The phone app is a one-time $4.99 per platform after a 30-day trial. The trial started 2026-11-13, so buy the license before about 2026-12-13; once the trial lapses, delivery to that phone stops without any error. To prove the path at any time: Integrations → Pushover → Test!.
PLAN §1 names Brevo as the alert path. Routing alerts through Brevo would add a second account and API key to the alert path, only to deliver the same email.
When la-droplet-monitor goes DOWN
- Convert the alert time to UTC. healthchecks.io shows it in your local zone (e.g.
22:55:30 -0700), but the droplet logs in UTC (05:55:30). - Read the ping body.
Connection refused (127.0.0.1:3000)means the server process wasn’t listening./health/db returned 503means PostgreSQL.a deploy has held /run/la-deploy.lockmeans a deploy is stuck.used for 3 runs in a roworMB went out to swapis the D-002 resize trigger: checkfree -mandps aux --sort=-rss | headbefore resizing, in case one runaway process is the cause.reboot required since …means a kernel update has waited a week. Disk, backup-age, loopback, andla-cf-ufw-refreshfailures name themselves. - Check whether it’s still down:
curl -s https://api.legendary-arena.com/healthandsudo systemctl start la-health.service. If the next timer run passed, healthchecks.io has already marked the check up again. - Line the alert up with the journals (use
sudo, oroperatorsees only its own entries):
sudo journalctl -t la-deploy --since "<UTC time - 10 min>" --until "<UTC time + 10 min>" # a deploy running?
sudo journalctl -u la-server --since ... --until ... | grep -E "listening|Stopped|exited" # restart or crash?
sudo journalctl -u la-server-standby --since ... --until ... # was :3001 covering?
sudo grep "<yyyy/mm/dd hh:mm>" /var/log/nginx/error.log # did users see errors?
A restart with no deploy around it means a crash: read the la-server lines just before the stop. If nginx has no upstream errors in that minute, users weren’t affected.
If the check never runs at all. systemctl status la-health showing status=203/EXEC means systemd can’t execute the unit’s ExecStart binary, so no check ran and no ping went out. On 2026-11-13 this turned out to have stopped la-health, la-backup and la-cf-ufw-refresh together. PowerShell is a snap on the droplet, and their units had been installed with a versioned /snap/powershell/<rev>/… path that a snap refresh deleted. The last backup was 2026-10-28. Repair it with grep -lE '/snap/(bin/pwsh|powershell/[0-9]+/)' /etc/systemd/system/*.service | xargs -r sudo sed -i -E 's#(/snap/bin/pwsh|/snap/powershell/[0-9]+/opt/powershell/pwsh)#/snap/powershell/current/opt/powershell/pwsh#', then sudo systemctl daemon-reload, and start each service once. The provisioners now install the snap’s current path, which snapd keeps pointed at the live revision. They avoid /snap/bin/pwsh: it goes through snap run, which moves the process into its own scope, so the script’s lines would leave journalctl -u <unit>.
If the check dumps core. Job for la-health.service failed because a fatal signal was delivered causing the control process to dump core, with The temporary HOME directory '/tmp/pwsh-root-…' already exists but does not have the expected user-only permission in journalctl -u la-health, means the unit has no HOME. systemd sets none for a root service, and PowerShell 7.7.0 (snap revision 456, published 2026-11-13) refuses to start without one; 7.6.9 tolerated it. The provisioners now write Environment=HOME=/root into all three units. Repair an older box with for u in la-health la-backup la-cf-ufw-refresh; do sudo grep -q '^Environment=HOME=' /etc/systemd/system/$u.service || sudo sed -i '/^Type=oneshot$/a Environment=HOME=/root' /etc/systemd/system/$u.service; done, then sudo systemctl daemon-reload.
Worked example (2026-10-21 05:55 UTC). A DOWN alert with Connection refused landed 1 s into the ~4 s restart of the 9dcde5f8 deploy. The standby served traffic, so nginx logged no errors, and the next run passed. It was a false alarm, and lab #43 closed the gap: the check now waits on the deploy lock.
What to watch
The timer catches “down”. These signs catch “unhealthy”, the slow failures that came before Render’s freezes:
- Restart loops:
systemctl show la-server -p NRestartsshould stay at 0 between deploys.journalctl -u la-server --since "1 hour ago" | grep -c Startedcounts the starts. - CPU load:
uptimeload average should stay well under 1 per core (under 2 on this 2-vCPU box). Sustained load near or above 2 starves the socket loop and freezes clients. - Memory:
la-healthnow alerts on the memory and swap halves of the D-002 trigger (see Summary), and every ping body carries the reading.free -mshows it by hand. CPU above ~50% under real traffic is the one half still watched by hand. - Query load:
pg_stat_statementsis preloaded (30-postgres.ps1), sosudo -u postgres psql -d la -c "select calls, round(total_exec_time) as ms, left(query, 80) from pg_stat_statements order by total_exec_time desc limit 10"shows where database time goes.select pg_stat_statements_reset()starts a fresh window. - DB connections:
sudo -u postgres psql -c "select count(*) from pg_stat_activity where datname='la'".
journald basics
journalctl -u la-server -f # follow the app log
journalctl -u la-server --since "30 min ago" # recent window
journalctl -u la-server -p err --since today # errors only
journalctl -u la-health -u la-backup -n 20 # the timers' last runs
systemctl list-timers 'la-*' # when each timer last ran and next runs
What this does not cover
The on-box check calls 127.0.0.1 directly, so it never tests Nginx, TLS, Cloudflare or DNS. The healthchecks.io switch covers the “whole droplet is gone” case. The public path is covered by an external monitor: UptimeRobot (free plan, created 2026-10-12) checks https://api.legendary-arena.com/health/db every 5 minutes and emails on failure. It was green against Render before cutover and followed api. to the droplet with no change. It watched /health until 2026-10-13, when it moved to /health/db (engine D-24711), so it now fails on a dead database as well as on a dead server, Nginx, TLS, Cloudflare or DNS.
References
- Plan §4.6 (
monitoring.mdstub), §5 (script manifest), and §9 (Cutover) - Backup and Restore
:
50-backup.ps1andbackup.env