Monitoring
Monitoring
Migration in progress.
api.legendary-arena.comand its PostgreSQL are moving off Render onto a self-hosted DigitalOcean Ubuntu droplet fronted by Cloudflare, perdocs/PLAN.md. Until decommission, Render stays warm as the rollback target.
Summary
Render’s dashboard and automatic restarts go away at cutover, so two signals replace them:
- On the box:
60-monitor.ps1runs every 5 minutes and pings a dead-man’s-switch service. - From outside: an uptime monitor hits
https://api.legendary-arena.com/health.
The first catches a sick box. The second catches anything between users and the box (DNS, Cloudflare, Nginx, TLS). Both alert even when the box is completely dead, because a missing ping is itself an alert.
Mechanics
On-box check (la-health.timer, every 5 minutes):
la-serveris active.GET http://127.0.0.1:<PORT>/healthreturns 200 (PORTfrom/etc/la/.env).- The root disk is below
MONITOR_DISK_ALERT_PERCENT(85). - The newest dump in
/var/backups/lais younger thanMONITOR_BACKUP_MAX_AGE_HOURS(26).
All pass → POST MONITOR_HEALTHCHECK_URL. Any fail → POST <url>/fail with the reasons, and the run exits non-zero, so it also shows up in systemctl --failed and the journal. Configuration lives in /etc/la/monitor.env (root-only).
Ping service. Any dead-man’s-switch service works. healthchecks.io’s free tier is enough: create one check with a 5-minute period and a 10-minute grace, and copy its ping URL into MONITOR_HEALTHCHECK_URL. A second check can take BACKUP_HEALTHCHECK_URL with a 1-day period. Send alerts to email; route them through Brevo if you want them in the same inbox as the other business alerts (PLAN §1).
External uptime check. In UptimeRobot or healthchecks.io, add an HTTP check on https://api.legendary-arena.com/health, expecting 200 every 5 minutes. This is dashboard setup with no script. Before cutover it watches Render; after cutover, the droplet.
What to watch by hand (no script):
journalctl -u la-serverfor restart loops.- Load average below the CPU count (
uptime; 4 on the 8 GB droplet). - Postgres connections:
sudo -u postgres psql -c "select count(*) from pg_stat_activity". - The dashboard’s Server Runtime Health tile (event-loop lag), which keeps working after cutover because it reads from the server.
Interactions
- Deploy Server
— the
/healthendpoint and thela-serverunit being checked. - Backup and Restore — the backup-age check watches its output.
- PostgreSQL Maintenance — the usual response to the disk alert.
Edge Cases
- No URL configured → results go to the journal only. Set
MONITOR_HEALTHCHECK_URLbefore cutover; without it nobody hears about a failure. - A failed ping is not a failed check. The ping service alerts on the missing ping, so the script logs and moves on.
- Before the first backup the backup-age check fails by design. Run
50-backup.ps1once right after installing.
Execute
sudo pwsh -File /opt/la-lab/infra/scripts/60-monitor.ps1 # one check now
sudo pwsh -File /opt/la-lab/infra/scripts/60-monitor.ps1 -Install # register la-health.timer
Verify
systemctl list-timers la-health.timer --no-pager
journalctl -u la-health --no-pager -n 20 # "All checks passed."
Then confirm the ping service shows the check as up, and that stopping the server (systemctl stop la-server) produces an alert within one period.
References
infra/scripts/60-monitor.ps1— owning artifact.docs/PLAN.md— §1 monitoring decision, §6.1 Phase 4 exit criteria (“uptime alert path proven”).