Legendary Arena Lab

Monitoring

wiki

Monitoring

Migration in progress. api.legendary-arena.com and its PostgreSQL are moving off Render onto a self-hosted DigitalOcean Ubuntu droplet fronted by Cloudflare, per docs/PLAN.md . Until decommission, Render stays warm as the rollback target.

Summary

Render’s dashboard and automatic restarts go away at cutover, so two signals replace them:

  1. On the box: 60-monitor.ps1 runs every 5 minutes and pings a dead-man’s-switch service.
  2. From outside: an uptime monitor hits https://api.legendary-arena.com/health.

The first catches a sick box. The second catches anything between users and the box (DNS, Cloudflare, Nginx, TLS). Both alert even when the box is completely dead, because a missing ping is itself an alert.

Mechanics

On-box check (la-health.timer, every 5 minutes):

  • la-server is active.
  • GET http://127.0.0.1:<PORT>/health returns 200 (PORT from /etc/la/.env).
  • The root disk is below MONITOR_DISK_ALERT_PERCENT (85).
  • The newest dump in /var/backups/la is younger than MONITOR_BACKUP_MAX_AGE_HOURS (26).

All pass → POST MONITOR_HEALTHCHECK_URL. Any fail → POST <url>/fail with the reasons, and the run exits non-zero, so it also shows up in systemctl --failed and the journal. Configuration lives in /etc/la/monitor.env (root-only).

Ping service. Any dead-man’s-switch service works. healthchecks.io’s free tier is enough: create one check with a 5-minute period and a 10-minute grace, and copy its ping URL into MONITOR_HEALTHCHECK_URL. A second check can take BACKUP_HEALTHCHECK_URL with a 1-day period. Send alerts to email; route them through Brevo if you want them in the same inbox as the other business alerts (PLAN §1).

External uptime check. In UptimeRobot or healthchecks.io, add an HTTP check on https://api.legendary-arena.com/health, expecting 200 every 5 minutes. This is dashboard setup with no script. Before cutover it watches Render; after cutover, the droplet.

What to watch by hand (no script):

  • journalctl -u la-server for restart loops.
  • Load average below the CPU count (uptime; 4 on the 8 GB droplet).
  • Postgres connections: sudo -u postgres psql -c "select count(*) from pg_stat_activity".
  • The dashboard’s Server Runtime Health tile (event-loop lag), which keeps working after cutover because it reads from the server.

Interactions

Edge Cases

  • No URL configured → results go to the journal only. Set MONITOR_HEALTHCHECK_URL before cutover; without it nobody hears about a failure.
  • A failed ping is not a failed check. The ping service alerts on the missing ping, so the script logs and moves on.
  • Before the first backup the backup-age check fails by design. Run 50-backup.ps1 once right after installing.

Execute

sudo pwsh -File /opt/la-lab/infra/scripts/60-monitor.ps1            # one check now
sudo pwsh -File /opt/la-lab/infra/scripts/60-monitor.ps1 -Install   # register la-health.timer

Verify

systemctl list-timers la-health.timer --no-pager
journalctl -u la-health --no-pager -n 20      # "All checks passed."

Then confirm the ping service shows the check as up, and that stopping the server (systemctl stop la-server) produces an alert within one period.

References