Legendary Arena Lab

Fail2ban and Unattended Upgrades

foundations

Migration in progress. api.legendary-arena.com and PostgreSQL are moving from Render to a self-hosted DigitalOcean Ubuntu host behind Cloudflare. Render remains rollback-ready until decommission phase.

Fail2ban and Unattended Upgrades

Enable intrusion mitigation and automatic security patching that were previously abstracted by managed hosting.

Summary

This page owns infra/scripts/07-fail2ban-upgrades.ps1, which:

  • installs fail2ban and unattended-upgrades
  • enables and starts fail2ban service
  • enables unattended upgrades on a daily schedule
  • adds the PGDG repository (apt.postgresql.org, suite <codename>-pgdg) to unattended-upgrades’ allowed origins, in its own drop-in /etc/apt/apt.conf.d/51la-pgdg-unattended-upgrades
  • turns on an automatic reboot at a fixed quiet hour, 10:30 UTC (LA_AUTO_REBOOT_TIME), for days an update needs one, in its own drop-in /etc/apt/apt.conf.d/52la-auto-reboot

PostgreSQL updates from PGDG

PostgreSQL 18 comes from PGDG (PostgreSQL Setup ), not from Ubuntu. Ubuntu’s 50unattended-upgrades only allows Ubuntu’s own pockets, so before the drop-in, PostgreSQL minor and security releases waited for someone to run apt upgrade. APT appends list entries across files, so the drop-in adds one origin and leaves the packaged file alone.

What it upgrades: only installed packages from PGDG (postgresql-18, postgresql-client-18, postgresql-common, libpq5 and their siblings). It never moves to a new major: PostgreSQL 19 is a different package, postgresql-19, and nothing installs it. A major upgrade stays a planned pg_upgradecluster.

What happens on the box: the package’s maintainer scripts restart the cluster, a few seconds of downtime, during the apt-daily-upgrade window (06:00 UTC plus up to 60 minutes of random delay, so late evening US Pacific). la-server rides through it: its unit only orders itself After=postgresql.service (no Requires=), so systemd doesn’t stop it, and the server’s pg pool drops the dead idle connections and opens new ones (the pool’s error listener keeps the process up). Requests in flight during those seconds fail, and a la-health run that lands in them can send one DOWN that clears on the next run. provision-smoke proves the restart with apt-get install --reinstall postgresql-18, which runs the same maintainer scripts: the postmaster start time changes, la-server keeps its PID, and /health/db answers 200 again.

Automatic reboot at 10:30 UTC

Kernel and libc security updates install unattended but only take effect after a reboot. On 2026-11-13 two arrived within a day (kernel 6.8.0-147, then libc6 and 6.8.0-149). So when an update leaves /var/run/reboot-required, unattended-upgrades schedules shutdown -r 10:30 for that day. The droplet’s clock is UTC, so 10:30 UTC is 2:30 AM Pacific in winter and 3:30 AM in summer.

The time is after the 06:00–07:00 UTC upgrade run that sets the flag, and after the 09:17 UTC backup. The reboot costs about 15 s: Nginx has no standby across it, so live sockets drop and clients reconnect. la-server, PostgreSQL and the la-* timers are enabled units and come back by themselves.

Two edges:

  • A deploy that happens to be building at 10:30 is cut off. Its half-built release directory never goes live, the live release and live-commit are untouched, and the next push redeploys.
  • If the reboot ever doesn’t happen, 60-monitor.ps1 still warns about the pending reboot in its ping body and fails after 7 days (Monitoring ).

To change the hour, re-run 07 with LA_AUTO_REBOOT_TIME=HH:MM (UTC). A malformed value is refused before anything changes. To see a reboot that’s already scheduled for today: cat /run/systemd/shutdown/scheduled.

Execute

sudo pwsh -File infra/scripts/07-fail2ban-upgrades.ps1

Verification

# fail2ban running
sudo systemctl status fail2ban --no-pager
sudo fail2ban-client status

# unattended upgrades configured
sudo systemctl status unattended-upgrades --no-pager
sudo cat /etc/apt/apt.conf.d/20auto-upgrades

# PGDG is an allowed origin
sudo unattended-upgrade --dry-run --debug 2>&1 | grep -m1 'Allowed origins'

# what unattended-upgrades actually installed, and when
sudo grep -h 'Packages that will be upgraded' /var/log/unattended-upgrades/unattended-upgrades.log | tail -5

Expected:

  • fail2ban service active
  • unattended upgrades enabled for package list update and upgrade install
  • the allowed origins include o=apt.postgresql.org,a=noble-pgdg

Operational note

Automatic patching can trigger package-level service restarts. Ensure the app remains managed by systemd so service restarts are recoverable.

References

  • Plan section 4.1 (fail2ban-unattended-upgrades.md stub)
  • Plan section 5.1 (Base security services)