category: infrastructure

axiom: ALL psycopg2 connection pools to remote PostgreSQL (Elestio) MUST include keepalives=1, keepalives_idle=30, keepalives_interval=10, keepalives_count=5. Without this, idle connections are silently closed by PG server after ~30min, causing health check failures and service restart loops.

source: failure_010 pre-mortem 2026-02-21


category: resilience

axiom: get_conn() must validate the returned connection with a liveness check (SELECT 1). On failure: close entire pool, set pool=None, recreate pool, return fresh connection. This prevents cascading failures from stale connections poisoning the pool.


category: operations

axiom: The watchdog correctly detects and restarts unhealthy services. BUT the 5-minute polling gap creates a vulnerability window. Primary fix is always making the SERVICE itself resilient (keepalives, self-healing pool). Watchdog is the last line of defense, not the first.