did!dZddlZddlZddlZddlZddlZddlmZmZddlm Z ddl m Z m Z ddl mZmZmZmZej$eZdZdZd Zd ZGd d Zdd Zy)u GCCWatchdog — Heartbeat Monitor and Auto-Respawn for All 5 Command Centres =========================================================================== Runs as a separate daemon (in its own tmux session or background process). Polls heartbeat files every 30 seconds. Respawns any centre whose heartbeat is stale (older than STALE_THRESHOLD_SECONDS = 120 s). Author: Genesis Parallel Builder Created: 2026-02-26 N)datetimetimezone)Path)AnyOptional)CENTRES_CONFIGGCC_HEARTBEATS_DIRSTALE_THRESHOLD_SECONDS launch_onez gcc-watchdog <ceZdZdZeefdededdfdZdededdfd Z dd Z d e de fd Z d e deefd ZdeddfdZdeddfdZdefdZy) GCCWatchdogz Monitors all 5 command centres. Respawns if heartbeat stale > 2 min. Tracks per-centre respawn counts and enforces a cooldown between successive respawns to avoid thrashing. poll_intervalstale_thresholdreturnNcn||_||_d|_tDcic]}|dd c}|_tDcic]}|dd c}|_t j t j|jt j t j|jycc}wcc}w)NTnamer) rrrunningr_respawn_counts _last_respawnsignalSIGTERM_signal_handlerSIGINT)selfrrcs =/mnt/e/genesis-system/core/gemini_command_centres/watchdog.py__init__zGCCWatchdog.__init__.s +. GU/U& 1 /U%3: !AfItO:   fnnd&:&:; fmmT%9%9: 0V: s B- B2signumframec>tjd|d|_y)Nu+GCCWatchdog received signal %d — stoppingF)loggerinfor)rr!r"s rrzGCCWatchdog._signal_handler@s A6J ctjdtt|j|j |j rtD]}|d}t|dz }|js(tjd||j|L|j|r>|j|}tjd||xsd|j|tjd|tj|j|j rtjdy ) u Continuous monitoring loop. Every WATCHDOG_POLL_INTERVAL seconds: 1. Check each centre's heartbeat file. 2. If stale → respawn (subject to cooldown and max attempts). 3. Log summary. uBGCCWatchdog started — monitoring %d centres, poll=%ds, stale=%dsrz.jsonu/GCCWatchdog: no heartbeat for %r — respawningu:GCCWatchdog: %r heartbeat stale (age=%.0fs) — respawningzGCCWatchdog: %r OKzGCCWatchdog stoppedN)r$r%lenrrrrr existswarning_maybe_respawn _is_stale_heartbeat_agedebugtimesleep)rcentrerhb_fileages rrunzGCCWatchdog.runFs   P        ll( =f~,$u~=~~'NNI4''/>>'*--g6CNNT r ''/LL!5t<) =, JJt)) */ll2  )*r&r3cH|j|}|y||jkDS)z Return True if the heartbeat file's last-written timestamp is older than stale_threshold seconds. Args: hb_file: Path to the heartbeat JSON file. Returns: True if stale, False if fresh. T)r.r)rr3r4s rr-zGCCWatchdog._is_staless-!!'* ;T))))r&c t|dd5}tj|}dddjdd}|rIt j |}t j tj|z jS |jj}tj|z S#1swYxYw#tjttf$rY^wxYw#t$rYywxYw)a Return the age in seconds of the timestamp recorded inside the heartbeat file. Falls back to file mtime if JSON parse fails. Args: hb_file: Path to heartbeat JSON. Returns: Age in seconds, or None on error. rzutf-8)encodingN timestamp)openjsonloadgetr fromisoformatnowrutc total_secondsJSONDecodeError ValueErrorOSErrorstatst_mtimer0)rr3fhdatats_strlast_tsmtimes rr.zGCCWatchdog._heartbeat_ages gsW5 %yy} %XXk2.F"008 X\\2W<KKMM LLN++E99;& & % % $$j':     s:C B>A$C 0C.>CC C+*C+. C:9C:r2cp|d}tj}|jj|}|.||z tkr"t j d|t||z z y|jj|d}|tk\rt jd|ty|j|y)z Respawn a centre if not in cooldown and under max attempts. Args: centre: Centre config dict from CENTRES_CONFIG. rNz5GCCWatchdog: %r in respawn cooldown (%.0fs remaining)ruIGCCWatchdog: %r has exceeded max respawn attempts (%d) — NOT respawning) r0 monotonicrr?RESPAWN_COOLDOWNr$r%rMAX_RESPAWN_ATTEMPTSerror_respawn)rr2rrAlastcounts rr,zGCCWatchdog._maybe_respawnsf~nn!!%%d+  t/? ? KKG C$J/  $$((q1 ( ( LL[$   fr&cF|d}t|d}|jj|ddz|j|<tj|j |<|r*t jd||j|tyt jd|y) zs Kill old tmux session and relaunch daemon. Args: centre: Centre config dict. rF)rverboserz)GCCWatchdog: respawned %r (attempt %d/%d)z!GCCWatchdog: failed to respawn %rN) r rr?r0rOrr$r%rQrR)rr2rsuccesss rrSzGCCWatchdog._respawnsf~$6%)%9%9%=%=dA%F%JT"#'>>#34  KK;$$T*$   LL3T r&c |jrdnd|j|jtDcgc]}|d c}Dcic];}||jj |d|j j |d=c}dScc}wcc}w)z Return current watchdog status including per-centre respawn counts. Returns: Status dict. rstoppedrr) respawn_count last_respawn)watchdogrrcentres)rrrrrr?r)rrrs r get_statuszGCCWatchdog.get_statuss&*\\ y!//#33 1??1QvY?   %)%9%9%=%=dA%F$($6$6$:$:4$@  @ s B AB rN)__name__ __module__ __qualname____doc__WATCHDOG_POLL_INTERVALr intr rrr5rboolr-rfloatr.dictr,rSr`r&rrr&s46;;;  ;$c#$ )+Z**$* dx>Td@t6 D r&rctjtjdtjt }|j y)z-Entry point to start the GCCWatchdog process.z1%(asctime)s [%(levelname)s] %(name)s: %(message)s)levelformatstreamN)logging basicConfigINFOsysstdoutrr5)r^s r run_watchdogrus3 llBzz }H LLNr&ra)rer=rprrsr0rrpathlibrtypingrr$core.gemini_command_centres.launcherrr r r getLoggerrbr$rfWATCHDOG_TMUX_SESSIONrQrPrrurkr&rr{sn   '    8 $&I I Xr&