!in2dZddlmZddlZddlZddlZddlmZmZddl m Z m Z m Z ddl m Z eGddZeGd d ZeGd d Zd ZdZGddZy)u core/evolution/scar_aggregator.py Story 8.05: ScarAggregator — L3 Failure Pattern Collector Collects, deduplicates, and clusters failure patterns from Qdrant L3 (genesis_scars collection). Produces ScarReport with clustered analysis of recurring failure modes to feed the Nightly Epoch. VERIFICATION_STAMP Story: 8.05 Verified By: parallel-builder (claude-sonnet-4-6) Verified At: 2026-02-25 Tests: 11/11 Coverage: 100% ) annotationsN) dataclassfield)datetime timedeltatimezone)Anyc:eZdZUdZded<ded<ded<ded<y ) ScarClusterz@A cluster of similar failure scars grouped by cosine similarity.str cluster_idrepresentative_scarint member_countfloat avg_severityN__name__ __module__ __qualname____doc____annotations__7/mnt/e/genesis-system/core/evolution/scar_aggregator.pyr r !sJOrr cFeZdZUdZded<eeZded<dZded<y ) ScarReportz@Aggregated report of all scar clusters from the lookback window.r total_scars)default_factorylist[ScarCluster]clustersrnew_since_last_epochN) rrrrrrlistr!r"rrrrr+s&J"'"=H= !#!rrcDeZdZUdZded<ded<ded<ded<ded <y ) _ScarRecordz8Internal representation of a scar retrieved from Qdrant.r scar_idtextrseverity timestamp list[float]vectorNrrrrr%r%9sB L ION rr%zC/mnt/e/genesis-system/data/observability/scar_aggregation_log.jsonlg333333?c|eZdZdZ d d dZdddZdddZddZddZddZ dd Z dd Z e dd Z y)ScarAggregatora  Collects, deduplicates, and clusters failure scars from Qdrant L3. Usage ----- agg = ScarAggregator(qdrant_client=client, last_epoch_timestamp="2026-02-24T02:00:00Z") report = agg.aggregate(lookback_days=7) top = agg.get_top_clusters(n=5) NcL||_||_|xst|_d|_y)a^ Parameters ---------- qdrant_client: Injected Qdrant client. When None, a real QdrantClient is constructed lazily on first call to aggregate(). In tests, pass a mock. last_epoch_timestamp: ISO 8601 timestamp of the last completed Nightly Epoch. Used to compute new_since_last_epoch. When None, defaults to 7 days ago (conservative fallback). log_path: Override path for the JSONL aggregation log. Useful in tests (pass tmp_path / "scar_aggregation_log.jsonl"). N)_client_last_epoch_ts_DEFAULT_LOG_PATH _log_path _last_report)self qdrant_clientlast_epoch_timestamplog_paths r__init__zScarAggregator.__init__Zs(*% *>&;*;/3rctjtjt |z }|j |}|s(t dgd}|j|||_|S|j|}|j|}t t|||}|j|||_|S)a? Query Qdrant for scars within the lookback window, cluster them, write the report to the JSONL log, and return the ScarReport. Parameters ---------- lookback_days: How many days back to query (default 7). Returns ------- ScarReport tzdaysr)rr!r") rnowrutcr _fetch_scarsr _write_logr3_cluster_scars_count_new_since_last_epochlen)r4 lookback_dayscutoffscarsreportr! new_counts r aggregatezScarAggregator.aggregatexs. 1NN!!&)AQRSF OOF # &D M&&u-44U; E !*  " rc|j tdt|jjdd}|d|S)uf Return the top-N clusters sorted by member_count DESC. Must call aggregate() first; raises RuntimeError if not yet called. Parameters ---------- n: Maximum number of clusters to return. Returns ------- list[ScarCluster] — at most n items, sorted by member_count DESC. Nz@No report available. Call aggregate() before get_top_clusters().c|jSN)r)cs rz1ScarAggregator.get_top_clusters..s !..rT)keyreverse)r3 RuntimeErrorsortedr!)r4nsorted_clusterss rget_top_clusterszScarAggregator.get_top_clusterssP    $R !    & &(  r""rc |j} |jdddd\}}g}|D]}|jxsi}|j dd} t j |jdd} | |krQ |jxsg} t| ts | rt| jd ng} |jtt!|j"|j d dt%|j d d || |S#t$rgcYSwxYw#ttf$rYwxYw)z Retrieve all scars from Qdrant genesis_scars collection whose timestamp >= cutoff. Returns a list of _ScarRecord objects. genesis_scarsi'T)collection_namelimit with_payload with_vectorsr)Z+00:00rr'r(g?)r&r'r(r)r+) _get_clientscroll Exceptionpayloadgetr fromisoformatreplace ValueErrorAttributeErrorr+ isinstancer#valuesappendr%r idr) r4rFclientresults_rGpointrcts_strtsr+s rr@zScarAggregator._fetch_scarssK!!#  /!! 'JGQ$& Emm)rG[[b1F ++FNN3,IJ; \\'RFfd+5;fmmo.q1 LLM VR0"7;;z3#?@$!  ! 4 = I /  s#D*D0 D-,D-0EEc  g}g}|D]}d}t|D]B\}}|j|j|}|tk\s,||j |d}n|rX|j |g|j |jg} t|D]p\}} t | d} t d| Dt| z } | j td|d| jt| t| d r| S) uL Greedy cosine-similarity clustering. For each scar (in order): - Check against the representative vector of each existing cluster. - If similarity >= CLUSTER_THRESHOLD → add to that cluster. - Otherwise → start a new cluster. Returns a list of ScarCluster dataclasses. FTc|jSrMr()ss rrOz/ScarAggregator._cluster_scars..s QZZr)rPc34K|]}|jywrMru).0ms r z0ScarAggregator._cluster_scars.. s6!**6scluster_04dr rrr) enumerate_compute_cosine_similarityr+CLUSTER_THRESHOLDrkmaxsumrDr r'round) r4rGr!representative_vectorsscarplacedidxrep_vecsimresultmembersrepavg_sevs rrBzScarAggregator._cluster_scarss-/46 ;DF )*@ A  W55dkk7K++SM((.!F   '&--dkk: ;%'%h/ LCg#78C6g66WEG MM!)#c3(+!$W!&w!2    rc|jr1 tj|jjdd}n1tj t jtdz }d}|D]<} tj|jjdd}||kDr|dz }>|S#t$r4tj t jtdz }YwxYw#ttf$rYwxYw)z Count scars whose timestamp is strictly after the last epoch timestamp. If last_epoch_timestamp is None, uses 7 days ago as a conservative boundary (same as the default lookback). r^r_r:r<r) r0rrerfrgr>rr?rr)rh)r4rGepoch_dtcountrscar_dts rrCz*ScarAggregator._count_new_since_last_epochs    M#11''//X>  ||x||4ya7HHH D "00NN**39X%QJE   ! M#<<8<<89!;LL M/  s#/B479C44:C10C14DDc tjj|j}|rtj|dt j tjj|jt|j|j|jDcgc]1}|j|j|j |j"d3c}d}t%|jdd5}|j't)j*|d zd d d y cc}w#1swYy xYw) z~ Append the ScarReport as a JSONL entry to the aggregation log. Creates parent directories if needed. T)exist_okr:r~)r)r cluster_countr"r!azutf-8)encoding N)ospathdirnamer2makedirsrr>rr? isoformatrrDr!r"r rrropenwritejsondumps)r4rHlog_dirrNentryfhs rrAzScarAggregator._write_log3s ''//$..1  KK$ /"6@@B!-- 1$*$?$?  #$,,+,+@+@$%NN$%NN    $..# 8 /B HHTZZ&- . / / / /s,6D0 ?(D55D>c|j |jS ddlm}|dd|_|jS#t$r}t d|d}~wwxYw)zV Return the injected client, or lazily construct a real QdrantClient. Nr) QdrantClientz"qdrant-b3knu-u50607.vm.elestio.appi)hostportz=qdrant_client package not installed and no mock was injected.)r/r5r ImportErrorrR)r4rexcs rr`zScarAggregator._get_clientOsc << #<<   2'9DL||  O  s; A AAc,|r|rt|t|k7rytdt||D}tjtd|D}tjtd|D}|dk(s|dk(ry|||zz S)z Compute cosine similarity between two vectors. Returns a float in [0, 1] for unit vectors, or [-1, 1] in general. Returns 0.0 for zero-length vectors to avoid division by zero. gc3,K|] \}}||zywrMr)rxrbs rrzz.us6DAq!a%6sc3&K|] }||z ywrMr)rxrs rrzz.v4q1u4c3&K|] }||z ywrMr)rxrs rrzz.wrr)rDrzipmathsqrt)vec_avec_bdotnorm_anorm_bs rrz)ScarAggregator._compute_cosine_similarityjsESZ3u:%=6Cu$56634e44534e445 S=FcMfvo&&r)NNN)r5r r6 str | Noner7rreturnNone)r)rErrr))rTrrr )rFrrlist[_ScarRecord])rGrrr )rGrrr)rHrrr)rr )rr*rr*rr)rrrrr8rJrVr@rBrCrAr` staticmethodrrrrr-r-Osw"+/# 44)4 4  4<"H#<.`)V</86''rr-)r __future__rrrr dataclassesrrrrrtypingr r rr%r1rr-rrrrs"# (22   "" "  Zm'm'r