i7dZddlZddlZddlmZmZddlmZmZmZm Z ddl m Z m Z m Z ejdZ ddeded ed eefd Zejd ej(Zejd ej(Zdededed efdZdeded ed eefdZdeed efdZ dde dede eded ed ee f dZ d dee dede ede eeefd ee f dZ d!dee de de ed ee fdZy)"u5 MODULE 3: SMART CHUNKER ======================== Splits ExtractedContent into overlapping Chunk objects, preserving heading hierarchy context and keeping code blocks intact. VERIFICATION_STAMP Story: M3 (Stories 3.01–3.05) Verified By: parallel-builder Verified At: 2026-02-26 Tests: 17/17 Coverage: 100% N)datetimetimezone)AnyDictListOptional)ChunkExtractedContentPlatformConfigz (?<=[.!?])\s+text chunk_sizeoverlapreturnc|j}|sgSt||kr|gStj|}|Dcgc]#}|js|j%}}g}g}d}d} | t|krQ|| } t| } | |kDr`|r$|j dj |g}d}d} | | kr-| |z} | | | }|j || | kr| |z n| } | | kr-| dz } ||rdndz| z}||kDr|r|j dj |g}d}t |D],}|jd||t|dzz }||k\s,n|}td|Dtdt|dz z}|j | |t|dkDrdnd| zz }| dz } | t|krQ|r |j dj ||Scc}w)aSplit text into overlapping chunks at sentence boundaries. Rules: - Prefer splitting at sentence endings (. ! ? followed by whitespace). - Each chunk is at most ``chunk_size`` characters. - Consecutive chunks share up to ``overlap`` characters at their boundary. - Empty input returns an empty list. - Text shorter than chunk_size returns a single chunk. r c32K|]}t|ywN)len).0ss (/mnt/e/genesis-system/core/kb/chunker.py zchunk_text..fs<c!f>'..1 2 < < >>cg}d}tj|D][}|||j}|r|jd|f|jd|j f|j }]||d}|r|jd|fggddfd }|D]\} } t | } | rJ| |kDr|j| 1| z|kDr || g| Fj| | z ]t| ||} | D]4} t | }|z|kDr r|j| |z 6|Dcgc]}|js|c}Scc}w)u Split *text* into chunks while keeping fenced code blocks (```…```) intact. Strategy: 1. Tokenise the text into alternating prose / code-fence segments. 2. Pack segments greedily into chunks of at most chunk_size chars. 3. When a code block fits in the current chunk, add it whole. 4. When a code block is too large for even an empty chunk, emit it as its own (oversized) chunk rather than splitting it mid-block. 5. Apply overlap at sentence boundaries between consecutive prose chunks. rFTNcRr jdjgdy)Nr)rr)r&r( current_partssrflushz,_split_preserving_code_blocks..flushs%  MM"''-0 1  r>)r r)rN) _CODE_FENCE_REfinditerr,rgroupr-rr2r)r r rsegmentslast_endmprosetailrCis_codesegseg_len prose_chunkspcpc_lencr&r(rBs @@@r_split_preserving_code_blocksrSs"(*HH  $ $T *Xaggi(  OOUEN +qwwy)*557   ?D  &F!MK!& c( # c"w&3!$ % $$S)w& &cj'RL" &R'*4G$$R(v%  &'&4 G +!A ++ +s E3,E3 lines_beforecNg}|D]}tjd|}|st|jd}|jdj }|Dcgc] }|d|ks |}}|j ||f|sydj d|DScc}w)zDReturn the active heading hierarchy as a string like '# H1 > ## H2'.r3rrrAz > c36K|]\}}d|zd|yw)#rN)rlvltxts rrz+_extract_heading_context..s#IxsCs 1SE*Is)rematchrrFrrr)rT heading_stacklinerIleveltitlehs r_extract_heading_contextrcs!M1 HH)4 0  OEGGAJ$$&E(5F11QFMF  % 01  ::I=I II Gs  B",B"contentplatform customer_idcn|jj}|sgS|j}g}g}g} |D]`} tjd| } | r$|r"t | } |j | |f| g}n|j | | j | b|rt | } |j | |fg} |D]m\}}dj|j}|s(t|||}|D]4}|js| j ||jf6ot| }g}t| D]l\}\}}t|j||}t||j|||j||||t|j  }|j |n|S)a+Chunk content preserving heading hierarchy context. - Each Chunk's ``heading_context`` contains the nearest parent headings. - Prefers to split at heading boundaries. - Keeps fenced code blocks intact. - chunk_id is a deterministic SHA-256 of (source_url + chunk_index + text). z ^#{1,6}\s+ chunk_idr4rerfrar heading_contextr5 total_chunksmetadata)r r splitlinesr\r]rcrrrSr enumerater=urlr radictrm)rdrerfr rr linessectionscurrent_section_lines seen_linesr_ is_headingctx raw_chunks heading_ctx sec_lines section_text sub_chunkssctotalresultidxchunk_text_valrjchunks rchunk_with_headingsrs <<   D  OO E-/H')J XXmT2 /*:6C OOS"78 9%)F ! ! ( ( .$ &z2345)+J"*= Yyy+113  2<WU  =Bxxz!!; ";< = =  OEF.7 .C* *k>$W[[#~F{{#--''**+    e Mr>r&extra_metadatac tjtjj d}g}|D]}t |j }||d<|j|d<|r|j|t|j|j|||j|j|j|j|j| }|j!||S)a Apply platform and customer tags to chunks for multi-tenant isolation. - Returns NEW Chunk objects (originals are not mutated). - Adds ``ingested_at`` (ISO 8601) and ``source_url`` to metadata. - Merges ``extra_metadata`` into each chunk's metadata. )tzz%Y-%m-%dT%H:%M:%SZ ingested_atr4ri)rnowrutcstrftimerqrmr4updater rjrar rkr5rlr) r&rerfrnow_isorrnew_metataggeds r tag_chunksr&sllhll+445IJGF#'#7")!&!1!1  OON +^^''#++!11))++    f'* Mr>contentsconfigcg}|D]k}||jr|jjs-t||j||j|j }|j |m|S)zChunk multiple extracted contents using platform config. - Uses config.chunk_size and config.chunk_overlap for all documents. - Skips None or empty-text contents gracefully. - Returns a flat list of all resulting Chunks. )rdrerfr r)r rrnamer chunk_overlapextend)rrrf all_chunksrd page_chunkss r chunk_batchrQsz!J ' ? ||7<<#5#5#7 )[[#((((   +& ' r>))Nrr)NNr)__doc__r8r\rrtypingrrrrcore.kb.contractsr r r compilerstrintlistr2 MULTILINE _HEADING_RErDr=rSrcrrrrYr>rrs  ',,EE2::./ P PPP #Y Pnbjj-r||< -r||<?#?C?s?s? C, C,C,C, #Y C,L J49 J J."& E EE#E E  E  %[ E^"&/3 $ K$$#$T#s(^, $  %[ $\"&#$ # %[ r>