şi ) JdZddlZddlZddlZddlZddlZddlZddlmZm Z ddl m Z m Z ejeZdZdZdZgdZd efd Zdad Zd ed eefd Zde d efdZdeed eeefdZ ddee deded ee fdZdZdZ dd ede eded eefdZ y)a MODULE 4: Gemini Embedder ========================== Converts Chunk objects into EmbeddedChunk objects with 3072-dim vectors using the gemini-embedding-001 model. Stories implemented: 4.01 - Single Text Embedder 4.02 - Batch Embedder with Rate Limiting 4.03 - Embedding Cache (Redis) 4.04 - Build Embedding Text Optimizer Usage: from core.kb.embedder import embed_text, embed_batch, embed_with_cache, build_embedding_text N)ListOptional)Chunk EmbeddedChunkzgemini-embedding-001i i@)returncvtjdd}|sd}tjj|rrt |5}|D]W}|j }|j ds%|jdddj j d}nddd|S|S#1swY|SxYw) z9Load GEMINI_API_KEY from environment or secrets.env file.GEMINI_API_KEYz(/mnt/e/genesis-system/config/secrets.envzGEMINI_API_KEY==rz'"N)osgetenvpathexistsopenstrip startswithsplit)key secrets_pathfhlines )/mnt/e/genesis-system/core/kb/embedder.py _load_api_keyr1s ))$b )C A 77>>, 'l# rD::zembed_text..oEt47?E429500503zembed_text failed after rz attempts: )r ValueErrorr$ enumerate _RETRY_DELAYSloggerwarninglentimesleepmodels embed_content EMBED_MODELlist embeddingsvalues Exceptionstrany RuntimeError)r&clientlast_excattemptdelayresultexcr1s @r embed_textrQRs tzz|FGG  F%)H#QC-$78  NN>]I[]b c JJu  ]]00!1F))!,334 4&  "3}#5#9":+hZP  #hGE/DEE  sAC(( D1!DDDchunkc|jrd|jdnd}|jr*|jjrd|jdnd}||fDcgc]}|s| }}|rdj|dz}nd}||jd|j }t jdd|}t jdd|}t|tkDr |d t}|Scc}w) a- Build optimised text for embedding. Format: [{platform}] [{heading_context}] {title}\n{text} - Heading context is included when present. - Total output is capped at MAX_EMBEDDING_CHARS (8000 chars). - Excessive whitespace (multiple spaces/newlines) is normalised to single. []r   z[ \t]+z\n{3,}z N) platformheading_contextrjointitler&resubr>MAX_EMBEDDING_CHARS)rR platform_tag heading_tagp prefix_partsprefixraws rbuild_embedding_textre~s-2NNQu~~&a(L272G2GELaLaLgLgLiAe++,A.oqK ,k:@!aA@L@,'#- HU[[MEJJ< 0C &&C %C &&FC (C 3x%%&&' J!As C,%C,textsct}d}tdgtzD]\}}|r,tj d||t j | |jjt|}|jDcgc]}t|jc}cSt!d|cc}w#t$r,}t|tfddDr|}Yd}~Ȃd}~wwxYw)zEmbed multiple texts in a single API call using Gemini's batch endpoint. Returns list of vectors (same order as input texts). Retries on transient errors (429, 500, 503). Nrz&_embed_texts_batch: retry %d after %dsr(c3&K|]}|v ywr,r-r.s rr2z%_embed_texts_batch..r3r4r5z)_embed_texts_batch failed after retries: )r$r:r;r<r=r?r@rArBrCrErDrFrGrHrIrJ) rfrKrLrMrNrOerPr1s @r_embed_texts_batchrjs  F$(H#QC-$78  NNCWe T JJu  ]]00!1F-3,=,=>qDN> > B8*M NN? #hGE/DEE   s0/B:B5!B:5B:: C/!C*)C**C/chunks batch_sizemax_rpmc g}t|}g}td||D](}||||z}tj}|D cgc] } || z dks | }} t||k\r>d||dz z dz} | dkDr+tj d| tj | |D cgc] } t| } } t| } |jtjt|| D])\}}| |jt||t+t!||z|}tj d ||+|Scc} wcc} w#t$r}tjd|g} | D]Z} | jt|#t$r1}tjd|| jdYd}~Td}~wwxYwYd}~d}~wwxYw) ah Embed multiple chunks with true API batching, rate limiting, and progress. - Sends up to batch_size texts per API call (massive speedup vs 1-at-a-time). - Rate-limits to max_rpm requests per minute. - Retries on transient errors (429, 500, 503) with exponential backoff. - Returns list of EmbeddedChunk (chunk + vector + model name). rgN@g?zRate limit: sleeping %.1fsz4Batch embed failed (%s), falling back to 1-at-a-timezSingle embed failed: %sN)rRvectorembedding_modelzEmbedded %d/%d chunks...)r>ranger? monotonicr<infor@rerjrGr=appendrQziprrCmin)rkrlrmembeddedtotalrequest_timestamps batch_start batch_chunksnowt sleep_forcrfvectorsrPr& single_excrRrodones r embed_batchrs%'H KE&(Qz2%= kK*,DE nn);NAsQw~aNN ! "g -&8&; ;: V M6M3 PRUV  L NNEs K K L V NNOQT U U M VsJ$D -AD7=AE# D4D//D47 E EE # F ,FF )2ix)NzOredis://default:e2ZyYYr4oWRdASI2CaLc-@redis-genesis-u50607.vm.elestio.app:26379)!__doc__rrloggingrr\r?typingrrcore.kb.contractsrr getLogger__name__r<rC VECTOR_DIMr^r;rHrr r$floatrQrerjintrrrrr-r%rrs6   !2   8 $ %   s   %S%T%[%X#HOd3iODe,=O:8 K888 - 8~ # $f1 1}11 %[ 1r%