i-UdZddlZddlZddlmZddlmZddlmZm Z m Z m Z m Z m Z ddlmZmZej"ddZej"d d Zd Zgd Zd Zej.dZdaeeed<defdZdedefdZdedefdZefde edede!fdZ" d'dedeede fdZ#defdedeedede!fdZ$dddefde e%dedeed e!d!e%dede efd"Z&efdedefd#Z'd'ded$eeddfd%Z(d'ded$eeddfd&Z)y)(u Qdrant Store — KB Ingestion Pipeline Module 5 ============================================== Upserts, deletes, searches, and reports stats for embedded KB chunks in the genesis_memories Qdrant collection. Stories implemented: 5.01 — upsert_vectors (batch upsert, deterministic UUID5 IDs) 5.02 — delete_platform (delete by platform, optionally customer-scoped) 5.03 — search_platform (scoped semantic search) 5.04 — get_platform_stats (collection + per-platform counts) 5.05 — Integration test coverage in tests/kb/test_m5_qdrant_integration.py Usage: from core.kb.qdrant_store import upsert_vectors, delete_platform, search_platform, get_platform_stats N)Optional) QdrantClient)DistanceFieldConditionFilter MatchValue PointStruct VectorParams)Chunk EmbeddedChunk QDRANT_URLz/https://qdrant-b3knu-u50607.vm.elestio.app:6333QDRANT_API_KEY@7b74e6621bd0e6650789f6662bca4cbf4143d3d1d710a0002b3b563973ca6876genesis_memories) hubspotghlxerotelnyxstripenotionairtablezapiermonday salesforcedz$12345678-1234-5678-1234-567812345678_qdrant_clientreturncFttttdatS)z&Get or create Qdrant client singleton.<)urlapi_keytimeout)rrr r-/mnt/e/genesis-system/core/kb/qdrant_store.py _get_clientr&Is%*nVXY r$chunk_idcHttjt|S)z>Convert a chunk_id string to a deterministic UUID5 hex string.)struuiduuid5_UUID5_NAMESPACE)r's r%_chunk_id_to_uuidr-Qs tzz*H5 66r$ecc |j}|j|j|j|j|j |j |j|jd|j|jd S)z5Build the Qdrant point payload from an EmbeddedChunk. PLATFORM_KB) platform customer_idtitle source_urltextheading_context chunk_indexembedding_modeltyper' total_chunks) chunkr1r2r3r4r5r6r7r8r'r:)r.r;s r%_build_payloadr<Vsj HHENN((&&  00((--NN**  r$embedded_chunks collectionc ^|syt}|Dcgc]@}tt|jj|j t |B}}d}tdt|tD]0}|||tz}|j||d|t|z }2|Scc}w)a Upsert embedded chunks into Qdrant. - Point IDs are deterministic UUID5 derived from chunk_id. - Payload includes all KB metadata fields. - Batch upsert in groups of _BATCH_SIZE for efficiency. Returns: Count of upserted points (0 for empty input). r)idvectorpayloadT)collection_namepointswait) r&r r-r;r'rAr<rangelen _BATCH_SIZEupsert)r=r>clientr.rDtotal_upsertedibatchs r%upsert_vectorsrNms  ]F"    !2!2399"2& FN 1c&k; /%q1{?+ jT J#e*$% !sAB*r1r2ctdt|g}|&|jtdt|t|S)z@Build a Qdrant filter for platform (and optionally customer_id).r1valuekeymatchr2must)rrappendr)r1r2 conditionss r%_build_platform_filterrYsL :Zh-GHJ }J[4Q R  z ""r$ct}t||}|j||d}|j}|dk(ry|j||d|S)z Delete all vectors for a platform (optionally scoped to a customer). Counts matching points before deletion to return the actual deleted count. Returns: Count of deleted points. TrC count_filterexactr)rCpoints_selectorrE)r&rYcountdelete)r1r2r>rJflt count_result pre_counts r%delete_platformrdsg]F ; 7C<< TX<YL""IA~ MM"  r$g333333? query_vectortop_kscore_thresholdc t}t||}|j|||||d}g} |jD]} | jxsi} | j t | jt| jd| jdd| jdd| jdd| jdd| jd dd | S) a Semantic search scoped to a specific platform's KB vectors. Optionally narrows scope to a specific customer_id for multi-tenant isolation. Returns: List of result dicts with keys: id, score, title, text, source_url, platform, heading_context T)rCquery query_filterlimitrh with_payloadr3r5r4r1r6)r@scorer3r5r4r1r6) r&rY query_pointsrDrBrWr)r@roundrpget) rfr1r2rgrhr>rJraresultshitspointrBs r%search_platformrws"]F ; 7C!!"' "G D  --%2 %((mu{{A. Wb1 FB/%kk,;#KK B7#*;;/@"#E     Kr$c t}|j|}|jxsd}i}tD]O}t t dt |g}|j||d}|j}|dkDsK|||<Q||d |d S) a Return vector counts per known platform and overall collection stats. Returns: { "total": int, "collection": str, "dimension": 3072, "platforms": {"hubspot": N, "ghl": N, ...} } rCrr1rPrRrUTr[ )totalr> dimension platforms)r&get_collection points_count_KNOWN_PLATFORMSrrrr_) r>rJinfor{platform_countsr1raresultr_s r%get_platform_statsrs]F   rJcs r%create_test_collectionr*s2+-A"#Hr$cD|xs t}|j|y)zDelete a test collection.ryN)r&delete_collectionrs r%drop_test_collectionr3s+-A 3r$)N)*__doc__osr*typingr qdrant_clientrqdrant_client.modelsrrrrr r core.kb.contractsr r getenvr r COLLECTIONrrHUUIDr,r__annotations__r&r)r-dictr<listintrNrYrdfloatrwrrrr#r$r%rs" &3 RYY5 F   499CD *.&-\777 }2!$-($$ $Z"& # ## #  #""& # P"&  ,u+,,#,  ,  ,  , $Z,f*4$3$$VsH\4JVZ4S4(<2H4TX4r$