i'pdZddlmZmZmZmZddlZddlZddlZ ddl m Z ddl m Z ddlmZGddZy) a} Vector Store - Structured Multi-View Indexing Implementation (Section 3.2) Paper Reference: Section 3.2 - Structured Indexing Implements the three structured indexing dimensions: - Semantic Layer: Dense vectors v_k in R^d (embedding-based similarity) - Lexical Layer: Full-text search with Tantivy FTS - Symbolic Layer: Metadata R_k = {(key, val)} (structured filtering via SQL) )ListOptionalDictAnyN) MemoryEntry)EmbeddingModel) get_configc^eZdZdZ ddeedeedeedeeeeffdZ dd Z dd Z d e eeefde e fd Zd e e ddfdZddedede e fdZ dde edede e fdZ d dee edeedeedee edeede e f dZde e fdZddZddZy)! VectorStorea Structured Multi-View Indexing - Storage and retrieval for Atomic Entries Paper Reference: Section 3.2 - Structured Indexing Implements M(m_k) with three structured layers: 1. Semantic Layer: Dense embedding vectors for conceptual similarity 2. Lexical Layer: Full-text search via Tantivy FTS index 3. Symbolic Layer: SQL-based metadata filtering with DataFusion Ndb_pathembedding_model table_namestorage_optionsct}|xs |j|_|xs t|_|xs |j |_d|_d|_|jjd|_ |jr'tj|j||_ nEtj|jdtj|j|_ |j!y)NF)zgs://zs3://zaz://)rT)exist_ok)r lancedb_pathr rr memory_table_namertable_fts_initialized startswith_is_cloud_storagelancedbconnectdbosmakedirs _init_table)selfr r rrconfigs [/mnt/e/genesis-system/.venv/lib/python3.12/site-packages/simplemem/database/vector_store.py__init__zVectorStore.__init__ s5&"5"5 .B.2B$@(@(@ %"&!8!89T!U  ! !oodllOTDG KK t 4oodll3DG returnctjtjdtjtjdtjtjdtjtjtjdtjtjdtjtjdtjtjtjdtjtjtjdtjtjd tjtj |j jg }|j|jjvrE|jj|j| |_ td |jy |jj|j|_ td |jy )z&Initialize table schema and FTS index.entry_idlossless_restatementkeywords timestamplocationpersonsentitiestopicvector)schemazCreated new table: zOpened existing table: N)par.fieldstringlist_float32r dimensionrr table_names create_tablerprint open_table)rr.s r rzVectorStore._init_table:s}RYY[1/=RXXbiik%:;biik2RYY[1BHHRYY[$9:RXXbiik%:;"))+.bhhrzz|T5I5I5S5ST   ??$''"5"5"7 7--doof-MDJ ''89 :++DOOr"c6|jry |jr*|jjdddt dn*|jjddddt d d|_y#t $r}t d |Yd}~yd}~wwxYw) zAInitialize Full-Text Search index on lossless_restatement column.Nr&FT) use_tantivyreplacez1FTS index created (native mode for cloud storage)en_stem)r:tokenizer_namer;z FTS index created (Tantivy mode)zFTS index creation skipped: )rrrcreate_fts_indexr7 Exception)res r _init_fts_indexzVectorStore._init_fts_indexSs   6%% ++* % , IJ ++* $#, , 89$(D ! 6 04 5 5 6sA'A77 BBBresultscg}|D]} |jt|d|dt|jdxsg|jdxsd|jdxsdt|jdxsgt|jdxsg|jd xsd |S#t$r}t d |Yd}~d}~wwxYw) z/Convert LanceDB results to MemoryEntry objects.r%r&r'r(Nr)r*r+r,)r%r&r'r(r)r*r+r,z!Warning: Failed to parse result: )appendrlistgetr?r7)rrBentriesrr@s r _results_to_entrieszVectorStore._results_to_entriesns A !":-./E-F!%aeeJ&7&=2!>"#%% "4"<!"z!2!:d $QUU9%5%; eeGn4   " 9!=> sB5C C# CC#rGc R|sy|Dcgc]}|j}}|jj|}g}t||D]\}}|j |j |j|j |jxsd|jxsd|j|j|jxsd|jd |jj|tdt!|d|j"s|j%yycc}w)zBatch add memory entries.N) r%r&r'r(r)r*r+r,r-zAdded z memory entries)r&r encode_documentsziprDr%r'r(r)r*r+r,tolistraddr7lenrrA)rrGentry restatementsvectorsdatar-s r add_entrieszVectorStore.add_entriess @GHu22H H&&77 E '2 ME6 KK %,1,F,F %!&!6B % 4"$}} %"[[.B$mmo    t s7|nO45$$  "%/IsD$querytop_kcv |jjdk(rgS|jj|d}|jj |j j |j}|j|S#t$r}td|gcYd}~Sd}~wwxYw)z Semantic Layer Search - Dense vector similarity. Paper Reference: Section 3.1 Retrieves based on v_k = E_dense(S_k) where S_k is the lossless restatement. rT)is_queryzError during semantic search: N) r count_rowsr encode_singlesearchrNlimitto_listrIr?r7)rrVrW query_vectorrBr@s r semantic_searchzVectorStore.semantic_searchs zz$$&!+ //==ed=SLjj'' (;(;(=>DDUKSSUG++G4 4  21#6 7I s#BA3B B8B3-B83B8r'cF |r|jjdk(rgSdj|}|jj|j |j }|j |S#t$r}td|gcYd}~Sd}~wwxYw)z Lexical Layer Search - Full-text search via Tantivy FTS. Paper Reference: Section 3.1 Retrieves based on BM25 text matching using LanceDB native FTS. r zError during keyword search: N) rrZjoinr\r]r^rIr?r7)rr'rWrVrBr@s r keyword_searchzVectorStore.keyword_searchs tzz446!; HHX&Ejj''.44U;CCEG++G4 4  1!5 6I s# A=AA== B BB B r*timestamp_ranger)r+c  |jjdk(rgSt||||gsgSg}|r8dj|Dcgc]}d|d c}}|j d|d|r'|j dd} |j d| d|r8dj|D cgc]} d| d c} }|j d |d|r|\} } |j d | d | dd j|} |jj j| d }|r|j|}|j}|j|Scc}wcc} w#t$r} td| gcYd} ~ Sd} ~ wwxYw)a Symbolic Layer Search - SQL-based metadata filtering. Paper Reference: Section 3.1 Retrieves based on R_k = {(key, val)} for structured constraints. Uses DataFusion SQL expressions with array_has_any for list columns. rz, 'z"array_has_any(persons, make_array(z))z''zlocation LIKE '%z%'z#array_has_any(entities, make_array(ztimestamp >= 'z' AND timestamp <= 'z AND T) prefilterz Error during structured search: N) rrZanyrcrDr;r\wherer]r^rIr?r7)rr*rer)r+rW conditionspvalues safe_locationr@ start_timeend_time where_clauserVrBs r structured_searchzVectorStore.structured_searchs& zz$$&!+ (HEF Jg#>as!H#>?!!$Fvhb"QR ( 0 0d ; !!$4]O2"FGh#?as!H#?@!!$Gxr"RS'6$ H!!$ZL0DXJaP#<< 3LJJ%%'--ld-KE E*mmoG++G4 41$?$@$  4QC8 9I sGE*E*E* E AE*( E%5B*E* E** F 3FF F ct|jjj}|j|S)zGet all memory entries.)rto_arrow to_pylistrI)rrBs r get_all_entrieszVectorStore.get_all_entriess.**%%'113''00r"cN|jjtdy)zBOptimize table after bulk insertions for better query performance.zTable optimizedN)roptimizer7rs r rxzVectorStore.optimizes   r"c|jj|jd|_|j t dy)z&Clear all data and reinitialize table.FzDatabase clearedN)r drop_tablerrrr7rys r clearzVectorStore.clear s5 4??+ %   !r")NNNN)r#N)))NNNNN)__name__ __module__ __qualname____doc__rstrrrrr!rrArrrIrUintr`rdtuplerrrvrxr|r"r r r sz"&48$(48 #".1SM  "$sCx.1 4?2664S#X+?DDU,#4 #4##>ST+=N(12S *- k 0(,+/"&(,# 5$s)$5"%53- 5 49% 5 } 5 k 5n1k!21 ! "r"r )rtypingrrrrrrpyarrowr/simplemem.models.memory_entryrsimplemem.utils.embeddingrsimplemem.configr r rr"r rs0-, 54'}"}"r"