diy$dZddlZddlZddlZddlZddlmZddlmZedZ e dz dz Z e jdd e dz e d z e d z e d z e d z e dz e dz e dz e dz dz e dz dz e dz dz e dz dz e dz e dz e dz e dz dz e dz dz dz e dz e dz dz e d z gZ hd!Z hd"Zd3d#ed$efd%Zd&ed$efd'Zd(Zd)Zd*ed$efd+Zd4d,ed-ed$efd.Zd/Zd0Zd1Zed2k(reyy)5a/ LightRAG Corpus Intelligence Layer ==================================== Indexes all Genesis deep_think files, KG axioms, and strategic documents. Enables global synthesis queries over the full Genesis knowledge corpus. Usage: # Test index (5 files): python3 core/lightrag_corpus.py --test # Full index (runs everything): python3 core/lightrag_corpus.py --full # Query: python3 core/lightrag_corpus.py --query "What are the dominant strategic patterns?" # List indexable files: python3 core/lightrag_corpus.py --list-files N)Path)partialz/mnt/e/genesis-systemKNOWLEDGE_GRAPHlightrag_indexT)parentsexist_okplans deep_thinkdeep_think_resultscoreSunaivadocs ConversationsRECEPTIONISTAIstrategyresearch verticalsswarm_generatedTRADIESpromptsn8nhive swarm_resultsdataagentsforksz mcp-serversloopzMission Briefs>.git.venvdistvenvbuild __pycache__ node_modulesr>.md.py.txt.json max_filesreturnclgtdtffd }tD]X}|jstD]<}|j d|D]#}|||st |k\scccS>ZtjdD]}|||st |k\scSS)zMGather all indexable corpus files from CORPUS_DIRS plus root-level .md files.fpcdt|vrytd|jDrytd|jDryt|j}|vryj |j |y)z0Add a file to the list if it passes all filters.rNc3>K|]}|jdyw).N) startswith.0parts -/mnt/e/genesis-system/core/lightrag_corpus.py z1get_corpus_files.._add..Ms9ts#9sc3,K|] }|tvywN) SKIP_DIRSr1s r4r5z1get_corpus_files.._add..Os6Tty 6s)stranypartsresolveaddappend)r,abs_pathfilesseens r4_addzget_corpus_files.._addIso s2w &  99 9  6RXX6 6 rzz|$ t    R*z*.md) setr CORPUS_DIRSexistsINCLUDE_EXTENSIONSrgloblen GENESIS_ROOTglob)r)rBdextr,r@rAs @@r4get_corpus_filesrODs E 5D  !xxz % !Cgg#i( !RUy!8 L ! !!' R Uy0L LrCr,cH |jddS#t$rYywxYw)z6Read a file safely, returning empty string on failure.zutf-8ignore)encodingerrors) read_text Exception)r,s r4read_file_saferWjs,||WX|>> s  !!c ddlm}m}ddlm m tjjdxs@tjjd xstjjd s td d fd }dd l m} fd}|dd|}tt ||}|fS#t $r8}td|tdtjdYd}~d}~wwxYw)zDBuild and return a LightRAG instance pointed at our index directory.r)LightRAG QueryParam)gemini_complete_if_cache gemini_embedz$ERROR: lightrag-hku not importable: zFInstall with: /mnt/e/genesis-system/.venv/bin/pip install lightrag-hkuNGEMINI_API_KEY_NEWGEMINI_API_KEYGOOGLE_API_KEYzKWARNING: No Gemini API key found. Set GEMINI_API_KEY_NEW or GEMINI_API_KEY.c DKdd|||xsgd|d{S7w)Nzgemini-2.0-flash)modelprompt system_prompthistory_messagesapi_keyrg)rcrdrekwargsrfr[s r4llm_model_funcz!build_rag..llm_model_funcsA- $'-3       s   ) EmbeddingFunccFKj|d{S7w)N)rf)func)textsrfr\s r4embedding_func_with_keyz*build_rag..embedding_func_with_keys#!&&ug&>>>>s !!ii) embedding_dimmax_token_sizerl) working_dirriembedding_func)NN)lightragrYrZlightrag.llm.geminir[r\ ImportErrorprintsysexitosenvirongetlightrag.utilsrjr9LIGHTRAG_INDEX_DIR) rYrZerirjrn embed_funcragrfr[r\s @@@r4 build_ragrrs1N +, , ::>>* + , ::>>* +  [\ -? $J *+%! C  ?U  4QC89 VW  sB== C>.C99C>cBK|jd{|S7w)z2Initialize LightRAG storages (required for v1.4+).N)initialize_storages)rs r4init_ragrs" ! ! ### J$s r@c K|jd{ddgd}|D]}t|}t|jdkr|dxxdz cc<8 d|d|}|j |d{|d xxdz cc<t d |j d t|d d |S77=#t$rH}|dj|j d|t d|j d|Yd}~d}~wwxYww)z0Insert files into LightRAG index. Returns stats.Nr)indexedskippedrS2rr]z[FILE: z] rz Indexed: z (,z chars)rSz: z ERROR indexing ) rrWrJstripainsertrvnamerVr>)r@rstatsr,contentdocr~s r4 index_filesrs!  ! ! ###a2 6E 6 $ w}} " $ )  !   6B4uWI.C++c" " " )  !  Ky3w<*:'B C 6 L$ # 6 (O " "bggYb#4 5 %bggYb4 5 5 6sLD B4AD B88B697B80D 6B88 D >D?D D  D querymodecKt\}}|jd{|j|||d{}|S7(7w)a Run a synthesis query over the indexed corpus. Modes: global - Synthesizes across entire corpus (best for patterns/strategy) local - Focuses on most relevant entities hybrid - Combines both naive - Simple similarity search Nrparam)rraquery)rrrrZresults r4 query_corpusrsM kOC ! ! ###::e:4+@:A AF M$ As!!AA !AAAAcKtdtd}tdt|d|D]}td|t\}}t ||d{}tdt j |d |d d kDrItd td |jd|dd{}td||S|S7z7#t$r}td|Yd}~|Sd}~wwxYww)z0Index first 5 corpus files and run a test query.z%=== LightRAG Test Index (5 files) ===)r)z Selected z files:z - Nz Index stats: indentrrz === Test Query ===z2Query: What is Genesis System and what does it do?z+What is Genesis System and what does it do?naiverrzResult: z Query error: ) rvrOrJrrjsondumpsrrV)r@frrZrrr~s r4run_test_indexrs 12 q )E Ic%j\ )*  QCj kOCeS) )E ODJJuQ78 9: Y! $% BC '::= g.&F IfX& ' L5L *  ' M!% & & L 'sOA!D#C$AD*C"C C"D C"" D+C?9D?DDcKtdt}tdt|t\}}t ||d{}t||d|d|dt t d}t dz }|jtj|d td |td tj|d |S7w) zIndex all corpus files.z"=== LightRAG Full Corpus Index ===zTotal files to index: NrrrS) total_filesrrrS index_dirzindex_manifest.jsonrrz Manifest saved: z Final stats: ) rvrOrJrrr9r} write_textrr)r@rrZrmanifest manifest_paths r4run_full_indexrs ./  E "3u:, /0kOCeS) )E5z##/+, H')>>MTZZ;< }o ./ M$**U156 78 L *sA C C BCc~ddl}|jd}|jddd|jddd |jd td |jd tdgdd|jddd|j }|j r7t }tdt||D]}td|y|jrtjty|jrtjty|jrttjt!|j|j"}td|j"j%d|jdt|y|j'y)Nrz"LightRAG Corpus Intelligence Layer) descriptionz--test store_truez Index 5 files and run test query)actionhelpz--fullzFull corpus indexz--queryzRun a synthesis query)typerz--modeglobal)rlocalhybridrzQuery mode (default: global))rdefaultchoicesrz --list-fileszList all corpus fileszTotal corpus files: z rz [z QUERY]:  )argparseArgumentParser add_argumentr9 parse_args list_filesrOrvrJtestasynciorunrfullrrrrupper print_help)rparserargsr@rrs r4mainrsl  $ $1U $ VF <^_ ? f rC__main__r7)r)__doc__ryrwrrpathlibr functoolsrrKr}mkdirrFr8rHintlistrOr9rWrrdictrrrrr__name__rgrCr4rs( +, !$558HH5$$7<''696?"##j0##j0##k1##&779956O+6H$w.= 6H$##) 0  5##t#Lt/d T4* c  C  6.> zFrC