i\c SrSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSK r SSK J r J r SSK J r SSKJrJrJrJrJr \ "S5r\S- S- r\R,R/S\"\S - S - 55 \R2"\R4S S S 9 \R6"S5rSrSrSr\R@RCSS5r"Sr#S\S\\4Sjr$S\S\4Sjr%Sr&S8S\\S\'S\\\\44Sjjr(S9S\S\\SS4Sjjr)S\S\\\\44Sjr*\4S \S!\'S\\4S"jjr+S#\\S\\\,4S$jr-\4S \S%\'S\\,4S&jjr.S\S'\'S\'4S(jr/S:S)jr0S\S*\\S+\\\,S,\\\4S\'4 S-jr1S.r2S/\\\4S0\'SS4S1jr3S;S2\\\\4S3\4S4\4S\\\44S5jjr5S6r6\7S7:Xa\6"5 gg)r<r?s r! mark_ingestedrFsS   #   # C  KKM s A Ac SSKJn SSKJn U"5nSnSn/SQS4HenU(aURXS9nOAURU5nUH*nURXR/S9nURn O U(a OMg U(dgU"5n U RU5n /n UHn U R[U S 5(a U RO[U R!S S 55[U S 5(a U R"OU R!S S5[U S 5(a U R$OU R!S S5S .5 M UU U['U R)55U S.$![a [ R S5 gf=f![a GM]f=f![a Uc GM8GMf=f)z Fetch transcript using youtube-transcript-api. Returns dict with keys: video_id, transcript (str), language, word_count, segments (list of {text, start, duration}). Returns None if no transcript is available. r)YouTubeTranscriptApi) TextFormatterzPyoutube-transcript-api is not installed. Fix: pip install youtube-transcript-apiNen)rJzen-USzen-AUzen-GB) languagestextstartduration)rLrNrO)r$ transcriptlanguage word_countsegments)youtube_transcript_apirH!youtube_transcript_api.formattersrI ImportErrorloggererrorfetchlist language_code Exceptionformat_transcriptr2hasattrrLstrgetrNrOlensplit) r$rHrIytttranscript_datarQ lang_list availablet formatter full_textrSentrys r!fetch_transcriptrks?C  COH8>  "%))H)"J HHX. "A!*-))HHY)*Z#$?? #?* I++OX$+E7$;$;U[[7TUAV*1%*D*D%))T^`aJb  !)//+,  _   6  0%! !    sF E7-F.&F7 F.7FF F+&F.*F++F.. GGrL max_wordsc:UR5nU(dUR5(aU/$/$/n[S[U5U5H@nSR X$XA-5nUR5(dM/UR U5 MB U=(d USS/$)zn Split transcript text into chunks of approximately max_words words. Each chunk will be its own Qdrant point. r Ni )rbstriprangerar3r2)rLrlwordschunksichunks r! chunk_textrus JJLE v-2- F 1c%j) ,1=12 ;;== MM% -  "d5Dk]"r#textsc LSSKJn UR[S9 /nUH,nUR [ USS9nUR US5 M. U$![aHn[RSUS35 UVs/sHn[U[S 9PM Os snfsnsSnA$SnAff=f) z Generate 768-dim embeddings using Gemini text-embedding-004. Falls back to deterministic hash-based vectors if Gemini is unavailable. rN)api_keyretrieval_document)modelcontent task_type embeddingzGemini embedding failed (z%). Using hash-based fallback vectors.)dim) google.generativeai generativeai configurer embed_contentGEMINI_EMBEDDING_MODELr2r\rWwarning _hash_vector VECTOR_DIM)rvgenaivectorsrLresultergs r! embed_textsrs @+ /D((,.)F NN6+. /  @21#5Z[\9>?A QJ/??@s*AA B#B8BBB#B#r~cP[R"URS55R5nUS-[ U5-S-nX#-n[ R "SUS3USUS-5n[SU55S-nUS :XaS /U-$UVs/sHowU- PM sn$s snf) zIDeterministic pseudo-vector from SHA-512 hash. Last-resort fallback only.utf-8r>fNc3*# UH oU-v M g7fNr&).0vs r! _hash_vector..#s%f1ufsg?rg)hashlibsha512encodedigestrastructunpacksum)rLr~hrepeatexpandedvaluesnormrs r!rrst{{7+,335AAgQ1 $FzH ]]Qse1:x #'': ;F %f% % ,D qyus{$ %fHf %% %sB# chunk_idxcUSU3RS5n[R"U5R5n[ USSS5S-$)z"*,,|Q"?#+<< #=#G#G#I'6   >, &7 G yV \(=QR v;r#c^SSKJn SSKJn U"5nU"URUR S9$)z1Get configured Qdrant client from elestio_config.r) QdrantClient) QdrantConfig)urlrx) qdrant_clientrr+rrrx)rrcfgs r!get_qdrant_clientros$*+ .C CGGS[[ 99r#record chunks_countc[R"[R5R S5n[ SUS3- n[ R SSS9 SUSS[[R"553S USURS S 5URS S 5URS [US55URSS5URS5URSS5URSS5U[[R"[R5R5SSURSS 5/S.n[USSS9nUR[R "USS9S-5 SSS5 ["R%SUSS UR&35 g!,(df  N7=f)!zi Write a KG entity JSONL entry for this video to KNOWLEDGE_GRAPH/entities/youtube_intel_YYYY-MM-DD.jsonl z%Y-%m-%dyoutube_intel_z.jsonlT)parentsexist_ok yt_intel_r$ryoutube_video_intelrrMrrr/unknownpositionrQrJrRrrintel)rtyper$rrrr/rrQrR chunks_storedqdrant_collectionrtagsarencodingF) ensure_ascii NzKG entity written for z -> )rrrrstrftimeKG_ENTITIES_DIRmkdirrtimer`r'rropenwritejsondumpsrWrr)rrtoday entity_fileentityrs r!write_kg_entityr|s~ LL & / / ;E!nUG6$BBK$6&,-Qs499;/?.@A%:&GR( >26ZZ _VJ=O-PQ**Xy1JJz*JJz40jjq1%.||HLL1;;=GVZZ"%=>F" k3 1Q  66=> 2 LL)&*<)=T+BRBRASTU 2 1s (F44 G video_records use_qdrantdry_runc*[U5SSSSS/S.nSnU(aU(d[5n[U5 USLnU(aU(d [ 5n[USS9GHkupU RS 5n U (d[RS US 35 M9U RS S 5n U RSS5n [RSUS[U5SU SU SU RSS5SU SS3 5 Sn [U 5n U (d@[RSU S35 US==S- ss'USRU U SS.5 MUS ==S- ss'U S!nU S"n[RS#US$U35 U(aUSRU U S%UUS&.5 GMK[U S'5n[RS([U5S)35 SnU(aWU(aPU(aI[U5n0U EUUS*.En[XZUUU5nUS+==U- ss'[RS,US-35 0U EUUS*.En[!UU=(d [U5S/9 US0==S- ss'U(a[#X:U S:waU OSS29 USRU U S5UUUS6.5 U[U5:dGMU[$R&"S75 GMn U(aU(aUR)5 U$![a&n[R SUS35 SnSnAGNSnAff=f![a&n[RSU SU35 SnAGN@SnAff=f![a&n[R S.U SU35 SnAGNFSnAff=f![a&n[RS1U SU35 SnAGNFSnAff=f![a&n[RS3U S4U35 SnAGN_SnAff=f!U(aU(aUR)5 fff=f)8a Core ingestion loop. Args: video_records: List of dicts with at minimum 'video_id'. May also include title, channel_name, source, position, video_url. use_qdrant: Whether to embed and store in Qdrant. dry_run: If True, fetch transcripts but do not store anything. conn: Open PostgreSQL connection. If None, opens its own. Returns: Summary dict. r)totaltranscripts_fetched no_transcript qdrant_chunks kg_entitiesfailed video_resultsNzQdrant init failed: z. Will skip vector storage.Fr)rNr$zRecord z missing video_id, skipping.rrMr/r[/z ] Ingesting z (z pos=r?z): <z Transcript fetch error for z: z No transcript available for .rr)r$rstatusrrRrQz Transcript: z words, lang= dry_run_ok)r$rr rRrQrPz Chunked into z segments.)rQrRrz Qdrant: z chunks stored.z Qdrant upsert failed for )rrz KG entity write failed for )r/z Failed to mark z as ingested: success)r$rr rRrQrrg?)rarrr\rWrXr.rr`rrrkr2rurrrrFrsleepclose)rrrr:resultsrr owns_connrsrr$rr/rdrRrQrrchunks_upsertedrmeta kg_records r! ingest_videosrsR(]# GM' -/M $] 3  I "f"=:IAzz*-H+GHIJJw+EZZ)4F KKA3aM*+<zB85J!< =Ss N  #O P"28"<#!?zKL(A-((// ("-1  ) *a / *(6J&z2H KK. M(L M(// ("*", ( 1  =>F KK/#f+jA B Om P)&1G $,&0D '7%$'OO,?,KK*_,=_ MN P (",  8V3v;W &!+& T!$6YCV\`a O $ + +$#($) - 3}%% 3C;H  JJL Na  LL/s2MN OJ 4 P!>xj1#NOO P^!PLL#>xj1#!NOOP P!>xj1#NOO P!TNN%6xjqc#RSST"  JJL9sK1B,O0 L$C$O04AM=2N /O07N= (O06O01 L!;LL!$ M.M O0MO0 N!N<O0NO0 N:N5/O05N::O0= O-O("O0(O--O00"Pc [R"S[RS9nURSS9nUR SSSS9 UR S S S S S 9 UR SSSS9 UR SSS/SS9 UR S[ SSS9 UR SSSS9 UR SSSS9 UR 5n/nUR(ar[RS5 [5n[XBRURS 9nUR5 [RS#['U5S$35 GOUR((aUR(HOn[+U5nU(a"UR-US%S%S&[/U5S'.5 M7[R1S(U35 MQ [RS)['U5S*35 GOMUR2(Ga;[5UR25nUR75(d.[R!S+U35 ["R$"S"5 [9US,S-9n [:R<"U 5n SSS5 [?W [@5(aU nOs[?U [B5(a3S.U ;a-U S.H#nUR-US%S%S/[/U5S'.5 M% O+[R!S05 ["R$"S"5 [RS1['U5S2URDS335 U(d9[R1S45 [G[:RH"S5S5S5S6.55 g[KUURL(+URNS79n U RQ5V V s0sHupU S8:wdM X_M nn n [G[:RH"US9S:95 [RS;U S<S=U S>S?U S@SAU SBSC3 5 g![a9n[R!S!U35 ["R$"S"5 SnAGN SnAff=f!,(df  GN=fs sn n f)DNzGIngest YouTube transcripts into Genesis Qdrant 'research_reports' + KG.)r6formatter_classT)requiredz --pending store_truezIIngest all videos from youtube_intel table with transcript_ingested=FALSE)actionhelpz--urls+ URL_OR_IDz,Specific YouTube URLs or video IDs to ingest)nargsmetavarrz --from-jsonFILEzOJSON file with list of video records (output from youtube_watchlist_scraper.py))r!rz--source watch_history watch_laterz9Filter --pending by source (watch_history or watch_later))choicesrz--limitrz8Max videos to ingest when using --pending (default: 500))rdefaultrz --dry-runz:Fetch transcripts but do not store to Qdrant or PostgreSQLz --no-qdrantz.Skip Qdrant embedding (only write KG entities)z2Loading pending videos from youtube_intel table...)r/r0z Failed to load pending records: rzFound z pending videos.rMmanual)r$rrr/rz!Could not extract video ID from: z Prepared z videos from CLI arguments.zJSON file not found: rr video_ids json_importz>JSON file must be a list of records or have a 'video_ids' key.zLoaded z videos from r zNo videos to ingest.r)rrr)rrrr)indentzIngestion complete: rr rz transcripts, rz Qdrant chunks, rz KG entities.))argparseArgumentParserRawDescriptionHelpFormatteradd_mutually_exclusive_group add_argumentr parse_argspendingrWrr.rCr/r0rr\rXsysexitraurlsr"r2r'r from_jsonrexistsrrload isinstancerZr8rprintrr no_qdrantritems)parser source_groupargsrr:rrvid json_pathrr rkrsummarys r!mainrD9s  $ $] <<F 666EL X  ;   ^  -0 H    G   I   =    D+-M || HI $&D/[[ M JJL  fS/00@AB I"9-C$$ #$&&!0!5 &!B9+NO#  iM 233NOP ( !! LL0 < = HHQK )g .!99QNaPQ -. djj1QQRSTU#~~% G!( H13GtqtGH $**WQ '( KK w'<=>a@P?QQ_ ? # $$4W]5K4LM [y  LL;A3? @ HHQKK 2/ .@Is0+8PQ  Q/Q Q.QQ Q__main__)Nrr)rN)TFN)8__doc__r,rrloggingosrrr3rrrpathlibrtypingrrrr r GENESIS_ROOTrpathinsertr_ basicConfigINFO getLoggerrWrrCHUNK_MAX_TOKENSenvironr`rrr"r'r.rrCrFrkrufloatrrrrrrrboolrrD__name__r&r#r!rVs&P  '33 '( !22Z?3|f,/??@A ,, >    3 4 '   02[\4    9c9c9F69@3-@/2@ $sCx.@@#x}4@s@xS#X'?@N,< #S #S #S #(@tCy@T$u+%6@,(2 &s & &d5k &0000 Q)) I)$u+  ) 38n )  )X:VDcNV#V$VL RS#X'RRR  #s(^ Rr~B zFr#