&i68dZddlZddlZddlZddlZddlZddlZddlZddlm Z ddl m Z ddl m Z mZddlZej dZdZej&j)dd Zd d iZe d Zd-d edede efdZd.d edede efdZd.d edede efdZd ede de e fdZd ede de e fdZd ede efdZ d/d ede e defdZ!e"dk(rejFejHde%ejLdkDrejLdndZ'e dZ(e(jSd d !e!e'e(Z*e+d"e+d#e*d$e+d%e%e*d&e+d'e%e*d(e+d)e*d*e+d+e*d&dd,yy)0u Genesis Clone Pipeline — Content Extraction Module =================================================== Extracts content from target websites using a layered fallback approach: Layer 1: Jina Reader API (free, fast, LLM-ready markdown) Layer 2: Crawl4AI (open-source, deep crawl, images/CSS/structure) Layer 3: Playwright screenshot (visual reference) Layer 4: Raw requests fallback (basic HTML) No SQLite. Output is file-based or passed in-memory. N)Path)Optional)urlparseurljoinzclone_pipeline.extractorzhttps://r.jina.ai/ JINA_API_KEYz User-AgentzoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36z3/mnt/e/genesis-system/scripts/clone_pipeline/outputurltimeoutreturnc8t|}ddd}tr dt|d< tjd|t j |||}|j |j}t|tr|j dij d xs$|j dxs|j d }t|tr2t|d kDr$tjd t|d |S|jr$t|jd kDr |jStjd t|ddy#tjj$rtjd|d|Yytjj $r9}tjd|j"j$d|Yd}~yd}~wt&$r"}tjd|Yd}~yd}~wwxYw)u Extract page content as clean Markdown via Jina Reader API. Prepends https://r.jina.ai/ to the target URL — no SDK required. Returns markdown string on success, None on failure. zapplication/jsonmarkdown)AcceptzX-Return-FormatzBearer Authorizationz[Jina] Fetching: headersr datacontentdz[Jina] Extracted z charsz&[Jina] Unexpected response structure: Nz[Jina] Timeout after zs for z [Jina] HTTP z for z[Jina] Failed: ) JINA_BASErloggerinforequestsgetraise_for_statusjson isinstancedictstrlentextwarning exceptionsTimeout HTTPErrorresponse status_code Exception)r r jina_urlrresprres 9/mnt/e/genesis-system/scripts/clone_pipeline/extractor.pyextract_with_jinar-2sSE"H$%G%,\N#;   'u-.||HgwG yy{ dD !$((3'88F#'88I&  '3'CL3,> /G ~VDE 99TYY#-99 ?D $3?PQR    & &.wivcUCD    ( (ajj&<&<%=U3%HI ,-s6C(E! /E!<$E!!8HH7/G++ H7HH max_pagesc vK ddlm}m}m}ddlm}|dd}|ddd}d ggidd }|| 4d {} | j || d {} | jr7| jxsd |d<| jxsi|d<d|d<| jrJ| jjdgD]+} | jdd } | s|dj| -| jrc| jjdg} | d dDcgc]5}|jdd jdr|jdd 7c}|d<tj!dt#|ddt#|dd|cd d d d {Sd d d d {tj%dy 77rcc}w767'#1d {7swY7xYw#t&$rtj%dYy t($r"}tj%d|Yd }~y d }~wwxYww)z$Internal async crawl using Crawl4AI.r)AsyncWebCrawler BrowserConfigCrawlerRunConfig)LLMExtractionStrategyTF)headlessverbose2)word_count_thresholdexclude_external_linksremove_overlay_elementsr)r imageslinksmetadata pages_crawled)configN)r r>r r<r=r:srcinternalhrefhttpr;z[Crawl4AI] OK: z chars, z imagesz%[Crawl4AI] Crawler returned no resultz3[Crawl4AI] Not installed. Run: pip install crawl4aiz[Crawl4AI] Failed: )crawl4air0r1r2crawl4ai.extraction_strategyr3arunsuccessr r<mediarappendr; startswithrrr r" ImportErrorr()r r.r0r1r2r3 browser_cfgrun_cfg result_datacrawlerresultimgr@rAlnkr+s r,_crawl4ai_asyncrThs58MMF#T5A "!##($(   #+6 # #'"<!ggeR0'188=> <<%||// B?H7?},037762.99&A+,K(  %c+j*A&B%C8;x012';#7 # # # # #: >?= #@",% # # # # #@ LM ,QC01sH9:G,GG,GG A;GAG:G;G G,GG,!H9" G,-G.G,H9G, GGG,G,G)G G)%G,,H6 H9 H6H1,H91H66H9cb tj}|jrdddl}|jj 5}|j tjt||}|jdcdddS|jt||S#1swYyxYw#t$rt}tjd| tjt||cYd}~S#t$r&}tjd|Yd}~Yd}~yd}~wwxYwd}~wwxYw)aZ Deep crawl a website using Crawl4AI. Returns dict with keys: - markdown: str (extracted text as markdown) - images: list[str] (image URLs) - links: list[str] (internal page links) - metadata: dict (title, description, etc.) - pages_crawled: int Returns None if Crawl4AI is not installed or fails. rN<r z[Crawl4AI] Event loop error: z"[Crawl4AI] Final fallback failed: )asyncioget_event_loop is_runningconcurrent.futuresfuturesThreadPoolExecutorsubmitrunrTrQrun_until_completer(rr")r r.loop concurrentpoolfuturer+e2s r,crawl_with_crawl4airfs%%' ??  %##668 1DKKi!@}}R}0  1 1 **?3 +JK K  1 1 6qc:; ;;sI>? ?  NN?tD E4  sfAB1sF.E!DE!E "DDDE EE 6E7E E(E <E=E EE 2 E!>E ?E!F.E!D,D=4D75D=:E <D==E E E E E  E! EE EE!F.E!!F+?F.F+ F&!F.&F++F. output_dirc |jdd tjt||S#t$rddl}|j j5}|jtjt||} |jdcdddcYS#t$r+}tjd|Yd}~dddYyd}~wwxYw#1swYYyxYwwxYw)z Take a full-page screenshot of a URL using Playwright. Saves to output_dir/screenshot_original.png Returns the path if successful, None otherwise. Tparentsexist_okrNrVrWz%[Playwright] Thread fallback failed: ) mkdirrXr_r RuntimeErrorr[r\r]r^rQr(rr")r rrbrcrdr+s r,screenshot_with_playwrightrsTD1 {{7ZHII  !    2 2 4 [[ 9#zJF }}R}0     !FqcJK        sQ4'C)+CB$ C)$ C-CC C)CCC% C)%C)c tjd|tj|td}|j |j S#t$r"}tjd|Yd}~yd}~wwxYw)za Basic HTML extraction via requests as last resort. Returns raw HTML string or None. z[Raw] Fetching HTML: rBrz[Raw] Failed: N) rrrrDEFAULT_HEADERSrr!r(r")r r*r+s r,extract_raw_htmlrsi  +C512||C"E yy s+,sAA A=A88A=c 8ddggidd|d}tjd|t|}|r]+> z\s+i@raw_htmlz[Extractor] Raw HTML fallback (z.[Extractor] ALL extraction methods failed for r~z [Extractor] Screenshot skipped: )rrr-r rfrrresubstriperrorrrr(r") r rrQ jina_content crawl_datarrr!r~r+s r,extract_contentr#s9*# F KK7u=>$S)LL)C/)z&,"# 23|3D2EWMN GH)- *..4Z =S9TWZ9Z!+J!7F: )~~h;F8 (nnWb9F7O!+ B!?F: *4F& ' KK:3vj?Q;R:SSZ[ \ KKO P#C(D!%vvvj#t4vvfc40668%)%4[z".8*+ =c$i[PQ McUST C8jIO,/,@() M6M C NN=aSA B B M Cs G-- H6HH__main__z%(levelname)s %(message)s)levelformatr?zhttps://example.comz8/mnt/e/genesis-system/scripts/clone_pipeline/output/testTrz Extraction result:z Method: rz Markdown length: r z Images found: r:z Screenshot: r~z First 500 chars of content: r)))N),__doc__ossysrtimeloggingrXhashlibpathlibrtypingr urllib.parserrr getLoggerrrenvironrrrSCREENSHOT_DIRrintr-rrTrfrrrr__name__ basicConfigINFOr argvr outputrrQprintr,rsB  *   5 6 ! zz~~nb1  )KL/3//hsm/l:s:s:8D>:zSS$L&C&d&xPT~&RCThtn8 # (3- &LL(4.LDLf zGgll3NOSXX*#((1+0EC L MF LLL- S& )F " Jv123 45 F:$6 78 9: S!123 45 N6"345 67 +F:,>t,D+E FGr