ğiB(dZddlZddlZddlZddlZddlmcmZddl m Z m Z ddl m Z ddl m Z ddlmZddlZddlmZmZej(eZ d(ded ed ed e ej2d ef d ZddiZded eefdZded eefdZ d)deededed eded eef dZ d*dedededed e ef dZ deedeedeed eefdZ! d+d ed ed e"efd!Z#ejHd"Z%d#ed efd$Z&d%eed&e'eefd eefd'Z(y),at MODULE 1: URL Fetcher ====================== Fetches web pages, parses sitemaps, handles concurrency, deduplication, robots.txt, and optional Browserless fallback. Stories: 1.01 - HTTP Fetcher (aiohttp) 1.02 - Sitemap Parser (XML) 1.03 - Concurrent Page Fetcher (Semaphore + polite delay) 1.04 - Browserless Fallback Fetcher (Playwright stub) 1.05 - URL Filter and robots.txt 1.06 - Content Hash Deduplication 1.07 - Integration Tests (see tests/kb/test_m1_fetcher_integration.py) VERIFICATION_STAMP Story: 1.01 - 1.07 Verified By: parallel-builder Verified At: 2026-02-26T00:00:00Z Tests: 15/15 Coverage: ~95% N)datetimetimezonefnmatch)Optional)urlparse) FetchedPagePlatformConfigurltimeout user_agentsessionreturnc Ktjtjj d}d|i}|du} |rBt j d}t j||t j|} |j|4d{}|j} |jjdd } t|j} |jd d{} t|| | | | | cdddd{|r||j!d{SSS77F#t$rd } YQwxYw787#1d{7swYnxYwn#t"j$$rIt&j)d |t|d dd i| cY|r||j!d{7SSSt j*$rQ} t&j)d|| t|d dd i| cYd} ~ |r||j!d{7SSSd} ~ wwxYw |r||j!d{7yyy#|r||j!d{7wwwxYww)uFetch a single URL and return FetchedPage. Handles HTTP errors (4xx, 5xx) gracefully — returns FetchedPage with error status and empty html. Callers may pass an existing session for connection reuse; if none is supplied a temporary session is created. %Y-%m-%dT%H:%M:%SZz User-AgentNF)ssltotal) connectorheadersr z Content-Typereplaceerrorsr html status_code content_typer fetched_atzTimeout fetching %sizClient error fetching %s: %sr)rnowrutcstrftimeaiohttp TCPConnector ClientSession ClientTimeoutgetstatusrdicttext Exceptionr closeasyncio TimeoutErrorloggerwarning ClientError)r r r rrr _owns_sessionrresprr resp_headersrexcs (/mnt/e/genesis-system/core/kb/fetcher.py fetch_pager7-sphll+445IJJZ(GtOM0" ,,7I++#--G<G $ {{3'  4"kk #||//C #DLL1 !%)!<ED90D71D95E E$E E$I*-E .I*5E$7D99 EEEE E$ I*E E E E$#I$:H"II*6F97I*?H"'H8H"9I=I*HI*H""I&I*=I>I*I'I!I''I*smz+http://www.sitemaps.org/schemas/sitemap/0.9 xml_contentc|r|jsgS tj|}|j }d|vr|jddn|}g}|dk(r|jD]r}d|j vr|j jddn |j }|dk(sA|jxsdj}|sb|j|tn|jD]r}d|j vr|j jddn |j } | dk(sA|jxsdj}|sb|j|tt} g} |D])} | | vs| j| | j| +| S#tj$rtj dgcYSwxYw)zExtract URLs from sitemap XML. Handles: - Standard sitemaps () - Sitemap index files () - Malformed XML (returns empty list) u.Malformed sitemap XML — returning empty list} sitemapindexlocr) stripET fromstring ParseErrorr/r0tagsplititerr*appendsetadd) r9rootrClocalurls sitemap_el sitemap_localr>url_el url_localseendedupedus r6_parse_sitemap_xmlrSys k//1 }}[) ((C"%*CIIcN2 #ED ))+ %J=@JNN=RJNN005b9XbXfXfM%!,"335KK$  %iik %F58FJJ5F ((-b1FJJIE!{{(b//1KK$  %UDG  D= HHQK NN1  NE ==GH sF*G  G  sitemap_urlcKt|d{}|jdk7s |js#tj d||jgSt |j}|sgS t j|j}d|jvr|jjddn |j}|dk(}|rVg}t}|D]C}t|d{} | D])} | |vs|j| |j| +E|S|S7#t j$rd}YtwxYw7Vw)zFetch and parse XML sitemap, return list of URLs. Handles sitemap index files by recursively fetching child sitemaps. Returns deduplicated list of page URLs (not sitemap URLs). Nz&Failed to fetch sitemap %s (status %d)r;r<r=F)r7rrr/r0rSr@rArCrDrBrG fetch_sitemaprHrF) rTpageraw_urlsrIrJis_index all_page_urlsrP child_url child_pagesrRs r6rWrWs0 K( (D 3dii?dN^N^_ !$)),H  }}TYY'+.$((?s#B'^+#% ! ,I -i 88K  ,D=HHQK!((+ , ,  O? ) ==9sGED$AE.AD' !E+E, E:+E'D?<E>D??ErK concurrencydelay_ms batch_sizec tKg}|dz t|}td||D]K}||||z}tj|t j d|dz} t j | t j4d{d td tffd } |D cgc] } | |  } } ttj| d{} |j| dddd{|t z}td | D}tjd |dz||||ddl}t#d|d|d|dt| d |j$dNtjdt|||S7cc} w77#1d{7swYxYww)a2Fetch multiple URLs with bounded concurrency, polite delay, and batched progress. Uses asyncio.Semaphore to limit concurrent connections. Adds a polite delay between each request to avoid hammering servers. Processes URLs in batches of *batch_size* to limit peak memory and log progress. @@rF)rlimitr)rr Nr rcK4d{t|d{}dkDrtjd{|cdddd{S7O7;77 #1d{7swYyxYww)N)r rr)r7r-sleep)r rXdelay_s semaphorerr s r6 _fetch_onezfetch_pages.._fetch_onesn$  !+C'!RRD{%mmG444    R4    sfA7AA7A"A!A"AA" A7A A7A"A" A7"A4(A+ )A40A7c3@K|]}|jdk(sdyw)rVN)r).0ps r6 zfetch_pages..sBqQ]]c-ABsz=fetch_pages: batch %d-%d done (%d/%d total, %d ok this batch)rkz [fetch-batch] /z fetched (z ok)T)fileflushz+fetch_pages: fetched %d URLs (%d requested))lenranger- Semaphorer#r$r%r&strr listgatherextendsumr/infosysprintstderr)rKr^r_r r` all_resultsr batch_start batch_urlsrrir tasks batch_resultsdoneokr{rgrhrs ` @@@r6 fetch_pagesrs&(KG IEQz2  + j(@A %%k2 ((U+/J (())8  . . c k  1;;Z_;E; w~~u'=!=>M   } - . . S// BMB B K !OT4  tfAeWJrd!C F!?BF8F#!F8#F5 )F,*F5 1F8browserless_urlbrowserless_tokenwait_msc K ddlm}|4d{}|r|d|n|} |jj|d{}|jd{} tjtjjd} | j|dd{} tj|d z d{| j!d{} | r | j"nd } d }| r*| j%d{}|j'd d }t)|| | |i| | j+d{|j+d{cdddd{S7b76#t$r2}t j d||Yd}~dddd{7yd}~wwxYw7_77777~7h7Y#| j+d{7|j+d{7wxYw#1d{7swYyxYw#t,$rt j/dYyt$r!}t j d||Yd}~yd}~wwxYww)uFetch a URL using headless Chrome via Browserless CDP. Falls back gracefully if Playwright/Browserless is unavailable — returns None. Primary path is aiohttp fetcher; this is the fallback for SPAs. r)async_playwrightNz?token=z%Browserless connect failed for %s: %sri0u)r rbrVrz content-typeru=Playwright not installed — Browserless fallback unavailablez#Browserless fetch failed for %s: %s)playwright.async_apirchromiumconnect_over_cdpr+r/r0new_pagerr rr!r"gotor-rfcontentr( all_headersr'r r, ImportErrordebug)r rrrrpw ws_endpointbrowserr5rXrr3rrr headers_dicts r6fetch_page_browserlessrs'9#% & &L]_-W5F4GHcrK  " < <[ II !))++D &%\\(,,7@@AUV !YYsFY;;mmGf$4555!\\^+-1dkks ! )-)9)9);#;L#/#3#3NB#GL" +!-) jjl""mmo%%; & & &J FSQ  & & & ,<5+$<#%; &8jjl""mmo%%; & & &>  TU +G  G G "G#+G G&G 5HG H G !H% H1G 2H6I%7H:E== F8F3H! H,F/-H2I%3F88H>G G G G H H H H!G$ "H:G= ;HHH H HHI%HI"7I%9I"II%I""I%include_patternsexclude_patternscg}|D]`t}|jxsd|rtfd|D}|s8tfd|D}|rP|jb|S)zFilter URLs by include/exclude glob patterns. Patterns are matched against the URL path (not the full URL). If include_patterns is empty, all URLs are included by default. Exclude patterns take precedence over include patterns. roc3RK|]}t|xs t| ywNrrlpatpathr s r6rnzfilter_urls..]s/S!7WT3%77$'c3RK|]}t|xs t| ywrrrs r6rnzfilter_urls..es/  C  3s!3 3 r)rranyrF) rKrrresultparsedincludedexcludedrr s @@r6 filter_urlsrJsF#{{!c +H '     c-. Mbase_urlcKt|}|jd|jd}t|d{}|jdk7s |j s t St }g}|j|j jD]}|j}|r|jdr'd|vr,|jd\} } } | jj} | j} | dk(r;| jdD cgc] } | jj"}} | d k(stfd |D} | s| s|j| |S7Dcc} ww) zFetch and parse robots.txt, return set of disallowed paths. Handles: - User-agent specific rules (matched case-insensitively) - Wildcard user-agent (*) - Missing robots.txt (returns empty set) z://z /robots.txtNrV#:z user-agent,disallowc3\K|]#}|dk(xs|k(xsj|%yw)*N) startswith)rlaua_lowers r6rnz#check_robots_txt..s9SCAMCX-@-@-CCs),)rschemenetlocr7rrrGlower splitlinesr?r partitionrDrrH)rr r robots_urlrX disallowedcurrent_agentsraw_lineline directive_valuerappliesrs @r6check_robots_txtrps]h FMM?#fmm_K@JJ' 'D 3diiu 5J "N!HII((*&~~ts+  d? "nnS1 1eOO%++-    $9>S9IJAaggioo/JNJ * $'G5u%+&. ? (*Ks46FE=C&F %FFF%F(FFz\s+rctjd|j}tj|j ddj S)zSHA-256 hash of normalized HTML content. Normalization collapses all whitespace runs to a single space and strips leading/trailing whitespace, so minor formatting changes do not produce a different hash.  zutf-8rr)_WHITESPACE_REsubr?hashlibsha256encode hexdigest)r normalizeds r6compute_content_hashrsG ##C.446J >>*++GI+F G Q Q SSrpages known_hashescKg}|D]i}t|j}|j|j}||k7r|j |Jt j d|jk|Sw)uFilter out pages whose content hash matches known hashes. Returns only changed or new pages. Args: pages: List of freshly fetched FetchedPage objects. known_hashes: Mapping of url → sha256 hash from the PG store. zSkipping unchanged page: %s)rrr'r rFr/r)rrchangedrX current_hash previous_hashs r6filter_unchangedrsn"$GB+DII6 $((2 = ( NN4 LL6 A B NsA3A5)zGenesisBot/1.0N)rVr2)z/wss://browserless-genesis-u50607.vm.elestio.appri ) GenesisBot))__doc__r-rloggingrexml.etree.ElementTreeetree ElementTreer@rrrtypingr urllib.parserr#core.kb.contractsr r getLogger__name__r/ruintr%r7 _SITEMAP_NSrvrSrWrrrrGrcompilerrr)rrr6rs'. ""'!9   8 $&/3 @" @" @"@"g++ , @"  @"P 7 .C.DI.b%S%T#Y%\ 4 s)444 4  4  + 4zM 2 222 2 k 2r# s)#3i#3i# #Y #P#--- X-hF#TsTsT  sCx. +r