/i%ddlmZmZddlmZmZmZmZddlm Z ddl m Z ddl m Z ddlZddlmZej"d Zd ed ed efd ZGddeZGddeZy))ABCabstractmethod)OptionalDictAnyTuple)MarkdownGenerationResult)CustomHTML2Text)RelevantContentFilterN)urljoinz+!?\[([^\]]+)\]\(([^)]+?)(?:\s+"([^"]*)")?\)baseurlreturnc|jdr|S|jdr|jdr|dd|zS||zSt||S)z"Fast URL joining for common cases.)http://https://mailto:z///N) startswithendswithr )rrs a/mnt/e/genesis-system/.venv/lib/python3.12/site-packages/crawl4ai/markdown_generation_strategy.py fast_urljoinrsS ~~>?  ~~c == 9s? "cz 4 ceZdZdZ ddeedeeeefde defdZ e dded ed eeeefdeed e d e f d Z y)MarkdownGenerationStrategyz7Abstract base class for markdown generation strategies.Ncontent_filteroptionsverbosecontent_sourcecD||_|xsi|_||_||_y)N)rrr r!)selfrrr r!s r__init__z#MarkdownGenerationStrategy.__init__s&-}"  ,r input_htmlbase_urlhtml2text_options citationsrc y)z/Generate markdown from the selected input HTML.N)r#r%r&r'rr(kwargss rgenerate_markdownz,MarkdownGenerationStrategy.generate_markdown)s r)NNF cleaned_html)NNT)__name__ __module__ __qualname____doc__rr rstrrboolr$rr r,r*rrrrsA;?,0, - !67 -$sCx.) - -  -6::>      $DcN3  !!67     "    rrceZdZdZ ddeedeeeefdeffd Z ddedede eeffd Z dd eded eeeefdeeeefdeed e de fd ZxZS)DefaultMarkdownGeneratora Default implementation of markdown generation strategy. How it works: 1. Generate raw markdown from cleaned HTML. 2. Convert links to citations. 3. Generate fit markdown if content filter is provided. 4. Return MarkdownGenerationResult. Args: content_filter (Optional[RelevantContentFilter]): Content filter for generating fit markdown. options (Optional[Dict[str, Any]]): Additional options for markdown generation. Defaults to None. content_source (str): Source of content to generate markdown from. Options: "cleaned_html", "raw_html", "fit_html". Defaults to "cleaned_html". Returns: MarkdownGenerationResult: Result containing raw markdown, fit markdown, fit HTML, and references markdown. rrr!c,t|||d|y)NF)r r!)superr$)r#rrr! __class__s rr$z!DefaultMarkdownGenerator.__init__Js %P^_rmarkdownr&rc@i}i}g}d}d}tj|D]}|j|||j|j \} } } |r)| j ds| |vrt || || <|| } | |vrOg} | r| j| | r| | k7r| j| || rddj| zndf|| <|dz }|| d} |j|jdj ds| d| d nd | d| d |j}|j||d dj|}d g}|jdt|jdD|dj|fS)a Convert links in markdown to citations. How it works: 1. Find all links in the markdown. 2. Convert links to citations. 3. Return converted markdown and references markdown. Note: This function uses a regex pattern to find links in markdown. Args: markdown (str): Markdown text. base_url (str): Base URL for URL joins. Returns: Tuple[str, str]: Converted markdown and references markdown. rr )rrrz: z - r.!⟨u⟩z![u⟩]Nz ## References c3>K|]\}\}}d|d||dyw)r=u⟩  Nr*).0rnumdescs r zFDefaultMarkdownGenerator.convert_links_to_citations..s3  [c4#d3%vR ( c|ddS)Nr rr*)xs rzEDefaultMarkdownGenerator.convert_links_to_citations..s1Q4PQ7r)key) LINK_PATTERNfinditerappendstartgroupsrrjoingroupendextendsorteditems)r#r:r&link_map url_cachepartslast_endcountermatchtextrtitlerBrAconverted_text referencess rconvert_links_to_citationsz3DefaultMarkdownGenerator.convert_links_to_citationsRs* !**84 #E LL(U[[]; <$||~ D#u/Q Ri'%1(C%@IcNn("KK&DEMKK%!(T$D1A*Ar R 1 3-"C LL{{1~005&C5$$s3%t,  yy{H3 #6  Xhi()..  $*8>>+;AR$S  rwwz222rr%r'r(c  t|}ddddddddd} |r| j|n;|r| j|n'|jr| j|j|jdi| |sd}nt |t s t |} |j |} | jd d } | } d} |r |j| |\} } d}d}|s |jrK |xs |j}|j|}d jd |D}|j |}t| xsd| xsd| xsd|xsd|xsdS#t$r} dt | } Yd} ~ d} ~ wwxYw#t$r} | } d t | } Yd} ~ d} ~ wwxYw#t$r} dt | }d}Yd} ~ d} ~ wwxYw#t$r(} dt | }t||dddcYd} ~ Sd} ~ wwxYw)a Generate markdown with citations from the provided input HTML. How it works: 1. Generate raw markdown from the input HTML. 2. Convert links to citations. 3. Generate fit markdown if content filter is provided. 4. Return MarkdownGenerationResult. Args: input_html (str): The HTML content to process (selected based on content_source). base_url (str): Base URL for URL joins. html2text_options (Optional[Dict[str, Any]]): HTML2Text options. options (Optional[Dict[str, Any]]): Additional options for markdown generation. content_filter (Optional[RelevantContentFilter]): Content filter for generating fit markdown. citations (bool): Whether to generate citations. Returns: MarkdownGenerationResult: Result containing raw markdown, fit markdown, fit HTML, and references markdown. )baseurlrFT) body_widthignore_emphasis ignore_links ignore_images protect_linkssingle_line_break mark_code escape_snobr.z#Error converting HTML to markdown: Nz ```z```zError generating citations: r?c3>K|]}dj|yw)z
{}
N)format)r@ss rrCz=DefaultMarkdownGenerator.generate_markdown..s .67..q1.rDzError generating fit markdown: ) raw_markdownmarkdown_with_citationsreferences_markdown fit_markdownfit_htmlzError in markdown generation: r*)r updater update_params isinstancer3handle Exceptionreplacer^rfilter_contentrNr )r#r%r&r'rrr(r+hdefault_optionsrlermrnro filtered_html error_msgs rr,z*DefaultMarkdownGenerator.generate_markdownsIG GG G<G71G<7G<)NNr-)r.)r.NNNT)r/r0r1r2rr rr3rr$rr^r4r r, __classcell__)r9s@rr6r67s(;?,0, ` !67`$sCx.)` `.0@3@3'*@3 sCx@3J6:,0:>ppp$DcN3 p $sCx.) p !!67 pp "prr6)abcrrtypingrrrrmodelsr html2textr content_filter_strategyr re urllib.parser compilerIr3rrr6r*rrrse#--,&: rzzHI  s     :M9Mr