/idZddlmZmZddlmZmZmZmZm Z m Z ddl m Z ddl Z ddlZddlmZmZddlmZmZddlZdd lmZmZddlZddlZGd d eZGd d eZGddeZGddeZy)z Table extraction strategies for Crawl4AI. This module provides various strategies for detecting and extracting tables from HTML content. The strategy pattern allows for flexible table extraction methods while maintaining a consistent interface. )ABCabstractmethod)DictListOptionalAnyUnionTuple)etreeN) LLMConfigcreate_llm_config)perform_completion_with_backoff sanitize_html)ThreadPoolExecutor as_completedcjeZdZdZdZedejdee e e ffdZ d de de de fd Z y ) TableExtractionStrategyz Abstract base class for all table extraction strategies. This class defines the interface that all table extraction strategies must implement. It provides a consistent way to detect and extract tables from HTML content. c `|jdd|_|jdd|_y)z Initialize the table extraction strategy. Args: **kwargs: Additional keyword arguments for specific strategies verboseFloggerN)getrr)selfkwargss U/mnt/e/genesis-system/.venv/lib/python3.12/site-packages/crawl4ai/table_extraction.py__init__z TableExtractionStrategy.__init__s(zz)U3 jj40 elementreturnc y)ad Extract tables from the given HTML element. Args: element: The HTML element (typically the body or a container element) **kwargs: Additional parameters for extraction Returns: List of dictionaries containing table data, each with: - headers: List of column headers - rows: List of row data (each row is a list) - caption: Table caption if present - summary: Table summary attribute if present - metadata: Additional metadata about the table Nrrrs rextract_tablesz&TableExtractionStrategy.extract_tables's" rlevelmessagetagc h|jr&t|j|d}|r |d||d|yyy)z#Helper method to safely use logger.N)r%r&r!)rgetattr)rr$r%r&r log_methods r_logzTableExtractionStrategy._log:s: ;; eT:J>7>v> rN)TABLE)__name__ __module__ __qualname____doc__rrr Elementrrstrrr#r*r!rrrrsZ1 emm $tCQTH~BV  $?#??#?rrceZdZdZfdZdej deee e ffdZ dej de fdZ dej dee e ffdZxZS) DefaultTableExtractiona^ Default table extraction strategy that implements the current Crawl4AI table extraction logic. This strategy uses a scoring system to identify data tables (vs layout tables) and extracts structured data including headers, rows, captions, and summaries. It handles colspan and rowspan attributes to preserve table structure. c t|di||jdd|_|jdd|_|jdd|_y)a Initialize the default table extraction strategy. Args: table_score_threshold (int): Minimum score for a table to be considered a data table (default: 7) min_rows (int): Minimum number of rows for a valid table (default: 0) min_cols (int): Minimum number of columns for a valid table (default: 0) **kwargs: Additional parameters passed to parent class table_score_thresholdmin_rowsrmin_colsNr!)superrrr5r7r8)rr __class__s rrzDefaultTableExtraction.__init__KsM "6"%+ZZ0G%K" :q1  :q1 rrrc g}|jd|j}|jd}|D]}|j||s |j |}|j dkDr)t |jdg|j kra|jdkDrbt |jdgxs5|jdr"td|jdgDnd}||jkr|j||S#t$r*} |jdd t| d Yd } ~ d } ~ wwxYw) a> Extract all data tables from the HTML element. Args: element: The HTML element to search for tables **kwargs: Additional parameters (can override instance settings) Returns: List of dictionaries containing extracted table data r5.//table)r5rrowsheadersc32K|]}t|ywNlen.0rows r z8DefaultTableExtraction.extract_tables..xsOSCOerrorzError extracting table data: TABLE_EXTRACTN) rr5xpath is_data_tableextract_table_datar7rBr8maxappend Exceptionr*r1) rrr tables_datascore_thresholdtablestable table_data col_countes rr#z%DefaultTableExtraction.extract_tablesZsA !**%X>XYz* E!!%!O!%!8!8!?J}}q(S1K-Lt}}-\ }}q($' y"(E$F%S]SaSabhSiCOJNN624NOOop"%t}}4$&&z2! * !IIg)Fs1vh'OQ`as& ADA0DD E "EE rSc d}t|jddkD}t|jddkD}|r|dz }|r|dz }t|jd}|dkDr|dz }|s|jdr|dz }t|jddkDr|d z}|jd d j}|d vr|d z}|jd }|sy|D cgc]} t| jd} } | rAt | t| z t fd| Dt| z } | dkr|dz }|jdr|dz }|jdr|dz }t d|D} t d|j D} | | dzz }|dkDr|d z }n |dkDr|dz }t d|j D}||dzz }| r/t|dk\r!t | t| z dk\r|dz }|jd|j}||k\Scc} w)a[ Determine if a table is a data table (vs. layout table) using a scoring system. Args: table: The table element to evaluate **kwargs: Additional parameters (e.g., table_score_threshold) Returns: True if the table scores above the threshold, False otherwise r.//thead.//tbodyr .//th .//tr[1]/thr<role>none presentation.//trF .//td|.//thc3.K|] }|z dzyw)rZNr!)rDcavg_colss rrFz7DefaultTableExtraction.is_data_table..sAAL1,Asz .//captionsummaryc3K|]P}|jdD]:}tdj|jj <Ryw)rcr_N)rJrBjoinitertextstrip)rDrEcells rrFz7DefaultTableExtraction.is_data_table..sS  -0   (..0 1 1 sAAc3 K|]}dyw)r Nr!)rD_s rrFz7DefaultTableExtraction.is_data_table..s c3DK|]}|jdsdyw)zdata-r N) startswith)rDattrs rrFz7DefaultTableExtraction.is_data_table..sPttw7OPs  g?r5)rBrJrlowersumiterdescendantsattribr5)rrSrscore has_thead has_tbodyth_countr^r=rE col_countsvariance total_text total_tags text_ratio data_attrs thresholdrfs @rrKz$DefaultTableExtraction.is_data_tablesa J/014  J/014  QJE  QJEu{{7+, a< QJEEKK 6  u{{:& '! + QJEyy$**, + + QJE{{7#?CDc#))M23D D :Z8HAjAAC OSH!|  ;;| $ QJE 99Y  QJE   .s)SC)rGNzColumn r\) row_count column_count has_headers has_caption has_summaryidclassr>r=captionrgmetadata) rJrkr text_contentintextendrNrBrMrangebool)rrSrrgr> thead_rows header_cellsrltextr first_rowr=rErow_data max_columns aligned_rowsalignedirs rrLz)DefaultTableExtraction.extract_table_datas ++12(/'!*""$R))Ir*002[[/ %a=..w7L$ 1((*002dhhy!45v/0 1  J/I%aL..}=5D,,.446D!$((9a"89GNND6G#345 ;;<= &CH '* 2((*002dhhy!45 01 2 H% &'.c'l-1C)D) )q   )C,;'2$+C2H*IIG    ( ) ;?05k0BC11CGC\*' +OtEKK 4N/O==   99T?"YYt_HTN 99W  % ' 2HW     #Ds=K()r,r-r.r/rr r0rrr1rr#rrKrL __classcell__r:s@rr3r3Bsp 2(emm($tCQTH~BV(TO"5==O"tO"bS  S $sCx.S rr3cDeZdZdZdej deeee ffdZ y)NoTableExtractionz A strategy that does not extract any tables. This can be used to explicitly disable table extraction when needed. rrc gS)z Return an empty list (no tables extracted). Args: element: The HTML element (ignored) **kwargs: Additional parameters (ignored) Returns: Empty list r!r"s rr#z NoTableExtraction.extract_tables2s  rN) r,r-r.r/r r0rrr1rr#r!rrrr+s, emm $tCQTH~BV rrcneZdZdZdZ d(deedeedede deded ed e ffd Z d e jd e eeeffdZded efdZded e fdZded ee e je e je e je ffdZded ee ee ffdZdede edeed efdZde eded e efdZd)dededede d eeeff d Zd!e eeefd e eeeffd"Zded e eeeffd#Zd e jded e e jfd$Zd%ed e fd&Zd%ed eeeffd'ZxZS)*LLMTableExtractionag LLM-based table extraction strategy that uses language models to intelligently extract and structure table data, handling complex cases like rowspan, colspan, and nested tables. This strategy uses an LLM to understand table structure semantically and convert it to structured data that can be easily consumed by pandas DataFrames. aw*You are a specialized table extraction system that converts complex HTML tables into structured JSON data. Your primary goal is to handle difficult, irregular HTML tables that cannot be easily parsed by standard tools, transforming them into clean, tabulated data. ## Critical Requirements **IMPORTANT**: You must extract **EVERY SINGLE ROW** from the table, regardless of size. Tables often contain hundreds of rows, and omitting data is unacceptable. The reason we use an LLM for this task is because these tables have complex structures that standard parsers cannot handle properly. ## Output Format **Your response must be valid JSON**. The output must be properly formatted, parseable JSON with: - Proper escaping of quotes in strings - Valid JSON syntax (commas, brackets, etc.) - No trailing commas - Proper handling of special characters ## Table Structure Every table should be extracted as a JSON object with this structure: ```json { "headers": ["Column 1", "Column 2", ...], "rows": [ ["Row 1 Col 1", "Row 1 Col 2", ...], ["Row 2 Col 1", "Row 2 Col 2", ...], // ... continue for ALL rows ... ], "caption": "Table caption if present", "summary": "Table summary attribute if present", "metadata": { "row_count": , "column_count": , "has_headers": , "has_merged_cells": , "nested_tables": , "table_type": "data|pivot|matrix|nested" } } ``` ## Handling Complex Structures ### Why This Matters Standard HTML parsers fail on tables with: - Complex colspan/rowspan arrangements - Nested tables - Irregular structures - Mixed header patterns Your job is to intelligently interpret these structures and produce clean, regular data. ### Colspan (Merged Columns) When a cell spans multiple columns, duplicate the value across all spanned columns to maintain rectangular data structure. Example HTML: ```html Quarterly Report Total ``` Becomes: ["Quarterly Report", "Quarterly Report", "Quarterly Report", "Total"] ### Rowspan (Merged Rows) When a cell spans multiple rows, duplicate the value down all affected rows. Example with many rows: ```html Category A Item 1 $100 Item 2 $200 ``` Result structure (response must be valid JSON): ```json { "headers": ["Category", "Item", "Price"], "rows": [ ["Category A", "Item 1", "$100"], ["Category A", "Item 2", "$200"], ["Category A", "Item 3", "$300"], ["Category A", "Item 4", "$400"], ["Category A", "Item 5", "$500"], // ... ALL 50 rows must be included ... ["Category A", "Item 50", "$5000"] ], "metadata": { "row_count": 50, "column_count": 3, "has_headers": true, "has_merged_cells": true, "nested_tables": false, "table_type": "data" } } ``` ### Nested Tables For tables containing other tables: 1. Extract the outer table structure 2. Represent nested tables as a JSON string or structured representation 3. Ensure the data remains usable ## Complete Examples ### Example 1: Large Table with Complex Structure Input HTML (abbreviated for documentation): ```html
Department 2024 Performance
Q1 Q2 Q3 Q4
Sales Region North $1.2M $1.5M $1.8M
Region South $0.9M $1.1M $1.3M
Engineering Team Alpha 85% 88% 92%
``` Output (showing structure with all rows) - must be valid JSON: ```json { "headers": ["Department", "Team/Region", "Q1", "Q2", "Q3", "Q4"], "rows": [ ["Sales", "Region North", "$1.2M", "$1.5M", "$1.8M"], ["Sales", "Region South", "$0.9M", "$1.1M", "$1.3M"], ["Sales", "Region East", "$1.1M", "$1.4M", "$1.6M"], ["Sales", "Region West", "$1.0M", "$1.2M", "$1.5M"], ["Sales", "Region Central", "$0.8M", "$1.0M", "$1.2M"], // ... ALL 15 Sales rows must be included ... ["Engineering", "Team Alpha", "85%", "88%", "92%"], ["Engineering", "Team Beta", "82%", "85%", "89%"], ["Engineering", "Team Gamma", "88%", "90%", "93%"], // ... ALL 20 Engineering rows must be included ... // ... Continue for EVERY row in the table ... ], "caption": "", "summary": "", "metadata": { "row_count": 235, "column_count": 6, "has_headers": true, "has_merged_cells": true, "nested_tables": false, "table_type": "data" } } ``` ### Example 2: Pivot Table with Hundreds of Rows Input structure: ```html
Product ID Jan Feb
PROD-001 1,234 1,456
``` Output must include ALL rows and be valid JSON: ```json { "headers": ["Product ID", "Jan", "Feb", "Mar", "Apr", "May", "Jun", "Jul", "Aug", "Sep", "Oct", "Nov", "Dec"], "rows": [ ["PROD-001", "1,234", "1,456", "1,789", "2,012", "2,234", "2,456", "2,678", "2,890", "3,123", "3,345", "3,567", "3,789"], ["PROD-002", "2,345", "2,567", "2,789", "3,012", "3,234", "3,456", "3,678", "3,890", "4,123", "4,345", "4,567", "4,789"], ["PROD-003", "3,456", "3,678", "3,890", "4,123", "4,345", "4,567", "4,789", "5,012", "5,234", "5,456", "5,678", "5,890"], // ... ALL 500+ rows MUST be included ... ["PROD-547", "9,876", "10,098", "10,321", "10,543", "10,765", "10,987", "11,210", "11,432", "11,654", "11,876", "12,098", "12,321"] ], "metadata": { "row_count": 547, "column_count": 13, "has_headers": true, "has_merged_cells": false, "nested_tables": false, "table_type": "pivot" } } ``` ## Critical Data Integrity Rules 1. **COMPLETENESS**: Extract EVERY row, no matter how many (10, 100, 1000+) 2. **ACCURACY**: Preserve exact values, including formatting 3. **STRUCTURE**: Maintain consistent column count across all rows 4. **VALIDATION**: Ensure output is valid JSON that can be parsed 5. **ESCAPING**: Properly escape quotes and special characters in cell values ## Special Handling Instructions ### Large Tables - Never abbreviate or summarize - Never use "..." to indicate omitted rows - Process every row even if it takes significant time - The metadata row_count must match actual extracted rows ### Complex Merged Cells - Track rowspan/colspan values carefully - Ensure proper cell duplication - Maintain data alignment across all rows ### Data Types - Keep numbers as strings to preserve formatting - Preserve currency symbols, percentages, etc. - Handle empty cells as empty strings "" ### Error Prevention - If a cell contains quotes, escape them properly - Handle newlines within cells appropriately - Ensure no JSON syntax errors ## Output Validation Before returning results: 1. Verify JSON is valid and parseable 2. Confirm row count matches actual data 3. Check that all rows have same column count 4. Ensure all data is preserved without truncation ## JSON Schema Definition Your output must conform to the following JSON schema (OpenAPI 3.0 format): { "components": { "schemas": { "ExtractedTable": { "type": "object", "required": [ "headers", "rows", "metadata" ], "properties": { "headers": { "type": "array", "description": "Column headers for the table", "items": { "type": "string" }, "minItems": 1 }, "rows": { "type": "array", "description": "All table rows - must include every single row", "items": { "type": "array", "items": { "type": "string" }, "minItems": 1 } }, "caption": { "type": "string", "description": "Table caption if present", "default": "" }, "summary": { "type": "string", "description": "Table summary attribute if present", "default": "" }, "metadata": { "type": "object", "required": [ "row_count", "column_count", "has_headers", "has_merged_cells", "nested_tables", "table_type" ], "properties": { "row_count": { "type": "integer", "description": "Actual count of rows extracted", "minimum": 0 }, "column_count": { "type": "integer", "description": "Number of columns in the table", "minimum": 1 }, "has_headers": { "type": "boolean", "description": "Whether table has identified headers" }, "has_merged_cells": { "type": "boolean", "description": "Whether table contains colspan or rowspan" }, "nested_tables": { "type": "boolean", "description": "Whether table contains nested tables" }, "table_type": { "type": "string", "enum": ["data", "pivot", "matrix", "nested"], "description": "Classification of table structure" } } } } } } } } **CRITICAL**: Your response must be a valid JSON object that conforms to this schema. The entire purpose of using an LLM for this task is to handle complex HTML tables that standard parsers cannot process correctly. Your value lies in intelligently interpreting complex structures and returning complete, clean, tabulated data in valid JSON format. llm_config css_selector max_triesenable_chunkingchunk_token_thresholdmin_rows_per_chunkmax_parallel_chunksrc  ~t |d d|i| ||_|js9tt j ddt j d|_||_td||_||_ ||_ td||_ td||_ | jdi|_y ) a Initialize the LLM-based table extraction strategy. Args: llm_config: LLM configuration for the extraction css_selector: Optional CSS selector to focus on specific page areas max_tries: Maximum number of retries if LLM fails to extract tables (default: 3) enable_chunking: Enable smart chunking for large tables (default: True) chunk_token_threshold: Token threshold for triggering chunking (default: 3000) min_rows_per_chunk: Minimum rows per chunk (default: 10) max_parallel_chunks: Maximum parallel chunk processing (default: 5) verbose: Enable verbose logging **kwargs: Additional parameters passed to parent class rDEFAULT_PROVIDERzopenai/gpt-4o-miniOPENAI_API_KEY)provider api_tokenr  extra_argsNr!)r9rrrosgetenvrrMrrrrrrr) rrrrrrrrrrr:s rrzLLMTableExtraction.__init__s0 33F3%/#57KL))$45DO )Q *.%:""%a);"<#&q*=#>  **\26rrrc d |jd|j}|rD|j||}|s|jdd|gSdj d|D}nt j |d}d|jvr |jr|jd d gS|jr|jd d t||jr@|j|r/|jr|jd d |j|Sd t|d}td|jdzD]Y} |jr(|dkDr#|jd d|d|jdt!|j"j$|j&dz|zdz|j"j(|j"j*d|j"j,|j"j.|j"j0|j2 }|ru|j4rh|j4dj6j8} |jrG|jddt;| t=| t>r|jdd| dddt=| t>rtAjB| } n| } t=| tDr!d| vr| d} nd| vr| d} n d | vr| d } n| g} t=| tFrXt| dk(rJt=| dtFr7| d} t=| tFr"t| dk(rt=| dtFr7t=| tFs| g} |jr|jdd!t| d"g} | D]V} |jI| r!| jK|jM| 5|jsB|jdd#| X| r1|jr!|jd d$t| d%|| cS||jkr$|jr|jdd&|d'*|jr |jdd(|jd)gcS\gS#t@jN$rP} |jr!|jd*d+|d,t?| ||jkrYd} ~ gcYd} ~ cSd} ~ wtP$r} |jrW|jd*d-|d,t?| ||jk(r'ddl)}|jdd.|jU||jkrddl+}|jXdYd} ~ WgcYd} ~ cSd} ~ wwxYw)/a Extract tables from HTML using LLM. Args: element: The HTML element to search for tables **kwargs: Additional parameters Returns: List of dictionaries containing extracted table data rwarningz$No elements found for CSS selector: r_c3JK|]}tj|dyw)unicodeencodingN)r tostring)rDelems rrFz4LLMTableExtraction.extract_tables..s"jPT5>>$#K#K"js!#rrz tags found in HTML contentz*Found table tags in HTML, content length: z9Content exceeds token threshold, using chunked extractionzFGENERATE THE TABULATED DATA from the following HTML content: ```html R ``` Return only a JSON array of extracted tables following the specified format.r Retry attempt /z for table extraction z{ MAKE SURE TO EXTRACT ALL DATA, DO NOT LEAVE ANYTHING FOR BRAVITY, YOUR GOAL IS TO RETURN ALL, NO MATTER HOW LONG IS DATAT rprompt_with_variablesrbase_url json_response base_delay max_attemptsexponential_factorrrdebugzLLM response type: zLLM response preview: Nz...resultrRdatazParsed z table(s) from LLM responsezTable failed validation: zSuccessfully extracted z tables using LLM on attempt z%No valid tables extracted on attempt z , retrying...z No valid tables extracted after z attemptsrHzJSON parsing error on attempt : z)Error in LLM table extraction on attempt z Traceback: )-rr_css_to_xpath_selectr*rir rrtrrBr_needs_chunking_extract_with_chunkingrrrrrrTABLE_EXTRACTION_PROMPTrrbackoff_base_delaybackoff_max_attemptsbackoff_exponential_factorrchoicesr%contenttype isinstancer1jsonloadsdictlist_validate_table_structurerN_ensure_table_formatJSONDecodeErrorrO traceback format_exctimesleep)rrrrselected_elements html_content user_promptattemptresponserrPvalidated_tablesrSrVrrs rr#z!LLMTableExtraction.extract_tablessmzz.$2C2CD  !% 9 9'< P $ )'KL>%Z[ 77"jXi"jjL!>>'IFL <--/ /|| &$IKI << IIf J3|K\J]^ _   D$8$8$F|| &"]^..|< <|MP Q 23h Gg <T?V/ V/BV*!V*"V/*V/rc d|jjjvr/tjd}t |j |S t |dzS#YxYw)zz Estimate token count for text. Uses tiktoken for OpenAI models, simple approximation for others. gptz gpt-3.5-turbo)rrrttiktokenencoding_for_modelrBencode)rrrs r_estimate_tokensz#LLMTableExtraction._estimate_tokensse  006688#66G8??401194yA~   s AA&&A*rc|jsy|j|}||jkD}|jr%|r#|j dd|d|jd|S)zJ Check if table HTML needs chunking based on token count. FrzTable needs chunking: z tokens > z threshold)rrrrr*)rr token_count needs_chunks rrz"LLMTableExtraction._needs_chunkingse##++L9 !D$>$>> <RUV_R`Qaamnqr}n~m@LM N|| &"QR >;. . !MM'2M" *$$S) *..KK"33K@ 2C~~cI>H..x8J"':Z'G$JdJd'd!44[BTVZ[  j)'/Z"&*&;&;K&H:&U#"))(3#z1# 2$ K % g 6 &.C!((-.#nn]YO 00>PR]^J MM* % v;?++FD4K4KLF << IIfV 5TU V{""rrrrcdg}|r|j||jd|j||jd|r|j||jddj|S)zb Create a complete table HTML chunk with headers, body rows, and optional footer. zzzz
r_)rNrri)rrrr html_partss rrz%LLMTableExtraction._create_chunk_html#st [    k *)$)$*%    k **%wwz""rr r7c|S)zm Rebalance chunks to ensure minimum rows per chunk. Merge small chunks if necessary. r!)rr r7s rrz$LLMTableExtraction._rebalance_chunks7s  rr chunk_index total_chunksrc |jr|jdd|dzd|d}|sd}d|dzd|d |d t|d }td|jdzD]!} |jr-|dkDr(|jdd |d|jd |dzt |j j|jdz|z|j j|j jd|j j|j j|j j|j }|r;|jr.|jdj j"} t%| t&rt)j*| } n| } t%| t,r!d| vr| d} nd| vr| d} n d| vr| d} n| g} t%| t.rXt1| dk(rJt%| dt.r7| d} t%| t.r"t1| dk(rt%| dt.r7t%| t.s| g} | D]+} |j3| s||j5| dccS|ddcS$|ddS#t6$ru} |jr$|jdd|dzdt'| ||jkrt9j:dYd} ~ |dt'| dcYd} ~ cSd} ~ wwxYw)z6 Process a single chunk with the LLM. rzProcessing chunk r rr_zs IMPORTANT: This table has NO headers. Return an empty array for 'headers' field and extract all rows as data rows.z6Extract table data from this HTML chunk. This is part z of zA of a larger table. Focus on extracting the data rows accurately.z ```html rrz for chunk rTrrrrRr)rrSNrHzError processing chunk rrrSrH)rr*rrrrrrrrrrrrrrr%rrr1rrrrrBrrrOrr) rrrrrheader_context chunk_promptrrrrPrSrVs r_process_chunkz!LLMTableExtraction._process_chunk@s& << IIf 1+/1B!L>R STNAo d<.1..<-=>zM P Q 23? XG> X<z9LLMTableExtraction._merge_chunk_results..s}a)@r)keyrSrr=rrTchunked chunk_countrzMerged z chunks into table with z rows)sortrcopyrrBrr*) rrr valid_chunks merged_tabler chunk_resultrSr=s r_merge_chunk_resultsz'LLMTableExtraction._merge_chunk_resultss @A$1CaAEE'NC CI$Aw/446 ( "L )E99VR(D OOD !  " ( V14H Z -.2 Z +25l2C Z / << IIfL(9'::RSVW_S`Raafg h~5Ds C)C)c|jr|jddt|d|j|\}}|jr|jddt|dt|dk(rN|jr|jdd|j |ddd|}|j d r|d gSgS|jr,|jdd t|d |j d g}t|j 5}t|Dcic].\}}|j|j||t|||0} }}t| D]\} | | } | jd}|jr%|jdd| dzdt|d|j|^ ddd|jr|jdd|j|Scc}}w#t$re} |jr0|jdd| dzdt|dt| |j| dt| dYd} ~ d} ~ wwxYw#1swYxYw)zH Extract tables using chunking and parallel processing. rz-Starting chunked extraction for content with z charactersrz chunk(s) for processingr z6Processing as single chunk (no parallelization needed)rrSz Processing z" chunks in parallel (max workers: )) max_workers<)timeoutzChunk rz completed successfullyrHz processing failed: Nrz(All chunks processed, merging results...)rr*rBrrrrr enumeratesubmitrrrNrOr1r*) rrr rrrexecutorrchunkfuturesfuturerrVs rrz)LLMTableExtraction._extract_with_chunkingsy << IIf McR^N_M``kl m#77 E << IIfV 5MN O v;! || &"Z[((Aq+FF(. 7(;F7O$ C C << IIf CK=8Z[_[s[sZttuv w D,D,D E g!*& 1Au 3 3UAs6{KXZ[[G 'w/ g%fo g#]]2]6F|| &F;?2C1S[MQh*ij!((0 g g( << IIf H J((77-!g|| 'VK!O3DAc&k]Rfgjklgmfn+op!((t^abc^d)effg g gsJI5&3G>I50AHI5>I5 I2 AI-'I5-I22I55I>c |}|jdr d|ddd}nr|jdr d|ddd}nWd|vr%|jd}|d }|d}d |d |d}n.d|vr%|jd}|d }|d}d |d |d}nd |} |j|S#t$r-}|j d d|dt |gcYd}~Sd}~wwxYw)am Convert CSS selector to XPath and select elements. This is a basic implementation - for complex CSS selectors, consider using cssselect library. Args: element: Root element to search from css_selector: CSS selector string Returns: List of selected elements #z .//*[@id='r Nz'].z.//*[contains(@class, 'z')]rz.//z[contains(@class, 'z[@id='rz&XPath conversion failed for selector 'z': )rrsplitrJrOr*r1) rrrrJparts element_name class_nameid_valuerVs rrz'LLMTableExtraction._css_to_xpath_selects,  " "3 ' ab!1 2"5E  $ $S )-l12.>-?sCE L  &&s+E 8LqJ,'::,cJE L  &&s+E 8LQxH,vhZr:E,(E ==' '  IIi#I,WZ[^_`[aZb!c dI sB"" C+"C CCrSct|tsyd|vsd|vry|jd}t|tsyt|tr`t |dk(rRt|dtr?|d|d<|d}t|tr"t |dk(rt|dtr?|jd}t|tsyg}|D]{}t|trXt |dk(rJt|dtr7|d}t|tr"t |dk(rt|dtr7|j |}||d<|jdgD]}t|tryy)z Validate that the table has the required structure. Args: table: Table dictionary to validate Returns: True if valid, False otherwise Fr>r=r rT)rrrrrBrN)rrSr>r= cleaned_rowsrEs rrz,LLMTableExtraction._validate_table_structureso%& E !V5%8))I&'4($'CLA,=*WUVZY]B^&qzE) I&G$'CLA,=*WUVZY]B^ yy $%  %CS$'CHMjQQU>V!fS$'CHMjQQU>V    $  % %f 99VR( Cc4( rc,|jdg|jdg|jdd|jdd|jdid}|dsi|d<|d}d|vrt|d|d<d |vrt|d|d <d |vrt|d|d <t|d}|d kDrZt|dD]I\}}t||kr|dg|t|z zz|d|<0t||kDs?|d ||d|<K|S) z Ensure the table has all required fields with proper defaults. Args: table: Table dictionary to format Returns: Properly formatted table dictionary r>r=rr_rgrrrrrrN)rrBrr0)rrSformatted_tablerrUrrEs rrz'LLMTableExtraction._ensure_table_formatOsUyyB/IIfb)yyB/yyB/ *b1  z**,OJ '#:. h &$'(?$@H[ !  )'*?9+E'FH^ $  (&*?9+E&FH] # 23 q=#OF$;< A3s8i'14ty3s8?S7T1TOF+A.X )14ZiOF+A.  Ar)NNr]Ti rprF)T)r,r-r.r/rrr r1rrrr r0rrrr#rrr rrrrrr*rrrrrrs@rrr@sf`R 48/3"#)-.2+-,-!&)7%i0)7'})7 )7#' )7 ), )7 &) )7'*)7)7V`emm`$tCQTH~BV`DSS C D ;@S;@U4 CVX\]b]j]jXkmqrwrrnACGDG>H;@zF#F#tCy$9OF#P#c#d3i#V^_bVc#hk#(S ST#YX;X;3X;cX;`dX;ptuxz}u}p~X;t"$tCH~2F"4PTUXZ]U]P^K_"H183184S#X;O18f+EMM++QUV[VcVcQd+Z/t//b+$+4S>+rr)r/abcrrtypingrrrrr r lxmlr rertypesr rutilsrrrconcurrent.futuresrrrrrr3rrr!rrrIsi$:: /A ? *?c*?Zf 4f R/*z0zr