/id d dl mZmZ d dlZd dlmZmZmZmZm Z m
Z
m Z d dl m
Z
mZ d dlZd dlZd dlmZmZ ddlmZmZmZmZmZ ddlmZm Z m Z m Z m Z dd l dd
l m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z' dd l( dd l(m)Z) dd l* dd l*m+Z+m,Z,m-Z-m.Z. dd
l/m0Z0m1Z1 d dl2m3Z3 d dl4Z5d dl6Z6d dl7m8Z8 d dl9m:Z:m;Z; G d de Z< G d de< Z= G d de< Z> G d de< Z? G d de< Z@ G d d e@ ZA G d d e@ ZB G d d e@ ZC G d! d"e@ ZD eEeF eGd# D ci c] } | d$vs| d% eH| d& c} ZI e6j d' ZK e6j d( ZLd)eeMeMf d*eeMeMf fd+ZN G d, d-e< ZOyc c} w ). )ABCabstractmethodN)AnyListDictOptionalTuplePatternUnion)ThreadPoolExecutor as_completed)IntFlagauto )PROMPT_EXTRACT_BLOCKS&PROMPT_EXTRACT_BLOCKS_WITH_INSTRUCTION&PROMPT_EXTRACT_SCHEMA_WITH_INSTRUCTIONJSON_SCHEMA_BUILDER_XPATH PROMPT_EXTRACT_INFERRED_SCHEMA)DEFAULT_PROVIDERDEFAULT_PROVIDER_API_KEYCHUNK_TOKEN_THRESHOLD OVERLAP_RATEWORD_TOKEN_RATE)*)
sanitize_htmlescape_json_string perform_completion_with_backoffextract_xml_data split_and_parse_json_objectssanitize_input_encode merge_chunks)
TokenUsage)
get_deviceload_HF_embedding_model load_text_multilabel_classifiercalculate_batch_size) LLMConfigcreate_llm_config)partial
BeautifulSoup)htmletreec
e Zd ZdZd defdZedededeeee f fd Z
dedee deeee f fd Z dedee deeee f fd
Z y )
ExtractionStrategyz<
Abstract base class for all extraction strategies.
input_formatc || _ d| _ | j j | _ |j dd | _ y)a
Initialize the extraction strategy.
Args:
input_format: Content format to use for extraction.
Options: "markdown" (default), "html", "fit_markdown"
**kwargs: Additional keyword arguments
z<|DEL|>verboseFN)r1 DEL __class____name__namegetr3 )selfr1 kwargss X/mnt/e/genesis-system/.venv/lib/python3.12/site-packages/crawl4ai/extraction_strategy.py__init__zExtractionStrategy.__init__6 s6 ) NN++ zz)U3 urlr- returnc y)z
Extract meaningful blocks or chunks from the given HTML.
:param url: The URL of the webpage.
:param html: The HTML content of the webpage.
:return: A list of extracted blocks or chunks.
N r9 r> r- qr: s r; extractzExtractionStrategy.extractD s
r= sectionsc g }t 5 }|D cg c]! } |j | j ||fi |# }}t | D ]! } |j | j # ddd |S c c}w # 1 sw Y |S xY w)z
Process sections of text in parallel by default.
:param url: The URL of the webpage.
:param sections: List of sections (strings) to process.
:return: A list of processed JSON blocks.
N)r submitrD r
extendresult)
r9 r> rE rC r: extracted_contentexecutorsectionfuturesfutures
r; runzExtractionStrategy.runO s
! :X ( c7EfEG 'w/
:!(( 9
: : !
: ! s A9&A41A94A99Bc j K ddl } |j | j ||g|i | d{ S 7 w)a{
Async version: Process sections of text in parallel using asyncio.
Default implementation runs the sync version in a thread pool.
Subclasses can override this for true async processing.
:param url: The URL of the webpage.
:param sections: List of sections (strings) to process.
:return: A list of processed JSON blocks.
r N)asyncio to_threadrO )r9 r> rE rC r: rQ s r; arunzExtractionStrategy.aruna s6 &W&&txxhMMfMMMMs *313N)markdown)
r6
__module__ __qualname____doc__strr< r r r r rD rO rS rA r= r; r0 r0 1 s 4S 4
3
c
Dc3h r- r? c d|dgS )zJ
Extract meaningful blocks or chunks from the given HTML.
r )indexcontentrA rB s r; rD z NoExtractionStrategy.extractu s -..r= rE c R t | D cg c] \ }}|g |d
c}}S c c}}w )Nr\ tagsr] ) enumerate)r9 r> rE rC r: irL s r; rO zNoExtractionStrategy.run{ s4 ( 1
7 8
s #N)
r6 rU rV rW rX r r r rD rO rA r= r; rZ rZ p sZ /3 /c /Dc3h
t) d
| d t; t j | j z
z d z yy)a
Initialize the strategy with clustering parameters.
Args:
semantic_filter (str): A keyword filter for document filtering.
word_count_threshold (int): Minimum number of words per cluster.
max_dist (float): The maximum cophenetic distance on the dendrogram to form clusters.
linkage_method (str): The linkage method for hierarchical clustering.
top_k (int): Number of top categories to extract.
r Nr3 Fdirectz#[LOG] Loading Extraction Model for z device.batchz([LOG] Loading Multilabel Classifier for z[LOG] Model loaded z, models/reuters, took secondsrA ) superr< numpysemantic_filterword_count_thresholdmax_distlinkage_methodtop_k
sim_thresholdtimetimerr8 r3 arraybuffer_embeddingsget_embedding_methodr$ devicer' default_batch_sizeprinttyper% tokenizermodeltoevalr& nlprX ) r9 rk rl rm rn ro
model_namerp r: np_r5 s r; r< zCosineStrategy.__init__ s * "6" . $8!
,
*YY[
zz)U3 !)" $,! l #7t{{"C << 7 8H8H7IR S &=Z%H"
dkk"
$+!!)" << z." s
ad r= T)keyreverseN) lensklearn.metrics.pairwiser get_embeddingsflattenziprp sortrH )
r9 r rk r r query_embeddingdocument_embeddings similaritiesdocsim
filtered_docsremaining_docsr s
r; filter_documents_embeddingsz*CosineStrategy.filter_documents_embeddings sg y>J & Y 1,J> --.?@C # 1 1) < )
2
') < 8
Sd((( #J
}
* !$I| < C+++ c
N
ND A 0Q*s=?Q2Q!R S ,99a9
9 [j))'
:s 4 D 1 D$
D* sentencesc | j j dv r"ddl}| | j }g }t dt | | D ] }||||z }| j
|ddd }|j D
ci c]! \ } }
| |
j | j # }} }
|j 5 | j d
i |} ddd j j d j j } |j | t! j" | | _ | j$ S | j j d k( ro| | j }g }t dt | | D ], }||||z }| j | } |j | . t! j" | | _ | j$ S c c}
} w # 1 sw Y xY w) z
Get BERT embeddings for a list of sentences.
Args:
sentences (List[str]): A list of text chunks (sentences).
Returns:
NumPy array of embeddings.
)cpugpucudampsr NTpt)padding
truncationreturn_tensorsr )dimr rA )rv ry torchrw ranger rz itemsr| no_gradr{ last_hidden_statemeanr rj appendr vstackrt )
r9 r
batch_size
bypass_bufferr all_embeddingsrb batch_sentences
encoded_inputr tensor model_output
embeddingss
r; r z CosineStrategy.get_embeddings* s ;; < < ! ! 4 4
N1c)nj9
2"+AJ"? $#TdSW !/ !
DQCVCVCX!4?CC4;;//!
!
]]_ ?#-4::#>
#>L? * ; ; @ @Q @ G K K M S S U
%%j1
2" &(YY~%>D " %%% [[
&! ! 4 4
N1c)nj9
2"+AJ"? !ZZ 8
%%j1
2
&(YY~%>D "%%%3!
? ?s 7&G0GG c ddl m}m} ddlm} t j
| _ | j |d } ||d } ||| j } ||| j d }|S )
z
Perform hierarchical clustering on sentences and return cluster labels.
Args:
sentences (List[str]): A list of text chunks (sentences).
Returns:
NumPy array of cluster labels.
r )linkagefcluster)pdistT)r cosine)methoddistance) criterion)
scipy.cluster.hierarchyr r scipy.spatial.distancer rq rr r rn rm ) r9 r r r r r distance_matrixlinkedlabelss r; hierarchical_clusteringz&CosineStrategy.hierarchical_clusteringa sc >0YY[
(($(G
H5 1D1DE&$--:F
r= clustersc i }|j D ]D \ }}dj | }t |j }|| j k\ s@|||< F |S )a
Filter clusters to remove those with a word count below the threshold.
Args:
clusters (Dict[int, List[str]]): Dictionary of clusters.
Returns:
Dict[int, List[str]]: Filtered dictionary of clusters.
)r joinr splitrl )r9 r filtered_clusters
cluster_idtexts full_text
word_counts r; filter_clusters_by_word_countz,CosineStrategy.filter_clusters_by_word_countz sf !) !1 6 JI Y__ ./J T 6 6605!*- 6 ! r= r> r- c V t j } |j | j }| j || j }|sg S | j | }t j }i }t
| D ]) \ } }
|j |
g j || + | j | } t | D cg c]# } t | g dj | | d% }
} | j rt d| j | j j dv r;| j# |
D cg c] }|d c} }t% |
| D ]
\ }}
|
|d< | j r%t dt j |z
dd |
S c c} w c c}w )
a
Extract clusters from HTML content using hierarchical clustering.
Args:
url (str): The URL of the webpage.
html (str): The HTML content of the webpage.
Returns:
List[Dict[str, Any]]: A list of processed JSON blocks.
r r_ u [LOG] 🚀 Assign tags using )r r r r r] r` u" [LOG] 🚀 Categorization done in z.2frh )rq r r4 r rk r ra
setdefaultr r sortedintr r3 rx rv ry r~ r )r9 r> r- rC r: t text_chunksr r r\ labelr idx cluster_listclusters r; rD zCosineStrategy.extract s
IIK djj* 66 --
I --k:
IIK %f - FLE5 r * 1 1+e2D E F ! > >x H
/0
#hsxx@QRU@V7W X
<< 1$++ ? @ ;; < <XX Ngwy 1 NOF"%lF";
("'
(& << 6tyy{Qs6K8T UC
Os
(F! F&rE c \ | j || j j | fi |S )z
Process sections using hierarchical clustering.
Args:
url (str): The URL of the webpage.
sections (List[str]): List of sections (strings) to process.
Returns:
)rD r4 r )r9 r> rE rC r: s r; rO zCosineStrategy.run s) t||C x!8CFCCr= )N
g?ward z&sentence-transformers/all-MiniLM-L6-v2g333333?) )NFN)r6 rU rV rW r< r rX r r r r r r r rD rO
__classcell__r5 s @r; rd rd s , ;Tn MO6* c6*586*FI6*
c6*r DI5& c5&nc 2! S$s)^,!
c49n !2F 3 F c F Dc3h /, e.g., "ollama/llama3.3".
api_token: The API token for the provider.
base_url: The base URL for the API request.
api_base: The base URL for the API request.
extra_args: Additional arguments for the API request, such as temperature, max_tokens, etc.
r1 r r r
extra_argsg eANrA )ri r< r r) r osenvironr8 r r extract_typer force_json_responser chunk_token_threshold overlap_rateword_token_rateapply_chunkingr r3 usagesr# total_usager r r r )r9 r r r extraction_typer r r r r1 r r3 r r r r r: r5 s r; r< z LLMExtractionStrategy.__init__ s X >|>v>$ /) **..)AB DO ' + (D #6 %:%S>S" ( . , **\26""),D & %<
"
r= c t j | j }|j }|| j v r/||| j
ur t
d| d| j | t | ! || y)zHandle attribute setting. Setting '' is deprecated. N) inspect signaturer<
parameters_UNWANTED_PROPSdefaultAttributeErrorri __setattr__)r9 r7 valuesig
all_paramsr5 s r; r z!LLMExtractionStrategy.__setattr__N sw
.^^
4'' 'ED9I9Q9Q,Q 9TF2CDDXDXY]D^C_!`a a
D%(r= r> ixr- r? c | j rt d| d| |t t | d}t }| j
r| j
|d< t }| j dk( r6| j r*t j | j d |d< t }| j dk( r| j st }|D ] }|j d |z d
z || } t | j j || j j" | j j$ | j& | j( | j j* | j j, | j j. }t1 |j2 j4 |j2 j6 |j2 j8 |j2 j: r |j2 j: j< ni |j2 j> r |j2 j> j< ni }| j@ jC | | jD xj4 |j4 z
c_ | jD xj6 |j6 z
c_ | jD xj8 |j8 z
c_ |jF d
jH jJ } d}
| j& rt jL | }
tO |
tP rYtS |
dk( rGtO tU |
jW d
tT r tU |
jW d
}
n<|
g}
n8tO |
tT r(|
}
n%tY dg| d }
t jL |
}
|
D ] } d| d< | j rt dtS |
d|d| |
S # tZ $ rM t] |jF d
jH jJ \ } }
| }
|
r|
jC d
ddg|
d Y |w xY w# tZ $ r6}| j rt d| |ddgt_ | dgcY d}~S d}~ww xY w)a
Extract meaningful blocks or chunks from the given HTML using an LLM.
How it works:
1. Construct a prompt with variables.
2. Make a request to the LLM using the prompt.
3. Parse the response and extract blocks or chunks.
Args:
url: The URL of the webpage.
ix: Index of the block.
html: The HTML content of the webpage.
Returns:
A list of extracted blocks or chunks.
[LOG] Call LLM for - block index: URLHTMLREQUESTr r indentSCHEMA{}r
json_responser
base_delay max_attemptsexponential_factorcompletion_tokens
prompt_tokens total_tokenscompletion_tokens_detailsprompt_tokens_detailsr Nr blocksFerrorTr\ r r` r] [LOG] Extractedblocks from URL: block index: [LOG] Error in LLM extraction: )0r3 rx r r r r r r r jsondumpsr r replacer r r r r r r backoff_base_delaybackoff_max_attemptsbackoff_exponential_factorr# usager r r r __dict__r r r r choicesmessager] loads
isinstancedictr listvaluesr Exceptionr rX )r9 r> r r- variable_valuesprompt_with_variablesvariableresponser, r] r r parsedunparsedes r; rD z LLMExtractionStrategy.extractY s " << 'u,>;; +3..*R*R*[*[>>77 '/nn&J&J&S&S
E
KK u %
. .%2I2I I . * *e.A.A A * ) )U-?-? ? )
"**1-55== ++ !ZZ 0F!&$/ v;!+
4
;PQR;SUY0Z%)&--/%:1%=F '-XF#FD1!' .xj' B8 LF !ZZ /F # +E%*E'N+ ||%K&"
M'
#? $$Q'//77$ MM"#dWIRZ[
( ||7s;< ! $I "1v
sE %G.Q C3O/ 'Q /AQQ QQ R+R<RRc 8 t |||| j }|S )z[
Merge documents into sections based on chunk_token_threshold and overlap.
)docs target_sizeoverlapword_token_ratio)r" r )r9 r r r@ rE s r; _mergez LLMExtractionStrategy._merge s' ! . ! 1 1
r= rE c ~ | j || j t | j | j z }g }| j j
j
d rat | D ]Q \ }}t | j | }|j ||t | t j d S |S t d 5 }t | j | }t | D cg c]! \ }}|j ||t | # } }}t! | D ]" }
|j |
j# $ d d d |S c c}}w # t$ $ rD} | j& rt) d| |j+ ddd gt- | d
Y d } ~ ~d } ~ ww xY w# 1 sw Y |S xY w) a7
Process sections sequentially with a delay for rate limiting issues, specifically for LLMExtractionStrategy.
Args:
url: The URL of the webpage.
sections: List of sections (strings) to process.
Returns:
A list of extracted blocks or chunks.
r@ zgroq/g ? ) max_workerszError in thread execution: r Tr r! N)rB r r r r r
startswithra r* rD rH r! rq sleepr rG r
rI r5 r3 rx r rX ) r9 r> rE merged_sectionsrJ r rL extract_funcrK rM rN r< s r; rO zLLMExtractionStrategy.run s ++ & &22T5F5FFG &
?? # # . .w 7 ( 9
G &t||S 9 !(( %:7%CD
3
F ! / $2
h &t||S 9 (1 'A#G OOL"6KG6TU
+7 3 F
)00 A
. ! + % << !$?s"C D)00)*)-)0 +.q6
. ! sH %F28&E F20 E"F2 F2" F/+:F*%F2*F//F22F<c 8 K ddl m} | j rt d| d| |t t | d}t }| j r| j |d< t }| j dk( r6| j r*t j | j d |d
< t }| j dk( r| j st }|D ] }|j d |z d z || } || j j" || j j$ | j j& | j( | j* | j j, | j j. | j j0
d{ }t3 |j4 j6 |j4 j8 |j4 j: |j4 j< r |j4 j< j> ni |j4 j@ r |j4 j@ j> ni } | jB jE | | jF xj6 | j6 z
c_ | jF xj8 | j8 z
c_ | jF xj: | j: z
c_ |jH d jJ jL }
d} | j( rt jN |
} tQ | tR rYtU | dk( rGtQ tW | jY d tV r tW | jY d } n<| g} n8tQ | tV r(| } n%t[ dg|
d } t jN | } | D ] } d| d< | j rt dtU | d|d| | S 7 o# t\ $ rM t_ |jH d jJ jL \ }
}|
} |r| jE dddg|d Y w xY w# t\ $ r6}| j rt d| |ddgta | dgcY d}~S d}~ww xY ww)a
Async version: Extract meaningful blocks or chunks from the given HTML using an LLM.
How it works:
1. Construct a prompt with variables.
2. Make an async request to the LLM using the prompt.
3. Parse the response and extract blocks or chunks.
Args:
url: The URL of the webpage.
ix: Index of the block.
html: The HTML content of the webpage.
Returns:
A list of extracted blocks or chunks.
r aperform_completion_with_backoffr r
r r r r r r r r r Nr r r Fr Tr! r" r# r$ r% )1utilsrM r3 rx r r r r r r r r& r' r r r( r r r r r r r) r* r+ r# r, r r r r r- r r r r r. r/ r] r0 r1 r2 r r3 r4 r r5 r rX )r9 r> r r- rM r6 r7 r8 r9 r, r] r r r: r; r< s r; aextractz LLMExtractionStrategy.aextract s " < << 'u,>;; +3..*R*R*[*[>>77 '/nn&J&J&S&S
E
KK u %
. .%2I2I I . * *e.A.A A * ) )U-?-? ? )
"**1-55== ++ !ZZ 0F!&$/ v;!+
4
;PQR;SUY0Z%)&--/%:1%=F&,XF#FD1!' -xj' B8 LF !ZZ /F # +E%*E'N+ ||%K&"
MI
b
#? $$Q'//77$ MM"#dWIRZ[
(
||7s;< ! $I "1v
sp C*R-B Q
O<EQ !C3O? 'Q ;R<Q ?AQQ QQ R!+R R
RRRc K ddl }| j || j t | j | j z }g }t | D cg c]! \ }}| j
||t | # }}} |j |ddi d{ } | D ]^ }
t |
t r;| j rt d|
|j dddgt |
d N|j |
` |S c c}}w 7 ow) a
Async version: Process sections with true parallelism using asyncio.gather.
Args:
url: The URL of the webpage.
sections: List of sections (strings) to process.
Returns:
A list of extracted blocks or chunks.
r NrD return_exceptionsTzError in async extraction: r r! )rQ rB r r r ra rO r! gatherr1 r5 r3 rx r rX rH ) r9 r> rE rQ rI rJ r rL tasksresultsrI s r; rS zLLMExtractionStrategy.arun s ++ & &22T5F5FFG &
) 9
G
MM#r#8#A B
'FFF
1F&),<< 7x@A!((!"!%!( #&v; "((0
1 ! 1
Gs AD&C=;DDA0Dc D t d t ddddd t d t ddd| j j d t d
dd| j j d t d dd| j j d t d t d
dddddd
ddd d t d t | j d D ]= \ }}t |dd|j d d|j d d|j d ? y)zHPrint a detailed token usage report showing total and per-request usage.z
=== Token Usage Summary ===Typez<15r Countz>12z ------------------------------
Completionz>12,PromptTotalz
=== Usage History ===z Request #z<10z0------------------------------------------------r N)rx r r r r ra r )r9 rb r, s r;
show_usagez LLMExtractionStrategy.show_usage s/
-.
Agc]+,
h
c"!D$4$4$F$Ft#LMN
#a 0 0 > >t DEF
Qt / / < >R ||.!c .!T#Y .!4S#X;O .!`r= r c e Zd ZdZdZdeeef f fd Zdedede eeef fdZ
e defd Z e d
efd Z
e d
efd Zd
Zd Zd Zd Zd Zd Zdede e de eeef fdZe defd Ze defd Ze defd ZdddZed)d ed ed ed edef
d! Zed"d d e d d fd ed ed ed ed#d$d%ed&ede fd' Z e d*d ed ed ed ed#d$de f d( Z xZ S )+ JsonElementExtractionStrategya
Abstract base class for extracting structured JSON from HTML content.
How it works:
1. Parses HTML content using the `_parse_html` method.
2. Uses a schema to define base selectors, fields, and transformations.
3. Extracts data hierarchically, supporting nested fields and lists.
4. Handles computed fields with expressions or functions.
Attributes:
DEL (str): Delimiter used to combine HTML sections. Defaults to '
'.
schema (Dict[str, Any]): The schema defining the extraction rules.
verbose (bool): Enables verbose logging for debugging purposes.
Methods:
extract(url, html_content, *q, **kwargs): Extracts structured data from HTML content.
_extract_item(element, fields): Extracts fields from a single element.
_extract_single_field(element, field): Extracts a single field based on its type.
_apply_transform(value, transform): Applies a transformation to a value.
_compute_field(item, field): Computes a field value using an expression or function.
run(url, sections, *q, **kwargs): Combines HTML sections and runs the extraction strategy.
Abstract Methods:
_parse_html(html_content): Parses raw HTML into a structured format (e.g., BeautifulSoup or lxml).
_get_base_elements(parsed_html, selector): Retrieves base elements using a selector.
_get_elements(element, selector): Retrieves child elements using a selector.
_get_element_text(element): Extracts text content from an element.
_get_element_html(element): Extracts raw HTML from an element.
_get_element_attribute(element, attribute): Extracts an attribute's value from an element.
r c ` t | di | || _ |j dd | _ y)z
Initialize the JSON element extraction strategy with a schema.
Args:
schema (Dict[str, Any]): The schema defining the extraction rules.
r3 FNrA )ri r< r r8 r3 r9 r r: r5 s r; r< z&JsonElementExtractionStrategy.__init__ s. "6" zz)U3 r= r> html_contentr? c | j | }| j || j d }g }|D ] }i } d| j v r1| j d D ] }
| j ||
} | | | |
d < ! | j || j d } | j | | sw|j
| |S )a
Extract structured data from HTML content.
How it works:
1. Parses the HTML content using the `_parse_html` method.
2. Identifies base elements using the schema's base selector.
3. Extracts fields from each base element using `_extract_item`.
Args:
url (str): The URL of the page being processed.
html_content (str): The raw HTML content to parse and extract.
*q: Additional positional arguments.
**kwargs: Additional keyword arguments for custom extraction.
Returns:
List[Dict[str, Any]]: A list of extracted items, each represented as a dictionary.
baseSelector
baseFieldsr7 fields) _parse_html_get_base_elementsr _extract_single_field
_extract_itemupdater )
r9 r> ra rC r: parsed_html
base_elementsrT elementitemfieldr
field_datas
r; rD z%JsonElementExtractionStrategy.extract s * &&|4 // ^4
$ %GDt{{* ![[ 6 4E 6 6w FE (.3U6]+4 ++GT[[5JKJ KK
#t$ % r= c y)z*Parse HTML content into appropriate formatNrA r9 ra s r; rf z)JsonElementExtractionStrategy._parse_html=
r= selectorc y)(Get all base elements using the selectorNrA r9 rk rt s r; rg z0JsonElementExtractionStrategy._get_base_elementsB rs r= c y)z%Get child elements using the selectorNrA r9 rm rt s r;
_get_elementsz+JsonElementExtractionStrategy._get_elementsG rs r= c Z |d dk( r7| j ||d }|r|d nd }|r| j ||d S i S |d dk( r8| j ||d }|D cg c] }| j ||d c}S |d dk( r8| j ||d }|D cg c] }| j ||d c}S | j || S c c}w c c}w # t $ rD}| j
r t
d|d d
t | |j d cY d }~S d }~ww xY w) Nry nestedrt r re r3 nested_listzError extracting field r7 : r ) rz ri _extract_list_itemrh r5 r3 rx rX r8 )r9 rm ro nested_elementsnested_elementelementselr< s r; _extract_fieldz,JsonElementExtractionStrategy._extract_fieldL sS (V}("&"4"4WeJ>O"P7F !3D & &&~uXG V}& --guZ7HIOWX//E(ODXXV}
- --guZ7HIJRSB**2uX?SS--gu= =
Y T (||/f
bQIJ99Y' ' (sR C !C " C>C C
C D*&9D% D*%D*c d|v r.| j ||d }|s|j d S |d }n|}d}|d dk( r| j | }n|d dk( r| j ||d }na|d dk( r| j | }nG|d d k( r?| j | }t j |d
| }|r|j d nd}d |v r| j ||d }||S |j d S )
a
Extract a single field based on its type.
How it works:
1. Selects the target element using the field's selector.
2. Extracts the field value based on its type (e.g., text, attribute, regex).
3. Applies transformations if defined in the schema.
Args:
element: The base element to extract the field from.
field (Dict[str, Any]): The field definition in the schema.
Returns:
Any: The extracted field value.
rt r r Nry text attributer- regexpatternr transform) rz r8 _get_element_text_get_element_attribute_get_element_htmlresearchgroup_apply_transform)r9 rm ro selectedr r matchs r; rh z3JsonElementExtractionStrategy._extract_single_fielde s " ))'53DEH yy++ {H H =F "**84E
6]k
)//% :LME
6]f
$**84E
6]g
%))(3DIIeI .5E&+EKKNE % ))%{1CDE )uCuyy/CCr= c R i }|D ] }| j || }||||d < ! |S )Nr7 )rh r9 rm re rn ro r s r; r z0JsonElementExtractionStrategy._extract_list_item sC ,E..w>E &+U6]# , r= c i }|D ]: }|d dk( r| j || }n| j || }|3|||d < < |S )a
Extracts fields from a given element.
How it works:
1. Iterates through the fields defined in the schema.
2. Handles computed, single, and nested field types.
3. Updates the item dictionary with extracted field values.
Args:
element: The base element to extract fields from.
fields (List[Dict[str, Any]]): The list of fields to extract.
Returns:
Dict[str, Any]: A dictionary representing the extracted item.
ry computedr7 )_compute_fieldr r s r; ri z+JsonElementExtractionStrategy._extract_item sc " ,EV}
* ++D%8 ++GU; &+U6]#
, r= c |dk( r|j S |dk( r|j S |dk( r|j S |S )a
Apply a transformation to a value.
How it works:
1. Checks the transformation type (e.g., `lowercase`, `strip`).
2. Applies the transformation to the value.
3. Returns the transformed value.
Args:
value (str): The value to transform.
transform (str): The type of transformation to apply.
Returns:
str: The transformed value.
lowercase uppercasestrip)lowerupperr )r9 r r s r; r z.JsonElementExtractionStrategy._apply_transform sE " #;;=
+
%;;=
'
!;;= r= c d|v rt |d i | S d|v r |d | S y # t $ rD}| j r t d|d dt | |j d cY d }~S d }~ww xY w)N
expressionfunctionzError computing field r7 r~ r )r} r5 r3 rx rX r8 )r9 rn ro r< s r; r z,JsonElementExtractionStrategy._compute_field s (u$E, /T::u$(uZ(.. % (||.uV}oRAxHI99Y' ' (s & & A39A.(A3.A3rE c ` | j j | } | j ||fi |S )a4
Run the extraction strategy on a combined HTML content.
How it works:
1. Combines multiple HTML sections using the `DEL` delimiter.
2. Calls the `extract` method with the combined HTML.
Args:
url (str): The URL of the page being processed.
sections (List[str]): A list of HTML sections.
*q: Additional positional arguments.
**kwargs: Additional keyword arguments for custom extraction.
Returns:
List[Dict[str, Any]]: A list of extracted items.
)r4 r rD )r9 r> rE rC r:
combined_htmls r; rO z!JsonElementExtractionStrategy.run s. $
h/
t||C 9&99r= c y)z Get text content from elementNrA r9 rm s r; r z/JsonElementExtractionStrategy._get_element_text rs r= c y)z Get HTML content from elementNrA r s r; r z/JsonElementExtractionStrategy._get_element_html rs r= r c y)z Get attribute value from elementNrA r9 rm r s r; r z4JsonElementExtractionStrategy._get_element_attribute rs r= r r r Nr- schema_typequerytarget_json_examplec ddl m} |dk( r|nt }d| }d| d}|r|d| z
}|r |d| d z
}|r|s|d
z
}n|s|r|d z
}n |s|s|d z
}|d
z
}dj ||g S )z
Build the prompt for schema generation. Shared by sync and async methods.
Returns:
str: Combined system and user prompt
r )JSON_SCHEMA_BUILDERCSSa# You specialize in generating special JSON schemas for web scraping. This schema uses CSS or XPATH selectors to present a repetitive pattern in crawled HTML, such as a product in a product list or a search result item in a list of search results. We use this JSON schema to pass to a language model along with the HTML content to extract structured data from the HTML. The language model uses the JSON schema to extract data from the HTML and retrieve values for fields in the JSON schema, following the schema.
Generating this HTML manually is not feasible, so you need to generate the JSON schema using the HTML content. The HTML copied from the crawled website is provided below, which we believe contains the repetitive pattern.
# Schema main keys:
- name: This is the name of the schema.
- baseSelector: This is the CSS or XPATH selector that identifies the base element that contains all the repetitive patterns.
- baseFields: This is a list of fields that you extract from the base element itself.
- fields: This is a list of fields that you extract from the children of the base element. {name, selector, type} based on the type, you may have extra keys such as "attribute" when the type is "attribute".
# Extra Context:
In this context, the following items may or may not be present:
- Example of target JSON object: This is a sample of the final JSON object that we hope to extract from the HTML using the schema you are generating.
- Extra Instructions: This is optional instructions to consider when generating the schema provided by the user.
- Query or explanation of target/goal data item: This is a description of what data we are trying to extract from the HTML. This explanation means we're not sure about the rigid schema of the structures we want, so we leave it to you to use your expertise to create the best and most comprehensive structures aimed at maximizing data extraction from this page. You must ensure that you do not pick up nuances that may exist on a particular page. The focus should be on the data we are extracting, and it must be valid, safe, and robust based on the given HTML.
# What if there is no example of target JSON object and also no extra instructions or even no explanation of target/goal data item?
In this scenario, use your best judgment to generate the schema. You need to examine the content of the page and understand the data it provides. If the page contains repetitive data, such as lists of items, products, jobs, places, books, or movies, focus on one single item that repeats. If the page is a detailed page about one product or item, create a schema to extract the entire structured data. At this stage, you must think and decide for yourself. Try to maximize the number of fields that you can extract from the HTML.
# What are the instructions and details for this schema generation?
zJ
HTML to analyze:
```html
z%
```
z4
## Query or explanation of target/goal data item:
z,
## Example of target JSON object:
```json
z
```a IMPORTANT: To remind you, in this process, we are not providing a rigid example of the adjacent objects we seek. We rely on your understanding of the explanation provided in the above section. Make sure to grasp what we are looking for and, based on that, create the best schema..a IMPORTANT: Please remember that in this process, we provided a proper example of a target JSON object. Make sure to adhere to the structure and create a schema that exactly fits this example. If you find that some elements on the page do not match completely, vote for the majority.zIMPORTANT: Since we neither have a query nor an example, it is crucial to rely solely on the HTML content provided. Leverage your expertise to determine the schema based on the repetitive patterns observed in the content.a IMPORTANT:
0/ Ensure your schema remains reliable by avoiding selectors that appear to generate dynamically and are not dependable. You want a reliable schema, as it consistently returns the same data even after many page reloads.
1/ DO NOT USE use base64 kind of classes, they are temporary and not reliable.
2/ Every selector must refer to only one unique element. You should ensure your selector points to a single element and is unique to the place that contains the information. You have to use available techniques based on CSS or XPATH requested schema to make sure your selector is unique and also not fragile, meaning if we reload the page now or in the future, the selector should remain reliable.
3/ Do not use Regex as much as possible.
Analyze the HTML and generate a JSON schema that follows the specified format. Only output valid JSON schema, nothing else.
)promptsr r r )r- r r r r prompt_templatesystem_content user_contents r; _build_schema_promptz2JsonElementExtractionStrategy._build_schema_prompt s 11<1E-Kd ( ), UV[U\ ] ]L NObNcch i iL , {
{L. }
}L 2 @
@L {{NL9::r= r r r( r r c ddl m} t j j D ]# \ } }
t | t
d| d|
t j | ||| } ||j | d|j |j | } t j | j d j j S # t $ r }
t! d t# |
d}
~
ww xY w)
ak
Generate extraction schema from HTML content and optional query (sync version).
Args:
html (str): The HTML content to analyze
query (str, optional): Natural language description of what data to extract
provider (str): Legacy Parameter. LLM provider to use
api_token (str): Legacy Parameter. API token for LLM provider
llm_config (LLMConfig): LLM configuration object
**kwargs: Additional args passed to LLM processor
Returns:
dict: Generated schema following the JsonElementExtractionStrategy format
r )r Nr r Tr r7 r r r r r Failed to generate schema: )rN r r] _GENERATE_SCHEMA_UNWANTED_PROPSr localsr r r r r r& r0 r. r/ r] r5 rX )r- r r r r r r r: r r7 r/ promptr9 r< s r; generate_schemaz-JsonElementExtractionStrategy.generate_schema? s 2 ; : Z Z ` ` b SMD'x~)$y6Gy%QRR S /CCD+W\^qr D6#,,&, "$..#,,!
H ::h . .q 1 9 9 A AB B D 9#a& BC C Ds $A!C C+C&&C+c K ddl m} |
t }t j | ||| } ||j
|d|j |j | d{ }t j |j d j j S 7 :# t $ r } t dt | d} ~ ww xY ww)a
Generate extraction schema from HTML content (async version).
Use this method when calling from async contexts (e.g., FastAPI) to avoid
issues with certain LLM providers (e.g., Gemini/Vertex AI) that require
async execution.
Args:
html (str): The HTML content to analyze
schema_type (str): "CSS" or "XPATH"
query (str, optional): Natural language description of what data to extract
target_json_example (str, optional): Example of desired JSON output
llm_config (LLMConfig): LLM configuration object
**kwargs: Additional args passed to LLM processor
Returns:
dict: Generated schema following the JsonElementExtractionStrategy format
r rL NTr r r )rN rM r) r] r r r r r& r0 r. r/ r] r5 rX )
r- r r r r r: rM r r9 r< s
r; agenerate_schemaz.JsonElementExtractionStrategy.agenerate_schemam s 6 < *,J.CCD+W\^qr D =#,,&, "$..#,,!
H ::h . .q 1 9 9 A AB B D 9#a& BC C Ds: +C/B B 9B CB B?#B::B??C)NN)r NNN)!r6 rU rV rW r4 r rX r r< r rD r rf rg rz r rh r ri r r rO r r r r staticmethodr r) r2 r r r r s @r; r] r] sk > C 4tCH~ 4++&)+
d38n +Z
s
(2(DT62 (:s :d3i :$tCQTH~BV :*
C
C
HI'#
=;3 =;S =; =;be =;qt =; =;~ !#'"3"5 +D+D+D +D ! +D
+D
+D +D
+D +DZ !#'"& ,D,D,D ,D ! ,D
,D
,D ,Dr= r] c z e Zd ZdZdeeef f fd ZdefdZdefdZ defdZ
d efd
Z d efd Z d efd
Z
xZS )JsonCssExtractionStrategya
Concrete implementation of `JsonElementExtractionStrategy` using CSS selectors.
How it works:
1. Parses HTML content with BeautifulSoup.
2. Selects elements using CSS selectors defined in the schema.
3. Extracts field data and applies transformations as defined.
Attributes:
schema (Dict[str, Any]): The schema defining the extraction rules.
verbose (bool): Enables verbose logging for debugging purposes.
Methods:
_parse_html(html_content): Parses HTML content into a BeautifulSoup object.
_get_base_elements(parsed_html, selector): Selects base elements using a CSS selector.
_get_elements(element, selector): Selects child elements using a CSS selector.
_get_element_text(element): Extracts text content from a BeautifulSoup element.
_get_element_html(element): Extracts the raw HTML content of a BeautifulSoup element.
_get_element_attribute(element, attribute): Retrieves an attribute value from a BeautifulSoup element.
r c 0 d|d< t | |fi | y Nr- r1 ri r< r` s r; r< z"JsonCssExtractionStrategy.__init__ !'~
*6*r= ra c t |d S )Nlxmlr+ rr s r; rf z%JsonCssExtractionStrategy._parse_html s \622r= rt c $ |j | S r selectrw s r; rg z,JsonCssExtractionStrategy._get_base_elements s !!(++r= c $ |j | S r r ry s r; rz z'JsonCssExtractionStrategy._get_elements s ~~h''r= r? c & |j d S )NTr )get_textr s r; r z+JsonCssExtractionStrategy._get_element_text s d ++r= c t | S r )rX r s r; r z+JsonCssExtractionStrategy._get_element_html s 7|r= r c $ |j | S r r8 r s r; r z0JsonCssExtractionStrategy._get_element_attribute {{9%%r= )r6 rU rV rW r rX r r< rf rg rz r r r r r s @r; r r sd *+tCH~ +3 3, ,(s (
,C , C & &r= r c e Zd Zdeeef f fd ZdefdZd Zd Z d Z
d Z d Z d
Z
d efd Zd efd
ZdefdZdefdZdefdZd Z xZS )JsonLxmlExtractionStrategyr c d|d< t | |fi | i | _ i | _ i | _ |j dd | _ |j dd | _ ddlm }m
} ddl m } || _ || _
|| _ y ) Nr- r1 use_cachingToptimize_common_patternsr )r. r- CSSSelector)ri r< _selector_cache _xpath_cache
_result_cacher8 r r r r. r- lxml.cssselectr html_parser)r9 r r: r. r- r r5 s r; r< z#JsonLxmlExtractionStrategy.__init__ s !'~
*6* ! "::mT:(.
3Mt(T% %.
&r= ra c | j j dd }| j j || S # t $ r}| j rt d| | j j | cY d}~S # t $ rD}| j rt d| | j j d cY d}~cY d}~S d}~ww xY wd}~ww xY w)z&Parse HTML content with error recoveryT)recoverremove_blank_textz8Error parsing HTML, falling back to alternative method: Nz Critical error parsing HTML: r- )r.
HTMLParser
fromstringr5 r3 rx r Element)r9 ra parserr< e2s r; rf z&JsonLxmlExtractionStrategy._parse_html s 2ZZ**44*PF::(( v> > 2||PQRPSTU
2''22<@@
2<< 9">?zz))&111
2 2sK 8; CC A?9C? C 4C<C =CCC CCc P | j s|S ddl} |j d| r|S |j }t | dk r|S t | dkD rKt d |D r9|D cg c]' }|j
d s|j
d s&|) }}|r|d S |S c c}w )
z8Optimize common selector patterns for better performancer Nztd:nth-child\(\d+\)r r c 3 d K | ]( }|j d xs |j d * yw).#N)rG ).0ps r; z@JsonLxmlExtractionStrategy._optimize_selector.. s* !XQ!,,s"3"Hq||C7H"H!Xs .0r r )r r r r r anyrG )r9 selector_strr partsr specific_partss r; _optimize_selectorz-JsonLxmlExtractionStrategy._optimize_selector s ,, 299+\ : ""$ u:? u:>c!XRW!X X). YA!,,s2Cq||TWGXa YN Y %b)) Zs ,'B#B#c * j r j j j j < d fd }|S # t
$ r+} j rt d d| ddcY d}~S d}~ww xY w)z6Create a selector function that handles all edge casesc d }j rL| j dd xs t t | }| d
}|j v rj | S g } | }|s|rj | }|r| j
| }|sd v rj | }|sj | }|sT j }|rB|d }t j d| }|r%|j d } | j
d| }j r|r|j |< |S # t $ r(}
j rt d
d
|
Y d }
~
|S d }
~
ww xY w) Nid z:: nth-childr z^(\w+)r .//Error applying selector '': )r r8 rX hashr _make_context_sensitive_xpathxpath_handle_nth_child_selector_fallback_class_id_searchr r r r r5 r3 rx )rm context_sensitive cache_key
element_idrT
context_xpathr last_part tag_matchtag_namer< compiledoriginal_selectorr r9 r s r;
selector_funczKJsonLxmlExtractionStrategy._create_selector_function..selector_func s ##!(T2!6!L#d7m:LJ#-,b ?I D$6$66 # 1 1) << $P &w /G #'8(,(J(J5RY(Z
(&-mmM&BG ';:K+K&*&E&EgO`&aG '&*&D&DWN_&`G '$5$;$;$=E $,1"I ,.HHY ,J #,/8q/AH.5mmc(.R s 2 r= N)T)r r r pathr r5 r3 rx )r9 r r r< r r r s `` @@@r; _create_selector_functionz4JsonLxmlExtractionStrategy._create_selector_function s ( ( ( 22<@LE >'' 5H MME /4D l +3
3
j ! >||2<.A3GH > = >s 7A B' B
B
Bc |j d r|S |j d rd| }nd| } |j | |S # d|j d d cY S xY w# Y yxY w)z1Convert absolute XPath to context-sensitive XPathdescendant-or-self::/r r r N)rG r r )r9 r rm r s r; r z8JsonLxmlExtractionStrategy._make_context_sensitive_xpathT s 67 $"#E7
"%eW
4
m,$$
4 U[[ -b 1 233 s' A# A# A A A# A# #A'c ddl }g } |j d| }|r|j d }|j d| dd d j }|rA |j d| }|r|j d nd } |j d
| d | }|S |j d
| d }|S # t $ r%}
| j rt d
|
Y d}
~
|S d}
~
ww xY w)z2Special handling for nth-child selectors in tablesr Ntd:nth-child\((\d+)\)r z
td:nth-child()r z(\w+)r .//td[]//]z#Error handling nth-child selector: ) r r r r r r r5 r3 rx ) r9 rm r r rT r col_numremaining_selectorr r r< s r; r z5JsonLxmlExtractionStrategy._handle_nth_child_selectork s A BII 6 EE ++a. &2%7%7-yPQ8RTU%VWY%Z%`%`%b"% !* (4F GI5>yq 1CH %mmfWIS
,K LG &mmfWIQ,? @G
A||;A3?@ As BB/ B/ / C 8CC c x g } ddl } |j d| } |j d| }|D ]( }|j d| d }|j | * |D ]( } |j d| d }
|j |
* |S # t $ r%} | j
rt
d | Y d} ~ |S d} ~ ww xY w)
z!Fallback to search by class or IDr Nz\.([a-zA-Z0-9_-]+)z#([a-zA-Z0-9_-]+)z.//*[contains(@class, 'z')]z
.//*[@id='z']z#Error in fallback class/id search: )r findallr rH r5 r3 rx ) r9 rm r rT r
class_matches
id_matches
class_name
class_resultsid_name
id_resultsr< s r; r z4JsonLxmlExtractionStrategy._fallback_class_id_search s A &BJJ'Get child elements using the selector with context sensitivityTr$ r% r9 rm rt r s r; rz z(JsonLxmlExtractionStrategy._get_elements s **84
W==r= r? c
dj d |j d D }|S # t $ rN}| j rt d| |j j
cY d}~S # Y Y d}~yxY wd}~ww xY w)z$Extract normalized text from elementr c 3 ^ K | ]% }|j s|j ' y wr r )r r s r; r z?JsonLxmlExtractionStrategy._get_element_text.. s W!QWWYAGGIWs -- .//text()zError extracting text: Nr )r r r5 r3 rx text_contentr )r9 rm r r< s r; r z,JsonLxmlExtractionStrategy._get_element_text sv 88Ww}}[/IWWDK ||/s34
++-3355
s2 (+ BA= A2,B2A:4A=:A==Bc | j j |dd S # t $ r$}| j rt d| Y d}~yd}~ww xY w)z)Get HTML string representation of elementunicoder- )encodingr zError serializing HTML: Nr )r. tostringr5 r3 rx )r9 rm r< s r; r z,JsonLxmlExtractionStrategy._get_element_html sM ::&&w6&R R ||045 s A
AA
r c |j | S # t $ r'}| j rt d| d| Y d}~yd}~ww xY w)zGet attribute value safelyzError getting attribute 'r N)r8 r5 r3 rx )r9 rm r r< s r; r z1JsonLxmlExtractionStrategy._get_element_attribute sF ;;y) ) ||1)CsCD s A >Ac R | j r| j j yy)zClear caches to free memoryN)r r clear)r9 s r;
_clear_cachesz(JsonLxmlExtractionStrategy._clear_caches s" $ $ & r= )r6 rU rV r rX r r< rf r r
r r r r" rg rz r r r r4 r r s @r; r r s 'tCH~ '$2 2 .M>^.:62 C C >s >
C
C 'r= r c | e Zd Zdeeef f fd ZdefdZd ZdefdZ defdZ
d efd
Z d efd Z d efd
Z
xZS ) JsonLxmlExtractionStrategy_naiver c > d|d< t | |fi | i | _ y r )ri r< r r` s r; r< z)JsonLxmlExtractionStrategy_naive.__init__ s' !'~
*6* ! r= ra c Z ddl m} |j d } |j || S )Nr r. T)r )r r. r r )r9 ra r. r s r; rf z,JsonLxmlExtractionStrategy_naive._parse_html s, !!!$/ u f55r= c j vr%ddlm} | fd}| j < j S # t $ r9} j rt d d| d }| j < Y d}~Ld}~ww xY w)zCGet a selector function that works within the context of an elementr r c
| }|r|S
j }|j d r|}nd|j d }| j | }|r|S d v r{dd l} |j
d }|rb|j
d } j dd d j }|r| j d
| d | S | j d
| d S j }t | dkD r |d r| j d
|d S g S # t $ r)} j rt d d| g cY d } ~ S d } ~ ww xY w)Nr z./r r r r r r r r r r r r r )
r rG lstripr r r r r r r r5 r3 rx )
rm rT r r r r r sub_selectorr r< r r r9 s
r; select_funczCJsonLxmlExtractionStrategy_naive._get_selector..select_func sb +""*7"3"#*N !)
!++,BC,1M /1 c1B0C,DM")--
">"#*N ',6 %$-BII.F $UE $*/++a./;/A/A#q/I"/M/S/S/U #/+2==6'#l^9\+]$]+2==6'!9L+M$M !- 2 2 4 u:>eBi#*==3uRyk1B#C C ! $ " << !$=l^3qc"R S ! "s< D AD A)D D 9D D E E :E Er
r c g S r rA )rm s r;
fallback_funczEJsonLxmlExtractionStrategy_naive._get_selector..fallback_func" s Ir= N)r r r r5 r3 rx )r9 r r r> r< r@ r s `` @r; r" z.JsonLxmlExtractionStrategy_naive._get_selector s t 3 3 3 2<
C&|4,"\ 6A$$\2 ##L11
C<< 6|nCsKL 6C$$\2
Cs A B/BBrt c 4 | j | } || S r r% r& s r; rg z3JsonLxmlExtractionStrategy_naive._get_base_elements) s **84
[))r= c 4 | j | } || S r r% r( s r; rz z.JsonLxmlExtractionStrategy_naive._get_elements- s **84
W%%r= r? c ^ dj |j d j S Nr r+ r r r r s r; r z2JsonLxmlExtractionStrategy_naive._get_element_text1 # www}}[1288::r= c 6 ddl m} |j |d S )Nr r9 r. r/ )r r. r0 )r9 rm r. s r; r z2JsonLxmlExtractionStrategy_naive._get_element_html4 s u~~g ::r= r c $ |j | S r r r s r; r z7JsonLxmlExtractionStrategy_naive._get_element_attribute8 r r= )r6 rU rV r rX r r< rf r" rg rz r r r r r s @r; r6 r6 sf "tCH~ "
6 6
B2H* *&s &;C ;;C ;& &r= r6 c e Zd ZdZdeeef f fd ZdefdZdefdZ ded efd
Z
ded efd Z defd Z d efd
Z
d efdZdefdZ xZS )JsonXPathExtractionStrategya
Concrete implementation of `JsonElementExtractionStrategy` using XPath selectors.
How it works:
1. Parses HTML content into an lxml tree.
2. Selects elements using XPath expressions.
3. Converts CSS selectors to XPath when needed.
Attributes:
schema (Dict[str, Any]): The schema defining the extraction rules.
verbose (bool): Enables verbose logging for debugging purposes.
Methods:
_parse_html(html_content): Parses HTML content into an lxml tree.
_get_base_elements(parsed_html, selector): Selects base elements using an XPath selector.
_css_to_xpath(css_selector): Converts a CSS selector to an XPath expression.
_get_elements(element, selector): Selects child elements using an XPath selector.
_get_element_text(element): Extracts text content from an lxml element.
_get_element_html(element): Extracts the raw HTML content of an lxml element.
_get_element_attribute(element, attribute): Retrieves an attribute value from an lxml element.
r c 0 d|d< t | |fi | y r r r` s r; r< z$JsonXPathExtractionStrategy.__init__R r r= ra c , t j | S r )r- r rr s r; rf z'JsonXPathExtractionStrategy._parse_htmlV s |,,r= rt c $ |j | S r )r rw s r; rg z.JsonXPathExtractionStrategy._get_base_elementsY s **r= css_selectorr? c 0 d|v r|S | j | S )z'Convert CSS selector to XPath if neededr )_basic_css_to_xpath)r9 rO s r;
_css_to_xpathz)JsonXPathExtractionStrategy._css_to_xpath\ s! , '' 55r= c d|v r%|j d }ddj | z S d|v r%|j d }ddj | z S d|z S )z.Basic CSS to XPath conversion for common casesz > z//r r )r r )r9 rO r s r; rQ z/JsonXPathExtractionStrategy._basic_css_to_xpathb sd L &&u-E#((5/) ) , &&s+E$))E** *l""r= c r | j | }|j d sd|z }|j | S )Nr )rR rG r )r9 rm rt r s r; rz z)JsonXPathExtractionStrategy._get_elementsl s8 ""8, $%KE}}U##r= c ^ dj |j d j S rD rE r s r; r z-JsonXPathExtractionStrategy._get_element_textr rF r= c 0 t j |d S )Nr. rH )r. r0 r s r; r z-JsonXPathExtractionStrategy._get_element_htmlu s ~~g ::r= r c $ |j | S r r r s r; r z2JsonXPathExtractionStrategy._get_element_attributex r r= )r6 rU rV rW r rX r r< rf rg rR rQ rz r r r r r s @r; rK rK ; s ,+tCH~ +- -+ +6# 6# 6 # # #$s $ ;C ;;C ;& &r= rK z
z\x02xz\x08z(?Fix common JSON-escape goofs coming from LLMs or manual edits.z\\b\\\\z Regex for 'u ' won’t compile after fix: N)
r _WB_FIXsub translate_CTRL
_NEEDS_ESCAPEr compiler
ValueError)r safer patr< s r; _sanitize_schemare s
D lln
skk&#&007 - ^ JJsO U
K xx ^ {5'1Nqc RSY] ] ^s A77B!
B B!c e Zd ZU dZ G d de Zej Zej Z ej Z ej Z
ej Zej Zej$ Zej( Zej, Zej0 Zej4 Zej8 Z ej< Z ej@ Z!ejD Z#ejH Z%ejL Z'ejP Z)ejT Z+ejX Z-ej\ Z/ej` Z1 ed Z2i ddddd d
d d d
ddddddddddddddd d d d d d!d"d#d$d%d&d'd(d)d*d+Z3e4e5e5f e6d,< e7jp e7jr z Z:d-d-d.Z;ejx fd/d0d1d2dd3e=e>e4e5e5f e?e@e5e5f f d4e5d5d/f fd6ZAd e5d7e5d5e?e4e5eBf fd8ZCd7e5d5e5fd9ZDeEd/d/d/d:d;e5de=e?e5 d?e=eF d5e4e5e5f f d@ ZG xZHS )ARegexExtractionStrategya
A lean strategy that finds e-mails, phones, URLs, dates, money, etc.,
using nothing but pre-compiled regular expressions.
Extraction returns::
{
"url": "",
"label": "",
"value": "",
"span": [start, end]
}
Only `generate_schema()` touches an LLM, extraction itself is pure Python.
c e Zd Z e Z e Z e Z e Z e Z e Z e Z
e Z e Z e Z
e Z e Z e Z e Z e Z e Z e Z e Z e Z e Z e Z e Zeez ez ez ez e z e
z e z e z e
z ez ez ez ez ez ez ez ez ez ez ez Zy)RegexExtractionStrategy._BN)r6 rU rV r EMAIL
PHONE_INTLPHONE_USr IPV4IPV6UUIDCURRENCY
PERCENTAGENUMBERDATE_ISODATE_USTIME_24H POSTAL_US POSTAL_UKHTML_COLOR_HEXTWITTER_HANDLEHASHTAGMAC_ADDRIBAN CREDIT_CARDNOTHINGALLrA r= r; _Bri s= & &
& & & & & & &
& & & & & & & & & & & & & J )C /$ 6 = D
#
$&,
-/7
8:A
BDL
M
#
$&4
57E
F
!
! $(
( +6
6 r= r r emailz[\w.+-]+@[\w-]+\.[\w.-]+
phone_intlz\+?\d[\d .()-]{7,}\dphone_usz#\(?\d{3}\)?[ -. ]?\d{3}[ -. ]?\d{4}r> zhttps?://[^\s\"'<>]+ipv4z(?:\d{1,3}\.){3}\d{1,3}ipv6z"[A-F0-9]{1,4}(?::[A-F0-9]{1,4}){7}uuidzG[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}currencyu, (?:USD|EUR|RM|\$|€|£)\s?\d+(?:[.,]\d{2})?
percentagez\d+(?:\.\d+)?%numberz%\b\d{1,3}(?:[,.\s]\d{3})*(?:\.\d+)?\bdate_isoz\d{4}-\d{2}-\d{2}date_usz\d{1,2}/\d{1,2}/\d{2,4}time_24hz.\b(?:[01]?\d|2[0-3]):[0-5]\d(?:[:.][0-5]\d)?\b postal_usz\b\d{5}(?:-\d{4})?\b postal_ukz$\b[A-Z]{1,2}\d[A-Z\d]? ?\d[A-Z]{2}\bhtml_color_hexz#[0-9A-Fa-f]{6}\btwitter_handlez @[\w]{1,15}z#[\w-]+z$(?:[0-9A-Fa-f]{2}:){5}[0-9A-Fa-f]{2}z [A-Z]{2}\d{2}[A-Z0-9]{11,30}zH\b(?:4\d{12}(?:\d{3})?|5[1-5]\d{14}|3[47]\d{13}|6(?:011|5\d{2})\d{12})\b)hashtagmac_addriban credit_cardDEFAULT_PATTERNSzUse llm_config insteadr Nfit_html)customr1 r r r1 r? c ( t | dd|i| | j j D ci c]9 \ }}t | j
|j
j |z r||; }}}|rGt |t r|j | n%|j |D ci c] \ }}||
c}} |j D ci c]& \ }}|t j || j ( c}}| _ yc c}}w c c}}w c c}}w )a
Args:
patterns: Custom patterns overriding or extending defaults.
Dict[label, regex] or list[tuple(label, regex)].
input_format: "html", "markdown" or "text".
**kwargs: Forwarded to ExtractionStrategy.
r1 NrA )ri r< r r getattrr r r r1 r2 rj r ra _FLAGS _compiled)
r9 r r r1 r: r rxmergedlblr5 s
r; r< z RegexExtractionStrategy.__init__ s =l=f=
00668"
Rtww ,22W<
G"
"
&$'
f%
f =73sBw => =CLLN.
18bCB , ,.
"
>.
s >D%
D
+Dr] c
g }| j j D ]^ \ }}|j | D ]E }|j |||j d |j |j
gd G ` |S )Nr )r> r r span)r r finditerr r startend) r9 r> r] rC kwoutr crems r; rD z RegexExtractionStrategy.extract4 s| $& ....0 JE3\\'*
"!&!"!"AEEG 4
r= c ^ | j dk( r|S t |d j dd S )Nr r r Tr )r1 r, r )r9 r] s r; _plain_textz#RegexExtractionStrategy._plain_textG s2 &N Wf-66s$6GGr= )r examplesr r r- r r r c t j D ]' }||v st | dt j | |
t }d| d| d}d|dd dg}|r"|j d |j |r&|j d
d j
|dd z d
j
| } t |j d
j
|| g d|j |j | }
|
j d j j } | j dd } t j dd| } t# j$ | } | j+ D ] \ }} t j, | | S # t& $ r}
t) d| |
d}
~
ww xY w# t j. $ r}t) d| d| dd}~ww xY w)u
Ask an LLM for a single page-specific regex and return
{label: pattern} ── ready for RegexExtractionStrategy(custom=…)
z is deprecated, NzaYou are an expert Python-regex engineer.
Return **one** JSON object whose single key is exactly "u ", and whose value is a raw-string regex pattern that works with the standard `re` module in Python.
Strict rules (obey every bullet):
• If a *user query* is supplied, treat it as the precise semantic target and optimise the pattern to capture ONLY text that answers that query. If the query conflicts with the sample HTML, the HTML wins.
• Tailor the pattern to the *sample HTML* – reproduce its exact punctuation, spacing, symbols, capitalisation, etc. Do **NOT** invent a generic form.
• Keep it minimal and fast: avoid unnecessary capturing, prefer non-capturing `(?: … )`, and guard against catastrophic backtracking.
• Anchor with `^`, `$`, or `\b` only when it genuinely improves precision.
• Use inline flags like `(?i)` when needed; no verbose flag comments.
• Output must be valid JSON – no markdown, code fences, comments, or extra keys.
• The regex value must be a Python string literal: **double every backslash** (e.g. `\\b`, `\\d`, `\\\\`).
Example valid output:
{"z5": "(?:RM|rm)\\s?\\d{1,3}(?:,\\d{3})*(?:\\.\\d{2})?"}z```htmli z```z
## Query
z ## Examples
r^ r r Tr r z\bz(? ? o (3B-0H H I;;z* / (("(++z8.D"E ** ((
ll1o%%--kk&%(ff+Wc: O ::c?L
%**, NHC
N
3 N O >se DE3 N O 88
N #6se3qc!BCM
Ns0 F .F' F$F F$'G:G G)Ir6 rU rV rW r r rj Emailrk PhoneIntlrl PhoneUSr Urlrm IPv4rn IPv6ro Uuidrp Currencyrq
Percentagerr Numberrs DateIsort DateUSru Time24hrv PostalUSrw PostalUKrx HexColorry
TwitterHandlerz Hashtagr{ MacAddrr| Ibanr}
CreditCardr AllNothingr r rX __annotations__r
IGNORECASE MULTILINEr r r~ r r r r r< r rD r r r( r r r s @r; rg rg s, &
W
> XXE]]I[[GVVCWWDWWDWWD[[H]]JYYF[[GZZF[[G\\H\\H&&H&&MZZG[[GWWD^^JVVCUG
(6 ( 2 ( A ( 2
( 5 ( @ ( e ( J ( , ( C ( /! (" 5# ($ L% (( 2) (* B+ (, /- (. >/ (0 &B:f7 (d38n < ]]R\\
)F,-O
!
JN & !
!
tCH~tE#s(O/D DEF !
!
!
L3 4S#X;O &H3 H3 H
$(,*.
S S S } S
49% S Y'
S
c3h S S r= rg )Pabcr r r typingr r r r r r
r concurrent.futuresr r
r& rq enumr r r r r r r r configr r r r r rN r r r r r r! r" modelsr# model_loaderr$ r% r& r' typesr( r) functoolsr* rj r r bs4r, r r- r. r0 rZ rd r r] r r r6 rK mapchrr ordr_ ra r\ r` rX re rg )cs 0r; r s # C C C ? v v 0 <N <N~
-
.`D' `DL p . plwD$6 wDr
-& = -&^M'!> M'^_&'D _&B>&"? >&@
),Cr(;Q1q?Pr#a& Q
"**W
01
T#s(^ S#X (M 0 M 3 Rs E=)E=