iNfdZddlmZddlmZddlmZmZmZddl m Z e GddeZ dgZ y) z Processor class for MarkupLM. ) TensorType)ProcessorMixin) BatchEncodingPaddingStrategyTruncationStrategy)auto_docstringc eZdZdZfdZe ddedeezezdeeze zde dzde d e dzd edzd edzd ed ededededee zdzde fdZ xZS)MarkupLMProcessorTc&t|||y)N)super__init__)selffeature_extractor tokenizer __class__s l/mnt/e/genesis-system/.venv/lib/python3.12/site-packages/transformers/models/markuplm/processing_markuplm.pyr zMarkupLMProcessor.__init__s *I6Nadd_special_tokenspadding truncation max_lengthstridepad_to_multiple_ofreturn_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping return_lengthverbosereturn_tensorsreturnc |jr:| td||| td|j|}|d}|d}n| td|| td||jrt|tr|g}|j did||n|d ||ndd|d |d |d |d |d| d| d| d| d| d|d|d|d|d|d||}|S)a html_strings (`str` or `list[str]`, *optional*): Raw HTML strings to parse and process. When `parse_html=True` (default), these strings are parsed to extract nodes and xpaths automatically. If provided, `nodes`, `xpaths`, and `node_labels` should not be provided. Required when `parse_html=True`. nodes (`list[list[str]]`, *optional*): Pre-extracted HTML nodes as a list of lists, where each inner list contains the text content of nodes for a single document. Required when `parse_html=False`. Should not be provided when `parse_html=True`. xpaths (`list[list[str]]`, *optional*): Pre-extracted XPath expressions corresponding to the nodes. Should be a list of lists with the same structure as `nodes`, where each XPath identifies the location of the corresponding node in the HTML tree. Required when `parse_html=False`. Should not be provided when `parse_html=True`. node_labels (`list[list[int]]`, *optional*): Labels for the nodes, typically used for training or fine-tuning tasks. Should be a list of lists with the same structure as `nodes`, where each label corresponds to a node. Optional and only used when `parse_html=False`. questions (`str` or `list[str]`, *optional*): Question strings for question-answering tasks. When provided, the tokenizer processes questions as the first sequence and nodes as the second sequence (text_pair). If a single string is provided, it is converted to a list to match the batch dimension of the parsed HTML. NzDMake sure to pass HTML strings in case `parse_html` is set to `True`zUPlease don't pass nodes, xpaths nor node labels in case `parse_html` is set to `True`nodesxpathsz@You have passed HTML strings but `parse_html` is set to `False`.zIMake sure to pass nodes and xpaths in case `parse_html` is set to `False`text text_pair node_labelsrrrrrrrrrrrrr r!) parse_html ValueErrorr isinstancestrr)r html_stringsr$r%r( questionsrrrrrrrrrrrrr r!kwargsfeaturesencoded_inputss r__call__zMarkupLMProcessor.__call__s\ ??# !ghh F$6+:Q k--l;HW%Eh'F' !cdd} !lmm  T__)S)&K ' '3 (4e$  $  2    " "   2 #8 #8 '@ (B $: (! "# $*' ,r)NNNNNTFNNNNNFFFFTN)__name__ __module__ __qualname__r*r rboolr-rrintrrr3 __classcell__)rs@rr r s0J7#'056:!%)--1-1*/+0',#26)Z!Zo-Z3J!33Z$JZZ $JZ $d{Z $d{Z$(Z %)!Z"!%#Z$%Z&'Z(j(4/)Z, -ZZrr N) __doc__ file_utilsrprocessing_utilsrtokenization_utils_baserrrutilsrr __all__r)rrrAsD%.YY#aaaH  r