ieTdZddlmZddlZddlmZddlmZmZmZddl m Z ddl m Z dd lmZdd lmZmZmZmZmZmZdd lmZmZdd lmZdd lmZddlmZm Z m!Z!m"Z"ddl#m$Z$e"jJe&Z'GddejPZ)GddejPZ*GddejPZ+GddejPZ,GddejPZ-GddejPZ.GddejPZ/Gdd ejPZ0Gd!d"ejPZ1 dAd#ejPd$ejdd%ejdd&ejdd'ejddzd(e3d)e3fd*Z4Gd+d,ejPZ5Gd-d.ejPZ6Gd/d0eZ7Gd1d2ejPZ8e Gd3d4eZ9e Gd5d6e9Z:e Gd7d8e9Z;e d9:Gd;dd?e9Z=gd@Z>y)BzPyTorch MarkupLM model.)CallableN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward)TransformersKwargsauto_docstringcan_return_tuplelogging)MarkupLMConfigc*eZdZdZfdZddZxZS)XPathEmbeddingszConstruct the embeddings from xpath tags and subscripts. We drop tree-id in this version, as its info can be covered by xpath. c t||j|_tj|j |jz|j |_tj|j|_ tj|_ tj|j |jzd|j z|_ tjd|j z|j |_tjt!|jDcgc],}tj"|j$|j .c}|_tjt!|jDcgc],}tj"|j(|j .c}|_ycc}wcc}w)N)super__init__ max_depthrLinearxpath_unit_hidden_size hidden_sizexpath_unitseq2_embeddingsDropouthidden_dropout_probdropoutReLU activationxpath_unitseq2_inner inner2emb ModuleListrange Embeddingmax_xpath_tag_unit_embeddingsxpath_tag_sub_embeddingsmax_xpath_subs_unit_embeddingsxpath_subs_sub_embeddings)selfconfig_ __class__s j/mnt/e/genesis-system/.venv/lib/python3.12/site-packages/transformers/models/markuplm/modeling_markuplm.pyr!zXPathEmbeddings.__init__1sZ )))+63P3PSWSaSa3acicucu)v&zz&"<"<= '')$&IIf.K.Kdnn.\^_bhbtbt^t$u!1v'9'9#96;M;MN(* t~~.  VAA6C`C`a ) %*,t~~.  VBBFDaDab * &   s 11G1Gc g}g}t|jD]^}|j|j||dddd|f|j|j||dddd|f`t j |d}t j |d}||z}|j|j|j|j|}|S)Ndim) r/r"appendr2r4torchcatr-r)r+r,)r5xpath_tags_seqxpath_subs_seqxpath_tags_embeddingsxpath_subs_embeddingsixpath_embeddingss r9forwardzXPathEmbeddings.forwardKs " "t~~& eA ! ( ()I)F)Fq)I.YZ\]_`Y`Ja)b c ! ( ()J)G)G)J>Z[]^`aZaKb)c d e!& *?R H % *?R H03HH>>$,,ttG`G`aqGr7s*tu)NN)__name__ __module__ __qualname____doc__r!rG __classcell__r8s@r9rr+s  4 rHrcXeZdZdZfdZedZeddZ ddZxZ S)MarkupLMEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.cdt|||_tj|j |j |j|_tj|j|j |_ |j|_ t||_ tj|j|j |_tj |j |j"|_tj$|j&|_|j+dt-j.|jj1dd|j|_tj|j|j |j2|_ y)N) padding_idxeps position_idsrr;F) persistent)r r!r6rr0 vocab_sizer% pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingsr"rrFtype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsr'r(r)register_bufferr?arangeexpandrRr5r6r8s r9r!zMarkupLMEmbeddings.__init__`s:  !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c )) / 7%'\\&2H2H&J\J\%]"f&8&8f>S>STzz&"<"<=   ELL)G)GHOOPWXej  "..#%<<  * *F,>,>DL\L\$  rHc|jdd}|d}tj|dz||zdztj|j}|j dj |S)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor Nr;rdtypedevicer)sizer?rblongrh unsqueezerc) inputs_embedsrR input_shapesequence_lengthrUs r9&create_position_ids_from_inputs_embedsz9MarkupLMEmbeddings.create_position_ids_from_inputs_embedsxsp$((*3B/ %a.|| !O_{:Q>ejjYfYmYm %%a(// <>*::;5::V\7N  !![[44uzzd;'4>>*::;5::V\8N)"66|D $ : :> J00P%(;;>SSVff ^^J/ \\*- rH)r)NNNNNN) rIrJrKrLr! staticmethodroryrGrMrNs@r9rPrP]sLQ 0==" 8 8"1rHrPcneZdZfdZdej dej dej fdZxZS)MarkupLMSelfOutputc(t|tj|j|j|_tj |j|j|_tj|j|_ yNrS) r r!rr#r%denser_r`r'r(r)rds r9r!zMarkupLMSelfOutput.__init__s` YYv1163E3EF f&8&8f>S>STzz&"<"<= rH hidden_states input_tensorreturncr|j|}|j|}|j||z}|SNrr)r_r5rrs r9rGzMarkupLMSelfOutput.forward7 =1  ]3 }|'CD rHrIrJrKr!r?TensorrGrMrNs@r9rr1> U\\RWR^R^rHrcVeZdZfdZdej dej fdZxZS)MarkupLMIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yr) r r!rr#r%intermediate_sizer isinstance hidden_actstrr intermediate_act_fnrds r9r!zMarkupLMIntermediate.__init__s] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $rHrrcJ|j|}|j|}|Sr)rrr5rs r9rGzMarkupLMIntermediate.forwards& =1 00? rHrrNs@r9rrs#9U\\ellrHrcneZdZfdZdej dej dej fdZxZS)MarkupLMOutputc(t|tj|j|j |_tj|j |j|_tj|j|_ yr) r r!rr#rr%rr_r`r'r(r)rds r9r!zMarkupLMOutput.__init__s` YYv779K9KL f&8&8f>S>STzz&"<"<= rHrrrcr|j|}|j|}|j||z}|Srrrs r9rGzMarkupLMOutput.forwardrrHrrNs@r9rrrrHrcVeZdZfdZdej dej fdZxZS)MarkupLMPoolerct|tj|j|j|_tj |_yr)r r!rr#r%rTanhr+rds r9r!zMarkupLMPooler.__init__s9 YYv1163E3EF '')rHrrc\|dddf}|j|}|j|}|S)Nr)rr+)r5rfirst_token_tensor pooled_outputs r9rGzMarkupLMPooler.forwards6+1a40 #56  6 rHrrNs@r9rrs#$ U\\ellrHrcVeZdZfdZdej dej fdZxZS)MarkupLMPredictionHeadTransformcht|tj|j|j|_t |jtrt|j|_ n|j|_ tj|j|j|_ yr) r r!rr#r%rrrrr transform_act_fnr_r`rds r9r!z(MarkupLMPredictionHeadTransform.__init__s{ YYv1163E3EF f'' -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STrHrrcl|j|}|j|}|j|}|Sr)rrr_rs r9rGz'MarkupLMPredictionHeadTransform.forwards4 =1 --m< }5 rHrrNs@r9rrs$UU\\ellrHrc$eZdZfdZdZxZS)MarkupLMLMPredictionHeadct|t||_t j |j |jd|_t jtj|j|_ y)NT)bias) r r!r transformrr#r%rXdecoder Parameterr?r{rrds r9r!z!MarkupLMLMPredictionHead.__init__"s[ 8@yy!3!3V5F5FTR LLV->->!?@ rHcJ|j|}|j|}|Sr)rrrs r9rGz MarkupLMLMPredictionHead.forward+s$}5  ]3 rH)rIrJrKr!rGrMrNs@r9rr!sArHrcVeZdZfdZdej dej fdZxZS)MarkupLMOnlyMLMHeadcBt|t||_yr)r r!r predictionsrds r9r!zMarkupLMOnlyMLMHead.__init__3s 3F;rHsequence_outputrc(|j|}|Sr)r)r5rprediction_scoress r9rGzMarkupLMOnlyMLMHead.forward7s ,,_=  rHrrNs@r9rr2s#<!u||! !rHrmodulequerykeyvalueattention_maskscalingr)c tj||jdd|z}|||z}tjj |dtj j|j}tjj|||j}tj||} | jddj} | |fS)Nrr;)r=rg)ptrainingr) r?matmul transposer functionalsoftmaxfloat32torgr)r contiguous) rrrrrrr)kwargs attn_weights attn_outputs r9eager_attention_forwardr=s<<s}}Q':;gEL!#n4 ==((2U]](SVVW\WbWbcL==((6??([L,,|U3K''1-88:K  $$rHc eZdZfdZ d dej dej dzdedzdee de ej f dZ xZ S) MarkupLMSelfAttentionc$t||j|jzdk7r2t |ds&t d|jd|jd||_|j|_t|j|jz |_|j|jz|_ tj|j|j|_ tj|j|j|_ tj|j|j|_tj|j |_|j |_|jdz|_y)Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()g)r r!r%num_attention_headshasattr ValueErrorr6rrattention_head_size all_head_sizerr#rrrr'attention_probs_dropout_probr)attention_dropoutrrds r9r!zMarkupLMSelfAttention.__init__UsC     : : :a ?PVXhHi#F$6$6#78 445Q8   #)#=#= #&v'9'9F>> from transformers import AutoProcessor, MarkupLMModel >>> processor = AutoProcessor.from_pretrained("microsoft/markuplm-base") >>> model = MarkupLMModel.from_pretrained("microsoft/markuplm-base") >>> html_string = " Page Title " >>> encoding = processor(html_string, return_tensors="pt") >>> outputs = model(**encoding) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 4, 768] ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer;z5You have to specify either input_ids or inputs_embeds)rhrfrr)rgg?g)rurArBrUrrlT)rr r r)r pooler_outputrr)r6rr use_return_dictr%warn_if_padding_and_no_attention_maskrirhr?r}r{rjrkrrgrr%r&r rr)r5rurArBrrrUrlrr r rrmrhextended_attention_maskembedding_outputencoder_outputsrrs r9rGzMarkupLMModel.forward"sN2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y. Q#..*K  &',,.s3KTU U%.%:!! @T@T  !"ZZ FCN  !"[[EJJvVN"0":":1"="G"G"J"9"<"<4::"<"N#&)@#@H"L??))%)' + ,,  #/!5 ' *!,8< 8O O4UY )-')77&11   rH)T) NNNNNNNNNN)rIrJrKr!r+r-rrr? LongTensorrrr~r rGrMrNs@r9r#r# s& /0.2262637260426)-,0#'X ##d*X ((4/X ((4/ X ))D0 X ((4/ X &&-X ((4/X  $;X #TkX D[X  + +X X rHr#ceZdZfdZee ddejdzdejdzdejdzdejdzdejdzdejdzd ejdzd ejdzd ejdzd edzd edzdedzde eje zfdZ xZ S)MarkupLMForQuestionAnsweringct|||j|_t|d|_t j |j|j|_|jyNF)r() r r! num_labelsr#rrr#r% qa_outputsr'rds r9r!z%MarkupLMForQuestionAnswering.__init__sU   ++%fF ))F$6$68I8IJ rHNrurArBrrrUrlstart_positions end_positionsrr r rc | | n|jj} |j|||||||| | d }|d}|j|}|j dd\}}|j dj }|j dj }d}|| t|jdkDr|j d}t| jdkDr| j d} |jd}|jd|| jd|t|}|||}||| }||zd z }t||||j|j S) ae xpath_tags_seq (`torch.LongTensor` of shape `(batch_size, sequence_length, config.max_depth)`, *optional*): Tag IDs for each token in the input sequence, padded up to config.max_depth. xpath_subs_seq (`torch.LongTensor` of shape `(batch_size, sequence_length, config.max_depth)`, *optional*): Subscript IDs for each token in the input sequence, padded up to config.max_depth. Examples: ```python >>> from transformers import AutoProcessor, MarkupLMForQuestionAnswering >>> import torch >>> processor = AutoProcessor.from_pretrained("microsoft/markuplm-base-finetuned-websrc") >>> model = MarkupLMForQuestionAnswering.from_pretrained("microsoft/markuplm-base-finetuned-websrc") >>> html_string = " My name is Niels " >>> question = "What's his name?" >>> encoding = processor(html_string, questions=question, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**encoding) >>> answer_start_index = outputs.start_logits.argmax() >>> answer_end_index = outputs.end_logits.argmax() >>> predict_answer_tokens = encoding.input_ids[0, answer_start_index : answer_end_index + 1] >>> processor.decode(predict_answer_tokens).strip() 'Niels' ```NT rArBrrrUrlrr r rrr;r<) ignore_indexr)loss start_logits end_logitsrr)r6r0rr;splitsqueezerlenriclamp_rrrr)r5rurArBrrrUrlr<r=rr r rrrlogitsrBrC total_loss ignored_indexloss_fct start_lossend_losss r9rGz$MarkupLMForQuestionAnswering.forwards`&1%>> from transformers import AutoProcessor, AutoModelForTokenClassification >>> import torch >>> processor = AutoProcessor.from_pretrained("microsoft/markuplm-base") >>> processor.parse_html = False >>> model = AutoModelForTokenClassification.from_pretrained("microsoft/markuplm-base", num_labels=7) >>> nodes = ["hello", "world"] >>> xpaths = ["/html/body/div/li[1]/div/span", "/html/body/div/li[1]/div/span"] >>> node_labels = [1, 2] >>> encoding = processor(nodes=nodes, xpaths=xpaths, node_labels=node_labels, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**encoding) >>> loss = outputs.loss >>> logits = outputs.logits ```NTr?rr;rArHrr) r6r0rrSrrr:rrr)r5rurArBrrrUrlrUrr r rrrrrArKs r9rGz&MarkupLMForTokenClassification.forwards\&1% 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoProcessor, AutoModelForSequenceClassification >>> import torch >>> processor = AutoProcessor.from_pretrained("microsoft/markuplm-base") >>> model = AutoModelForSequenceClassification.from_pretrained("microsoft/markuplm-base", num_labels=7) >>> html_string = " Page Title " >>> encoding = processor(html_string, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**encoding) >>> loss = outputs.loss >>> logits = outputs.logits ```NTr?r regressionsingle_label_classificationmulti_label_classificationr;rW)r6r0rr)rS problem_typer:rgr?rjrrrrErrrrrr)r5rurArBrrrUrlrUrr r rrrrHrArKs r9rGz)MarkupLMForSequenceClassification.forwardhsZ&1%rqsG$ AA&!9G&6RR2   H %/ bii/ dpph   299  RYY  RYY  bii$ ryy "!"))!$% II% <<% % << % LL4' %  %%.5BII5r 2#.#N* bii* Z io i ir +r r ji #:i i X ] %<]  ] @ n (?n  n b rH