i,dZddlZddlmZddlZddlmZddlmZmZm Z ddl m Z ddl mZmZdd lmZdd lmZmZmZmZmZmZdd lmZdd lmZdd lmZmZddl m!Z!ejDe#Z$GddejJZ&eGddeZ'GddejJZ(GddejJZ)GddejJZ*GddejJZ+GddejJZ,GddejJZ-Gd d!ejJZ.Gd"d#eZ/Gd$d%ejJZ0Gd&d'ejJZ1Gd(d)ejJZ2eGd*d+e'Z3Gd,d-ejJZ4eGd.d/e'Z5Gd0d1ejJZ6ed23Gd4d5e'Z7eGd6d7e'Z8eGd8d9e'Z9eGd:d;e'Z:gd<Z;y)=zPyTorch ConvBERT model.N)Callable)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FNget_activation)GradientCheckpointingLayer)"BaseModelOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward)auto_docstringlogging)ConvBertConfigc eZdZdZfdZ d dej dzdej dzdej dzdejdzdej f d ZxZ S) ConvBertEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.ct|tj|j|j |j |_tj|j|j |_ tj|j|j |_ tj|j |j|_ tj|j|_|j#dt%j&|jj)dd|j#dt%j*|j,j/t$j0dy) N) padding_idxeps position_idsrF) persistenttoken_type_idsdtype)super__init__r Embedding vocab_sizeembedding_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_buffertorcharangeexpandzerosrsizelongselfconfig __class__s j/mnt/e/genesis-system/.venv/lib/python3.12/site-packages/transformers/models/convbert/modeling_convbert.pyr'zConvBertEmbeddings.__init__1s !||F,=,=v?T?Tbhbubuv#%<<0N0NPVPePe#f %'\\&2H2H&J_J_%`"f&;&;AVAVWzz&"<"<=   ELL)G)GHOOPWXej    ekk$*;*;*@*@*B%**Ubg  N input_idsr#r inputs_embedsreturnc2||j}n|jdd}|d}||jddd|f}|st|dr-|jddd|f}|j |d|}|}n:t j |t j|jj}||j|}|j|} |j|} || z| z} |j| } |j| } | S)Nr!rr#rr%device)r;rhasattrr#r9r7r:r<rHr,r.r0r1r5) r>rCr#rrD input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedr.r0 embeddingss rAforwardzConvBertEmbeddings.forwardAs,  #..*K',,.s3K ^  ,,Q  ^ J"%88;PP ^^J/ \\*- rB)NNNN) __name__ __module__ __qualname____doc__r'r7 LongTensor FloatTensorrO __classcell__r@s@rArr.sQ $.2260426 $##d*$((4/$&&- $ ((4/ $    $rBrcZeZdZUeed<dZdZejfdZ xZ S)ConvBertPreTrainedModelr?convbertTcbt||t|tr t j |j yt|trVt j|jd|jjt j |j yt|tryt j|jtj |jj"dj%dt j |j&yy)zInitialize the weightsmeanstdr!r N)r& _init_weights isinstanceSeparableConv1Dinitzeros_biasGroupedLinearLayernormal_weightr?initializer_rangercopy_rr7r8shaper9r#)r>moduler@s rAr`z%ConvBertPreTrainedModel._init_weightsns f% fo . KK $  2 3 LLSdkk6S6S T KK $  2 3 JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h KK-- .4rB) rPrQrRr__annotations__base_model_prefixsupports_gradient_checkpointingr7no_gradr`rVrWs@rArYrYhs/ "&*#U]]_ / /rBrYcZeZdZdZfdZdej dej fdZxZS)rbzSThis class implements separable convolution, i.e. a depthwise and a pointwise layerc t|tj|||||dzd|_tj||dd|_tj tj|d|_ |jjjjd|j|j jjjd|jy)NF) kernel_sizegroupspaddingrer)rtrer\r])r&r'rConv1d depthwise pointwise Parameterr7r:rerhdatargri)r>r? input_filtersoutput_filtersrtkwargsr@s rAr'zSeparableConv1D.__init__s   # 1$  =.aV[\LL^Q!?@  ""**9Q9Q*R ""**9Q9Q*RrB hidden_statesrEch|j|}|j|}||jz }|SN)rxryre)r>rxs rArOzSeparableConv1D.forwards0 NN= ) NN1  TYYrB rPrQrRrSr'r7TensorrOrVrWs@rArbrb|s']S U\\ellrBrbceZdZfdZ d dej dej dzdej dzdedzdeej ej dzff dZ xZ S) ConvBertSelfAttentioncjt||j|jzdk7r2t |ds&t d|jd|jd|j|j z}|dkr|j|_d|_n||_|j |_|j|_|j|jzdk7r t d|j|jzdz|_|j|jz|_ tj|j|j|_ tj|j|j|_ tj|j|j|_t||j|j|j|_tj|j|j|jz|_tj|j|j|_tj&|jdgt)|jdz dz dg |_tj,|j.|_y) Nrr*zThe hidden size (z6) is not a multiple of the number of attention heads ()rz6hidden_size should be divisible by num_attention_headsrs)rtrv)r&r' hidden_sizenum_attention_headsrI ValueError head_ratioconv_kernel_sizeattention_head_size all_head_sizerLinearquerykeyvaluerbkey_conv_attn_layerconv_kernel_layerconv_out_layerUnfoldintunfoldr3attention_probs_dropout_probr5)r>r?new_num_attention_headsr@s rAr'zConvBertSelfAttention.__init__s>     : : :a ?PVXhHi#F$6$6#78 445Q8  #)"<"<@Q@Q"Q "Q &$88DO'(D $'>D $$//DO & 7 7    8 8 8A =UV V$*$6$6$:R:R$RWX#X !558P8PPYYv1143E3EF 99V//1C1CDYYv1143E3EF #2 F&&(:(:Drrrr batch_sizerK_mixed_key_layermixed_value_layermixed_key_conv_attn_layermixed_query_layer query_layer key_layer value_layerconv_attn_layerrrattention_scoresattention_probs context_layerconv_outnew_context_layer_shapeoutputss rArOzConvBertSelfAttention.forwards%2$7$7! J ! ,"hh'<=O $ +@ A "hh}5O $ = 9 $($<$<]=T=TUVXY=Z$[!$=$G$G1$M! JJ}5',, D44d6N6N )Aq/ $((R9Q9QSWSkSklvv q (,, D44d6N6N )Aq/  ..)BDUV 22?C!MM*;b$BWBWYZ=[\!MM*;C,,];~ BHZHZ7[\'11!Q7BBDNNrR-- ..2++a/A5q9 . (11!Q7?? D..0E0E ~DS>STzz&"<"<= rBr input_tensorrEcr|j|}|j|}|j||z}|Srrr5r1r>rrs rArOzConvBertSelfOutput.forward7 =1  ]3 }|'CD rBrPrQrRr'r7rrOrVrWs@rArrs1> U\\RWR^R^rBrceZdZfdZ d dej dej dzdej dzdedzdeej ej dzff dZ xZ S) ConvBertAttentioncbt|t||_t ||_yr)r&r'rr>routputr=s rAr'zConvBertAttention.__init__!s& )&1 (0 rBNrrrrrEcj|j||||}|j|d|}|f|ddz}|S)Nrr)r>r)r>rrrr self_outputsattention_outputrs rArOzConvBertAttention.forward&sNyy   !    ;;|A F#% QR(88rBrrrWs@rArr s}14859). ||))D0 %||d2   $;  u||U..55 6 rBrcVeZdZfdZdej dej fdZxZS)rfct|||_||_||_|j|jz|_|j|jz|_tjtj|j|j |j |_ tjtj||_ yr) r&r' input_size output_size num_groups group_in_dim group_out_dimrrzr7emptyrhre)r>rrrr@s rAr'zGroupedLinearLayer.__init__9s $&$ OOt>!--@ll5;;t@Q@QSWSeSe#fg LL[!9: rBrrEct|jd}tj|d|j|j g}|j ddd}tj||j}|j ddd}tj||d|jg}||jz}|S)Nrr!rrs) listr;r7rrrrrrhrre)r>rrrs rArOzGroupedLinearLayer.forwardCs-,,./2 MM-"doot?P?P)Q R IIaA  LLDKK ( IIaA  MM!j"d.>.>? @  MrBrrWs@rArfrf8s#;U\\ellrBrfcVeZdZfdZdej dej fdZxZS)ConvBertIntermediatect||jdk(r0tj|j |j |_n1t|j |j |j|_t|jtrt|j|_ y|j|_ y)Nrrrr)r&r'rrrrintermediate_sizerrfra hidden_actstrr intermediate_act_fnr=s rAr'zConvBertIntermediate.__init__Os     !6#5#5v7O7OPDJ+!--6;S;S`f`q`qDJ f'' -'-f.?.?'@D $'-'8'8D $rBrrEcJ|j|}|j|}|Sr)rrr>rs rArOzConvBertIntermediate.forward\s& =1 00? rBrrWs@rArrNs# 9U\\ellrBrcneZdZfdZdej dej dej fdZxZS)ConvBertOutputct||jdk(r0tj|j |j |_n1t|j |j |j|_tj|j |j|_ tj|j|_ y)Nrrr)r&r'rrrrrrrfr1r2r3r4r5r=s rAr'zConvBertOutput.__init__cs     !6#;#;V=O=OPDJ+!33ASAS`f`q`qDJf&8&8f>S>STzz&"<"<= rBrrrEcr|j|}|j|}|j||z}|Srrrs rArOzConvBertOutput.forwardnrrBrrWs@rArrbs1 >U\\RWR^R^rBrceZdZfdZ d dej dej dzdej dzdej dzdedzdeej ej dzff d Z d Z xZ S) ConvBertLayercbt||j|_d|_t ||_|j |_|j|_|jr*|j st|dt ||_ t||_ t||_ y)Nrz> should be used as a decoder model if cross attention is added)r&r'chunk_size_feed_forward seq_len_dimr attention is_decoderadd_cross_attention TypeErrorcrossattentionr intermediaterrr=s rAr'zConvBertLayer.__init__vs '-'E'E$*62 ++#)#=#=  # #??4&(f ghh"3F";D 08$V, rBNrrrencoder_attention_maskrrEc:|j|||}|d}|dd}|jr>|rrrrrself_attention_outputsrrcross_attention_outputs layer_outputs rArOzConvBertLayer.forwards"&  /"0"  2!4(, ??4@4!12$=dVDDD'+&9&9 &%! ' # 7q9  7 ;;G0  # #T%A%A4CSCSUe  /G+rBcL|j|}|j||}|Sr)rr)r>rintermediate_outputrs rArz ConvBertLayer.feed_forward_chunks,"//0@A{{#68HI rB)NNNF) rPrQrRr'r7rrUrrrOrrVrWs@rArrus -"48596:). #||#))D0# %||d2 # !& t 3 #  $; # u||U..55 6#JrBrceZdZfdZ d dej dej dzdej dzdej dzdedzdedzd edzd ee zfd Z xZ S) ConvBertEncoderct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}w)NF) r&r'r?r ModuleListrangenum_hidden_layersrlayergradient_checkpointing)r>r?rr@s rAr'zConvBertEncoder.__init__sN  ]]5IaIaCb#caM&$9#cd &+#$dsA#Nrrrrroutput_hidden_states return_dictrEct|rdnd}|rdnd} |r|jjrdnd} t|jD]J\} } |r||fz}| |||||} | d}|s"| | dfz} |jjsB| | dfz} L|r||fz}|st d||| | fDSt ||| | S)Nrrrsc3$K|]}|| ywrr).0vs rA z*ConvBertEncoder.forward..s=s)last_hidden_stater attentionscross_attentions)r?r enumerater rr )r>rrrrrr r all_hidden_statesall_self_attentionsall_cross_attentionsi layer_module layer_outputss rArOzConvBertEncoder.forwards#7BD$5b4%64;;;Z;Zr`d(4 VOA|#$58H$H!(%&! M*!,M &9]1=MS>STrBrrEcl|j|}|j|}|j|}|Sr)rr!r1rs rArOz'ConvBertPredictionHeadTransform.forwards4 =1 --m< }5 rBrrWs@rArrs$UU\\ellrBrceZdZdZdeffd Z d dejdejdzdejfdZ xZ S) ConvBertSequenceSummarya Compute a single vector summary of a sequence hidden states. Args: config ([`ConvBertConfig`]): The config used by the model. Relevant arguments in the config class of the model are (refer to the actual config class of your model for the default values it uses): - **summary_type** (`str`) -- The method to use to make this summary. Accepted values are: - `"last"` -- Take the last token hidden state (like XLNet) - `"first"` -- Take the first token hidden state (like Bert) - `"mean"` -- Take the mean of all tokens hidden states - `"cls_index"` -- Supply a Tensor of classification token position (GPT/GPT-2) - `"attn"` -- Not implemented now, use multi-head attention - **summary_use_proj** (`bool`) -- Add a projection after the vector extraction. - **summary_proj_to_labels** (`bool`) -- If `True`, the projection outputs to `config.num_labels` classes (otherwise to `config.hidden_size`). - **summary_activation** (`Optional[str]`) -- Set to `"tanh"` to add a tanh activation to the output, another string or `None` will add no activation. - **summary_first_dropout** (`float`) -- Optional dropout probability before the projection and activation. - **summary_last_dropout** (`float`)-- Optional dropout probability after the projection and activation. r?cft|t|dd|_|jdk(rtt j |_t|drq|jret|dr(|jr|jdkDr |j}n |j}t j|j||_t|dd}|r t|nt j |_t j |_t|dr3|j"dkDr$t j$|j"|_t j |_t|d r5|j(dkDr%t j$|j(|_yyy) N summary_typelastattnsummary_use_projsummary_proj_to_labelsrsummary_activationsummary_first_dropoutsummary_last_dropout)r&r'getattrr&NotImplementedErrorrIdentitysummaryrIr)r* num_labelsrrr activation first_dropoutr,r3 last_dropoutr-)r>r? num_classesactivation_stringr@s rAr'z ConvBertSequenceSummary.__init__sU #FNFC    && %{{} 6- .63J3Jv78V=Z=Z_e_p_pst_t$// $00 99V%7%7EDL#F,@$GIZN3D$E`b`k`k`m[[] 62 38T8TWX8X!#F,H,H!ID KKM 61 2v7R7RUV7V " 6+F+F GD 8W 2rBNr cls_indexrEc|jdk(r |dddf}n|jdk(r |dddf}n|jdk(r|jd}n|jd k(r|Atj|d ddddf|jd dz tj }nX|j dj d}|jd |jdz z|jdfz}|jd |jd }n|jdk(rt|j}|j|}|j|}|j!|}|S)ak Compute a single vector summary of a sequence hidden states. Args: hidden_states (`torch.FloatTensor` of shape `[batch_size, seq_len, hidden_size]`): The hidden states of the last layer. cls_index (`torch.LongTensor` of shape `[batch_size]` or `[batch_size, ...]` where ... are optional leading dimensions of `hidden_states`, *optional*): Used if `summary_type == "cls_index"` and takes the last token of the sequence as classification token. Returns: `torch.FloatTensor`: The summary of the sequence hidden states. r'Nr!firstrr^rrr8.rr$)r!r()r&r^r7 full_likerkr<rr9rr;gathersqueezer/r4r1r3r5)r>rr8rs rArOzConvBertSequenceSummary.forward.sn    &"1b5)F   ' )"1a4(F   & ("''A'.F   + - !OO!#rr1*-!''+a/** &//3==bA %,,Uimmo6I-JmN`N`acNdMf-fg "))"i8@@DF   & (% %##F+f%(""6* rBr) rPrQrRrSrr'r7rUrTrOrVrWs@rAr$r$sQ2H~H<VZ)"..);@;K;Kd;R)   )rBr$c eZdZfdZdZdZe ddejdzdejdzdejdzdejdzd ejdzd e dzd e dzd e dzd e e zfdZ xZS) ConvBertModelc"t||t||_|j|j k7r/t j|j|j |_t||_ ||_ |jyr) r&r'rrNr*rrrembeddings_projectrencoderr? post_initr=s rAr'zConvBertModel.__init__\sl  ,V4  F$6$6 6&(ii0E0EvGYGY&ZD #&v.   rBc.|jjSrrNr,r>s rAget_input_embeddingsz"ConvBertModel.get_input_embeddingshs...rBc&||j_yrrE)r>rs rAset_input_embeddingsz"ConvBertModel.set_input_embeddingsks*/'rBNrCrr#rrDrr r rEc  T||n|jj}||n|jj}||n|jj}| | t d|#|j |||j } n!||j dd} n t d| \} } | |jn |j} |tj| | }|pt|jdr4|jjddd| f}|j| | }|}n&tj| tj| }|j!|| }|j||||}t|dr|j#|}|j%||||| }|S) NzDYou cannot specify both input_ids and inputs_embeds at the same timer!z5You have to specify either input_ids or inputs_embeds)rHr#rG)rCrr#rDrA)rrr r )r?rr use_return_dictr%warn_if_padding_and_no_attention_maskr;rHr7onesrIrNr#r9r:r<get_extended_attention_maskrArB)r>rCrr#rrDrr r r~rJrrKrHrLrMextended_attention_maskrs rArOzConvBertModel.forwardns2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "  6 6y. Q#..*K  &',,.s3KTU U!, J%.%:!! @T@T  !"ZZ FCN  !t(89*.//*H*HKZK*X'3J3Q3QR\^h3i0!A!&[ SY!Z"&"B"B>S^"_l>iv(  4- . 33MBM 2/!5# % rB)NNNNNNNN)rPrQrRr'rGrIrr7rTrUrrr rOrVrWs@rAr?r?Zs /0.237260426)-,0#':##d*:))D0:((4/ : &&- : ((4/ : $;:#Tk:D[: 3 3::rBr?cZeZdZdZfdZdej dej fdZxZS)ConvBertGeneratorPredictionszAPrediction module for the generator, made up of two dense layers.ct|td|_t j |j |j|_t j|j|j |_ y)Ngelur) r&r'r r3rr1r*r2rrrr=s rAr'z%ConvBertGeneratorPredictions.__init__sV (0f&;&;AVAVWYYv1163H3HI rBgenerator_hidden_statesrEcl|j|}|j|}|j|}|Sr)rr3r1)r>rTrs rArOz$ConvBertGeneratorPredictions.forwards3 #:;  6 }5 rB) rPrQrRrSr'r7rUrOrVrWs@rArQrQs+KJu/@/@UEVEVrBrQc4eZdZddiZfdZdZdZe ddejdzdejdzd ejdzd ejdzd ejdzd ejdzd e dzde dzde dzde e zfdZxZS)ConvBertForMaskedLMzgenerator_lm_head.weightz*convbert.embeddings.word_embeddings.weightct||t||_t ||_t j|j|j|_ |jyr) r&r'r?rZrQgenerator_predictionsrrr*r)generator_lm_headrCr=s rAr'zConvBertForMaskedLM.__init__sR  %f- %A&%I"!#6+@+@&BSBS!T rBc|jSrrZrFs rAget_output_embeddingsz)ConvBertForMaskedLM.get_output_embeddingss%%%rBc||_yrr\)r>r,s rAset_output_embeddingsz)ConvBertForMaskedLM.set_output_embeddingss !0rBNrCrr#rrDlabelsrr r rEc | | n|jj} |j|||||||| } | d} |j| } |j | } d}|Pt j }|| jd|jj|jd}| s| f| ddz}||f|zS|St|| | j| jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Nrr!rlosslogitsrr) r?rKrZrYrZrrrr)rrr)r>rCrr#rrDr`rr r r~rTgenerator_sequence_outputprediction_scoresrcloss_fctrs rArOzConvBertForMaskedLM.forwards(&1%! 667PQ 223DE  **,H-222t{{7M7MNPVP[P[\^P_`D'),CAB,GGF)-)9TGf$ Ev E$1??.99   rB NNNNNNNNN)rPrQrR_tied_weights_keysr'r]r_rr7rTrUrrrrOrVrWs@rArWrWs46bc&1.237260426*.)-,0#'3 ##d*3 ))D03 ((4/ 3 &&- 3 ((4/ 3   4'3  $;3 #Tk3 D[3   3 3 rBrWcZeZdZdZfdZdej dej fdZxZS)ConvBertClassificationHeadz-Head for sentence-level classification tasks.cht|tj|j|j|_|j |j n |j}tj||_ tj|j|j|_ ||_ yr) r&r'rrrrclassifier_dropoutr4r3r5r2out_projr?r>r?rmr@s rAr'z#ConvBertClassificationHead.__init__ s YYv1163E3EF )/)B)B)NF % %TZTnTn zz"45  &"4"4f6G6GH  rBrrEc |dddddf}|j|}|j|}t|jj|}|j|}|j |}|S)Nr)r5rr r?rrn)r>rr~rs rArOz"ConvBertClassificationHead.forwardse !Q' " LLO JJqM 4;;)) *1 - LLO MM! rBrrWs@rArkrk s&7 U\\ rBrkz ConvBERT Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ) custom_introc eZdZfdZe ddej dzdejdzdej dzdej dzdejdzdej dzd edzd edzd edzd e e zfd Z xZ S)!ConvBertForSequenceClassificationct|||j|_||_t ||_t ||_|jyr) r&r'r2r?r?rZrk classifierrCr=s rAr'z*ConvBertForSequenceClassification.__init__(sH   ++ %f- 4V< rBNrCrr#rrDr`rr r rEc | | n|jj} |j|||||||| } | d} |j| } d}||jj|j dk(rd|j_nl|j dkDrL|j tjk(s|j tjk(rd|j_nd|j_|jjdk(rIt}|j dk(r&|| j|j}n|| |}n|jjdk(r=t}|| jd|j |jd}n,|jjdk(rt}|| |}| s| f| ddz}||f|zS|St|| | j | j" S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrr#rrDrr r rr regressionsingle_label_classificationmulti_label_classificationr!rb)r?rKrZru problem_typer2r%r7r<rrr=rrrrrrr>rCrr#rrDr`rr r r~rsequence_outputrdrcrgrs rArOz)ConvBertForSequenceClassification.forward2s(&1%rCrr#rrDr`rr r r~ num_choicesrr} pooled_outputrdreshaped_logitsrcrgrs rArOz!ConvBertForMultipleChoice.forwardsZ&1%G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqM[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei (   r=#5#5b#9=;M;Mb;Q R  -- ))%'/!5#  "!*--o> / ++b+6  ')HOV4D%''!"+5F)-)9TGf$ Ev E("!//))   rBrh) rPrQrRr'rr7rTrUrrrrOrVrWs@rArrys.237260426*.)-,0#'X ##d*X ))D0X ((4/ X &&- X ((4/ X   4'X  $;X #TkX D[X  * *X X rBrc eZdZfdZe ddej dzdejdzdej dzdej dzdejdzdej dzd edzd edzd edzd e e zfd Z xZ S)ConvBertForTokenClassificationc`t|||j|_t||_|j |j n |j }tj||_ tj|j|j|_ |jyr)r&r'r2r?rZrmr4rr3r5rrrurCros rAr'z'ConvBertForTokenClassification.__init__s   ++%f- )/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ rBNrCrr#rrDr`rr r rEc | | n|jj} |j|||||||| } | d} |j| } |j | } d}|rCrr#rrDrrrr r r~rr}rdrr total_loss ignored_indexrg start_lossend_lossrs rArOz$ConvBertForQuestionAnswering.forward2s&1%rs' $ AA&19.63   H %77t/o//&bii4yBIIyx   0,299(RYY&7.7t3 bii3 lbii$`bii`FN+NNb299$G 1G G T0 O (?O  O dd 7d d NA %<A A HI #:I I X rB