iG"dZddlmZddlmZddlZddlmZddlmZm Z m Z ddl m Z dd lmZdd lmZmZmZdd lmZdd lmZmZdd lmZddlmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$ddl%m&Z&m'Z'ddl(m)Z)ddl*m+Z+ddl,m-Z-m.Z.m/Z/m0Z0ddl1m2Z2m3Z3ddl4m5Z5ddl6m7Z7e0jpe9Z:GddejvZ< d\dejvdejzdejzdejzdejzdzde>dzde>d e)e.fd!Z?Gd"d#ejvZ@Gd$d%ejvZAGd&d'ejvZBGd(d)ejvZCGd*d+ejvZDGd,d-ejvZEGd.d/eZFGd0d1ejvZGGd2d3ejvZHGd4d5ejvZIGd6d7ejvZJGd8d9ejvZKGd:d;ejvZLGd<d=ejvZMe/Gd>d?e'ZNee/d@AGdBdCe-ZOe/dDAGdEdFeNZPe/dGAGdHdIeNZQe/dJAGdKdLeNeZRe/GdMdNeNZSe/dOAGdPdQeNZTe/dRAGdSdTeNZUe/GdUdVeNZVe/GdWdXeNZWe/GdYdZeNZXgd[ZYy)]zPyTorch BERT model.)Callable) dataclassN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer) )BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputNextSentencePredictorOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward) ModelOutputTransformersKwargsauto_docstringlogging)can_return_tuplemerge_with_config_defaults)capture_outputs) BertConfigc eZdZdZfdZ d dej dzdej dzdej dzdejdzded ejf d Z xZ S) BertEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.ct|tj|j|j |j |_tj|j|j |_ tj|j|j |_ tj|j |j|_ tj|j|_|j#dt%j&|jj)dd|j#dt%j*|j,j/t$j0dy) N) padding_idxeps position_idsr'F) persistenttoken_type_ids)dtype)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_buffertorcharangeexpandzerosr/sizelongselfconfig __class__s b/mnt/e/genesis-system/.venv/lib/python3.12/site-packages/transformers/models/bert/modeling_bert.pyr6zBertEmbeddings.__init__8s !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c %'\\&2H2H&J\J\%]"f&8&8f>S>STzz&"<"<=   ELL)G)GHOOPWXej    ekk$*;*;*@*@*B%**Ubg  N input_idsr3r/ inputs_embedspast_key_values_lengthreturnc||j}n|jdd}|\}}||jdd|||zf}|t|drT|jj |j dd} t j| d|} | j ||}n:t j|t j|jj}||j|}|j|} || z} |j|} | | z} |j| } |j| } | S)Nr1r3rr')dimindex)r4device)rJr/hasattrr3rHshaperFgatherrIrKrYr;r?r=r@rD) rMrRr3r/rSrT input_shape batch_size seq_lengthbuffered_token_type_idsr? embeddingsr=s rPforwardzBertEmbeddings.forwardHsP  #..*K',,.s3K!, J  ,,Q0FVlIl0l-lmL  !t-.*.*=*=*D*D\EWEWXYEZ\^*_'*/,,7NTU]i*j'!8!?!? J!W!&[ SWSdSdSkSk!l   00;M $ : :> J"%:: "66|D"55 ^^J/ \\*- rQ)NNNNr) __name__ __module__ __qualname____doc__r6rF LongTensor FloatTensorintTensorrb __classcell__rOs@rPr*r*5sQ $.2260426&' (##d*(((4/(&&- ( ((4/ ( !$ ( (rQr*modulequerykeyvalueattention_maskscalingrDkwargsc ||jddz}tj||jdd|z}|||z}tj j |d}tj j|||j}tj||} | jddj} | |fS)Nr1r rW)ptrainingr') rJrFmatmul transposer functionalsoftmaxrDry contiguous) rmrnrorprqrrrDrs attn_weights attn_outputs rPeager_attention_forwardrss**R.D(<<s}}Q':;gEL!#n4 ==((2(>L==((6??([L,,|U3K''1-88:K  $$rQceZdZd fd Z d dej dej dzdedzdej dzdee de ej f d Z xZ S) BertSelfAttentionNc@t||j|jzdk7r2t |ds&t d|jd|jd||_|j|_t|j|jz |_|j|jz|_ |jdz|_ tj|j|j|_ tj|j|j|_tj|j|j|_tj |j"|_|j&|_||_||_yNrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()ru)r5r6r9num_attention_headsrZ ValueErrorrNriattention_head_size all_head_sizerrrLinearrnrorprBattention_probs_dropout_probrD is_decoder is_causal layer_idxrMrNrrrOs rPr6zBertSelfAttention.__init__sP     : : :a ?PVXhHi#F$6$6#78 445Q8  #)#=#= #&v'9'9F?*9*N*N'&=%C%C!>2 & "I{)@(M(M KK , ,.E) %8      % $}}C$,,..LL %  % ! \*k));;;;FFH L((rQFNNNN rcrdrer6rFrjrhr rr!tuplerbrkrls@rPrrs}#648(,.2 -)||-)))D0-) -)  t+ -) +, -) u|| -)rQrceZdZd fd Z d dej dej dzdej dzdedzdee de ej f d Z xZ S) BertCrossAttentionNct||j|jzdk7r2t |ds&t d|jd|jd||_|j|_t|j|jz |_|j|jz|_ |jdz|_ tj|j|j|_ tj|j|j|_tj|j|j|_tj |j"|_||_||_yr)r5r6r9rrZrrNrirrrrrrrnrorprBrrDrrrs rPr6zBertCrossAttention.__init__sC     : : :a ?PVXhHi#F$6$6#78 445Q8  #)#=#= #&v'9'9FB**4>>:(?(M(M KK , ,.E) %8      % $}}C$,,..LL %  % ! \"))#w;FFH L((rQrr) rcrdrer6rFrjrhrrr!rrbrkrls@rPrrs#4;?376: 2)||2) %00472)))D0 2) -t3 2) +, 2) u|| 2)rQrcneZdZfdZdej dej dej fdZxZS)BertSelfOutputc(t|tj|j|j|_tj |j|j|_tj|j|_ yNr-) r5r6rrr9denser@rArBrCrDrLs rPr6zBertSelfOutput.__init__&s` YYv1163E3EF f&8&8f>S>STzz&"<"<= rQr input_tensorrUcr|j|}|j|}|j||z}|SNrrDr@rMrrs rPrbzBertSelfOutput.forward,7 =1  ]3 }|'CD rQrcrdrer6rFrjrbrkrls@rPrr%1> U\\RWR^R^rQrceZdZd fd Z d dej dej dzdej dzdej dzdedzdej dzd ee d e ej fd Z xZ S) BertAttentionNct|||_|rtnt}|||||_t ||_y)Nrr)r5r6is_cross_attentionrrrMroutput)rMrNrrrattention_classrOs rPr6zBertAttention.__init__4s= "40B,HY#Fi9U $V, rQrrqrencoder_attention_maskrrrsrUc |js|n|}|j|f||||d|\}} |j||}|| fS)N)rrqrr)rrMr) rMrrqrrrrrsattention_outputrs rPrbzBertAttention.forward;sg04/F/FLb)2 * "7)+) *  * &, ;;'7G--rQ)FNFNNNNNrrls@rPrr3s-48:>;?(,.2.||.))D0. %0047 . !& 1 1D 8 .  . t+.+,. u|| .rQrcVeZdZfdZdej dej fdZxZS)BertIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yr) r5r6rrr9intermediate_sizerr hidden_actstrr intermediate_act_fnrLs rPr6zBertIntermediate.__init__Ss] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $rQrrUcJ|j|}|j|}|Sr)rrrMrs rPrbzBertIntermediate.forward[s& =1 00? rQrrls@rPrrRs#9U\\ellrQrcneZdZfdZdej dej dej fdZxZS) BertOutputc(t|tj|j|j |_tj|j |j|_tj|j|_ yr) r5r6rrrr9rr@rArBrCrDrLs rPr6zBertOutput.__init__bs` YYv779K9KL f&8&8f>S>STzz&"<"<= rQrrrUcr|j|}|j|}|j||z}|Srrrs rPrbzBertOutput.forwardhrrQrrls@rPrrarrQrceZdZd fd Z ddej dej dzdej dzdej dzdedzdej dzd ee d e ej fd Z d Z xZ S) BertLayerNct||j|_d|_t ||j ||_|j |_|j|_|jr.|j st|dt |d|d|_ t||_ t||_ y)Nr'rz> should be used as a decoder model if cross attention is addedFT)rrr)r5r6chunk_size_feed_forward seq_len_dimrr attentionadd_cross_attentionrcrossattentionr intermediaterr)rMrNrrOs rPr6zBertLayer.__init__ps '-'E'E$&v9J9JV_` ++#)#=#=  # #?? D6)g!hii"/##' #D  -V4 ( rQrrqrrrrrsrUc "|j||f||d|\}} |} |jr:|8t|dstd|d|j|d||fd|i|\} } | } t |j |j|j| } | S)N)rrrz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`r) rrrZrrrfeed_forward_chunkrr) rMrrqrrrrrsself_attention_output_rcross_attention_output layer_outputs rPrbzBertLayer.forwards$24>>  $ ,) $  $  q1 ??4@4!12 =dVDDD )<(;(;%%& ) !0 )  ) % "A 6 0  # #T%A%A4CSCSUe rQcL|j|}|j||}|Sr)rr)rMrintermediate_outputrs rPrzBertLayer.feed_forward_chunks,"//0@A{{#68HI rQrr)rcrdrer6rFrjrhr rr!rrbrrkrls@rPrros),48:>;?(,.2'||'))D0' %0047 ' !& 1 1D 8 '  ' t+'+,' u|| 'RrQrceZdZfdZ d dej dej dzdej dzdej dzdedzdedzd ej dzd e e d e ej e zfd Z xZS) BertEncoderc t|||_tjt |j Dcgc]}t||c}|_ycc}w)N)r) r5r6rNr ModuleListrangenum_hidden_layersrlayer)rMrNirOs rPr6zBertEncoder.__init__sF  ]]ERXRjRjLk#lqIf$B#lm #lsANrrqrrr use_cacherrsrUc t|jD]\} } | |||f|||d|}t||r|SdS)N)rrr)last_hidden_stater) enumeraterr) rMrrqrrrrrrsr layer_modules rPrbzBertEncoder.forwardsq )4 OA|(%(> /- M 9+/8O  >B  rQNNNNNN)rcrdrer6rFrjrhr boolrr!rrrbrkrls@rPrrsn48:>;?(,!%.2 || ))D0  %0047  !& 1 1D 8    $;  t+ +,  u|| H H rQrcVeZdZfdZdej dej fdZxZS) BertPoolerct|tj|j|j|_tj |_yr)r5r6rrr9rTanh activationrLs rPr6zBertPooler.__init__s9 YYv1163E3EF '')rQrrUc\|dddf}|j|}|j|}|S)Nr)rr)rMrfirst_token_tensor pooled_outputs rPrbzBertPooler.forwards6+1a40 #56  6 rQrrls@rPrrs#$ U\\ellrQrcVeZdZfdZdej dej fdZxZS)BertPredictionHeadTransformcht|tj|j|j|_t |jtrt|j|_ n|j|_ tj|j|j|_ yr) r5r6rrr9rrrrr transform_act_fnr@rArLs rPr6z$BertPredictionHeadTransform.__init__s{ YYv1163E3EF f'' -$*6+<+<$=D !$*$5$5D !f&8&8f>S>STrQrrUcl|j|}|j|}|j|}|Sr)rr r@rs rPrbz#BertPredictionHeadTransform.forwards4 =1 --m< }5 rQrrls@rPrrs$UU\\ellrQrc$eZdZfdZdZxZS)BertLMPredictionHeadct|t||_t j |j |jd|_t jtj|j|_ y)NT)bias) r5r6r transformrrr9r8decoder ParameterrFrIrrLs rPr6zBertLMPredictionHead.__init__s[ 4V<yy!3!3V5F5FTR LLV->->!?@ rQcJ|j|}|j|}|Sr)rrrs rPrbzBertLMPredictionHead.forwards$}5  ]3 rQrcrdrer6rbrkrls@rPr r sArQr cVeZdZfdZdej dej fdZxZS)BertOnlyMLMHeadcBt|t||_yr)r5r6r  predictionsrLs rPr6zBertOnlyMLMHead.__init__s /7rQsequence_outputrUc(|j|}|Sr)r)rMrprediction_scoress rPrbzBertOnlyMLMHead.forward s ,,_=  rQrrls@rPrrs#8!u||! !rQrc$eZdZfdZdZxZS)BertOnlyNSPHeadclt|tj|jd|_yNrv)r5r6rrr9seq_relationshiprLs rPr6zBertOnlyNSPHead.__init__s'  " &*<*rrr)rQrPr;r;>s~ &*D%  d ")26u((4/68<U..5<59M5**+d2926Je''(4/6rQr;a  The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in [Attention is all you need](https://huggingface.co/papers/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. To behave as an decoder the model needs to be initialized with the `is_decoder` argument of the configuration set to `True`. To be used in a Seq2Seq model, the model needs to initialized with both `is_decoder` argument and `add_cross_attention` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass. ceZdZddgZdfd ZdZdZeee dde jdzde jdzd e jdzd e jdzd e jdzd e jdzd e jdzde dzde dzde jdzdeedee jezfdZdZxZS) BertModelr*rct||||_d|_t ||_t ||_|r t|nd|_ |jy)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer FN) r5r6rNgradient_checkpointingr*rarencoderrpooler post_init)rMrNadd_pooling_layerrOs rPr6zBertModel.__init__fsU   &+#(0"6* ,=j(4  rQc.|jjSrrar;rMs rPget_input_embeddingszBertModel.get_input_embeddingsws...rQc&||j_yrrJ)rMrps rPset_input_embeddingszBertModel.set_input_embeddingszs*/'rQNrRrqr3r/rSrrrrrrsrUc L|jjr| | n|jj} nd} | rd|b||jjr4t t |jt |jnt |j}|du|duz r t d||j} |jd} n|j} |jd} ||jnd}| tj||| z| } |j|||||}|j||||| |\}}|j|f||||| | |d | }|j}|j |j!|nd}t#|||j$ S) NF)rNz:You must specify exactly one of input_ids or inputs_embedsr'r)rY)rRr/r3rSrT)rqrembedding_outputrrr)rqrrrrrr/)r pooler_outputr)rNrris_encoder_decoderrr rrYr[get_seq_lengthrFrGra_create_attention_masksrErrFrr)rMrRrqr3r/rSrrrrrrsrYr_rTrPencoder_outputsrrs rPrbzBertModel.forward}s" ;; ! !%.%:  @U@UII 0)4 8V8V$L $DlZ^ZeZeFfg!5   -t"; <YZ Z  %%F"+J"))F&,,Q/JETE`!?!?!Afg  !"\\*@BX[eBentuN??%)'#9 + 261M1M)#9-"7)+ 2N2 ..'$,,   )"7#9+)%    *;;8< 8O O4UY ;-'+;;  rQc|jjrt|j||||}nt|j||}|t|j|||}||fS)N)rNrSrqrr)rNrSrq)rNrSrqr)rNrrr)rMrqrrPrrrs rPrTz!BertModel._create_attention_maskssx ;; ! !/{{.-- / N7{{.-N " -%>{{.5&; & "555rQ)T) NNNNNNNNNN)rcrdre_no_split_modulesr6rLrNr%r&r"rFrjr rrr!rrrbrTrkrls@rPrBrBWsR*;7"/0 *..2.2,0-1596:(,!%.2K <<$&K  t+K  t+ K llT) K ||d* K  %||d2K !& t 3K K $;K  t+K +,K  u|| K KK  K Z 6rQrBz Bert Model with two heads on top as done during the pretraining: a `masked language modeling` head and a `next sentence prediction (classification)` head. c`eZdZdddZfdZdZdZee dde jdzd e jdzd e jdzd e jdzd e jdzd e jdzde jdzde e de e jezfdZxZS)BertForPreTraining&bert.embeddings.word_embeddings.weightcls.predictions.biaszcls.predictions.decoder.weightzcls.predictions.decoder.biasct||t||_t ||_|j yr)r5r6rBr(r#clsrGrLs rPr6zBertForPreTraining.__init__s4  f% '/ rQcB|jjjSrr^rrrKs rPget_output_embeddingsz(BertForPreTraining.get_output_embeddingsxx##+++rQc||jj_|j|jj_yrr^rrrrMnew_embeddingss rPset_output_embeddingsz(BertForPreTraining.set_output_embeddings,'5$$2$7$7!rQNrRrqr3r/rSlabelsnext_sentence_labelrsrUc |j|f||||dd|} | dd\} } |j| | \} } d}|u|st}|| jd|jj |jd}|| jdd|jd}||z}t || | | j| jS)am labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` next_sentence_label (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring) Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Example: ```python >>> from transformers import AutoTokenizer, BertForPreTraining >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased") >>> model = BertForPreTraining.from_pretrained("google-bert/bert-base-uncased") >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.prediction_logits >>> seq_relationship_logits = outputs.seq_relationship_logits ``` Trqr3r/rS return_dictNrvr1)r<r=r>rr)) r(r^rrrNr8r;rr))rMrRrqr3r/rSrirjrsoutputsrrrr! total_lossloss_fctmasked_lm_lossnext_sentence_losss rPrbzBertForPreTraining.forward sR$))  ))%'   *1!&48HH_m4\11  "5"A')H%&7&<& ,8*..2.2,0-1&*37A <<$&A  t+A  t+ A llT) A ||d* A  t#A #\\D0A +,A  u|| 7 7A A rQrYzP Bert Model with a `language modeling` head on top for CLM fine-tuning. c eZdZdddZfdZdZdZee dde jdzd e jdzd e jdzd e jdzd e jdzd e jdzde jdzde jdzde dzde dzde jdzde e jzdeedee jezfdZxZS)BertLMHeadModelrZr[r\ct|||jstj dt |d|_t||_|jy)NzLIf you want to use `BertLMHeadModel` as a standalone, add `is_decoder=True.`FrH r5r6rloggerwarningrBr(rr^rGrLs rPr6zBertLMHeadModel.__init__]sL    NNi jf> "6* rQcB|jjjSrr`rKs rPraz%BertLMHeadModel.get_output_embeddingsirbrQc||jj_|j|jj_yrrdres rPrgz%BertLMHeadModel.set_output_embeddingslrhrQNrRrqr3r/rSrrrirrrlogits_to_keeprsrUc  |d} |j|f||||||| | | dd | }|j}t| tr t | dn| }|j |dd|ddf}d}|*|j d|||jjd| }t|||j|j|j|jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]` NFT) rqr3r/rSrrrrrrm)logitsrir8)r<rrrr)r*r@)r(rrrislicer^ loss_functionrNr8rrrr)r*)rMrRrqr3r/rSrrrirrrr~rsrnr slice_indicesrr<s rPrbzBertLMHeadModel.forwardps0  I@I  A ))%'"7#9+) A  A  11 8B>SV8W~ot4]k -=!(;<=  %4%%pVFt{{OeOepiopD0#33!//))$55   rQ) NNNNNNNNNNNr)rcrdrertr6rargr$r"rFrjr rrirr!rrrbrkrls@rPrvrvRsf+S(> ,8*..2.2,0-1596:&*(,!%.2-.8 <<$&8  t+8  t+ 8 llT) 8 ||d* 8  %||d28 !& t 38  t#8 8 $;8  t+8 ell*8 +,8  u|| @ @8 8 rQrvceZdZdddZfdZdZdZee dde jdzd e jdzd e jdzd e jdzd e jdzd e jdzde jdzde jdzde e de e jezfdZxZS)BertForMaskedLMrZr[r\ct|||jrtj dt |d|_t||_|jy)NzkIf you want to use `BertForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention.FrxryrLs rPr6zBertForMaskedLM.__init__sR     NN1  f> "6* rQcB|jjjSrr`rKs rPraz%BertForMaskedLM.get_output_embeddingsrbrQc||jj_|j|jj_yrrdres rPrgz%BertForMaskedLM.set_output_embeddingsrhrQNrRrqr3r/rSrrrirsrUc >|j|f||||||dd| } | d} |j| } d} |Ft}|| jd|jj |jd} t | | | j| jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` T)rqr3r/rSrrrmrNr1r<rrr)) r(r^rrrNr8rrr))rMrRrqr3r/rSrrrirsrnrrrqrps rPrbzBertForMaskedLM.forwards($))   ))%'"7#9    "!* HH_5  ')H%&7&<& ,8*..2.2,0-1596:&*+ <<$&+  t++  t+ + llT) + ||d* +  %||d2+ !& t 3+  t#+ +,+  u|| ~ -+ + rQrzT Bert Model with a `next sentence prediction (classification)` head on top. c*eZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd ee d e eje zfd Z xZ S) BertForNextSentencePredictionct||t||_t ||_|j yr)r5r6rBr(rr^rGrLs rPr6z&BertForNextSentencePrediction.__init__s4  f% "6* rQNrRrqr3r/rSrirsrUc |j|f||||dd|}|d} |j| } d} |2t} | | jdd|jd} t | | |j |j S)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring). Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Example: ```python >>> from transformers import AutoTokenizer, BertForNextSentencePrediction >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased") >>> model = BertForNextSentencePrediction.from_pretrained("google-bert/bert-base-uncased") >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced." >>> next_sentence = "The sky is blue due to the shorter wavelength of blue light." >>> encoding = tokenizer(prompt, next_sentence, return_tensors="pt") >>> outputs = model(**encoding, labels=torch.LongTensor([1])) >>> logits = outputs.logits >>> assert logits[0, 0] < logits[0, 1] # next sentence was random ``` Trlr'Nr1rvr)r(r^rrrrr)) rMrRrqr3r/rSrirsrnrseq_relationship_scoresrrrps rPrbz%BertForNextSentencePrediction.forward sN$))  ))%'     "&((="9!  ')H!)*A*F*Fr1*Mv{{[]!_ *#*!//))   rQr)rcrdrer6r$r"rFrjrr!rrrbrkrls@rPrrs *..2.2,0-1&*= <<$&=  t+=  t+ = llT) = ||d* =  t#= +,=  u|| : := = rQrz Bert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c*eZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd ee d e eje zfd Z xZ S) BertForSequenceClassificationcnt|||j|_||_t ||_|j |j n |j}tj||_ tj|j|j|_ |jyr)r5r6 num_labelsrNrBr(classifier_dropoutrCrrBrDrr9 classifierrGrMrNrrOs rPr6z&BertForSequenceClassification.__init__Rs   ++ f% )/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ rQNrRrqr3r/rSrirsrUc |j|f||||dd|}|d} |j| } |j| } d} ||jj|j dk(rd|j_nl|j dkDrL|j tjk(s|j tjk(rd|j_nd|j_|jjdk(rIt} |j dk(r&| | j|j} n| | |} n|jjdk(r=t} | | jd|j |jd} n,|jjdk(rt} | | |} t| | |j |j" S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Trlr'N regressionsingle_label_classificationmulti_label_classificationr1r)r(rDrrN problem_typerr4rFrKrirsqueezerrrrrr)) rMrRrqr3r/rSrirsrnrrr<rps rPrbz%BertForSequenceClassification.forwardas$$))  ))%'      ]3 /  {{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#FNN$4fnn6FGD#FF3D))-JJ+- B @&++b/R))-II,./'!//))   rQr)rcrdrer6r$r"rFrjrr!rrrbrkrls@rPrrKs *..2.2,0-1&*; <<$&;  t+;  t+ ; llT) ; ||d* ;  t#; +,;  u|| 7 7; ; rQrc*eZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd ee d e eje zfd Z xZ S) BertForMultipleChoicec*t||t||_|j |jn |j }t j||_t j|jd|_ |jy)Nr') r5r6rBr(rrCrrBrDrr9rrGrs rPr6zBertForMultipleChoice.__init__su  f% )/)B)B)NF % %TZTnTn zz"45 ))F$6$6: rQNrRrqr3r/rSrirsrUc ||jdn|jd}|!|jd|jdnd}|!|jd|jdnd}|!|jd|jdnd}|!|jd|jdnd}|1|jd|jd|jdnd}|j|f||||dd|} | d} |j | } |j | } | jd|} d} |t }|| |} t| | | j| jS)a[ input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nr'r1Trlr) r[rrJr(rDrrrrr))rMrRrqr3r/rSrirs num_choicesrnrrreshaped_logitsr<rps rPrbzBertForMultipleChoice.forwardsT-6,Aiooa(}GZGZ[\G] >G>SINN2y~~b'9:Y] M[Mg,,R1D1DR1HImqM[Mg,,R1D1DR1HImqGSG_|((\->->r-BCei (   r=#5#5b#9=;M;Mb;Q R  $))  ))%'      ]3 / ++b+6  ')HOV4D("!//))   rQr)rcrdrer6r$r"rFrjrr!rrrbrkrls@rPrrs *..2.2,0-1&*N <<$&N  t+N  t+ N llT) N ||d* N  t#N +,N  u|| 8 8N N rQrc*eZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd ee d e eje zfd Z xZ S) BertForTokenClassificationcdt|||j|_t|d|_|j |j n |j }tj||_ tj|j|j|_ |jyNFrx)r5r6rrBr(rrCrrBrDrr9rrGrs rPr6z#BertForTokenClassification.__init__s   ++f> )/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ rQNrRrqr3r/rSrirsrUc H|j|f||||dd|}|d} |j| } |j| } d} | ))F$6$68I8IJ rQNrRrqr3r/rSstart_positions end_positionsrsrUc |j|f||||dd|} | d} |j| } | jdd\} } | jdj } | jdj } d}||t |j dkDr|jd}t |j dkDr|jd}| j d}|jd|}|jd|}t|}|| |}|| |}||zdz }t|| | | j| j S) NTrlrr'r1rw) ignore_indexrv)r< start_logits end_logitsrr)) r(rsplitrr~lenrJclamprrrr))rMrRrqr3r/rSrrrsrnrrrrro ignored_indexrp start_lossend_losss rPrbz BertForQuestionAnswering.forwardKs$))  ))%'   "!*1#)<<r<#: j#++B/::< ''+668   &=+D?'')*Q."1"9"9""==%%'(1, - 5 5b 9 (--a0M-33A}EO)//=AM']CH!,@J M:H$x/14J+%!!//))   rQrs)rcrdrer6r$r"rFrjrr!rrrbrkrls@rPrr?s*..2.2,0-1/3-13 <<$&3  t+3  t+ 3 llT) 3 ||d* 3 ,3 ||d*3 +,3  u|| ; ;3 3 rQr) rrrrYrrrrrvrBr')Nr)Zrfcollections.abcr dataclassesrrFrtorch.nnrrrr r- activationsr cache_utilsr r r generationr masking_utilsrrmodeling_layersrmodeling_outputsrrrrrrrrrmodeling_utilsrrprocessing_utilsr pytorch_utilsrutilsr r!r"r# utils.genericr$r%utils.output_capturingr&configuration_bertr( get_loggerrcrzModuler*rjfloatrrrrrrrrrrrr rrr#r'r;rBrYrvrrrrrr__all__r@rQrPrs$! AA&!CC)J9   G&6MMI5*   H %;RYY;H!% II% <<% % << % LL4' % T\ %%'(%8F) F)RJ)J)Z RYY .BII.> ryy   @*@F ")) D  "))" 299  !bii!&bii& 9299 9////2  7{7  7&  J6#J6 J6Z Y ,Y  Y x S )?S  S lI )I I X I $7I  I X M $7M  M `^ /^ ^ B8 !48 8 v@ 2@ @ F rQ