#iddlmZddlZddlmZddlmZmZmZddlm Z ddl m Z m Z ddlmZmZmZddlmZdd lmZmZdd lmZdd lmZmZmZmZmZmZm Z m!Z!dd l"m#Z#m$Z$dd l%m&Z&ddl'm(Z(ddl)m*Z*m+Z+m,Z,ddl-m.Z.m/Z/ddl0m1Z1ddl2m3Z3e,jhe5Z6GddejnZ8 dJdejndejrdejrdejrdejrdzde:dzde:de&e*fdZ;Gdd ejnZ<Gd!d"ejnZ=Gd#d$ejnZ>Gd%d&ejnZ?Gd'd(ejnZ@Gd)d*ejnZAGd+d,eZBe+Gd-d.e$ZCGd/d0ejnZDGd1d2ejnZEe+d34Gd5d6eCZFe+d74Gd8d9eCeZGe+Gd:d;eCZHGd<d=ejnZIe+d>4Gd?d@eCZJe+GdAdBeCZKe+GdCdDeCZLGdEdFejnZMe+GdGdHeCZNgdIZOy)K)CallableN)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FNgelu)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward)TransformersKwargsauto_docstringlogging)can_return_tuplemerge_with_config_defaults)capture_outputs) RobertaConfigc eZdZdZfdZ d dej dzdej dzdej dzdejdzded ejf d Z e d Z e dd Z xZS)RobertaEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.cTt|tj|j|j |j |_tj|j|j |_ tj|j |j|_ tj|j|_|jdt!j"|j$j'dd|jdt!j(|j*j-t j.d|j |_tj|j$|j |j0|_y) N) padding_idxeps position_idsr$F) persistenttoken_type_ids)dtype)super__init__nn Embedding vocab_size hidden_size pad_token_idword_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_buffertorcharangemax_position_embeddingsexpandzerosr,sizelongr)position_embeddingsselfconfig __class__s h/mnt/e/genesis-system/.venv/lib/python3.12/site-packages/transformers/models/roberta/modeling_roberta.pyr3zRobertaEmbeddings.__init__;s4 !||F,=,=v?Q?Q_e_r_rs%'\\&2H2H&J\J\%]"f&8&8f>S>STzz&"<"<=   ELL)G)GHOOPWXej    ekk$*;*;*@*@*B%**Ubg  "..#%<<  * *F,>,>DL\L\$  N input_idsr0r, inputs_embedspast_key_values_lengthreturnc|<||j||j|}n|j||j}||j}n|jdd}|\}}|t |drT|j j |jdd} tj| d|} | j ||}n:tj|tj|jj}||j|}|j|} || z} |j!|} | | z} |j#| } |j%| } | S)Nr.r0rr$)dimindexr1device)"create_position_ids_from_input_idsr)&create_position_ids_from_inputs_embedsrGhasattrr0rEshaperBgatherrFrHr,rXr9r;rIr<r@) rKrPr0r,rQrR input_shape batch_size seq_lengthbuffered_token_type_idsr; embeddingsrIs rNforwardzRobertaEmbeddings.forwardOsn  $#FFt//1G   $JJ=Z^ZjZjk  #..*K',,.s3K!, J  !t-.*.*=*=*D*D\EWEWXYEZ\^*_'*/,,7NTU]i*j'!8!?!? J!W!&[ SWSdSdSkSk!l   00;M $ : :> J"%:: "66|D"55 ^^J/ \\*- rOc|jdd}|d}tj|dz||zdztj|j}|j dj |S)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor Nr.r$rWr)rGrBrCrHrX unsqueezerE)rQr)r^sequence_lengthr,s rNrZz8RobertaEmbeddings.create_position_ids_from_inputs_embedssp$((*3B/ %a.|| !O_{:Q>ejjYfYmYm %%a(// <L==((6??([L,,|U3K''1-88:K  $$rOceZdZd fd Z d dej dej dzdedzdej dzdee de ej f d Z xZ S) RobertaSelfAttentionNc@t||j|jzdk7r2t |ds&t d|jd|jd||_|j|_t|j|jz |_|j|jz|_ |jdz|_ tj|j|j|_ tj|j|j|_tj|j|j|_tj |j"|_|j&|_||_||_yNrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()r)r2r3r7num_attention_headsr[ ValueErrorrLrjattention_head_size all_head_sizer~r4Linearrzr{r|r>attention_probs_dropout_probr@ is_decoder is_causal layer_idxrKrLrrrMs rNr3zRobertaSelfAttention.__init__sP     : : :a ?PVXhHi#F$6$6#78 445Q8  #)#=#= #&v'9'9F?*9*N*N'&=%C%C!>2 & "I{)@(M(M KK , ,.E) %8      % $}}C$,,..LL %  % ! \*k));;;;FFH L((rOFNNNN rorprqr3rBrurtr rrtuplercrwrxs@rNrrs}#648(,.2 -)||-)))D0-) -)  t+ -) +, -) u|| -)rOrceZdZd fd Z d dej dej dzdej dzdedzdee de ej f d Z xZ S) RobertaCrossAttentionNct||j|jzdk7r2t |ds&t d|jd|jd||_|j|_t|j|jz |_|j|jz|_ |jdz|_ tj|j|j|_ tj|j|j|_tj|j|j|_tj |j"|_||_||_yr)r2r3r7rr[rrLrjrrr~r4rrzr{r|r>rr@rrrs rNr3zRobertaCrossAttention.__init__sC     : : :a ?PVXhHi#F$6$6#78 445Q8  #)#=#= #&v'9'9FB**4>>:(?(M(M KK , ,.E) %8      % $}}C$,,..LL %  % ! \"))#w;FFH L((rOrr) rorprqr3rBrurtr rrrrcrwrxs@rNrrs#4;?376: 2)||2) %00472)))D0 2) -t3 2) +, 2) u|| 2)rOrcneZdZfdZdej dej dej fdZxZS)RobertaSelfOutputc(t|tj|j|j|_tj |j|j|_tj|j|_ yNr*) r2r3r4rr7denser<r=r>r?r@rJs rNr3zRobertaSelfOutput.__init__Us` YYv1163E3EF f&8&8f>S>STzz&"<"<= rOr input_tensorrScr|j|}|j|}|j||z}|SNrr@r<rKrrs rNrczRobertaSelfOutput.forward[7 =1  ]3 }|'CD rOrorprqr3rBrurcrwrxs@rNrrT1> U\\RWR^R^rOrceZdZd fd Z d dej dej dzdej dzdej dzdedzdej dzd ee d e ej fd Z xZ S)RobertaAttentionNct|||_|rtnt}|||||_t ||_y)Nrr)r2r3is_cross_attentionrrrKroutput)rKrLrrrattention_classrMs rNr3zRobertaAttention.__init__cs= "43E/K_#Fi9U '/ rOrr}rencoder_attention_maskrrrrSc |js|n|}|j|f||||d|\}} |j||}|| fS)N)rr}rr)rrKr) rKrr}rrrrrattention_outputrs rNrczRobertaAttention.forwardjsg04/F/FLb)2 * "7)+) *  * &, ;;'7G--rO)FNFNNNNNrrxs@rNrrbs048:>;?(,.2.||.))D0. %0047 . !& 1 1D 8 .  . t+.+,. u|| .rOrcVeZdZfdZdej dej fdZxZS)RobertaIntermediatect|tj|j|j |_t|jtrt|j|_ y|j|_ yr) r2r3r4rr7intermediate_sizerr hidden_actstrr intermediate_act_fnrJs rNr3zRobertaIntermediate.__init__s] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $rOrrScJ|j|}|j|}|Sr)rr)rKrs rNrczRobertaIntermediate.forwards& =1 00? rOrrxs@rNrrs#9U\\ellrOrcneZdZfdZdej dej dej fdZxZS) RobertaOutputc(t|tj|j|j |_tj|j |j|_tj|j|_ yr) r2r3r4rrr7rr<r=r>r?r@rJs rNr3zRobertaOutput.__init__s` YYv779K9KL f&8&8f>S>STzz&"<"<= rOrrrScr|j|}|j|}|j||z}|Srrrs rNrczRobertaOutput.forwardrrOrrxs@rNrrrrOrceZdZd fd Z ddej dej dzdej dzdej dzdedzdej dzd ee d e ej fd Z d Z xZ S) RobertaLayerNct||j|_d|_t ||j ||_|j |_|j|_|jr.|j st|dt |d|d|_ t||_ t||_ y)Nr$rz> should be used as a decoder model if cross attention is addedFT)rrr)r2r3chunk_size_feed_forward seq_len_dimrr attentionadd_cross_attentionrcrossattentionr intermediaterr)rKrLrrMs rNr3zRobertaLayer.__init__s '-'E'E$)&F>  $ ,) $  $  q1 ??4@4!12 =dVDDD )<(;(;%%& ) !0 )  ) % "A 6 0  # #T%A%A4CSCSUe rOcL|j|}|j||}|Sr)rr)rKrintermediate_outputrs rNrzRobertaLayer.feed_forward_chunks,"//0@A{{#68HI rOrr)rorprqr3rBrurtr rrrrcrrwrxs@rNrrs,,48:>;?(,.2'||'))D0' %0047 ' !& 1 1D 8 '  ' t+'+,' u|| 'RrOrcneZdZeZdZdZdZdZdZ dZ e e e dZej fdZxZS)RobertaPreTrainedModelrobertaT)r attentionscross_attentionsct||t|tr t j |j yt|tryt j|jtj|jjdjdt j |jyy)zInitialize the weightsr.r-N)r2 _init_weightsr RobertaLMHeadinitzeros_biasr'copy_r,rBrCr\rEr0)rKryrMs rNrz$RobertaPreTrainedModel._init_weightss f% fm , KK $  1 2 JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h KK-- .3rO)rorprqr% config_classbase_model_prefixsupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendrrr_can_record_outputsrBno_gradrrwrxs@rNrrsX L!&*#N"&%*1 U]]_//rOrceZdZfdZ d dej dej dzdej dzdej dzdedzdedzd ej dzd e e d e ej e zfd Z xZS)RobertaEncoderc t|||_tjt |j Dcgc]}t||c}|_ycc}w)N)r) r2r3rLr4 ModuleListrangenum_hidden_layersrlayer)rKrLirMs rNr3zRobertaEncoder.__init__sF  ]]uU[UmUmOn#o!L1$E#op #osANrr}rrr use_cacherrrSc t|jD]\} } | |||f|||d|}t||r|SdS)N)rrr)last_hidden_stater) enumeraterr) rKrr}rrrrrrr layer_modules rNrczRobertaEncoder.forwardsq )4 OA|(%(> /- M 9+/8O  >B  rONNNNNN)rorprqr3rBrurtr boolrrrrrcrwrxs@rNrrsq48:>;?(,!%.2 || ))D0  %0047  !& 1 1D 8    $;  t+ +,  u|| H H rOrcVeZdZfdZdej dej fdZxZS) RobertaPoolerct|tj|j|j|_tj |_yr)r2r3r4rr7rTanh activationrJs rNr3zRobertaPooler.__init__s9 YYv1163E3EF '')rOrrSc\|dddf}|j|}|j|}|SNr)rr!)rKrfirst_token_tensor pooled_outputs rNrczRobertaPooler.forward#s6+1a40 #56  6 rOrrxs@rNrrs#$ U\\ellrOra  The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in [Attention is all you need](https://huggingface.co/papers/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. To behave as an decoder the model needs to be initialized with the `is_decoder` argument of the configuration set to `True`. To be used in a Seq2Seq model, the model needs to initialized with both `is_decoder` argument and `add_cross_attention` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass. ) custom_introceZdZddgZdfd ZdZdZeee dde jdzde jdzd e jdzd e jdzd e jdzd e jdzd e jdzde dzde dzde jdzdeedee jezfdZdZxZS) RobertaModelr'rct||||_d|_t ||_t ||_|r t|nd|_ |jy)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer FN) r2r3rLgradient_checkpointingr'rbrencoderrpooler post_init)rKrLadd_pooling_layerrMs rNr3zRobertaModel.__init__;sU   &+#+F3%f- /@mF+d  rOc.|jjSrrbr9rKs rNget_input_embeddingsz!RobertaModel.get_input_embeddingsLs...rOc&||j_yrr0)rKr|s rNset_input_embeddingsz!RobertaModel.set_input_embeddingsOs*/'rONrPr}r0r,rQrrrrrrrSc L|jjr| | n|jj} nd} | rd|b||jjr4t t |jt |jnt |j}|du|duz r t d||j} |jd} n|j} |jd} ||jnd}| tj||| z| } |j|||||}|j||||| |\}}|j|f||||| | |d | }|j}|j |j!|nd}t#|||j$ S) NF)rLz:You must specify exactly one of input_ids or inputs_embedsr$r)rX)rPr,r0rQrR)r}rembedding_outputrrr)r}rrrrrr,)r pooler_outputr)rLrris_encoder_decoderr r rrXr\get_seq_lengthrBrCrb_create_attention_masksr+rr,rr)rKrPr}r0r,rQrrrrrrrXr`rRr6encoder_outputssequence_outputr%s rNrczRobertaModel.forwardRs" ;; ! !%.%:  @U@UII 0)4 8V8V$L $DlZ^ZeZeFfg!5   -t"; <YZ Z  %%F"+J"))F&,,Q/JETE`!?!?!Afg  !"\\*@BX[eBentuN??%)'#9 + 261M1M)#9-"7)+ 2N2 ..'$,,   )"7#9+)%    *;;8< 8O O4UY ;-'+;;  rOc|jjrt|j||||}nt|j||}|t|j|||}||fS)N)rLrQr}rr)rLrQr})rLrQr}r)rLrrr)rKr}rr6rrrs rNr:z$RobertaModel._create_attention_maskssx ;; ! !/{{.-- / N7{{.-N " -%>{{.5&; & "555rO)T) NNNNNNNNNN)rorprq_no_split_modulesr3r2r4r"r#rrBrur rrrrrrcr:rwrxs@rNr(r(,sR-n="/0 *..2.2,0-1596:(,!%.2K <<$&K  t+K  t+ K llT) K ||d* K  %||d2K !& t 3K K $;K  t+K +,K  u|| K KK  K Z 6rOr(zS RoBERTa Model with a `language modeling` head on top for CLM fine-tuning. c eZdZdddZfdZdZdZee dde jdzd e jdzd e jdzd e jdzd e jdzd e jdzde jdzde jdzde e e jdzde dzde jdzdee jzdeede e jezfdZxZS)RobertaForCausalLM)roberta.embeddings.word_embeddings.weight lm_head.biaszlm_head.decoder.weightzlm_head.decoder.biasct|||jstj dt |d|_t||_|jy)NzOIf you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.`Fr. r2r3rloggerwarningr(rrlm_headr-rJs rNr3zRobertaForCausalLM.__init__sL    NNl m#FeD $V,  rOc.|jjSrrIdecoderr1s rNget_output_embeddingsz(RobertaForCausalLM.get_output_embeddings||###rOc&||j_yrrKrKnew_embeddingss rNset_output_embeddingsz(RobertaForCausalLM.set_output_embeddings- rONrPr}r0r,rQrrlabelsrrrlogits_to_keeprrSc  |d} |j|f||||||| | | dd | }|j}t| tr t | dn| }|j |dd|ddf}d}|*|j d|||jjd| }t|||j|j|j|jS)am token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Example: ```python >>> from transformers import AutoTokenizer, RobertaForCausalLM, AutoConfig >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("FacebookAI/roberta-base") >>> config = AutoConfig.from_pretrained("FacebookAI/roberta-base") >>> config.is_decoder = True >>> model = RobertaForCausalLM.from_pretrained("FacebookAI/roberta-base", config=config) >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.logits ```NFT) r}r0r,rQrrrrr return_dict)logitsrTr6)lossrXrrrr)rrrrjslicerI loss_functionrLr6rrrrr)rKrPr}r0r,rQrrrTrrrrUroutputsr slice_indicesrXrYs rNrczRobertaForCausalLM.forwardsb  I@L  A ))%'"7#9+) A  A  11 8B>SV8W~ot4]k mA}a,?@A  %4%%pVFt{{OeOepiopD0#33!//))$55   rO) NNNNNNNNNNNr)rorprq_tied_weights_keysr3rMrRr!rrBrsrtrrrurjrrrrcrwrxs@rNr@r@s#N . $..237260426:>;?*.BF!%.2-.Q ##d*Q ))D0Q ((4/ Q &&- Q ((4/ Q  %0047Q !& 1 1D 8Q   4'Q uU%6%6784?Q $;Q  t+Q ell*Q +,Q  u|| @ @Q Q rOr@ceZdZdddZfdZdZdZee dde jdzd e jdzd e jdzd e jdzd e jdzd e jdzde jdzde jdzde e dee jezfdZxZS)RobertaForMaskedLMrArBrCct|||jrtj dt |d|_t||_|jy)NznIf you want to use `RobertaForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention.FrErFrJs rNr3zRobertaForMaskedLM.__init__?sR     NN1  $FeD $V,  rOc.|jjSrrKr1s rNrMz(RobertaForMaskedLM.get_output_embeddingsNrNrOc&||j_yrrKrPs rNrRz(RobertaForMaskedLM.set_output_embeddingsQrSrONrPr}r0r,rQrrrTrrSc t|j|f||||||dd| } | d} |j| } d} |a|j| j}t }|| j d|j j|j d} t| | | j| jS)a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` T)r}r0r,rQrrrWrNr.rYrXrr) rrItorXrrrLr6rrr)rKrPr}r0r,rQrrrTrr]r<prediction_scoresmasked_lm_lossloss_fcts rNrczRobertaForMaskedLM.forwardTs:$,,   ))%'"7#9    "!* LL9  YY0778F')H%&7&<&;?*.5 ##d*5 ))D05 ((4/ 5 &&- 5 ((4/ 5  %00475 !& 1 1D 85   4'5 +,5  u|| ~ -5 5 rOrac(eZdZdZfdZdZxZS)rz*Roberta Head for masked language modeling.ct|tj|j|j|_tj |j|j|_tj|j|j|_ tjtj|j|_yr)r2r3r4rr7rr<r= layer_normr6rL ParameterrBrFrrJs rNr3zRobertaLMHead.__init__s YYv1163E3EF ,,v'9'9v?T?TUyy!3!3V5F5FG LLV->->!?@ rOc |j|}t|}|j|}|j|}|Sr)rr rmrLrKfeaturesrxs rNrczRobertaLMHead.forwards; JJx  G OOA  LLOrOrorprqrrr3rcrwrxs@rNrrs4ArOrz RoBERTa Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c*eZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd e e d e eje zfd ZxZS) RobertaForSequenceClassificationct|||j|_||_t |d|_t ||_|jyNFrE) r2r3 num_labelsrLr(rRobertaClassificationHead classifierr-rJs rNr3z)RobertaForSequenceClassification.__init__sJ   ++ #FeD 3F; rONrPr}r0r,rQrTrrSc |j|f||||dd|}|d} |j| } d} ||j| j}|jj |j dk(rd|j_nl|j dkDrL|jtjk(s|jtjk(rd|j_nd|j_|jj dk(rIt} |j dk(r&| | j|j} n| | |} n|jj dk(r=t} | | jd |j |jd } n,|jj dk(rt} | | |} t!| | |j"|j$ S) a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Tr}r0r,rQrWrNr$ regressionsingle_label_classificationmulti_label_classificationr.rf)rrzrgrXrL problem_typerxr1rBrHrjrsqueezerrrrrr rKrPr}r0r,rQrTrr]r<rXrYrjs rNrcz(RobertaForSequenceClassification.forwards6$,,  ))%'   "!*1  YYv}}-F{{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#FNN$4fnn6FGD#FF3D))-JJ+- B @&++b/R))-II,./'!//))   rOr)rorprqr3r!rrBrsrtrrrrurrcrwrxs@rNrurus .237260426*.C ##d*C ))D0C ((4/ C &&- C ((4/ C   4'C +,C  u|| 7 7C C rOruc*eZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd e e d e eje zfd ZxZS) RobertaForMultipleChoicect||t||_t j |j |_t j|jd|_ |jy)Nr$) r2r3r(rr4r>r?r@rr7rzr-rJs rNr3z!RobertaForMultipleChoice.__init__sV  #F+ zz&"<"<= ))F$6$6: rONrPr0r}rTr,rQrrSc "||jdn|jd}|!|jd|jdnd} |!|jd|jdnd} |!|jd|jdnd} |!|jd|jdnd} |1|jd|jd|jdnd} |j| f| | | | dd|}|d}|j |}|j |}|jd|}d}|.|j |j}t}|||}t|||j|jS)a input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. Nr$r.T)r,r0r}rQrWrf) r\rrGrr@rzrgrXrrrr)rKrPr0r}rTr,rQr num_choicesflat_input_idsflat_position_idsflat_token_type_idsflat_attention_maskflat_inputs_embedsr]r%rXreshaped_logitsrYrjs rNrcz RobertaForMultipleChoice.forward sV-6,Aiooa(}GZGZ[\G] CLCXINN2,>?^bLXLdL--b,2C2CB2GHjnR`Rln11"n6I6I"6MNrvR`Rln11"n6I6I"6MNrv(   r=#5#5b#9=;M;Mb;Q R  $,,  *..,      ]3 / ++b+6  YY556F')HOV4D("!//))   rOr)rorprqr3r!rrBrsrtrrrrurrcrwrxs@rNrrs.22637*.0426P ##d*P ((4/P ))D0 P   4' P &&- P ((4/P +,P  u|| 8 8P P rOrc*eZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd e e d e eje zfd ZxZS) RobertaForTokenClassificationcdt|||j|_t|d|_|j |j n |j }tj||_ tj|j|j|_ |jyrw)r2r3rxr(rclassifier_dropoutr?r4r>r@rr7rzr-rKrLrrMs rNr3z&RobertaForTokenClassification.__init__as   ++#FeD )/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ rONrPr}r0r,rQrTrrSc ~|j|f||||dd|}|d} |j| } |j| } d} |W|j| j}t } | | j d|j|j d} t| | |j|jS)a- token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Tr|rNr.rf) rr@rzrgrXrrrxrrrrs rNrcz%RobertaForTokenClassification.forwardos2$,,  ))%'   "!*,,71  YYv}}-F')HFKKDOOr@rxout_projrs rNr3z"RobertaClassificationHead.__init__s YYv1163E3EF )/)B)B)NF % %TZTnTn zz"45  &"4"4f6G6GH rOc |dddddf}|j|}|j|}tj|}|j|}|j |}|Sr#)r@rrBtanhrrps rNrcz!RobertaClassificationHead.forwardsY Q1W  LLO JJqM JJqM LLO MM! rOrsrxs@rNryrys7IrOrycJeZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd ejdzd e e d e eje zfd ZxZS)RobertaForQuestionAnsweringct|||j|_t|d|_t j |j|j|_|jyrw) r2r3rxr(rr4rr7 qa_outputsr-rJs rNr3z$RobertaForQuestionAnswering.__init__sU   ++#FeD ))F$6$68I8IJ rONrPr}r0r,rQstart_positions end_positionsrrSc |j|f||||dd|} | d} |j| } | jdd\} } | jdj } | jdj } d}||t |j dkDr|jd}t |j dkDr|jd}| j d}|jd|}|jd|}t|}|| |}|| |}||zd z }t|| | | j| j S) a[ token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) Tr|rr$r.rhN) ignore_indexr)rY start_logits end_logitsrr) rrsplitrrlenrGclamprrrr)rKrPr}r0r,rQrrrr]r<rXrr total_loss ignored_indexrj start_lossend_losss rNrcz#RobertaForQuestionAnswering.forwards0$,,  ))%'   "!*1#)<<r<#: j#++B/::< ''+668   &=+D?'')*Q."1"9"9""==%%'(1, - 5 5b 9 (--a0M-33A}EO)//=AM']CH!,@J M:H$x/14J+%!!//))   rO)NNNNNNN)rorprqr3r!rrBrsrtrrrrurrcrwrxs@rNrrs.2372604263715> ##d*> ))D0> ((4/ > &&- > ((4/ > ))D0> ''$.> +,>  u|| ; ;> > rOr)r@rarrrurr(r)Nr)Pcollections.abcrrBtorch.nnr4rrrrr activationsr r cache_utilsr r r generationr masking_utilsrrmodeling_layersrmodeling_outputsrrrrrrrrmodeling_utilsrrprocessing_utilsr pytorch_utilsrutilsrrr utils.genericr!r"utils.output_capturingr#configuration_robertar% get_loggerrorGModuler'rufloatrrrrrrrrrrrr(r@rarrurrryr__all__rZrOrNrs,% AA&'CC)J9   G&6@@I50   H %g8 g8`!% II% <<% % << % LL4' % T\ %%'(%8F)299F)RJ)BIIJ)Z  .ryy.> "))  BII @-@F/_//2 RYY D BII   J6)J6 J6Z k /k  k \R /R R jBII, Q '=Q  Q h] 5] ] @C $:C C L ,K "8K K \ rO