#i{dZddlZddlmZddlmZmZmZddlmZ ddl m Z ddl m Z ddlmZmZmZmZmZmZmZdd lmZdd lmZdd lmZmZmZdd lmZd dl m!Z!m"Z"m#Z#m$Z$m%Z%ddl&m'Z'ejPe)Z*Gdde"Z+Gdde%Z,Gdde!Z-Gdde#Z.eGddeZ/Gdde$Z0edGdd e/e Z1eGd!d"e/Z2Gd#d$ejfZ4ed%Gd&d'e/Z5eGd(d)e/Z6eGd*d+e/Z7Gd,d-ejfZ8eGd.d/e/Z9gd0Z:y)1zPyTorch RoBERTa model.N)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)gelu)GenerationMixin),BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)Unpack)TransformersKwargsauto_docstringlogging)can_return_tuple)BertCrossAttentionBertEmbeddings BertLayer BertModelBertSelfAttention) RobertaConfigc eZdZfdZ d dej dzdej dzdej dzdej dzdef dZe d Z e d d Z xZ S) RobertaEmbeddingsct|||`|`|j|_t j |j|j|j|_y)N) padding_idx) super__init__ pad_token_idposition_embeddingsr"nn Embeddingmax_position_embeddings hidden_sizeselfconfig __class__s g/mnt/e/genesis-system/.venv/lib/python3.12/site-packages/transformers/models/roberta/modular_roberta.pyr$zRobertaEmbeddings.__init__-sT     $!..#%<<  * *F,>,>DL\L\$  N input_idstoken_type_ids position_ids inputs_embedspast_key_values_lengthc|<||j||j|}n|j||j}||j}n|jdd}|\}}|t |drT|j j |jdd} tj| d|} | j ||}n:tj|tj|jj}||j|}|j|} || z} |j!|} | | z} |j#| } |j%| } | S)Nr2rr)dimindexdtypedevice)"create_position_ids_from_input_idsr"&create_position_ids_from_inputs_embedssizehasattrr2expandshapetorchgatherzeroslongr3r<word_embeddingstoken_type_embeddingsr& LayerNormdropout) r,r1r2r3r4r5 input_shape batch_size seq_lengthbuffered_token_type_idsrH embeddingsr&s r/forwardzRobertaEmbeddings.forward8sn  $#FFt//1G   $JJ=Z^ZjZjk  #..*K',,.s3K!, J  !t-.*.*=*=*D*D\EWEWXYEZ\^*_'*/,,7NTU]i*j'!8!?!? J!W!&[ SWSdSdSkSk!l   00;M $ : :> J"%:: "66|D"55 ^^J/ \\*- r0c|jdd}|d}tj|dz||zdztj|j}|j dj |S)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor Nr7rr:r)r?rCarangerFr< unsqueezerA)r4r"rKsequence_lengthr3s r/r>z8RobertaEmbeddings.create_position_ids_from_inputs_embedshsp$((*3B/ %a.|| !O_{:Q>ejjYfYmYm %%a(// <r= __classcell__r.s@r/r r ,s  .2260426&' .##d*.((4/.&&- . ((4/ . !$ .`==" 8 8r0r c eZdZy)RobertaSelfAttentionNr]r^r_r0r/rfrfr0rfc eZdZy)RobertaCrossAttentionNrgrhr0r/rkrkrir0rkc eZdZy) RobertaLayerNrgrhr0r/rmrmrir0rmcneZdZeZdZdZdZdZdZ dZ e e e dZej fdZxZS)RobertaPreTrainedModelrobertaT) hidden_states attentionscross_attentionsct||t|tr t j |j yt|tryt j|jtj|jjdjdt j |jyy)zInitialize the weightsr7)rr7N)r# _init_weights isinstance RobertaLMHeadinitzeros_biasr copy_r3rCrRrBrAr2)r,moduler.s r/ruz$RobertaPreTrainedModel._init_weightss f% fm , KK $  1 2 JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h KK-- .3r0)r]r^r_r config_classbase_model_prefixsupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendrmrfrk_can_record_outputsrCno_gradrurcrds@r/rorosX L!&*#N"&%*1 U]]_//r0roc eZdZdfd ZxZS) RobertaModelc&t|||yN)r#r$)r,r-add_pooling_layerr.s r/r$zRobertaModel.__init__s v&r0)T)r]r^r_r$rcrds@r/rrs ''r0rzS RoBERTa Model with a `language modeling` head on top for CLM fine-tuning. ) custom_introc eZdZdddZfdZdZdZee dde jdzd e jdzd e jdzd e jdzd e jdzd e jdzde jdzde jdzde e e jdzde dzde jdzdee jzdeede e jezfdZxZS)RobertaForCausalLM)roberta.embeddings.word_embeddings.weight lm_head.biaszlm_head.decoder.weightzlm_head.decoder.biasct|||jstj dt |d|_t||_|jy)NzOIf you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.`Fr r#r$ is_decoderloggerwarningrrprwlm_head post_initr+s r/r$zRobertaForCausalLM.__init__sL    NNl m#FeD $V,  r0c.|jjSrrdecoderr,s r/get_output_embeddingsz(RobertaForCausalLM.get_output_embeddings||###r0c&||j_yrrr,new_embeddingss r/set_output_embeddingsz(RobertaForCausalLM.set_output_embeddings- r0Nr1attention_maskr2r3r4encoder_hidden_statesencoder_attention_masklabelspast_key_values use_cachecache_positionlogits_to_keepkwargsreturnc  |d} |j|f||||||| | | dd | }|j}t| tr t | dn| }|j |dd|ddf}d}|*|j d|||jjd| }t|||j|j|j|jS)am token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Example: ```python >>> from transformers import AutoTokenizer, RobertaForCausalLM, AutoConfig >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("FacebookAI/roberta-base") >>> config = AutoConfig.from_pretrained("FacebookAI/roberta-base") >>> config.is_decoder = True >>> model = RobertaForCausalLM.from_pretrained("FacebookAI/roberta-base", config=config) >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.logits ```NFT) rr2r3r4rrrrr return_dict)logitsr vocab_size)lossrrrqrrrsrh)rplast_hidden_statervrXslicer loss_functionr-rr rrqrrrs)r,r1rr2r3r4rrrrrrrroutputsrq slice_indicesrrs r/rPzRobertaForCausalLM.forwardsb  I@L  A ))%'"7#9+) A  A  11 8B>SV8W~ot4]k mA}a,?@A  %4%%pVFt{{OeOepiopD0#33!//))$55   r0) NNNNNNNNNNNr)r]r^r__tied_weights_keysr$rrrrrCr`ratupleboolTensorrXrrr rPrcrds@r/rrs#N . $..237260426:>;?*.BF!%.2-.Q ##d*Q ))D0Q ((4/ Q &&- Q ((4/ Q  %0047Q !& 1 1D 8Q   4'Q uU%6%6784?Q $;Q  t+Q ell*Q +,Q  u|| @ @Q Q r0rceZdZdddZfdZdZdZee dde jdzd e jdzd e jdzd e jdzd e jdzd e jdzde jdzde jdzde e dee jezfdZxZS)RobertaForMaskedLMrrrct|||jrtj dt |d|_t||_|jy)NznIf you want to use `RobertaForMaskedLM` make sure `config.is_decoder=False` for bi-directional self-attention.Frrr+s r/r$zRobertaForMaskedLM.__init__0sR     NN1  $FeD $V,  r0c.|jjSrrrs r/rz(RobertaForMaskedLM.get_output_embeddings?rr0c&||j_yrrrs r/rz(RobertaForMaskedLM.set_output_embeddingsBrr0Nr1rr2r3r4rrrrrc t|j|f||||||dd| } | d} |j| } d} |a|j| j}t }|| j d|j j|j d} t| | | j| jS)a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` T)rr2r3r4rrrrNr7rrrqrr) rprtor<rviewr-rr rqrr)r,r1rr2r3r4rrrrrsequence_outputprediction_scoresmasked_lm_lossloss_fcts r/rPzRobertaForMaskedLM.forwardEs:$,,   ))%'"7#9    "!* LL9  YY0778F')H%&7&<&;?*.5 ##d*5 ))D05 ((4/ 5 &&- 5 ((4/ 5  %00475 !& 1 1D 85   4'5 +,5  u|| ~ -5 5 r0rc(eZdZdZfdZdZxZS)rwz*Roberta Head for masked language modeling.ct|tj|j|j|_tj |j|j|_tj|j|j|_ tjtj|j|_y)N)eps)r#r$r'Linearr*denserIlayer_norm_eps layer_normrr ParameterrCrErzr+s r/r$zRobertaLMHead.__init__s YYv1163E3EF ,,v'9'9v?T?TUyy!3!3V5F5FG LLV->->!?@ r0c |j|}t|}|j|}|j|}|Sr)rrrrr,featuresrxs r/rPzRobertaLMHead.forwards; JJx  G OOA  LLOr0r]r^r___doc__r$rPrcrds@r/rwrws4Ar0rwz RoBERTa Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c*eZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd e e d e eje zfd ZxZS) RobertaForSequenceClassificationct|||j|_||_t |d|_t ||_|jyNFr) r#r$ num_labelsr-rrpRobertaClassificationHead classifierrr+s r/r$z)RobertaForSequenceClassification.__init__sJ   ++ #FeD 3F; r0Nr1rr2r3r4rrrc |j|f||||dd|}|d} |j| } d} ||j| j}|jj |j dk(rd|j_nl|j dkDrL|jtjk(s|jtjk(rd|j_nd|j_|jj dk(rIt} |j dk(r&| | j|j} n| | |} n|jj dk(r=t} | | jd |j |jd } n,|jj dk(rt} | | |} t!| | |j"|j$ S) a token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Trr2r3r4rrNr regressionsingle_label_classificationmulti_label_classificationr7r)rprrr<r- problem_typerr;rCrFrXrsqueezerrrrrqrr r,r1rr2r3r4rrrrrrrs r/rPz(RobertaForSequenceClassification.forwards6$,,  ))%'   "!*1  YYv}}-F{{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#FNN$4fnn6FGD#FF3D))-JJ+- B @&++b/R))-II,./'!//))   r0NNNNNN)r]r^r_r$rrrCr`rarrrrrrPrcrds@r/rrs .237260426*.C ##d*C ))D0C ((4/ C &&- C ((4/ C   4'C +,C  u|| 7 7C C r0rc*eZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd e e d e eje zfd ZxZS) RobertaForMultipleChoicect||t||_t j |j |_t j|jd|_ |jy)Nr) r#r$rrpr'Dropouthidden_dropout_probrJrr*rrr+s r/r$z!RobertaForMultipleChoice.__init__sV  #F+ zz&"<"<= ))F$6$6: r0Nr1r2rrr3r4rrc "||jdn|jd}|!|jd|jdnd} |!|jd|jdnd} |!|jd|jdnd} |!|jd|jdnd} |1|jd|jd|jdnd} |j| f| | | | dd|}|d}|j |}|j |}|jd|}d}|.|j |j}t}|||}t|||j|jS)a input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. Nrr7T)r3r2rr4rr) rBrr?rprJrrr<rr rqrr)r,r1r2rrr3r4r num_choicesflat_input_idsflat_position_idsflat_token_type_idsflat_attention_maskflat_inputs_embedsr pooled_outputrreshaped_logitsrrs r/rPz RobertaForMultipleChoice.forwardsV-6,Aiooa(}GZGZ[\G] CLCXINN2,>?^bLXLdL--b,2C2CB2GHjnR`Rln11"n6I6I"6MNrvR`Rln11"n6I6I"6MNrv(   r=#5#5b#9=;M;Mb;Q R  $,,  *..,      ]3 / ++b+6  YY556F')HOV4D("!//))   r0r)r]r^r_r$rrrCr`rarrrrr rPrcrds@r/rrs.22637*.0426P ##d*P ((4/P ))D0 P   4' P &&- P ((4/P +,P  u|| 8 8P P r0rc*eZdZfdZee d dejdzdejdzdejdzdejdzdejdzdejdzd e e d e eje zfd ZxZS) RobertaForTokenClassificationcdt|||j|_t|d|_|j |j n |j }tj||_ tj|j|j|_ |jyr)r#r$rrrpclassifier_dropoutrr'rrJrr*rrr,r-rr.s r/r$z&RobertaForTokenClassification.__init__Rs   ++#FeD )/)B)B)NF % %TZTnTn zz"45 ))F$6$68I8IJ r0Nr1rr2r3r4rrrc ~|j|f||||dd|}|d} |j| } |j| } d} |W|j| j}t } | | j d|j|j d} t| | |j|jS)a- token_type_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0,1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. This parameter can only be used when the model is initialized with `type_vocab_size` parameter with value >= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. TrrNr7r) rprJrrr<rrrrrqrrrs r/rPz%RobertaForTokenClassification.forward`s2$,,  ))%'   "!*,,71  YYv}}-F')HFKKDOO= 2. All the value in this tensor should be always < type_vocab_size. [What are token type IDs?](../glossary#token-type-ids) Trrrr7rVN) ignore_indexr)r start_logits end_logitsrqrr) rprsplitr contiguouslenr?clamprrrqrr)r,r1rr2r3r4rrrrrrr r  total_loss ignored_indexr start_lossend_losss r/rPz#RobertaForQuestionAnswering.forwards0$,,  ))%'   "!*1#)<<r<#: j#++B/::< ''+668   &=+D?'')*Q."1"9"9""==%%'(1, - 5 5b 9 (--a0M-33A}EO)//=AM']CH!,@J M:H$x/14J+%!!//))   r0)NNNNNNN)r]r^r_r$rrrCr`rarrrrrrPrcrds@r/rrs.2372604263715> ##d*> ))D0> ((4/ > &&- > ((4/ > ))D0> ''$.> +,>  u|| ; ;> > r0r)rrrrrrrro);rrCtorch.nnr'rrrrrx activationsr generationr modeling_outputsr r r r rrrmodeling_utilsrprocessing_utilsrutilsrrr utils.genericrbert.modeling_bertrrrrrconfiguration_robertar get_loggerr]rr rfrkrmrorrrModulerwrrrrr__all__rhr0r/r!s AA&).&@@-ll0   H %\8\8~ ,  .  9 /_//2'9'  k /k  k \R /R R jBII, Q '=Q  Q h] 5] ] @C $:C C L ,K "8K K \ r0