i!dZddlZddlZddlmZddlmZmZmZddlm Z ddl m Z ddl mZmZmZdd lmZdd lmZmZdd lmZdd lmZmZmZmZmZmZmZdd l m!Z!ddl"m#Z#m$Z$m%Z%ddl&m'Z'e$jPe)Z*dejVde,de,fdZ-Gddej\Z/Gddej`Z1GddeZ2GddeZ3Gddej`Z4Gdd ej`Z5e#Gd!d"e!Z6Gd#d$e6Z7Gd%d&e6Z8e#Gd'd(e6Z9e#d)*Gd+d,e6eZ:e#d-*Gd.d/e6Z;e#Gd0d1e6Z<Gd2d3e6Z=Gd4d5e6eZ>gd6Z?y)7zPyTorch MVP model.N)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput#Seq2SeqQuestionAnsweringModelOutputSeq2SeqSequenceClassifierOutput)PreTrainedModel)auto_docstringloggingtorch_compilable_check) MvpConfig input_ids pad_token_iddecoder_start_token_idc|j|j}|ddddfj|ddddf<||dddf<| td|j |dk(||S)z1 Shift input ids one token to the right. Nrrz1self.model.config.pad_token_id has to be defined.i) new_zerosshapeclone ValueError masked_fill_)rrr shifted_input_idss `/mnt/e/genesis-system/.venv/lib/python3.12/site-packages/transformers/models/mvp/modeling_mvp.pyshift_tokens_rightr*.s}"++IOO<(CRC0668ae4adLMM""#4#(HPUPZPZcgcncncucufS"o (11!4Lw|dkk9::r+)rN) __name__ __module__ __qualname____doc__intr5r=TensorrC __classcell__r7s@r)r-r-?sWFsF3Fmq ; ;?B ;V[VbVbeiVi ; ;r+r-cpeZdZdZ ddedededzdedzdedzdedzf fd Z dd ejd ejdzd e dzd ejdzdejdzdedejdzde ejejdze ejdzffdZ xZ S) MvpAttentionz=Multi-headed attention from 'Attention Is All You Need' paperN embed_dim num_headsdropout is_decoderbias layer_idxct|||_||_||_||z|_|j |z|jk7rt d|jd|d|j dz|_||_||_ tj||||_ tj||||_ tj||||_tj||||_y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).g࿩rT)r4r5rPrQrRhead_dimr&scalingrSrUrLineark_projv_projq_projout_proj)r6rPrQrRrSrTrUr7s r)r5zMvpAttention.__init__]s "" !Y. MMI %$.. 8MdnnM]$YKr3 }}d* $"ii 94@ ii 94@ ii 94@  )YTB r+ hidden_stateskey_value_statespast_key_valuesattention_mask attn_promptoutput_attentionscache_positionreturnc |du}|j\} } } |j||jz} d} |St|trA|j j |j} |r |j}n|j}n|}|r|n|}|rK|I| rGj|jj}|j|jj}n|j|}|j|}|j| d|j |j"j%dd}|j| d|j |j"j%dd}|T|s|nd}j'|||jd|i\}}|r)t|trd|j |j<|t)j*|dj-| ddd|gd }t)j*|dj-| ddd|gd }|\t)j.| d| |djdj1|j2}t)j*||gd }| |j zd|j"f}| j| | |j |j"j%dd} | j4|} |j4|}|j4|}|jd}t)j6| |j%dd}|j| |j z| |fk7r/t9d | |j z| |fd |j|{|j| d| |fk7r#t9d | d| |fd |j|j| |j | ||z}|j| |j z| |}t:j<j?|d }|r?|j| |j | |}|j| |j z| |}nd}t:j<jA||j@|jB }t)j6||}|j| |j z| |j"fk7r7t9d| |j | |j"fd |j|j| |j | |j"}|j%dd}|j5| | |jD}|jG|}||fS)z#Input shape: Batch x Time x ChannelNFr"rr2reTrdimz$Attention weights should be of size z , but is z!Attention mask should be of size ptrainingz `attn_output` should be of size )$sizer]rY isinstancer is_updatedgetrUcross_attention_cacheself_attention_cachelayerskeysvaluesr[r\viewrQrX transposeupdater=catrAzerostor<reshapebmmr&r functionalsoftmaxrRrlrPr^)r6r_r`rarbrcrdreis_cross_attentionrDtgt_len_ query_statesrocurr_past_key_valuescurrent_states key_states value_states prompt_mask proj_shapesrc_len attn_weightsattn_weights_reshaped attn_probs attn_outputs r)rCzMvpAttention.forwardzsN.T9',,.Wa{{=1DLL@   &/+>?,77;;DNNK %+:+P+P(+:+O+O('6$-?)] /"=*-44T^^DIIJ/66t~~FMML^4J;;~6L#b$..$--PZZ[\^_`J',,S"dnndmmT^^_`bcdL*7It+?+F+F dnn?OQ_>`,( L&*_FY*ZAEO..t~~>  "KN$9$9#r2r$JJ#W]^_J 99k!n&;&;CR&Ll%[abcL)#kk#q';q>;N;Nq;QRUUVdVkVkl !&K+Hr!SDNN*B > #((gt~~t}}U__`acde +|++Z8 'Z''4 +|++Z8 //!$yyz/C/CAq/IJ    3#7'"J J6dnn8LgW^7_6`a %%'(*   %""$a'(BB 7a'8R7SS\]k]p]p]r\st(,,S$..'7SVddL',,S4>>-A7GTL}},,\r,B  %1$5$5c4>>7T[$\ !055cDNN6JGU\]L$( !]]**<4<$>?"(";";99T^^V-C-CD99V33T^^D " T^^ <r+r_rbself_attn_promptrdNrfc|}|j||||\}}tjj||j|j}||z}|j |}|}|j |j|}tjj||j|j}|j|}tjj||j|j}||z}|j|}|jtjk(rtj|js#tj |jrEtj"|jj$dz }tj&|| |}||fS)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. self_attn_prompt (`torch.FloatTensor`): prompt of self attention of shape `(2, encoder_attention_heads, pro_len, head_dim)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )r_rbrcrdrji)minmax)rrr~rRrlrrrrrrr;r=float16isinfanyisnanfinforclamp)r6r_rbrrdresidualr clamp_values r)rCzMvpEncoderLayer.forwards|$!&*nn')(/ '5' # |  --mt||VZVcVc-d  =0 11-@  **488M+BC  --mt?V?Vaeanan-o /  --mt||VZVcVc-d  =0 --m<   %-- / KK & * * , M0J0N0N0P++m&9&9:>>EK!KK K<[YMl**r+F) rFrGrHrr5r= FloatTensorrrrCrLrMs@r)rrsz=y=**/ ++((++))++ ++ ++  $; ++ u  %"3"3d":: ; ++r+rceZdZddeffd Z ddej dej dzdej dzdej dzdej dzd ej dzd edzd edzd edzd ej dzde eje ejejfdzffdZ xZ S)MvpDecoderLayerNrct||j|_t |j|j |j d||_|j|_t|j|_ |j|_ tj|j|_t |j|j |j d||_tj|j|_tj$|j|j&|_tj$|j&|j|_tj|j|_y)NT)rPrQrRrSrU)rRrSrU)r4r5rrPrOdecoder_attention_headsrrrRr rrrrrr encoder_attnencoder_attn_layer_normrZdecoder_ffn_dimrrr)r6rrUr7s r)r5zMvpDecoderLayer.__init__0s %nn44,,  ~~ #F$>$>?"(";";$&LL$@!( NN  * *,,  (*||DNN'C$99T^^V-C-CD99V33T^^D " T^^ <r+r_rbencoder_hidden_statesencoder_attention_maskrcross_attn_promptrard use_cachererfc 0|} |j|||||| \}} tjj||j|j}| |z}|j |}d} |h|} |j ||||||\}} tjj||j|j}| |z}|j|}|} |j|j|}tjj||j|j}|j|}tjj||j|j}| |z}|j|}|f}|r|| | fz }|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, embed_dim)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. self_attn_prompt (`torch.FloatTensor`): prompt of self attention of shape `(2, decoder_attention_heads, pro_len, head_dim)`. cross_attn_prompt (`torch.FloatTensor`): prompt of cross attention of shape `(2, decoder_attention_heads, pro_len, head_dim)`. past_key_values (`Cache`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )r_rarbrcrdrerjN)r_r`rbrcrard) rrr~rRrlrrrrrrrr)r6r_rbrrrrrardrrerself_attn_weightscross_attn_weightsoutputss r)rCzMvpDecoderLayer.forwardLs>!,0>>'+)(/) ,:, ( ( --mt||VZVcVc-d  =0 11-@ " ,$H040A0A+!65- /"3 1B1 -M-MM11-4< >Gr+N) NNNNNNFTN) rFrGrHrr5r=rKr rrrrCrLrMs@r)rr/s=y=>/3596:0415(,).!%.2M||M t+M %||d2 M !& t 3 M  ,,- M!<<$.MM $;M$;M t+M u  %(9(95;L;L(L"MPT"TT UMr+rcleZdZdZdedededeffd Zdejdejfd Z xZ S) MvpClassificationHeadz-Head for sentence-level classification tasks. input_dim inner_dim num_classespooler_dropoutct|tj|||_tj ||_tj|||_y)Nrk)r4r5rrZdenseDropoutrRr^)r6rrrrr7s r)r5zMvpClassificationHead.__init__sD YYy)4 zzN3  )[9 r+r_rfc|j|}|j|}tj|}|j|}|j |}|Sr)rRrr=tanhr^)r6r_s r)rCzMvpClassificationHead.forwardsN ]3  =1  =1  ]3  m4 r+) rFrGrHrIrJrr5r=rKrCrLrMs@r)rrsL7 : : : :  :U\\ellr+rc`eZdZdZfdZdej deej fdZxZ S) MvpPromptz)Layer-wise prompt for encoder or decoder.c 8t||j|_||_||_|j |z|_tj|j|_ tj|j|j |_ tjtj|j |jtjtj|j|dz|j z|_y)Nrr2)r4r5 prompt_length num_layersrQrrXrrrR Embeddingprompt_embedding SequentialrZprompt_mid_dimGELU prompt_trans)r6rrrQr7s r)r5zMvpPrompt.__init__s #11$")3 zzFNN3 " V-A-A6>> RMM IIfnnf&;&; < GGI IIf++Z!^fnn-L M r+ prompt_idsrfc*|j|j|}|j|j|jdz|j |j }|j|}|jgdjd}|S)Nr2)rr2rr) rrrvrrrQrXrRpermutesplit)r6rprompts r)rCzMvpPrompt.forwardsw""4#8#8#DET//11DdnnVZVcVcdf% -33A6 r+) rFrGrHrIr5r=rKrrCrLrMs@r)rrs+3  %,,53Fr+rcBeZdZUeed<dZdZfdZedZ xZ S)MvpPreTrainedModelrmodelTct||t|tr t j |j yyr)r4 _init_weightsrnMvpForConditionalGenerationinitzeros_final_logits_bias)r6moduler7s r)rz MvpPreTrainedModel._init_weightss2 f% f9 : KK00 1 ;r+c|jj}tjgddddd|gg|j}|j ||d}|S)N)r r2r r2r<)rbr)rrr=tensorr<ne)r6 pad_tokenr dummy_inputss r)rzMvpPreTrainedModel.dummy_inputssWKK,, LL"2Q2q)4L!MVZVaVab 'll95" r+) rFrGrHr__annotations__base_model_prefixsupports_gradient_checkpointingrpropertyrrLrMs@r)rrs. &*#2 r+rceZdZdZddedej dzdedzffd Z dde jdzde jdzd e jdzd edzd edzd edzd e ezfdZxZS) MvpEncodera Transformer encoder consisting of *config.encoder_layers* self attention layers. Each layer is a [`MvpEncoderLayer`]. Args: config: MvpConfig embed_tokens (nn.Embedding): output embedding use_prompt (bool): whether to use prompt Nr embed_tokens use_promptct|||j|_|j|_|j }|j |_|j|_ |jrtj|nd|_ tj|j ||j|_t%|j||_tj(t+|j,Dcgc] }t/|c}|_tj2||_||_|r7|j8|_t;||j,|j<|_d|_ |jCycc}w)N?F)"r4r5rRencoder_layerdrop layerdroprr padding_idxmax_position_embeddingsmax_source_positionsscale_embeddingmathsqrt embed_scalerr vocab_sizerr-embed_positions ModuleListrangeencoder_layersrrsrlayernorm_embeddingrrrrrgradient_checkpointing post_init)r6rrrrPrr7s r)r5zMvpEncoder.__init__s4  ~~ 11NN !..$*$B$B!393I3I499Y/sLL):):ItGWGWX<  * *  mmeFLaLaFb$c_V%<$cd #%<< #: $ !'!5!5D $-%%..%D ! ',# %ds:F rrb inputs_embedsrdoutput_hidden_states return_dictrfc ||n|jj}||n|jj}||n|jj}| | t d|$|}|j } |j d| d}n-| |jdd} |dddddf}n t d||j||jz}|j|} || z} |j| } tjj| |j|j} |j rIt#j$|j&j)|j*} |j-| } |t/|j| |}|rdnd}|rdnd}t1|j2D]t\}}|r|| fz}d}|jr&t#j4g}||j6krd }|rd }n"|| ||j r |nd| }|d } |sl||d fz}v|r|| fz}|st9d| ||fDSt;| ||S)a8 Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NzDYou cannot specify both input_ids and inputs_embeds at the same timer"z5You have to specify either input_ids or inputs_embedsrj)rrrbFT)NN)rrdrrc3&K|] }|| ywrr.0vs r) z%MvpEncoder.forward..seqWXWdeslast_hidden_stater_ attentions)rrdruse_return_dictr&r$rvrmrrrr rr~rRrlrr=r>rr{r<rr enumeratersrandrrr)r6rrbrrdrrkwargsinput input_shape embed_posr_rrencoder_statesall_attentionsidx encoder_layerto_dropdropout_probability layer_outputss r)rCzMvpEncoder.forwardsP2C1N-TXT_T_TqTq$8$D $++JjJj &1%cd d  "E++K!r;r?;I  &',,.s3K!!Q(+ETU U   --i84;K;KKM((/ % 1 00?  --mt||VZVcVc-d  ??d&8&89<>Ne]NN$See e+>Vd  r+NF)NNNNNN)rFrGrHrIrrrrr5r= LongTensorrKrrrrCrLrMs@r)rrsy t8K`dgk`kF.2.226)-,0#'t ##d*t  t+t ((4/ t  $; t #Tk t D[t   t r+rcDeZdZdZddededzffd Z ddejdzdejdzdejdzd ejdzd e dzd ejdzd edzd edzdedzdedzdejdzde e zfdZxZS) MvpDecoderz Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`MvpDecoderLayer`] Args: config: MvpConfig embed_tokens (nn.Embedding): output embedding use_prompt (bool): whether to use prompt rrNc t|||j|_|j|_|j |_|j|_|jrtj|jnd|_ tj|j |j|j |_t%|j|j|_tj(t+|j,Dcgc]}t/||c}|_tj2|j|_||_|r]|j8|_t;||j,|j<|_t;||j,|j<|_ d|_!|jEycc}w)Nr)rUF)#r4r5rRdecoder_layerdroprrrrmax_target_positionsrrrrrrrrrr-rrr decoder_layersrrsrr rrrrrrr r )r6rrir7s r)r5zMvpDecoder.__init__sd  ~~ 11!..$*$B$B!8>8N8N499V^^4TWLL):):FNNDL\L\]<  * * NN mmSXY_YnYnSo$pa_Vq%I$pq #%<<#? $ !'!5!5D $-%%..%D ! &/%%..&D " ',# '%qs Grrbrrrarrrdrrrerfc b||n|jj}| | n|jj} ||n|jj}| | n|jj} | | t d|$|} |j }|jd|d}n-| |jdd}|dddddf} n t d||j||jz}|jr%|jr|rtjdd}|rd|b||jjr4t!t#|jt#|jnt#|j}||j%nd}| 2t'j(|||j d z|j* } t-|j||| | }||t/|j||| }|j1| |}||z}|j3|}t4j6j9||j8|j }|j:rZt'j(|j<j?|j*}|jA|}|jC|}| rdnd}|rdnd}|r|dnd}tE|jFD]\}}| r||fz }|jr%t'jHg}||jJkr?||||||j:r|nd|j:r|nd||||  }|d}|s{||d fz }|||dfz }| r||fz }| stMd|||||fDStO|||||S)aU Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.LongTensor` of shape `(batch_size, encoder_sequence_length)`, *optional*): Mask to avoid performing cross-attention on padding tokens indices of encoder input_ids. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks and in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same timer"zEYou have to specify either decoder_input_ids or decoder_inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...F)rrrr)rrrbrera)rrrbrrjr)rrrrardrrer2c3$K|]}|| ywrrrs r)rz%MvpDecoder.forward..ds=s)rrar_rcross_attentions)(rrdrrrr&r$rvrmrrr rllogger warning_onceis_encoder_decoderr r get_seq_lengthr=r>r<rrrr rr~rRrrr{rrrrsrrrr)r6rrbrrrarrrdrrrerrr r8 positionsr_rrrall_hidden_statesall_self_attnsall_cross_attentionsr$ decoder_layerr'r(s r)rCzMvpDecoder.forwards B2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKKst t  "E#//K!r;r?;I  &',,.s3K!!Q(+Ede e   --i84;K;KKM  & &4==##p" 0)4 8V8V$L $DlZ^ZeZeFfg!5  FUE`!?!?!Afg  !"\\&(>ATATUVAW(W`m`t`tN,;;'))+   ! ,1G1S%>{{+5&; & "((0FG % 1 00?  --mt||VZVcVc-d  ??d&8&89<:>(,26!%)-,0#'.2| ##d*|  t+|  %0047 | !& 0 04 7 |  | ((4/| $;|  $;| #Tk| D[|  t+|  : :| r+r,ceZdZdgZdddZdeffd ZdZdZdZ e dd e jd zd e jd zd e jd zd e jd zdee jd zded zde jd zde jd zded zded zded zded zde jd zdeezfdZxZS)MvpModelrz shared.weight)zencoder.embed_tokens.weightzdecoder.embed_tokens.weightrcNt|||j|j}}|j|_t j ||j||_t||j|_ t||j|_ |jyr)r4r5rrrrrrsharedrencoderr,decoderr )r6rrrr7s r)r5zMvpModel.__init__zs~  "("5"5v7H7HZ  ++ll:v~~{K !&&*;*;< !&&*;*;<  r+c|jSr)rAr6s r)get_input_embeddingszMvpModel.get_input_embeddingss {{r+c~||_|j|j_|j|j_yr)rArBrrCr6values r)set_input_embeddingszMvpModel.set_input_embeddingss) $(KK !$(KK !r+c*|jsJd|jd|jjjd|jjjd|jj jdy)NzHIf you want to use lightweight tuning, make sure that `use_prompt=True`.FT)rrequires_grad_rBrrCrrEs r)set_lightweight_tuningzMvpModel.set_lightweight_tuningsjj jj E" %%44T: %%44T: &&55d;r+Nrrbdecoder_input_idsdecoder_attention_maskencoder_outputsrardecoder_inputs_embedsrrdrrrerfc 6|D|B| tdt||jj|jj}| | n|jj } | | n|jj } | | n|jj} | | n|jj} ||j|||| | | }nI| rGt|ts7t|dt|dkDr|dndt|dkDr|dnd}|j|||d|||| | | | |  }| s||zSt|j|j |j"|j$|j&|j|j"|j$ S) a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Mvp uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_mvp._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. NzIf no `decoder_input_ids` or `decoder_inputs_embeds` are passed, `input_ids` cannot be `None`. Please pass either `input_ids` or `decoder_input_ids` or `decoder_inputs_embeds`.)rrbrrdrrrrr2r) rrbrrrarrrdrrre)rradecoder_hidden_statesdecoder_attentionsr4encoder_last_hidden_staterencoder_attentions)r&r*rrr rdrrrrBrnrlenrCrrrar_rr4)r6rrbrNrOrPrarrQrrdrrrerdecoder_outputss r)rCzMvpModel.forwardsV  $)>)F  U !34;;33T[[5W5W! 2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKK&'6'6 y 0 <.2.259:>:>(,26:>!%)-,0#'.2i ##d*i  t+i !++d2 i !& 0 04 7 i e//047 i i ((4/i  %0047i $;i  $;i #Tki D[i  t+i # #!i i r+r?ze The MVP Model with a language modeling head. Can be used for various text generation tasks. ) custom_introc!eZdZddiZdeffd Z ddededzded ejffd Z ded dfd Z d Z e dd ejdzdej dzdejdzdejdzdeej$dzdedzdej$dzdej$dzdejdzdedzdedzdedzdedzdej dzd eezfdZdej fdZxZS)rlm_head.weightzmodel.shared.weightrcxt||t||_|j dt j d|jjjftj|j|jjjd|_ |jy)NrrFrW)r4r5r?rregister_bufferr=rzrAr.rrZrlm_headr rs r)r5z$MvpForConditionalGeneration.__init__s  f%  0%++q$**BSBSBbBb>c2deyy1B1B1Q1QX]^  r+Nnew_num_tokenspad_to_multiple_of mean_resizingrfcLt||||}|j||Sr)r4resize_token_embeddings_resize_final_logits_bias)r6rcrdrenew_embeddingsr7s r)rgz3MvpForConditionalGeneration.resize_token_embeddingss.8I[]jk &&~6r+c6|jjd}||kr|jddd|f}nStjd||z f|jj}tj |j|gd}|j d|y)Nr"rrrhr)rr$r=rzr<ryra)r6rcold_num_tokensnew_bias extra_biass r)rhz5MvpForConditionalGeneration._resize_final_logits_biass//55b9 ^ +--a..@AHa.)H%IRVRhRhRoRopJyy$"8"8*!E1MH 0(;r+cn|jj|jjdyr)rrMrbrLrEs r)rMz2MvpForConditionalGeneration.set_lightweight_tuning'$ ))+ ##E*r+rrbrNrOrPrarrQlabelsrrdrrrec | | n|jj} | R| rtjdd} |7|5t | |jj |jj }|j||||||||| | | | | }|j|d|jz}d}| Ft}||jd|jj| jd}| s|f|ddz}||f|zS|St|||j|j|j |j"|j$|j&|j( S) a\ decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Mvp uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_mvp._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example of summarization: Fine-tuning a model ```python >>> import torch >>> from transformers import AutoTokenizer, MvpForConditionalGeneration >>> tokenizer = AutoTokenizer.from_pretrained("RUCAIBox/mvp") >>> model = MvpForConditionalGeneration.from_pretrained("RUCAIBox/mvp") >>> inputs = tokenizer( ... "Summarize: You may want to stick it to your boss and leave your job, but don't do it if these are your reasons.", ... return_tensors="pt", ... ) >>> labels = tokenizer("Bad Reasons To Quit Your Job", return_tensors="pt")["input_ids"] >>> loss = model(**inputs, labels=labels).loss >>> loss.backward() ``` Inference after the model fine-tuned ```python >>> with torch.no_grad(): ... generated_ids = model.generate(**inputs) >>> generated_text = tokenizer.batch_decode(generated_ids, skip_special_tokens=True) ``` NzJThe `use_cache` argument is changed to `False` since `labels` is provided.F) rbrNrPrOrarrQrrdrrrerr"r losslogitsrarSrTr4rUrrV)rrr5warningr*rr rrbrrrvrrrarSrTr4rUrrV)r6rrbrNrOrPrarrQrqrrdrrrerr lm_logitsmasked_lm_lossloss_fctoutputs r)rCz#MvpForConditionalGeneration.forward+sT&1%:>(,26:>*.!%)-,0#'.2x ##d*x  t+x !++d2 x !& 0 04 7 x e//047 x x ((4/x  %0047x   4'x $;x  $;x #Tkx D[x  t+x "  #x x thELLhr+rz Mvp model with a sequence classification/head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c~eZdZdeffd ZdZe ddejdzdejdzdejdzdejdzd e ejdzd ejdzd ejdzd ejdzd e dzde dzde dzde dzde ezfdZxZS)MvpForSequenceClassificationrc t||fi|t||_t |j |j |j |j|_|jyr) r4r5r?rrr num_labelsclassifier_dropoutclassification_headr )r6rrr7s r)r5z%MvpForSequenceClassification.__init__sZ *6*f% #8 NN NN     % % $   r+cn|jj|jjdyr))rrMrrLrEs r)rMz3MvpForSequenceClassification.set_lightweight_tunings& ))+   //6r+NrrbrNrOrPrrQrqrrdrrrfc | | n|jj} |d} |$|"td|jj|j |||||||| | | |  }|d}|j |jjj|j}ttj|jdjdk(d||ddfj|j!dd|j!ddddddf}|j#|}d}||jj$|jj&dk(rd |j_nv|jj&dkDrL|j(tj*k(s|j(tj,k(rd |j_nd |j_|jj$d k(rSt/}|jj&dk(r&||j1|j1}n|||}n|jj$d k(rGt3}||jd|jj&|jd}n,|jj$d k(rt5}|||}| s|f|ddz}||f|zS|St7|||j8|j:|j<|j>|j@|jB|jD S) a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Mvp uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_mvp._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Example of single-label classification: Fine-tuning a model on `num_labels` classes ```python >>> import torch >>> from transformers import AutoTokenizer, MvpForSequenceClassification >>> num_labels = 2 # for example, this is a binary classification task >>> tokenizer = AutoTokenizer.from_pretrained("RUCAIBox/mvp") >>> model = MvpForSequenceClassification.from_pretrained("RUCAIBox/mvp", num_labels=num_labels) >>> inputs = tokenizer("Classify: Hello, my dog is cute", return_tensors="pt") >>> labels = torch.tensor(1) # the real label for inputs >>> loss = model(**inputs, labels=labels).loss >>> loss.backward() ``` Inference after the model fine-tuned ```python >>> with torch.no_grad(): ... logits = model(**inputs).logits >>> predicted_class_id = logits.argmax() ``` NFz8Passing input embeddings is currently not supported for rbrNrOrPrrQrrdrrrrz7All examples must have the same number of tokens.r" regressionsingle_label_classificationmulti_label_classificationrs)#rrNotImplementedErrorr7rFreq eos_token_idr{r<rr=unique_consecutivesumnumelrvrmr problem_typerr;r?rJrsqueezerrrrarSrTr4rUrrV)r6rrbrNrOrPrrQrqrrdrrrrr_eos_masksentence_representationrurtryrzs r)rCz$MvpForSequenceClassification.forwards J&1%>KbKbJcd ** )/#9+'"7/!5#    << 8 89<<]=Q=QR  $ $X\\!_ 5 ; ; = B E #0! "<"A"A-BTBTUVBWY[]j]o]opr]s"t r1H# ))*AB  {{''/;;))Q./;DKK,[[++a/V\\UZZ5OSYS_S_chclclSl/LDKK,/KDKK,{{''<7"9;;))Q.#FNN$4fnn6FGD#FF3D))-JJ+- B 0F0F GUWY))-II,./Y,F)-)9TGf$ Ev E.#33")"?"?&99$55&-&G&G")"?"?&99  r+) NNNNNNNNNNNN)rFrGrHrr5rMrr=r*rKr\rrrrrCrLrMs@r)r~r~sN y 7.2.259:>:>26:>*.!%)-,0#'K ##d*K  t+K !++d2 K !& 0 04 7 K e//047 K ((4/K  %0047K   4'K $;K  $;K #TkK D[K  0 0K K r+r~ceZdZfdZdZe ddejdzdejdzdejdzdejdzde ejdzd ejdzd ejdzd ejdzd ejdzd e dzde dzde dzde dzde e zfdZxZS)MvpForQuestionAnsweringct||d|_|j|_t||_t j |j|j|_|jyr1) r4r5rr?rrrZ hidden_size qa_outputsr rs r)r5z MvpForQuestionAnswering.__init__Ss[   ++f% ))F$6$68I8IJ r+cn|jj|jjdyr))rrMrrLrEs r)rMz.MvpForQuestionAnswering.set_lightweight_tuning_s$ ))+ &&u-r+NrrbrNrOrPstart_positions end_positionsrrQrrdrrrfc | | n|jj} ||d} |j||||||| | | | |  }|d}|j|}|j dd\}}|j dj }|j dj }d}||t|jdkDr|j d}t|jdkDr|j d}|jd}|jd|}|jd|}t|}|||}|||}||zd z }| s||f|ddz}||f|zS|St||||j|j|j|j|j |j"|j$ S) a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Mvp uses the `eos_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_mvp._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. Example: Fine-tuning a model for extrative question answering, and our model also supports generative question answering using `BartForConditionalGeneration` ```python >>> import torch >>> from transformers import AutoTokenizer, MvpForQuestionAnswering >>> tokenizer = AutoTokenizer.from_pretrained("RUCAIBox/mvp") >>> model = MvpForQuestionAnswering.from_pretrained("RUCAIBox/mvp") >>> inputs = tokenizer( ... "Answer the following question: Who was Jim Henson? [SEP] Jim Henson was a nice puppet", ... return_tensors="pt", ... ) >>> target_start_index = torch.tensor([18]) >>> target_end_index = torch.tensor([19]) >>> loss = model(**inputs, start_positions=target_start_index, end_positions=target_end_index).loss >>> loss.backward() ``` Inference after the model fine-tuned ```python >>> with torch.no_grad(): ... outputs = model(**inputs) >>> answer_start_index = outputs.start_logits.argmax() >>> answer_end_index = outputs.end_logits.argmax() >>> predict_answer_tokens = inputs.input_ids[0, answer_start_index : answer_end_index + 1] >>> predict_answer = tokenizer.decode(predict_answer_tokens) ``` NFrrrr"rh) ignore_indexr2) rt start_logits end_logitsrarSrTr4rUrrV)rrrrrr contiguousrWrmrrrrarSrTr4rUrrV)r6rrbrNrOrPrrrrQrrdrrrrsequence_outputrurr total_loss ignored_indexry start_lossend_lossrzs r)rCzMvpForQuestionAnswering.forwardcsV&1%:>371526:>!%)-,0#'F <<$&F  t+F !++d2 F !& 0 04 7 F e//047 F ))D0F ''$.F ((4/F  %0047F $;F  $;F #TkF D[F 4 4!F F r+rc(eZdZdZfdZdZxZS)MvpDecoderWrapperz This wrapper class is a helper class to correctly load pretrained checkpoints when the causal language model is used in combination with the [`EncoderDecoderModel`] framework. cdt||t||_|j yr)r4r5r,rCr rs r)r5zMvpDecoderWrapper.__init__s&  !&)  r+c&|j|i|Sr)rC)r6argsrs r)rCzMvpDecoderWrapper.forwardst||T,V,,r+)rFrGrHrIr5rCrLrMs@r)rrs  -r+rceZdZddiZfdZdZdZdZe dde jdzd e jdzd e jdzd e jdzd e dzd e jdzde jdzdedzdedzdedzdedzde jdzdee jzdeezfdZxZS)MvpForCausalLMr_z!model.decoder.embed_tokens.weightcd|_d|_t| |t ||_t j|j|jd|_ |jy)NTFrW) rSr7r4r5rrrrZrrrbr rs r)r5zMvpForCausalLM.__init__sX $)!  &v. yy!3!3V5F5FUS  r+cB|jjjSrrrCrrEs r)rFz#MvpForCausalLM.get_input_embeddings szz!!...r+c:||jj_yrrrHs r)rJz#MvpForCausalLM.set_input_embeddingss*/ 'r+cn|jj|jjdyr)rorEs r)rMz%MvpForCausalLM.set_lightweight_tuningrpr+Nrrbrrrarrqrrdrrrelogits_to_keeprfc | | n|jj} | | n|jj} | | n|jj} |jj |||||||| | |  }|d}t | tr t| dn| }|j|dd|ddf}d}|Ft}||jd|jj|jd}| s|f|ddz}||f|zS|St|||j|j|j |j"S)ap labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, MvpForCausalLM >>> tokenizer = AutoTokenizer.from_pretrained("RUCAIBox/mvp") >>> model = MvpForCausalLM.from_pretrained("RUCAIBox/mvp") >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> list(logits.shape) [1, 8, 50267] ```N) rrbrrrarrrdrrrr"r)rtrurar_rr4)rrdrrrrCrnrJslicerbrrvrrrar_rr4)r6rrbrrrarrqrrdrrrerrrr_ slice_indicesrurtryrzs r)rCzMvpForCausalLM.forwardslP2C1N-TXT_T_TqTq$8$D $++JjJj &1%SV8W~ot4]k mA}a,?@A  ')HFKKDKK,B,BCV[[QS_UDY,F'+'7D7V# CV C0#33!//))$55   r+) NNNNNNNNNNNNr)rFrGrHr[r5rFrJrMrr=r*rKrr rrJrrrCrLrMs@r)rrsf*,OP /0+.2.2:>;?(,26*.!%)-,0#'.2-.P ##d*P  t+P  %0047 P !& 1 1D 8 P  P ((4/P   4'P $;P  $;P #TkP D[P  t+P ell*P 2 2!P P r+r)rrrr~r?r)@rIrr=rtorch.nnrrrrr activationsr cache_utilsr r r generationr masking_utilsrrmodeling_layersrmodeling_outputsrrrrrrrmodeling_utilsrutilsrrrconfiguration_mvpr get_loggerrFr5rKrJr*rr-ModulerOrrrrrrr,r?rr~rrr__all__rr+r)rs AA&!CC)J9.DD(   H % %,, c [^ ";BLL;6S2299S2l<+0<+~j0j\BII0 2*` #` Fh #h VN !N N b ^h"4o^h  ^hB ^ #5^  ^ BX 0X X x -* -i 'i X r+