iFddlmZddlmZddlZddlmZddlmZddl m Z ddl m Z dd lmZmZdd lmZdd lmZmZdd lmZdd lmZmZddlmZmZddlmZm Z ddl!m"Z"m#Z#ddl$m%Z%ddl&m'Z'm(Z(m)Z)m*Z*ddl+m,Z,m-Z-ddl.m/Z/m0Z0ddl1m2Z2e*jfe4Z5edGddejlZ7GddejlZ8GddejlZ9GddejlZ:Gd d!ejlZ;Gd"d#ejlZd'ej~d(e@d)ej~fd*ZA dEd+ejld,ej~d-ej~d.ej~d/ej~dzd0eBd1eBd2e%e'fd3ZCGd4d5ejlZDGd6d7eZEe(Gd8d9e#ZFe(Gd:d;eFZG dFdZIGd?d@eFeZJGdAdBeeFZKgdCZLy)G)Callable)OptionalN)nn) functional)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hubuse_kernel_func_from_hub)create_causal_mask) GenericForSequenceClassificationGradientCheckpointingLayer)MoeCausalLMOutputWithPastMoeModelOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)maybe_autocastmerge_with_config_defaults)OutputRecordercapture_outputs) JetMoeConfigRMSNormcheZdZddeddffd Zdej dej fdZdZxZ S) JetMoeRMSNormepsreturnNct|tjt j ||_||_y)z< JetMoeRMSNorm is equivalent to T5LayerNorm N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizer& __class__s f/mnt/e/genesis-system/.venv/lib/python3.12/site-packages/transformers/models/jetmoe/modeling_jetmoe.pyr*zJetMoeRMSNorm.__init__2s1 ll5::k#:; # hidden_statesc"|j}|jtj}|j dj dd}|tj ||jzz}|j|j|zS)NT)keepdim) dtypetor,float32powmeanrsqrtr/r.)r0r5 input_dtypevariances r3forwardzJetMoeRMSNorm.forward:sy#)) %((7  $$Q',,R,>% Ht?T?T4T(UU {{]--k:::r4c^t|jjd|jS)Nz, eps=)tupler.shaper/)r0s r3 extra_reprzJetMoeRMSNorm.extra_reprAs* ))*+6$2G2G1HIIr4)gư>) __name__ __module__ __qualname__floatr*r,TensorrBrF __classcell__r2s@r3r%r%0s7$$$$;U\\;ell;Jr4r%c eZdZUejed<d deffd Ze d dedzde dde dzde d e ffd Z ejed ZxZS)JetMoeRotaryEmbeddinginv_freqNconfigct||j|_|j|_||_|j j d|_|j}|jdk7rt|j}||j |\}|_ |jd|d|jd|jdy)N rope_typedefaultrPF) persistentoriginal_inv_freq) r)r*max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrQrope_parametersrScompute_default_rope_parametersrattention_scalingregister_bufferclone)r0rQdevice rope_init_fnrPr2s r3r*zJetMoeRotaryEmbedding.__init__Hs "("@"@$*$B$B! 44[A!%!E!E >>Y &.t~~>L+7 V+L($( ZeD 0(..2BuUr4r_z torch.deviceseq_lenr'z torch.Tensorc|jd}t|ddxs|j|jz}d}d|t j d|dtj j|tj|z zz }||fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PreTrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). rope_thetahead_dimNg?rr7r:)r_r:) rZgetattrr1num_attention_headsr,arangeint64r;rJ)rQr_rabasedimattention_factorrPs r3r[z5JetMoeRotaryEmbedding.compute_default_rope_parametersXs&%%l3fj$/c63E3EIcIc3c U\\!S!5;;?BB&X]XcXcBdgjj k )))r4cN|jddddfjj|jdddj |j }|dddddfj}t |j jtr/|j jdk7r|j jnd}t|d5|j|jzjdd}tj||fd }|j|jz}|j|jz} dddj |j  j |j fS#1swYAxYw) Nrr8r!mpscpuF) device_typeenabledr7rkre)rPrJexpandrEr;r_ isinstancetypestrr transposer,catcosr\sinr:) r0x position_idsinv_freq_expandedposition_ids_expandedrpfreqsembryrzs r3rBzJetMoeRotaryEmbedding.forwardvsR!MM$4-8>>@GG HZHZ[\H]_acdehhijiqiqr ,QaZ 8 > > @'1!((--'E!((--[`J`ahhmmfk  U C 5&,,.1F1L1L1NNYYZ[]^_E))UEN3C'')d444C'')d444C  5 vvAGGv$cff177f&;;;  5 5s BFF$N)NNN)rGrHrIr,rK__annotations__r"r* staticmethodrintrDrJr[no_gradrrBrLrMs@r3rOrOEsllV|V &*+/"*t#*(*t* ~u$ % **:U]]_ < (=a(@ADYZ[D\\ ++M;G'+ag*>> S6\4??;>CWCW`n`u`uvq+~F #((fdooF #dii/ r4)rGrHrI__doc__r"r*rBrLrMs@r3rrs | r4rc:eZdZdZdeffd ZdZdZdZxZ S) JetMoeMoAz A Sparsely gated mixture of attention layer with pairs of query- and output-projections as experts. Args: config: Configuration object with model hyperparameters. rQcht||j|_|j|_|j |jz|_|j|_ tjjtj|j |_t|j|j |j|_t|j|j|j |_t%|j |j|j|_y)Nr)r)r*rrr1r kv_channelsnum_key_value_headsrrr,rr+rrrrrrrrs r3r*zJetMoeMoA.__init__'s !33 ,,!--0J0JJ// HH&&u{{4??'CD 1$2B2BDOOUYUeUef243C3CTEUEUW[WfWfg&((**  r4c|j\}}}|jd|}|j|\}}}}} ||||f} ||} |j| |} t j ||z|j z|jf| j| j} | jd|| }|j|||j d}|| | fS)z Map inputs to attention experts according to routing decision and compute query projection inside each experts. r8rr) rrrrr,rrr1r:r_rr)r0rrrrrrrrr topo_inforrrrs r3mapz JetMoeMoA.map9s !, 0 0 2VX!))"h7 UYU`U`alUmRk; ]); [Q $K0 **=+F 6\DJJ &(8(8 9AUAU^l^s^s q*>O #((fdjj"E ]I55r4c|j\}}}}|jd|}|\}}} } ||} |j| | } | | dddfz} tj||z|j f| j | j} | jd|| }|j|||j }||jz}|S)zu Compute output projection inside each attention experts and merge the outputs of different experts. r8Nrr) rrrr,rrr:r_rrr)r0rrrrkr1rrrrrrrrs r3reducezJetMoeMoA.reducePs'2&6&6&8#VQ !))"k: FOCk; $$89 ++M;G(+ag*>> S6\4??;>CWCW`n`u`uvq+~F #((fdooF #dii/ r4ctd)Nz-This module doesn't support call and forward.)NotImplementedError)r0rs r3rBzJetMoeMoA.forwardfs!"QRRr4) rGrHrIrr"r*rrrBrLrMs@r3rrs$ | $6.,Sr4rc|dd|jddzf}|d|jddzdf}tj| |fdS)z*Rotates half the hidden dims of the input..Nr8r7rr)rEr,rx)r{x1x2s r3 rotate_halfrjsZ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''r4rotary_pos_embc|j|}|j|}||zt||zz}||zt||zz}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer)qrryrz unsqueeze_dimq_embedk_embeds r3apply_rotary_pos_embrqsY& -- &C -- &C3w;q>C/0G3w;q>C/0G G r4r5n_repr'c|j\}}}}|dk(r|S|dddddddddfj|||||}|j|||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) r!N)rErsr)r5rbatchrslenrds r3 repeat_kvrso 2?1D1D.E h z!!Qa"23::5BUW\^bdlmM  (;e(CT8 TTr4modulequerykeyvalueattention_maskscalingdropoutkwargsc t||j}t||j} tj||j dd|z} || |z} t j j| dtjj|j} t j j| ||j} tj| | } | j ddj} | | fS)Nr7rr8)rkr:)ptrainingr!)rnum_key_value_groupsr,matmulrwrrrr<r;r:rr contiguous) rrrrrrrr key_states value_states attn_weights attn_outputs r3eager_attention_forwardrs3 ; ;#D#DQB#D#O L#((6@@AF __\2<E+, r4r&ceZdZUeed<dZdZdgZdgZdZ dZ dZ dZ dZ eedeed geeed d Zej(fd ZxZS) JetMoePreTrainedModelrQmodelFr&rTr7)indexr!)rr5 attentionsc t||t|tr7t j |j d|jjyt|ttzr t j|jyy)zInitialize the weights.r)r>stdN) r) _init_weightsrtrinitnormal_r.rQinitializer_rangerrzeros_r)r0rr2s r3r9z#JetMoePreTrainedModel._init_weightsHs_ f% f3 4 LLSdkk6S6S T  I 5 6 KK $7r4)rGrHrIr"rbase_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_supports_attention_backendrr rr&_can_record_outputsr,rr9rLrMs@r3r2r26s &+#-.#4"5N""&(BNScklDmn+$_A> U]]_%%r4r2ceZdZdeffd Zeee ddejdzdejdzdejdzde dzdejdzd e dzd ejdzd eed efd ZxZS) JetMoeModelrQc .t|||j|_|j|_t j |j|j|j|_t jt|jDcgc]}t||c}|_ t|j|j|_t#||_d|_|j(|_|j+ycc}w)N)r&rQF)r)r* pad_token_id padding_idx vocab_sizer Embeddingr1 embed_tokens ModuleListrnum_hidden_layersr&layersr% rms_norm_epsnormrO rotary_embgradient_checkpointingr post_initrs r3r*zJetMoeModel.__init__Ts  !.. ++LL):):F&+#$*$?$?!  esDN input_idsrr|r inputs_embedsr,rrr'c D|du|duz r td|r|t|j}||j|}|F||j nd} t j | | |jdz|j}||jd}t|j|||||} |} |j| |} |jd|jjD]} | | f| | ||||d|} |j| } t| |S) Nz:You must specify exactly one of input_ids or inputs_embedsrKrr!)r_)rQrZrrrr|)rrrr,rr|)last_hidden_stater) ValueErrorr rQrPget_seq_lengthr,rhrEr_rrrVrSrRrUr)r0rYrr|rrZr,rrpast_seen_tokens causal_maskr5r decoder_layers r3rBzJetMoeModel.forwardes`  -t"; <YZ Z 0*$++>O   --i8M  !CRC^==?de "\\ "2]5H5H5K"KTaThThN  )33A6L(;;'))+%  & #oom\J![[)H4;;+H+HI M) $7* /#-)  M  -0 %++  r4)NNNNNNN)rGrHrIr"r*rr rr,r$rKr FloatTensorr0rrrrBrLrMs@r3rIrIRs|" .2.204(,26!%26; ##d*;  t+; &&- ;  ; ((4/ ; $;; ((4/; +,;  ;  ; r4rI gate_logitsrc|t|tsyt|trC|dj}tj|Dcgc]}|j |c}d}tj jjd}tj||d\}} tj jj| |} |>tj| jd} tj|d} n|j\} }|jd| |zz}|dddddddfj|| |||fjd||j }tj | j|zdtj |dz } |ddddddfj|| ||fjd|j |}tj ||zdtj |dz } tj | | j#dz}||zScc}w)a Computes auxiliary load balancing loss as in Switch Transformer - implemented in Pytorch. See Switch Transformer (https://huggingface.co/papers/2101.03961) for more details. This function implements the loss function presented in equations (4) - (6) of the paper. It aims at penalizing cases where the routing between experts is too unbalanced. Args: gate_logits: Logits from the `gate`, should be a tuple of model.config.num_hidden_layers tensors of shape [batch_size X sequence_length, num_experts]. num_experts: Number of experts top_k: The number of experts to route per-token, can be also interpreted as the `top-k` routing parameter. attention_mask (`torch.Tensor`, *optional*): The attention_mask used in forward function shape [batch_size X sequence_length] if not None. Returns: The auxiliary loss. Nrrrr8)rtrDr_r,rxr;rrrrone_hotr>rJrErsrrr)rcrrrcompute_device layer_gateconcatenated_gate_logitsrouting_weightsrselected_experts expert_masktokens_per_expertrouter_prob_per_expert batch_sizesequence_lengthrRexpert_attention_mask router_per_expert_attention_mask overall_losss r3load_balancing_loss_funcrss9:*[%"@+u%$Q..#(99^i-jPZjmmN.K-jpq#r hh))112JPR1SO**_eDA((%%--.> LK!JJ{'8'8':B"'O!C&4&:&:# O4::1=*B^_ 4AtT1 2 V& OUKX Y WR , R  "IIk&7&7&9SV8W~ot4]k mA}a,?@A  %4%% ;;11 D /%%  (( H !**X[[-EEE(#33!//))!//  r4) NNNNNNNNrF)rGrHrI_tied_weights_keysr*rrr,r$rKr rbr0rrrBrLrMs@r3rurus*,GH .2.204(,26*.!%26-.,1; ##d*;  t+; &&- ;  ; ((4/ ;   4'; $;; ((4/; ell*; #Tk;  #; ; r4ruc eZdZy)JetMoeForSequenceClassificationN)rGrHrIr.r4r3rrHsr4r)rurIr2r)r!)r)Nr7N)Mcollections.abcrtypingrr,rtorch.nnrrrr: activationsr cache_utilsr r generationr integrationsr r masking_utilsrmodeling_layersrrmodeling_outputsrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrrutils.output_capturingrr configuration_jetmoer" get_loggerrGrModuler%rOrrrrrrrKrrrJrr r&r2rIrDrsrur__all__r.r4r3rs*% $&!.)Q/[QKF&RRGE.   H %Y'JBIIJ(J(>