iJddlZddlZddlmZddlmZddlmZmZddl m Z m Z ddl m Z ddlmZd d lmZd d lmZmZmZmZmZmZmZmZmZd d lmZd dlmZej@e!Z"dZ#dZ$GddeZ%dZ&GddeZ'GddejPZ)Gdde)Z*Gdde)Z+e)e*e+dZ,GddeZ-Gdd eZ.Gd!d"eZ/Gd#d$eZ0Gd%d&eZ1Gd'd(eZ2Gd)d*eZ3gd+Z4y),N)nn)initialization)Cache StaticCache)_flash_attention_forward!flash_attn_supports_top_left_mask)PreTrainedModel)logging)GemmaForCausalLM) LlamaDecoderLayerLlamaForQuestionAnsweringLlamaForSequenceClassificationLlamaForTokenClassification LlamaModelLlamaPreTrainedModelLlamaRotaryEmbeddingapply_rotary_pos_emb repeat_kv) MistralMLP)DiffLlamaConfigzkajuma/DiffLlama-0.3B-handcutrc eZdZy) DiffLlamaMLPN__name__ __module__ __qualname__k/mnt/e/genesis-system/.venv/lib/python3.12/site-packages/transformers/models/diffllama/modular_diffllama.pyrr1r!rc>ddtjd|zzz S)Ng?g333333?g333333ӿ)mathexp) layer_idxs r"lambda_init_fnr(5s txxy 011 11r!c eZdZy)DiffLlamaRotaryEmbeddingNrr r!r"r*r*9r#r!r*c\eZdZdZddededzffd Z ddejde ejejfdejdzd ejdzd e dzd e d ejdzd e ejejdze ejdzffdZ xZS)DiffLlamaAttentionz=Multi-headed attention from 'Attention Is All You Need' paperNconfigr'ct|||_||_|-tj d|j jd|j|_|j|_ |j|_ t|d|j|jz|_ |j|_|j|jz|_|j |_d|_t%j&|j|j|jz|j(|_t%j&|j|j|jz|j(|_t%j&|j|j|jz|j(|_t%j&|j|jz|j|j(|_t3||_t%j6t9j:d|j<|jf|_t%j6t9j:d|j<|jf|_ t%j6t9j:d|j<|jf|_!t%j6t9j:d|j<|jf|_"t%jFd|jz|jHd |_%y) NzInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.head_dimT)biasr)sizer F)epselementwise_affine)&super__init__r-r'logger warning_once __class__rattention_dropout hidden_sizenum_attention_heads num_headsgetattrr/num_key_value_headsnum_key_value_groupsmax_position_embeddings is_causalrLinearattention_biasq_projk_projv_projo_projr( lambda_init Parametertorchnormallambda_std_dev lambda_q1 lambda_k1 lambda_q2 lambda_k2RMSNorm rms_norm_eps groupnormselfr-r'r8s r"r5zDiffLlamaAttention.__init__@sq  "      !8!8 9:,,  "(!9!9!--33 D4D4D4VW #)#=#= $(NNd6N6N$N!'-'E'E$ii 0 0$..4==2PW]WlWlm ii 0 0$2J2JT]]2Zagavavw ii 0 0$2J2JT]]2Zagavavw ii >@P@PW]WlWlm ))4ell1f6K6KSWS`S`Rb&cdell1f6K6KSWS`S`Rb&cdell1f6K6KSWS`S`Rb&cdell1f6K6KSWS`S`Rb&cdA $56;N;Nchir! hidden_statesposition_embeddingsattention_mask position_idspast_key_values use_cachecache_positionreturnc |j\} } } | } |j|} |j|}|j|}| j | | |j |j jdd} |j | | |j|j jdd}|j | | |j|j jdd}|\}}t| |||\} }|'|||d}|j|||j|\}}t||j}t||j}tjtj |ddd}|j#dddd}tj$| |jddt'j(|j z }|||z}t*j,j/|dtj0j3| j4}t*j,j7||j8|j:}tj<tj>|j@|jBzdtj0j3| j4}tj<tj>|jD|jFzdtj0j3| j4}||z |jHz}tj$||}tj |dd\}}|||zz }d|jHz |jK|z}|jddjM}|jO| | d}|jQ|}||fS) Nrr sincosr\dimrrcdtype)ptraining))r1rDrErFviewr<r/ transposer>rupdater'rr?rJcatchunkrepeatmatmulr%sqrtr functionalsoftmaxfloat32torfdropoutr9rhr&sumrMrNrOrPrHrS contiguousreshaperG)rUrVrWrXrYrZr[r\kwargsbsz target_len_q_len query_states key_states value_statesrar` cache_kwargs attn_weightslambda_1lambda_2 lambda_full attn_output attn_output1 attn_output2s r"forwardzDiffLlamaAttention.forwardasX+//1Z{{=1 [[/ {{=1 #((eT^^T]]S]]^_abc __S%1I1I4==Yccdeghi #((eT5M5Mt}}]gghiklm &S#7 jRUWZ#[ j  &#&snUL'6'='=j,X\XfXfht'u $J z4+D+DE  t/H/HI yy\1!!D"M #**1aA6 ||L*2F2Fq!2LMPTPYPYZ^ZgZgPhh  %'.8L}},,\r,WZZ[g[m[mn }},,\T=S=S^b^k^k,l 99UYYt~~'FBV[VcVcdehh    99UYYt~~'FBV[VcVcdehh    )D,<,<< ll<> %*[[aQ%G" l"[<%?? 4+++t~~k/JJ !++Aq1<<> !))#ub9 kk+. L((r!NNNNFN)rrr__doc__rintr5rJTensortuple LongTensorrboolr __classcell__r8s@r"r,r,=sGjj3:jJ/304(,26;)||;)#5<<#=>;) t+ ;) &&- ;)  ;);)((4/;) u||U\\D0% 2E2LL M;)r!r,ceZdZdZfdZ d dej deej ej fdejdzdejdzde dzd e d ejdzd eej dffd Z xZ S)DiffLlamaFlashAttention2aN DiffLlama flash attention module. This module inherits from `DiffLlamaAttention` as the weights of the module stays untouched. The only required change would be on the forward pass where it needs to correctly call the public API of flash attention and deal with padding tokens in case the input contains any of them. cBt||i|t|_yr)r4r5r _flash_attn_uses_top_left_mask)rUargsryr8s r"r5z!DiffLlamaFlashAttention2.__init__s# $)&) /P.Q+r!NrVrWrXrYrZr[r\r]c t|tr td|j\}} } |j |} |j |} |j |} | j|| |j|jjdd} | j|| |j|jjdd} | j|| |j|jjdd} |\}}t| | ||\} } |'|||d}|j| | |j|\} } | jdd} | jdd} | jdd} |jr |j nd}| j"}| j$j&dk7r| j$j&nd}|t(j*k(rt)j,|rt)j.|}nMt1|j2dr|j2j"}n |jj4j"}t6j9d |d | j;|} | j;|} | j;|} t)j<| dd \}}|j?dddd}|j?dddd}tA| | ||| ||tC|d d|jD|jF }tA| | ||| ||tC|d d|jD|jF }t)jH||gd }t)j<|dd \}}t)jJt)jL|jN|jPzdt(j*j;| j"}t)jJt)jL|jR|jTzdt(j*j;| j"}||z |jVz}|||zz }d|jVz |jY|z}|j[|| dj]}|j_|}|dfS)Nz`static` cache implementation is not compatible with `attn_implementation==flash_attention_2` make sure to use `sdpa` in the mean time, and open an issue at https://github.com/huggingface/transformersrr r_mpscpu _is_quantizedzThe input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in .rbsliding_window)rYruruse_top_left_maskrArdre)0 isinstancer ValueErrorr1rDrErFrir<r/rjr>rrkr'rhr9rfdevicetyperJrsis_autocast_enabledget_autocast_dtypehasattrr-weightr6r7rtrmrnrr=rrArlr&rvrMrNrOrPrHrSrxrwrG)rUrVrWrXrYrZr[r\rzr}r|r~rrrar`r dropout_rate input_dtype device_type target_dtype value_states1 value_states2rrrrrrs r"rz DiffLlamaFlashAttention2.forwardsT o{ 3}  &**, UA{{=1 [[/ {{=1 $((eT^^T]]S]]^_abc __S%1I1I4==Yccdeghi #((eT5M5Mt}}]gghiklm &S#7 jRUWZ#[ j  &#&snUL'6'='=j,X\XfXfht'u $J $--a3 ))!Q/ #--a3 15t--C #(( 2>2E2E2J2Je2Sl))..Y^ %-- '((5$77 D o6#{{00 #{{1177    >$  (??<8L#|4J'??<8L',{{<'J$ }%,,Q1a8 %,,Q1a8 /     % "4)94@"AAnn  0     % "4)94@"AAnn  ii| <"E %*[[aQ%G" l99UYYt~~'FBV[VcVcdehh    99UYYt~~'FBV[VcVcdehh    )D,<,<< "[<%?? 4+++t~~k/JJ !))#ub9DDF kk+. D  r!r) rrrrr5rJrrrrrrrrs@r"rrs R3704(,26u!||u!#5<<#=>u!((4/ u! &&- u!  u!u!((4/u! u||T! "u!r!rc8eZdZdZ d dej deej ej fdej dzdejdzdedzde d ejdzd eej ej dzeej dzffd Z y) DiffLlamaSdpaAttentiona DiffLlama attention module using torch.nn.functional.scaled_dot_product_attention. This module inherits from `DiffLlamaAttention` as the weights of the module stays untouched. The only changes are on the forward pass to adapt to SDPA API. NrVrWrXrYrZr[r\r]c |j\} } } |j|} |j|} |j|}| j | | |j |j jdd} | j | | |j|j jdd} |j | | |j|j jdd}|\}}t| | ||\} } |'|||d}|j| ||j|\} }t| |j} t||j}tjtj |ddd}|j#dddd}|}||ddddddd| j$df}|duxr| dkD}tj&j(j+| | |||j,r |j.nd|}tj |dd\}}tj0tj2|j4|j6zdtj8 j;| j<}tj0tj2|j>|j@zdtj8 j;| j<}||z |jBz}|||zz }d|jBz |jE|z}|jddjG}|j | | d}|jI|}|dfS) Nrr r_rbrdr) attn_mask dropout_prAre)%r1rDrErFrir<r/rjr>rrkr'rr?rJrlrmrnshaperrqscaled_dot_product_attentionrhr9r&rvrMrNrsrtrfrOrPrHrSrwrG)rUrVrWrXrYrZr[r\ryrzr}r|r~rrrar`r causal_maskrArrrrrrs r"rzDiffLlamaSdpaAttention.forward.s(&**, UA{{=1 [[/ {{=1 #((eT^^T]]S]]^_abc __S%1I1I4==Yccdeghi #((eT5M5Mt}}]gghiklm &S#7 jRUWZ#[ j  &#&snUL'6'='=j,X\XfXfht'u $J z4+D+DE  t/H/HI yy\1!!D"M #**1aA6 $  %%aA/E1A1A"1E/E&EFK 4'5EAI hh))FF   !04 d,,3 G &+[[aQ%G" l99UYYt~~'FBV[VcVcdehh    99UYYt~~'FBV[VcVcdehh    )D,<,<< "[<%?? 4+++t~~k/JJ !++Aq1<<> !&&sE26 kk+. D  r!r) rrrrrJrrrrrrr r!r"rr&s/304(,26B!||B!#5<<#=>B! t+ B! &&- B!  B!B!((4/B! u||U\\D0% 2E2LL MB!r!r)eagerflash_attention_2sdpac(eZdZdedeffd ZxZS)DiffLlamaDecoderLayerr-r'cdt|||t|j|||_y)N)r-r')r4r5DIFFLLAMA_ATTENTION_CLASSES_attn_implementation self_attnrTs r"r5zDiffLlamaDecoderLayer.__init__{s- +4V5P5PQY_ktur!)rrrrrr5rrs@r"rrzsvv3vvr!rcBeZdZdZdZej dZy)DiffLlamaPreTrainedModelFctj||t|trt j |j d|jjt j |jd|jjt j |jd|jjt j |jd|jjyy)Nr) r _init_weightsrr,initnormal_rMr-rLrNrOrP)rUmodules r"rz&DiffLlamaPreTrainedModel._init_weightss%%dF3 f0 1 LL))1dkk.H.H I LL))1dkk.H.H I LL))1dkk.H.H I LL))1dkk.H.H I 2r!N)rrr_supports_flex_attn_supports_attention_backendrJno_gradrr r!r"rrs*"'U]]_JJr!rc eZdZy)DiffLlamaModelNrr r!r"rrr#r!rc eZdZy)DiffLlamaForCausalLMNrr r!r"rrr#r!rc eZdZy)"DiffLlamaForSequenceClassificationNrr r!r"rrr#r!rc eZdZy)DiffLlamaForQuestionAnsweringNrr r!r"rrr#r!rc eZdZy)DiffLlamaForTokenClassificationNrr r!r"rrr#r!r)rrrrrr)5r%rJrrr cache_utilsrrmodeling_flash_attention_utilsrr modeling_utilsr utilsr gemma.modeling_gemmar llama.modeling_llamarrrrrrrrrmistral.modeling_mistralrconfiguration_diffllamar get_loggerrr6_CHECKPOINT_FOR_DOC_CONFIG_FOR_DOCrr(r*Moduler,rrrrrrrrrr__all__r r!r"rs!" &-i-3   24   H %5# : 2 3 _)_)DD!1D!NJ!/J!\ 1 "v-v J3 J Z  +  )G  $=  &A  r!