鬜iAAUddlZddlZddlZddlmZddlmZddlm Z ddl m Z m Z ddlmZdd lmZe j$eZ dd lmZmZmZmZmZdd lmZdd lmZ dd lmZ ddlm!Z"dZ#ejHjKddjMZ'dZ(e'e vZ)de*fdZde*fdZ!deddideddiejVeddiejVeddiejVedd iejVed!d iejVeddid"ded#d$iejXed%d&ejVed%d&iejVed'd&iejVed%d&id(dejVejZzed)d*d+iidejVejZzed)d,d+iidejVejZzed)d-d+iidejVejZzed)d.d+iidejVejZzed)d/d+iidejVejZzed)d0d+iid1 Z.e/e*e/ee*zee/eefzffe0d2<e0ejVed3d45iejVed3d45id6e.d7<d8Z1dUd9Z2dAddBdCddBdCddBdDZ3e/e*e/e*e*ffe0dE<iZ4e/e*edzfe0dF<dGe*dzdHe5fdIZ6 dUdGe*dJedzdHedzfdKZ7e4fdLe*dMe/e*edzffdNZ8dVdLe*dOe*dzdPe9e*zdzdHefdQZdRe:eezfdSZ;gdTZZd?Zd@Z2YwxYw)WN)Callable) ModuleType)version)ENV_VARS_TRUE_VALUESlogging)is_kernels_available)flash_attention_forward)DeviceLayerRepositoryModeregister_kernel_mappingreplace_kernel_forward_from_hub) get_kernel)use_kernel_forward_from_hub)FuncRepository)use_kernel_func_from_hubTFUSE_HUB_KERNELSYES layer_namecbtr t|StjdtdS)NFkernels hub usage is disabled through the environment USE_HUB_KERNELS=c|SNclss a/mnt/e/genesis-system/.venv/lib/python3.12/site-packages/transformers/integrations/hub_kernels.pyz-use_kernel_forward_from_hub..Ess)_kernels_enabled$_kernels_use_kernel_forward_from_hublogger warning_once_TRANSFORMERS_USE_HUB_KERNELS)rs rrr>s2 7 C C   XYvXwx # "r! func_namectrtr t|Ststj ddStj dt dS)Nzwuse_kernel_func_from_hub is not available in the installed kernels version. Please upgrade kernels to use this feature.rc|Srrfuncs rr z*use_kernel_func_from_hub..Tsr!)r"_has_use_kernel_func_from_hub!_kernels_use_kernel_func_from_hubr$r%r&)r's rrrGsV  =4Y? ?0##B% $##\]z\{|% $r!cudaz!kernels-community/deformable-detrMultiScaleDeformableAttention)repo_idrzkernels-community/moe Llama4TextMoezkernels-community/liger_kernels LigerRMSNormzkernels-community/rmsnormRMSNormzkernels-community/mlx_rmsnorm)r.rocmxpumpsnpuzmedmekk/triton-llama-mlpTritonLlamaMLPzkernels-community/megablocksMegaBlocksMoeMLPzahadnagy/megablocks)r.r4r5zkernels-community/activationFastGELU)r0rr QuickGELUNewGELUSiluGeluGeluTanh) r/r1r3MLPr9r:r;r<SiLUGeLUr?_KERNEL_MAPPINGzkernels-community/rotaryapply_rotary_transformers)r0r')r5r.rotary_pos_embcT|vxs"tfd|jDS)Nc3ZK|]"}t|txr t|$ywr) isinstancedicthas_key).0vkeys r zhas_key..s&[1z!T2Fwq#F[s(+)anyvalues)drMs `rrJrJs'ax[3[PQPXPXPZ[[[r!cp|t}t|drtds tdt |y)Nr5z0.10.2) MIN_VERSIONzfkernels uses an incompatible version. Please install the latest version with `pip install -U kernels`.)rCrJr ImportErrorr)mappings r$register_kernel_mapping_transformersrVs8 ?%G 7E "+?H+Ux  (r!c d}|S)Nc|Srrrs r decoratorz.use_kernel_forward_from_hub..decoratorsJr!rargskwargsrYs rrrs r!c d}|S)Nc|Srrr*s rrYz+use_kernel_func_from_hub..decoratorsKr!rrZs rrrs r!ceZdZdZy)r ctd)NzNLayerRepository requires `kernels` to be installed. Run `pip install kernels`. RuntimeError)selfr[r\s r__init__zLayerRepository.__init__sop pr!N)__name__ __module__ __qualname__rdrr!rr r s qr!r ctd)Nz^replace_kernel_forward_from_hub requires `kernels` to be installed. Run `pip install kernels`.rar[r\s rrrs l  r!ctd)NzVregister_kernel_mapping requires `kernels` to be installed. Run `pip install kernels`.raris rrr ssttr!ctd)Nzcregister_kernel_mapping_transformers requires `kernels` to be installed. Run `pip install kernels`.raris rrVrVs q  r!zkernels-community/causal-conv1d)r0rzkernels-community/mamba-ssm)z causal-conv1dz mamba-ssmzfalcon_mamba-ssm_HUB_KERNEL_MAPPING_KERNEL_MODULE_MAPPINGattn_implementationreturnc>|duxrtjd|duS)zJCheck whether `attn_implementation` matches a kernel pattern from the hub.Nz'^[^/:]+/[^/:]+(?:@[^/:]+)?(?::[^/:]+)?$)research)rns r is_kernelrss- 4' c II@BU V^b br!attention_wrappercrddlm}ddlm}d|vr|j ddn|}t |syt s tdd|vr%|j d\}}|j}n|}d}|j}|jd \}}}|j}|r|jnd} t|| } t| dr |t}|} n| t| |} |j!| |j!||d| S#t$r} td |d | d d} ~ wwxYw)a< Load and register the kernel associated to `attn_implementation`. Args: attn_implementation: A string, usually a kernel repo like "kernels-community/flash-mla". attn_wrapper: a callable for the wrapper around the attention implementation. In `transformers` we have a wrapper around the `flash_attn_var_len` call, and the same goes for `sdpa` and `eager`. They just prepare the arguments properly. This is mostly used for continious batching, where we want the `paged` wrapper, which calls the paged cache. r)ALL_MASK_ATTENTION_FUNCTIONS)ALL_ATTENTION_FUNCTIONS|r Nz`kernels` is either not installed or uses an incompatible version. Please install the latest version with `pip install -U kernels`.:@)revisionz-An error occurred while trying to load from 'z': .flash_attn_varlen_funcflash_attention_2) masking_utilsrvmodeling_utilsrwsplitrs_kernels_availablerTstrip partitionr Exception ValueErrorhasattrr getattrregister) rnrtrvrwactual_attn_namer0 kernel_name_revkernelekernel_functions rload_and_register_attn_kernelr%sf=8 ; - %--c37/**5  '/ //0F TI"))Y"E  ' (-@-B ,000O3DF'- $ $(GK ; =! (#'GK : ; 9 (#'GK 4 ; 9 ( '"&  ' ,'+ $ ;   ,s= AD>$E5EE- EE EEE*)E*r{rc ddlm}d||d}trgtjj d}t j|t jdk\rt||||St|||Std ) Nr) __version__ transformers) frameworkrr0kernelsz0.10.4)r{r user_agentrzFkernels is not installed, please install it with `pip install kernels`) rrrmetadatar pkg_versionparseget_kernel_hubrT)rr{rrrkernels_versions rrrsv-+R]^J#,,44Y?   _ -1B1B81L L!+'^hi i!+'R Rbccr! module_namesc8ttrgfd}|S)a This decorator attaches the target function as an attribute of the module. The function must already be decorated with @use_kernel_func_from_hub this decorator then wraps it as an nn.Module internally. When kernelize is later applied to the full model, the function can be accessed as a regular module attribute and kernelized just like any other layer. The kernelization is performed in place, modifying the module directly. c<|jfd}||_|S)NcF|g|i|D]}t|d|y)N rotary_fn)setattr)rcr[r\fnr orig_inits rnew_initz8use_kernelized_func..decorator..new_inits0 d ,T ,V ," /k2. /r!)rd)rrrrs @rrYz&use_kernelized_func..decoratorsLL  /    r!)rHr)rrYs` ruse_kernelized_funcrs#,)$~   r!) r rrrrVrrrrr)NN)=importlib.metadatarosrqcollections.abcrtypesr packagingrrutilsrrutils.import_utilsr flash_attentionr get_loggerrer$rr r rrrrrrr#rrTrr-r,environrupperr&rr"str INFERENCETRAINING TORCH_COMPILErCrI__annotations__rJrVrlrmboolrsrrintlistr__all__rr!rrs+ $,154   H %t  * .Y(,%%'JJNN3De$L$R$R$T!48LL## %C %" O;:*  O/* =-!=-!7(!;(!=-!5  D O2+  :1 :1! 11!:1!# 2 !3!33_:)6  !3!33_:*6  !3!33_:(6  !3!33_:vWX6  !3!33_:vWX6  !3!33_:z[\6 k|eOT#tFSL/DI^D_2_$_``a|~!6B]! 6B]! - ()\)^"CqQ:qI$AaP2T#tCH~-. 8:S*t"33493:$DH7719D7$7tPf, #, S*t:K5K0L, ^ dC d3: dsSySWGW dcm dd8nx&?4 O  .(-%.r!   qq u =! sUM!"M)M1IM!M M!MM!MM!MM!!"NN