7i0(RddlmZmZerddlZdZdZdZGddeZy) )PushToHubMixinis_torch_availableNcd} t|j}|jj }|dk(rt jjy|S#t$rtd|wxYw)z Infers the device type from the model parameters. Args: model: The model instance. Returns: The device type. z { "RMSNorm": { "cuda": "kernels-community/layer_norm:LlamaRMSNorm", ... }, ... } zPCannot determine model device, please provide a device to the mapping. Example: cudarocm) next parameters StopIteration ValueErrordevicetypetorchversionhip)modelEXAMPLE_MAPPINGparamdev_types \/mnt/e/genesis-system/.venv/lib/python3.12/site-packages/transformers/utils/kernel_config.py infer_devicers| O U%%'( ||  H6 ==   ( O  ^_n^o p   s AA-cddlm}|dvrtd||jdd}|jdd}|||||ii||<y)NrLayerRepositoryrrxpunpu5Only cuda, rocm, xpu and npu devices supported, got: :)repo_id layer_name)kernelsrr split)r"r repo_namemodecompatible_mappingrrepo_layer_namer!s radd_to_mappingr)9sn' 33PQWPXYZZooc*1-Oooc"1%G /* &z"cddlm}ddlm}|dvrt d||j dd}|j dd}|j dd } |||||| | ii||<y) Nr)Path)LocalLayerRepositoryrrrr /) repo_path package_namer")pathlibr,r#r-r r$) r"r r%r&r'r,r-r(r0repo_package_names radd_to_mapping_localr4Js, 33PQWPXYZZooc*1-O$Q'I!,R0 &y/.* &z"r*c8eZdZdZidfdZd dZdZdZd dZy) KernelConfigze Kernel configuration class. This class is used to configure the kernel mapping for a model. Fc.||_i|_||_yN)kernel_mappingregistered_layer_namesuse_local_kernel)selfr9r;s r__init__zKernelConfig.__init__ds,&(# 0r*NcHddlm}||||||ii|j|<y)Nrr)r!r"revision)r#rr9)r<r!registered_namer"r r&r?rs r update_kernelzKernelConfig.update_kernelis4+ o#)%0 O,r*c|jD]+\}}t|ds|j|j|<-y)Nkernel_layer_name) named_moduleshasattrrCr:)r<rnamemodules rstore_registered_layer_namesz)KernelConfig.store_registered_layer_namesvsA!//1 MLD&v234:4L4L++D1 Mr*c|d}|j|t|jts$t d|dt |j|jj D]\}}||jjvrt d|dt|trd|vsd|vsJt d|d |t|trT|j D]@\}}|d vrt d |t|tr d|vsd|vs1t d|d |t d |d|y )aJ Validates the kernel_mapping to ensure that: 1. Each layer_name in the mapping is registered in the model (i.e., the model contains a module with a matching kernel_layer_name). 2. Each kernel value is either a string of the form 'org/repo:layer_name' or a dict mapping device types ("cuda", "rocm", "xpu", "npu") to such strings. 3. Each device key in a dict is one of "cuda", "rocm", "xpu", or "npu". 4. Each repo_name is a valid repository and layer name in the format 'org/repo:layer_name' (i.e., a string containing both a slash and a colon). 5. If a local path is detected, it should be in the format '/abs/path:layer_name'. The absolute path must include the `package_name`, like "/home/user/layer_norm". Args: model: The model instance whose modules are checked for registered kernel_layer_name attributes. Raises: ValueError: If a layer_name is not registered in the model, if a device is not supported, or if a repo_name is not a valid 'org/repo:layer_name' string. a For single device form remote { "RMSNorm": "kernels-community/layer_norm:LlamaRMSNorm", ... }, For multiple devices form remote { "RMSNorm": { "cuda": "kernels-community/layer_norm:LlamaRMSNorm", "rocm": "kernels-community/layer_norm:LlamaRMSNorm", ... }, ... } For single device form local { "RMSNorm": "/abs/path:LlamaRMSNorm", ... }, For multiple devices form local { "RMSNorm": { "cuda": "/abs/path:LlamaRMSNorm", "rocm": "/abs/path:LlamaRMSNorm", ... }, ... } z7Kernel mapping must be a dict of the following format: z, got: zLayer z` is not registered in the model, please register it first using register_kernel_forward_from_hubr.rzKernel mapping for 'zl' must be a valid repo name with a layer name (e.g., 'org/repo:layer_name' or '/abs/path:layer_name'), got: rrz'Kernel mapping must follow the format: N) rH isinstancer9dictr ritemsr:valuesstr)r<rMAPPING_FORMATr"kernelr r%s rsanitize_kernel_mappingz$KernelConfig.sanitize_kernel_mapping{s # H ))%0$--t4I.IYY`aefjfyfyaz`{| #'"5"5";";"= l J!JZZabhai!jkk- lr*c ddlm}i}t|}|jj D]\}}|j r |j n |j}|r||jz}t|tr.|} |jst||| ||it||| ||yt|ts|j D]6\} } | |k7r |jst|| | ||(t|| | ||8||_y)aW Transforms a simple kernel_mapping of the form: { "RMSNorm": "kernels-community/layer_norm:LlamaRMSNorm", ... }, or for local path: { "RMSNorm": "/home/user/liger_kernels:LigerRMSNorm", ... }, into a nested mapping: { "RMSNorm": { "cuda": { Mode.INFERENCE: LayerRepository( repo_id="kernels-community/layer_norm", layer_name="LlamaRMSNorm", ) } } } or for local path: { "RMSNorm": { "cuda": { Mode.INFERENCE: LocalLayerRepository( repo_path=Path("/home/user/liger_kernels"), package_name="liger_kernels", layer_name="LigerRMSNorm", ) } } } that's compatible with the kernels library. The device is inferred from the model's parameters if not provided. The Mode is inferred from the model's training state. r)ModeN)r#rSrr9rLtrainingTRAINING INFERENCE TORCH_COMPILErJrNr;r)r4rK) r<rcompilerSr'current_devicer"rPr&r%r s rcreate_compatible_mappingz&KernelConfig.create_compatible_mappingsb !%e,"&"5"5";";"= f J$)NN4==Dd000&#&" ,,":~y$Pbc(^YPTVhiFD))/f%FI/ 00&z69dL^_,ZDRde f f*1r*r8)F) __name__ __module__ __qualname____doc__r=rArHrQrZr*rr6r6_s,')51  M QlfJ1r*r6)utilsrrrrr)r4r6r_r*rras37 F"*y1>y1r*