謜idZddlZddlZddlZddlZddlmZmZddlm Z ddl m Z ddl m Z ddlmZmZmZmZdd lmZd d lmZd d lmZer ddlZddlmZerdd lmZddlmZddl m!Z!m"Z"e r ddl#m$Z$ddl%m&Z&ejNe(Z)de*e+e,e-e,ej\ffdZ/ddde0e-zdzfdZ1 d=ddddde2de2de*e0e-e+fe0e-e+fff dZ3d>dej\ddfdZ4 d?ddde0e+e-ze+e-zfdzd e5e-dzddd!e2f d"Z6ddde0e-zdzde0dzddde0f d#Z7d$Z8de0dzd%e,e-fd&Z9d@de0dzd'e-d(e2dejte-ze+zfd)Z; d>ddd*e-dzd+e,e-dzde0d,e0dzd-ejxdzf d.Z=d/ej|d0e-d1e-dzd2e0de0f d3Z?ddd'e-dej|fd4Z@ dAdej\de0e+e-ze+e-zfdzd e5e-dzd5e,e,e-dzddde*e,e+e-ze0e+e-ze+e-zfe,e+e-ze,e+e0e-e+fe,e,e-e,e-e,e*e-ej\fff d6ZA dBdej\de0e+e-ze+e-zfdzd e5e-dzd7e2d8e2d9e2d5e,e,e-dzddfd:ZBd;ZCd<ZDy)Cz Some of the functions here are derived from the `accelerate` library, with some tweaks for better performances and simplicity/ease of use. N) OrderedDict defaultdict) TYPE_CHECKING) safe_open) save_file)is_accelerate_availableis_torch_availableis_torch_xpu_availablelogging)QuantizationMethod)is_deepspeed_zero3_enabled)is_fsdp_enabled)dispatch_model)get_max_memory)clean_device_mapget_max_layer_size)PreTrainedModel) HfQuantizerreturnc pt|dkr|ggfSg}g}|D]k}t|D cgc]\}\}} |j|dzs|!c} }}d} |j|| d|j|| dm|} t ||D]\}} | || ||z z } | ||fScc} }}w)a: Calculate the total size of a module, including its tied parameters. Args: tied_params (`List[str]`): The list of tied parameters. module_size (`int`): The size of the module without tied parameters. module_sizes (`Dict[str, int]`): A dictionary mapping each layer name to its size. modules_to_treat (`List[Tuple[str, nn.Module]]`): The list of named modules to treat. Returns: `Tuple[int, List[str], List[nn.Module]]`: The total size of the module, the names of the tied modules, and the tied modules. r.r)len enumerate startswithappendzip) tied_params module_size module_sizesmodules_to_treattied_module_names tied_modules tied_paramin_tied_module_indexmodule_size_with_tiestied_module_names `/mnt/e/genesis-system/.venv/lib/python3.12/site-packages/transformers/integrations/accelerate.pyget_module_size_with_tiesr-9s& ;!B""L!D 09:J0Knn91fq!zOdOdefilelOmQnopq  !12C!DQ!GH,->?BCD ((+K9J(K[$ $.>!?,zBZ!ZZ[ !"3\ AAos B1 B1 device_mapz&torch.device | int | str | dict | NonecXddlm}|6ts,|}|tjdk(r t d|}t |tjrd|i}nt |trP|dvrL |dk(r.ttjjdd }d |}dtj|i}n$t |tr|d kr td d|i}|*tr tdts td|S#t $rtd |d wxYw)Nr)*get_torch_context_manager_or_global_devicemetaaHYou are using `from_pretrained` with a meta device context manager or `torch.set_default_device('meta')`. This is an anti-pattern as `from_pretrained` wants to load existing weights. If you want to initialize an empty model on the meta device, use the context manager or global device with `from_config`, or `ModelClass(config)`)autobalancedbalanced_low_0 sequentialcuda LOCAL_RANKrzcuda:zWhen passing device_map as a string, the value needs to be a device name (e.g. cpu, cuda:0) or 'auto', 'balanced', 'balanced_low_0', 'sequential' but found rznYou can't pass device_map as a negative int. If you want to put the model on the cpu, pass device_map = 'cpu' z?DeepSpeed Zero-3 is not compatible with passing a `device_map`.zUsing a `device_map`, `tp_plan`, `torch.device` context manager or setting `torch.set_default_device(device)` requires `accelerate`. You can install it with `pip install accelerate`)modeling_utilsr0rtorchdevice RuntimeError isinstancestrintosenvironget ValueErrorr )r.r0device_in_context local_ranks r,check_and_set_device_maprF]sUK"<">FH  V 4 4G  ' *ell+*% J $;o)o V#  a!@A $ZL1 ell:67J J $ >A j)J % '^_ _&(Z  + PPZ|[\^  s 3A DD)modelr hf_quantizerzHfQuantizer | None buffers_only only_modulescFtt}tt}|rj}n fd}|}tdij }|D]\} } | |vr ||j | | } n| j } | j| z} | jd} tt| D]!}|dj| d|xx| z cc<#d| vr || jdddxx| z cc<|r|| xx| z cc<||fS)a Compute the size of each submodule of a given model (in bytes). Returns a tuple of 2 dicts, the fist one containing a mapping of all the modules and the corresponding size in bytes, and the 2nd one containing a mapping from all leaf modules (modules containing parameters, the end of the model graph) and the corresponding sizes. If `only_modules` is set to False, the first mapping will not only contain the size of all modules, but also the size of all parameters and buffers. c3vKjEd{jEd{y77wN)named_parameters named_buffers)rGsr, all_tensorsz)compute_module_sizes..all_tensorss4--/ / /**, , , 0 ,s959799all_tied_weights_keysNrrr) rr?rOgetattrkeysparam_element_size element_sizenumelsplitrangerjoinrsplit)rGrHrIrJall_module_sizesleaves_module_sizesiteratorrP tied_keysnameparam dtype_sizesize name_partsidxs` r,compute_module_sizesres9#3'%c*&&( -=6;@@BI+ e 9    #%88eLJ++-J{{}z)ZZ_ Z) AC SXXj#&67 8D @ 8 A $;  C 3A 6 74 ? 7 T "d * "#+& 0 00cHt||d\}}|jddS)zO Compute the total size of buffers in each submodule of a given model. T)rIr2r)rerB)rGrHr!r(s r, compute_module_total_buffer_sizerhs)+5,TROL!   B ""rf max_memoryno_split_module_classeslow_zeroc b|du}t|}tj|}|jdd|jddc}}t |Dcgc] }||dkDs |c}} | dk(r|S| dk(rQd}|rM|j D]:} t | ts|| xxdzcc<tjd| d nt||\} } | d |r| dz n| z} |g}nt |tttfs|g}d}t |dkDri}| jD]f\}}|d k(r |j|}|j j"}||vr ||vr|||<t|j t|k(sfnt |dkDrt%|j'nd}tt)| j't%t | dz }td t%||z}| |z } t+d |jD}|dd D] }t-|r |dk(r|dn| ||||<"|rHt%d| d t)t/d| Dcgc]}|| c}z }t-||d|d<|Scc}wcc}w)au Compute a `max_memory` dictionary for [`infer_auto_device_map`] that will balance the use of each available GPU. All computation is done analyzing sizes and dtypes of the model parameters. As a result, the model can be on the meta device (as it would if initialized within the `init_empty_weights` context manager). Args: model (`PreTrainedModel`): The model to analyze. max_memory (`Dict`, *optional*): A dictionary device identifier to maximum memory. Will default to the maximum memory available if unset. Example: `max_memory={0: "1GB"}`. no_split_module_classes (`set[str]`, *optional*): A set of layer class names that should never be split across device (for instance any layer that has a residual connection). hf_quantizer (`HfQuantizer`, *optional*): A quantizer for the model. low_zero (`bool`, *optional*): Minimizes the number of weights on GPU 0, which is convenient when it's used for other operations (like the Transformers generate function). NcpudiskrrF?z(We will use 90% of the memory on device z for storing the model, and 10% for the buffer to avoid OOM. You can set `max_memory` in to a higher value to use more memory (at your own risk).r2g?c3TK|] \}}t|ts|dkDs|"yw)rN)r=r?).0 device_id device_mems r, z&get_balanced_memory.."s-+iZPY[^E_dnqrdr s ((()rcopydeepcopypoprrSr=r?loggerinforelisttuplesetitems get_submodule __class____name__maxvaluessumsortedminrX)rGrirjrHrkuser_not_set_max_memoryaccelerator_max_memoryr(d num_deviceskeyr!leave_modules_sizesper_gpubufferno_split_childrenr_rb submodule class_name mean_leaves gpus_idx_listrdr&min_zeros r,get_balanced_memoryrs B)D0 +J!]]:6 ! % %eT 24J4N4NvW[4\DAq"8ZQVXY:ZZ[K FK00 1F vG/9/?/?/AMSb!dxC1Hjm'S]^aSbc 3dq,r*SqR]I^1_A*Q-1_-``aHjm4 1 A[z2`s J'!J': J,ct|trC|j}|dk7rt|||||dk(}n t |}|d|vr |dxxdzcc<||j |}|D]}t|t rtr@tjj|tjj|z }n?tjj|tjj|z }||xx|z cc<|||vst||||||<t||||}||j||S)zCompute the final `device_map` to use if we passed a value in ['auto', 'balanced', 'balanced_low_0', 'sequential']. Otherwise, we check for any device inconsistencies in the device_map. r6r5)rirjrHrkrmro)rirjrH)r.)r=r>_no_split_modulesrradjust_max_memoryr?r r:xpumemory_reservedmemory_allocatedr7rinfer_auto_device_mapvalidate_environment)rGr.rirHno_split_modulesinferred_max_memory device_name unused_memorys r,_get_device_mapr0so*c" 22  %"5%(8)$(88 # #1"<   %+>">  &$ . &  #"."@"@AT"U / rK+s+)+$)II$=$=k$JUYYMgMghsMt$tM$)JJ$>$>{$KejjNiNijuNv$vM#K0MA0%+*C367J;7WYcdoYp3q#K0 r+ *$4%    #  - - - D rfc(||||d}dtjtjvr|j|d<dtjtjvr.|,|j j tjk(rd|d<|`|j j tjk(r9t|tr)d|jvsd|jvrd|d<tsts t|fi|yyy)N)r. offload_dir offload_indexoffload_buffers skip_keys force_hooksTrmrnr)inspect signaturer parameters_skip_keys_device_placementquantization_config quant_methodr HQQ FBGEMM_FP8r=dictrrr)rGrHr.offload_folderrrdevice_map_kwargss r,accelerate_dispatchrls %&*  g''7BBB).)J)J+& **>:EEE  $  , , 9 9=O=S=S S+/-(  , , 9 9=O=Z=Z Z z4 ( j'') )Vz7H7H7J-J/3+,  %?%Au2 12&B rf param_namesc J|tj|dStjdj dt |j ddD}i}|D]=}|j|}|r||jn|jdd||<?|S)zT Expand a device map to return the correspondence parameter name to device. rm|c3(K|] }d|d yw)()N)rqks r,rtz$expand_device_map..srqAaSrsc:|jdt|fS)Nr)countr)xs r,z#expand_device_map..sQWWUX\[^_`[aLbrfT)rreverser2) rfromkeysrecompilerYrrSmatchgrouprB)r.rdevice_map_regexnew_device_mapr` device_matchs r,expand_device_maprs}}[%00zz rF:??+replacerrrpathseprYrWr~base_model_prefix)rGrrr.rrweight_mappingrris_offloaded_safetensors renamingsentrymeta_state_dictparam_device_map str_dtyperfolderrvweight_renaming_map target_name source_namedisk_offload_indexs r,accelerate_disk_offloadrsG& '$7/t;l@PQR@S@\@\]k@lI!(6\u*UN:[U\ \ **,,Z9M9M9OP8=8ICJ&&x4y  #y1A!1DPT'U'Z'Z'\^nop^qrJWW[[%%&6q&9&?&? &LSb&QRFAQR^A_AeAeAghA!RWW\\&!44hJhjt de aB0G0G YZ[ \^_ _  -@,E,E,G  ) [..3CK3PTZ3Z $.{$;*"        E]i  sGG +G">!G(4 G-weightrrrc| tdtjj||d}t ||i|t |j jdd}|||d||<|S)zWrite `weight` to disk inside `offload_folder`, and update `offload_index` accordingly. Everything is saved in `safetensors` format.aGThe current `device_map` had weights offloaded to the disk, which needed to be re-saved. This is either because the weights are not in `safetensors` format, or because the model uses an internal weight format different than the one saved (i.e. most MoE models). Please provide an `offload_folder` for them in `from_pretrained`.rrr2r)rCr@rrYrr>rr)rrrrsafetensor_filers r,offload_weightrs| !  ggll>k],3OPO {F#_5FLL!))(B7I6EValu!vM+ rfc||jd}tdt|Dcgc]}dj|d| c}dgz}|D]S}|j |}t |ds!|j j}|j|dk7r|dn|d}nt|d||} | Scc}w)a'Load `param_name` from disk, if it was offloaded due to the device_map, and thus lives as a meta parameter inside `model`. This is needed when resaving a model, when some parameters were offloaded (we need to load them from disk, to then resave them to disk in the correct shard...).rrNr2_hf_hookzd is on the meta device because it was offloaded, but we could not find the corresponding hook for it) rWrXrrYrhasattrr weights_maprrC) rGr module_partsrdmodules_to_check parent_nameparentrtruncated_param_nametensors r,load_offloaded_parameterrs ##C(LAFq#lJ[A\]#et!45]ac`dd'  $$[1 6: & //55K#-#5#5;Z\K\ Q6Gbmoq#r   l, ,  - .F M!^sB9tied_parametersc Dt|}|g}nt|tttfs|g}t|j }d|vr|j d|Dcgc] }|dvs| }}d|vrdg}nt|dkDr|ddg}ndg}t||d\} } |t|jdkDrut |jj} | D cgc]B} t|jjD cgc] \} }|| k(s | c}} | gzD}} } }ngg}t|jdt|jzt|jdz}||||| |||fScc}wcc}} wcc}} } w) zZ Initialize variables required for computing the device map for model allocation. rn)rmrnmpsrrmF)rJrecurse)rr=r{r|r}rSrrrerQrrr~rNnamed_childrenrO)rGrirjrrHdevicesr;gpus main_devicesr!r(groupstargetrrr"s r,_init_infer_auto_device_maprs& +J&"$ /$s1C D#:";:??$%G Wv!( JvF/,IF JD J }w TQQ' w *5,USOL! u** +a /44;;=>F%e&A&A&G&G&IYdaQRX[Y]c\ddeO "dO U # #E # 23 u##% & ' u""5"1 2 3    = K"Zs*! F+F&'F FFFFverbose clean_resultrc 34t|||||\}}} } } }}} t} d}tj|d}i}i}t | | |\}}t | dkDr~| j d\3}|rtd3d|Dcgc]}|3k7s |j3dzr|!}}t |dk(rJt | Dcgc].\}}t|tjjs+||f0c}}| |\}}| 3}|Dcgc],}t3fd|Drt3fd|Ds|.}}|rt |dkDrtd|t|Dcgc]}|Dcgc]}3dz|dzvs|c}c}}g}|rt |dkDrtd|||}|dk7r||nd }d}||| vr||z }|}t!||| | \}} }!| |||z|kr|r>d 3}"| r |"d | z }"n |"d |d z }"||"d|||z d z }"|"d|dz }"t|"||xx|z cc<|| 3<| D]J44| Dcgc]}|d c}vr.t#4fdt%| D}#| j |#|| 4<L|s>t|tjr$t'||}$|j)|d|$z||<t |dkDrt|||z|krh|r$td||d3d | d|||z d|d d}%t+| |!D]\4}&t-|&j/}'t |'dk(s|&j0j2|vrG|rtd4dt-|&j5d|'z}'|'D(cgc]\}}(4d||(f}'}}(t%| D)*cgc]\})\}}*|4k(s|)c}*}})d}#3|fg| d |#z|'z| |#dzd z} t | Dcgc].\}}t|tjjs+||f0c}}| |\}}d}%n|%r|r td|||z|k\r@t|tj6st|tj8rgnt-|j/}+|r!td||d3d|||z d|d t |+dk(s|j0j2|vr|rtdn|rtd3dt-|j5d|+z}+|+D(cgc]\}}(3d||(fc}(}| z} t | Dcgc].\}}t|tjjs+||f0c}}| |\}}U||dk(r||z||<|||z||<|dz }3|fg| z} t | dkDr~|j;D,cic]\}},|,dkDs ||,}}},|r t=| } |j)dd|j)ddz}-|-dkDrp|snd}.|j;D]0\}/}0|/dk(s|/dk(r|.r|j)|/d}1|0|-|1zk\s/d}.2t | dkDr|.st>jAd|-d |r?d!jCd"|j;D}2t>jEd#|2d$tG|| | Scc}wcc}}wcc}wcc}wcc}}wcc}wcc}(}wcc}*}})wcc}}wcc}(}wcc}}wcc},}w)%a; Compute a device map for a given model giving priority to GPUs, then offload on CPU and finally offload to disk, such that: - we don't exceed the memory available of any of the GPU. - if offload to the CPU is needed, there is always room left on GPU 0 to put back the layer offloaded on CPU that has the largest size. - if offload to the CPU is needed,we don't exceed the RAM available on the CPU. - if offload to the disk is needed, there is always room left on the CPU to put back the layer offloaded on disk that has the largest size. All computation is done analyzing sizes and dtypes of the model parameters. As a result, the model can be on the meta device (as it would if initialized within the `init_empty_weights` context manager). Args: model (`torch.nn.Module`): The model to analyze. max_memory (`Dict`, *optional*): A dictionary device identifier to maximum memory. Will default to the maximum memory available if unset. Example: `max_memory={0: "1GB"}`. no_split_module_classes (`set[str]`, *optional*): A set of layer class names that should never be split across device (for instance any layer that has a residual connection). verbose (`bool`, *optional*, defaults to `False`): Whether or not to provide debugging statements as the function builds the device_map. clean_result (`bool`, *optional*, defaults to `True`): Clean the resulting device_map by grouping all submodules that go on the same device together. offload_buffers (`bool`, *optional*, defaults to `False`): In the layers that are offloaded on the CPU or the hard drive, whether or not to offload the buffers as well as the parameters. rz Treating module rc32K|]}dz|dzvywrNrrqrr_s r,rtz(infer_auto_device_map..s=Q4#:S(=c32K|]}dz|dzvywrrrs r,rtz(infer_auto_device_map..s#Itde$QT*XY\_X_J_Itrz' Found the relevant tied param groups z4 So those parameters need to be taken into account rnNzPutting z and z (size=rz (available=z on c3:K|]\}\}}|k(s|ywrMr)rqr&r'r(r+s r,rtz(infer_auto_device_map..s",u91fq!_`dt_tQ,uszNot enough space on z to put z (space available z, needed size z).Fz Splitting rrTz?None of the tied module can be split, going to the next device.z, module size z6This module cannot be split, going to the next device.rmzCurrent model requires z bytes of buffer for offloaded layers, which seems does not fit any GPU's remaining memory. If you are experiencing a OOM later, please consider using offload_buffers=True. c34K|]\}}d|d|dyw)z - z: z bytes requiredNr)rqr;mems r,rtz(infer_auto_device_map..^s(! 6Afcd6("SE 1! sz{Based on the current allocation process, no modules could be assigned to the following devices due to insufficient memory: z These minimum requirements are specific to this allocation attempt and may vary. Consider increasing the available memory for these devices to at least the specified minimum, or adjusting the model config.)$rrrrrrrxprintrr=r:nnModuleanyallrr-nextrrhrBrr{r rrrN ParameterTensorr~rrywarningrYrz$check_tied_parameters_on_same_device)5rGrirjrrrrrHr r r r!r"r.current_devicedevice_memory_useddevice_buffer_sizes device_minimum_assignment_memorymax_layer_sizemax_layer_namesmoduler'mr tied_grouptied_param_groupsprr;current_max_sizecurrent_memory_reservedr*r#r$outputr)current_buffer_sizesplit_happened tied_moduletied_module_childrenrr&r(modules_childrenrnon_gpu_buffer_sizeis_buffer_fit_any_gpu gpu_devicegpu_max_memorygpu_memory_used devices_infor_r+s5 @@r,rrTs n $E:7NP_amn  JNw2')$'99I "l 2/.@ &4 #AZ l4DB >0,  #'9&'ADY'Y]m'm#D6*$&7%899F }A66F#/ -=@RSY@Z-Z,[[\]]FD**f v &*? ? & &Jt %6 6 #6F'G!'GG(,,u JZ@[,u(u%$(():;06 +, 6#z&"))'D&Fv|&\#.A.E.Efa.PSf.f#F+  { a $6v$>$LP`$`*7>+B*C8D6QVWhVij!!14Fv4N!N O~^s]ttvx#N145F 1U - +'+K,F,F,H'I$+,1[5J5J5S5SWn5nJ'7&8:;'+K,H,HQV,H,W'X[o'o$Sg'h41a,<+=Qqc)BA(F'h$'h8ABR8S$m$m91fq!WX\lWlQ$mno$p!F^$&'9(9:;*+''81'<'>?@!3E(8[1Jq%((//+B*C8D6Qc'*>W  ! #Z:L9Q9Q9S_+&#WZ]^W^&#+__%j1 -11%;>Q>U>UV\^_>``Q %*4*:*:*< 1 &JU"jF&:("4"8"8Q"G!%8?%JJ,0) 1 t9q=!6 NN)*=)>?()  (yy! EeEkEkEm!    %nw x )*E ]eX B fR(HH(i$m\J$U\`s [=[=)[=,\ 4\ 1\+ \ 4\ \ \  \!\\# \# ,\* 1\* \0$,\6 \6 1 \<?\< \ c||vr||Sdj|jddd}||k(rtd|dt||S)Nrruz-The `device_map` does not contain the module )rYrWrC_get_param_device)r`r. parent_params r,rArAms] %  88EKK,Sb12LuHqQRR z::rfc|D]Y}i}|D]}t||||<tt|jdkDsAtj d|d[y)a9 Check if tied parameters are on the same device Args: tied_params (`List[List[str]]`): A list of lists of parameter names being all tied together. device_map (`Dict[str, Union[int, str, torch.device]]`): A map that specifies where each submodule should go. rz*Tied parameters are on different devices: zC. Please modify your custom device map or set `device_map='auto'`. N)rArr}rryr%)rr. tie_paramtie_param_devicesr`s r,r&r&wsz!  LE'8 'K e $ L s$++-. /! 3 NN<=Nrr-rrFboolrerhr}rrrrr;rrrr$rrrrrAr&rrfr,rVs|   0 !' ;1!)/N0(   H %!B  3S 4 ? *+ !BH-)Q-VZ]`V`cgVg-d*. /1 /1&/1/1 /1  4S>4S> )* /1d#BII#EY#59/3)- f fS3Yc )*T1f!X_f' f  fR9 9s T!9t 9' 9  9x38$+DI&4$;CT^c^j^jmp^psv^v 9 9t93i$&9 9 Tk 9 ;;  9x5<<c3QU:fjos($535<<859/3.2)- C 99CS3Yc )*T1C!X_C$s)_t+ C ' C  sOsC#I sOIcNcOIsBII~ !  CP59/3!.2)-V 99VS3Yc )*T1V!X_V V  V  V$s)_t+V'Vr;rf