謜i*ddlZddlZddlmZddlmZddlZejjjejjjejjjejjjejjjejjjejjjejjj ejjj"ejjj$ejjj&ejjj(ejjj*ejjj,dZ d)dej0dededej4dzd ej0f d Z d)dej0d ed edej4dzd ej0f d Z dej0ded ej0fdZ dej0d ej0fdZ dej0d ej0fdZ dej0d ej0fdZd*dej0ded ej0fdZd+dej0dedej4dzd ej0fdZd+dej0dedej4dzd ej0fdZ d,dej0dedededej4dzd ej0f dZ d,dej0dedededej4dzd ej0f dZ d-dej0d ed edededej4dzd ej0fdZ d.dej0dedej4dzd ej0fdZ d/dej0ded edej4dzd ej0f dZdej0d ej0d ej0fd!Zd0d"Zd#Zd$Z d%Z!ed&Z"ed'Z#ed(Z$y)1N) defaultdict)contextmanager)uniform_normal_ constant_ones_zeros_eye_dirac_xavier_uniform_xavier_normal_kaiming_uniform_kaiming_normal_ trunc_normal_ orthogonal_sparse_tensorab generatorreturncDt|ddstd||||S|S)N_is_hf_initializedFr)rrrgetattrTORCH_INIT_FUNCTIONS)rrrrs W/mnt/e/genesis-system/.venv/lib/python3.12/site-packages/transformers/initialization.pyrr*s, 6/ 7#J/!qIVV MmeanstdcDt|ddstd||||S|S)NrFr)rr rr)rrr rs rrr2s- 6/ 7#I.vDcU^__ Mrvalc@t|ddstd||S|S)NrFr)r"r)rr"s rrr:s& 6/ 7#K0SAA Mrc<t|ddstd|S|S)NrFrrrs rrr@s$ 6/ 7#G,V44 Mrc<t|ddstd|S|S)NrFr rr%s rr r Fs$ 6/ 7#H-f55 Mrc<t|ddstd|S|S)NrFr rr%s rr r Ls$ 6/ 7#F+F33 Mrgroupsc@t|ddstd||S|S)NrFr )r(r)rr(s rr r Rs& 6/ 7#H-fVDD MrgaincBt|ddstd|||S|S)NrFr r*rrrr*rs rr r Xs* 6/ 7#$56vDT]^^ MrcBt|ddstd|||S|S)NrFr r,rr-s rr r ^s* 6/ 7#$45f4S\]] Mrmode nonlinearitycFt|ddstd|||||S|S)NrFrrr/r0rrrrr/r0rs rrrds6 6/ 7#$67 ad   MrcFt|ddstd|||||S|S)NrFrr2rr3s rrrrs6 6/ 7#$56 ad   MrcHt|ddstd||||||S|S)NrFr)rr rrrr)rrr rrrs rrrs3 6/ 7#O4V$CSTXYenoo MrcBt|ddstd|||S|S)NrFrr,rr-s rrrs+ 6/ 7#M26PYZZ MrsparsitycDt|ddstd||||S|S)NrFr)r7r rr)rr7r rs rrrs- 6/ 7#I.vc]fgg Mrotherct|dds/tj5|j|cdddS|S#1swY|SxYw)NrF)rtorchno_gradcopy_)rr9s rr=r=sC 6/ 7 ]]_ '<<& ' ' M ' Ms ?A ctjjj|\}}|dk(r|}n|dk(r|}n |dk(r||zdz }dz }|dk(r$t |t j |dz y|d k(r!t|t j |y|d k(r't j d |z}t|| |ytd |) Nfan_infan_outfan_avg?truncated_normalg۶%?)r normaluniformzinvalid distribution ) r;nninit_calculate_fan_in_and_fan_outrmathsqrtrr ValueError)rr/ distributionr?r@denomvariancebounds r_variance_scalingrRshhmmAA&IOFG x    '!Q&U{H))f$))H"58K"KL  !DIIh/0  " !h,'%'0?@@rc<t|ddst|dd|S)NrFr?rDr/rNrrRr%s r lecun_normal_rVs! 6/ 7&x>PQ Mrc<t|ddst|dd|S)NrFr?rErTrUr%s rdefault_flax_embed_init_rXs 6/ 7&xhG Mr) z torch.nn.initztorch.nn.modules.activationztorch.nn.modules.transformerztorch.nn.modules.linearztorch.nn.modules.lossztorch.nn.modules.batchnormztorch.nn.modules.convztorch.nn.modules.normalizationztorch.nn.modules.rnnztorch.nn.modules.sparsec #4Ktt} tD]x}|tjvstj|}t j D]9}t||st|||||<t||t|;zd|jD]*\}}|jD]\}}t|||,y#|jD]*\}}|jD]\}}t|||,wxYww)a Guard the `torch.nn.init` primitive functions to behave exactly like the functions in this file, i.e. be protected against the `_is_hf_initialized` flag to avoid re-init if the param was already loaded. Usually, all models are using the init from `transformers` which are already guarded, but just to make extra sure and for remote code, we also use this context manager. N) rdictTORCH_MODULES_TO_PATCHsysmodulesrkeyshasattrrsetattrglobalsitems) originals module_namemodule func_name functionsfuncs rguard_torch_init_functionsrisD!I11 IKckk)[1!5!:!:!<IIvy17>vy7Q &))4 79Y3GHI I "+!2 1 FI#,??#4 1 4 40 1 1!2 1 FI#,??#4 1 4 40 1 1s'DC6C&2C>D?DDc #rKddlm}d}tt} tD]m}|t j vst j |}tjD].}t||st|||||<t|||0o|j}||_ d|jD]*\}}|jD]\}}t|||,||_ y#|jD]*\}}|jD]\}}t|||,|_ wxYww)ac Disable weight initialization both at the torch-level, and at the transformers-level (`init_weights`). This is used to speed-up initializing an empty model with deepspeed, as we do not initialize the model on meta device with deepspeed, but we still don't need to run expensive weight initializations as we are loading params afterwards. PreTrainedModelcyNargskwargss r empty_funcz#no_init_weights..empty_func rN)modeling_utilsrmrrZr[r\r]rr^r_rr` init_weightsrb) rmrtrcrdrerforiginal_init_weightsrgrhs rno_init_weightsrysB0 D!I=1 ?Kckk)[1!5!:!:!<?Ivy17>vy7Q &))4 :>? ?!0 < <'1$ "+!2 1 FI#,??#4 1 4 40 1 1(=$ "+!2 1 FI#,??#4 1 4 40 1 1(=$s)D7C.6C./:C.)AD7.AD44D7c#tKddlm}d} |j}||_d||_y#|_wxYww)a Disable weight tying during loading with `from_pretrained`. This is needed as we want to have access to ALL weights in the state_dict during `from_pretrained`, and otherwise tying them would remove them from it, as it's called in `post_init` when instantiating. rkrlcyrorprqs rrtz"no_tie_weights..empty_func$rurN)rvrm tie_weights)rmrtoriginal_tie_weightss rno_tie_weightsr~s?0 ;.::&0# ';#&:#s 8,8 58)rCN)rk)rCN)rr? leaky_reluN)rrCgg@N)rkN)g{Gz?N)r?rE)%rKr\ collectionsr contextlibrr;rHrIrrrrr r r r r rrrrrrTensorfloat Generatorintstrr=rRrVrXr[riryr~rprrrs #%  &&xx}}$$(( XX]] hhmm"" HHMM  hhmm""xx}}44hhmm22 66xx}}44XX]]0088==,,xx}}$$&_c LL"-2EJ__W[E[ \\dh LL %27JO//\`J` \\ell5<< %,,5<< 5<<ELL %,, 5<<U\\ ELLZ^H^jojvjv 5<<uuY]G]iniuiu$(,  LL       %   \\  $(,  LL       %   \\  (,  LL         %  \\ (, LL % \\ cg LL$)05IN[_I_ \\%,,u|| A,  114!=!=H;;r