謜i9 ddlmZddlmZmZerddlZddlmZddlmcm Z eje Z dZdejdejfdZej"d ejd ej$dejfd ZGd d ej(ZGddej,j.ZGddej,j.ZGddej4ZddeedzfdZGddZy))should_convert_module)is_torch_availableloggingNquantized_weightsreturnc|j}|dtzdz tz}t|dk(r|f}n |g|dd}|dz }tj||j tj }|jtj }tt|d|zdz}t|D]3}||z}t||z|d} |d| |z xxx||| d|zzzccc5|S)a Packs a tensor of quantized weights into a compact format using 2 bits per value. Parameters: ----------- quantized_weights : torch.Tensor A tensor containing ternary quantized weights with values in {-1, 0, 1}. These values are adjusted to {0, 1, 2} before being packed. Returns: -------- torch.Tensor A packed tensor where each element stores 4 quantized values (each using 2 bits) in an 8-bit format. rNdevicedtyper) shapeVALUES_PER_ITEMlentorchzerosr uint8tominrange) roriginal_shaperow_dimpacked_tensor_shapepackedunpackeditistartends \/mnt/e/genesis-system/.venv/lib/python3.12/site-packages/transformers/integrations/bitnet.py pack_weightsr"s  ',,Na ?2Q6?JG >a&j&<);< [[,5F5M5MUZU`U` aF ##EKK0H _~a0G;q@ AB 2Y@G %'/>!#45#+8E##6!a%#??@ Mrrc||j}t|dk(r|dtz}|f}n|dtz}|g|dd}tj||j tj }ttD](}||dz}||dz}dd|zz} || zd|zz |||*|j|dz S)u2 Unpacks a tensor of quantized weights that were stored in a packed format using 2 bits per value. Parameters: ----------- packed : torch.Tensor A tensor containing packed weights where each element represents 4 quantized values (using 2 bits per value). dtype : torch.dtype The dtype of the returned Tensor Returns: -------- torch.Tensor A tensor of unpacked weights, where each value is converted from its packed 2-bit representation. Example: -------- packed = torch.tensor([[0b10100001, 0b00011000], [0b10010000, 0b00001010]], dtype=torch.uint8) # Unpack the values unpacked = unpack_weights(packed) # Resulting unpacked tensor print(unpacked) # Output: tensor([[ 0, -1], [-1, 1], [-1, 1], [-1, 1], [ 1, 0], [ 0, -1], [ 1, -1], [ 1, -1]]) Explanation of the example: --------------------------- Let's take the first value for example 0b10100001, we will only focus on the first column, because every element is unpacked across the first dimension - First 2 bits: `01` → 0 at [0][0] - Second 2 bits: `00` → -1 at [0][2] - Third 2 bits: `10` → 1 at [0][4] - Fourth 2 bits: `10` → 1 at [0][6] the second value of the same row (0b10010000) will give the values for [0][1], [0][3], [0][5], [0][7] We subtract 1 because during the packing process, it's easier to work with values like 0, 1, and 2. To make this possible, we add 1 to the original ternary weights (which are typically -1, 0, and 1) when packing them. When unpacking, we reverse this by subtracting 1 to restore the original ternary values. r rNr r) rrrrrr rrr) rr packed_shapeoriginal_row_dimunpacked_shaperrrr masks r!unpack_weightsr*7sb<\!"-=>{{>&--u{{SH ? #9LO#l1o%QU|%}!a%8s 9 ;;u  !!r#c eZdZ d dededededef fd Zejd dZ ejdZ d Z xZ S) BitLinear in_features out_featuresbias use_rms_norm rms_norm_epsct |||_||_||_|j dt j|tz|ft j||j dt jd|||r)|j dt j|||nd|_ d|_ |rddl m}||||_ yy) Nweightrr weight_scaler r/r LlamaRMSNormeps)super__init__rr-r.register_bufferrrrronesr/rms_normmodels.llama.modeling_llamar7) selfr-r.r/r rr0r1r7 __class__s r!r;zBitLinear.__init__}s  &(   KK0+>kk     JJ     l5Y_)` aDI  B(,GDM r#c.d|dz z }d|dz zdz }||jjddjjdz }||zj j||}|j t j|fS)a\ Activation function : Performs symmetric, per-token quantization on the input activations. Parameters: ----------- input : torch.Tensor Input activations to be quantized. num_bits : int, optional (default=8) Number of bits to use for quantization, determining the quantization range. Returns: -------- result : torch.Tensor Quantized activation tensor, with values mapped to an `int8` range. scale : torch.Tensor The per-channel scaling factors used to quantize the tensor. rr Tdimkeepdimh㈵>r)absmaxvaluesclamproundrrint8)r@inputnum_bitsQnQpscaleresults r!activation_quantzBitLinear.activation_quants$X\" # 8a< 1 $UYY[__T_:AAGGDGQQ%-&&(..r26yy$e++r#c|||zz }|SN)r@rO input_scaler5outs r!post_quant_processzBitLinear.post_quant_processs{\12 r#c|j|j|}|j}t||j}|j |\}}t j |j|j|}|j||j|}|j.||jjddj|z }|S)Nrr rC) r>r3r*rrUFlinearrr[r5r/view expand_as)r@rOww_quant input_quantrYys r!forwardzBitLinear.forwards == $MM%(E KK $**5#'#8#8#? [ HH[^^DJJ/ 9  # #At'8'8+ F 99 2&003 3Ar#)NNFư>)) __name__ __module__ __qualname__intboolfloatr;rcompilerUr[rf __classcell__rAs@r!r,r,|s ""(H(H(H (H(H(HT ]],,. ]] r#r,cNeZdZdZeej dZedZy) WeightQuanta Implements a custom autograd function for weight quantization. This performs ternary quantization (-1, 0, 1) based on scaling by the mean absolute value of the weights. It uses the Straight-Through Estimator (STE) for the backward pass. c |j}|j}d|jjj dz }||zj j dd|z }|j|S)Ng?rGrHrCr )rrnrImeanclamp_rMrLr)ctxr3rrSs r!rfzWeightQuant.forwardsr fjjl'')00T0::5.'')//A6>yyr#c&|j}|SrWclonerw grad_output grad_inputs r!backwardzWeightQuant.backward &&( r#N rirjrk__doc__ staticmethodrrorfr~rXr#r!rsrss; ]]  r#rscNeZdZdZeej dZedZy)ActQuanta8 Implements a custom autograd function for activation quantization. This performs symmetric 8-bit quantization (to the range [-128, 127]) based on the maximum absolute value along the last dimension (per-token/row scaling). It uses the Straight-Through Estimator (STE) for the backward pass. c$|j}|j}d|jjddjj dz }||zj jdd|z }|j|S)NrCTrDrGrHi) rrnrIrJrKrvrMrLr)rw activationrrSs r!rfzActQuant.forwards  %%' jnn&**r4*@GGNNSWNXX 5(//177cBUJ }}U##r#c&|j}|SrWryr{s r!r~zActQuant.backwardrr#NrrXr#r!rrs; ]]$$r#rc ReZdZ d dedededededef fd ZdZd ZxZ S) AutoBitLinearr-r.r/ online_quantr0r1c t ||||||_d|_|rddlm} | |||_|sD|j dtjd|||j|jyy)Nrr6r8r5r r4) r:r;rr>r?r7r<rr="_register_load_state_dict_pre_hook load_hook) r@r-r.r/r rrr0r1r7rAs r!r;zAutoBitLinear.__init__s{ lD9(  B(,GDM   !   3 3DNN Cr#c|dz|vrV||dzj|jjk7r-t||dz|jj||dz<|S)Nr3r])rr3r*)r@ state_dictprefixargskwargss r!rzAutoBitLinear.load_hook sh X * ,FXrrsapplyr3rr^r_r/r5)r@rOr3outputs r!rfzAutoBitLinear.forward+s == $MM%(E    &&t{{3F[[Fu%%3  d///F r#)TNNFFrg) rirjrkrlrmrnr;rrfrprqs@r!rrsg """DDD DDDD<  r#rmodules_to_not_convertc d}|jD]\}}t||stjd5t |t j rI|r|jdk(rt|j|j|jdu|jj|jj|jdk(|j|j }|jdk(r|j#dnt%|j|j|jdu|jj|jj|r |jnd|r |j nd }|j#d|j'||d }ddd|st(j+d |S#1swYxYw) aa Public method that replaces the linear layers of the given model with bitnet quantized layers. Args: model (`torch.nn.Module`): The model to convert, can be any `torch.nn.Module` instance. modules_to_not_convert (`list[str]`, *optional*, defaults to `None`): A list of nn.Linear weights to not convert. If a parameter path is in the list (e.g. `lm_head.weight`), the corresponding module will not be converted. quantization_config (`BitNetConfig`): The quantization config object that contains the quantization parameters. Fmeta autobitlinearNonline)r-r.r/r rrr0r1offlinerg)r-r.r/r rr0r1TzYou are loading your model using bitnet but no linear modules were found in your model. Please double check your model architecture, or submit an issue on github if you think this is a bug.) named_modulesrrr isinstancennLinear linear_classrr-r.r/r3rquantization_moder0r1requires_grad_r, set_submoduleloggerwarning)modelrquantization_confighas_been_replaced module_namemodule new_modules r!replace_with_bitnet_linearr;s$224) V$[2HI  \\& ! )&")),&+>+K+K+^!.$*$6$6%+%8%8#[[4%}}33$mm11&9&K&Kx&W%8%E%E%8%E%E "J+<< I"11%8!*$*$6$6%+%8%8#[[4%}}33$mm11I\%8%E%EbgI\%8%E%Ebf"J--e4##K<$(!7 ) ))@   LI ) )s E%GG c eZdZdZ ddeeeejfdejjdzdedzdeeejffdZ y) BitNetDeserializec||_yrW) hf_quantizer)r@rs r!r;zBitNetDeserialize.__init__vs (r#N input_dictrfull_layer_namer c |jD]\}}t|ts|d||< d}|j|}ddlm} d} |j } |O|M| ||\} } t| dr5t| dr)| j}|jd}|tz|k(rd} | r,|jtj}t|| }||iS) Nrr3r)get_module_from_nameFr.r-Tr])itemsrlistpopquantizers.quantizers_utilsrrhasattrr.rrrrrr*)r@rrrrkeyvalue key_weightr3rneeds_unpacking target_dtyper_ expected_out actual_out weight_uint8s r!convertzBitNetDeserialize.convertys%**, +JC%&"'( 3 +  +F||  !<,UODIFAv~.76=3Q &22 #\\!_ /<?&*O !99U[[1L#L EFF##r#NN) rirjrkr;dictstrrrTensorrModulerrXr#r!rrusl) )-&* $d5<<001$xx%$t $ c5<< $r#rr) rrutilsrrrtorch.nnrtorch.nn.functional functionalr^ get_loggerrirrrr"rorr*rr,autogradFunctionrsrrrrrrrrXr#r!rs?/##   H %#ELL#U\\#LA"5<<A" A" A"A"HT Tn%..)).u~~&&.7BII7t7d3i$>N7t $ $r#