iZ1pddlZddlmZmZddlmZddlmZmZddl m Z ddl m Z ddl mZmZd d lmZd d lmZmZmZmZd d lmZmZej4eZGd deZGddeZGddeZGddeZ GddeZ!Gdde Z"GddeZ#GddeZ$gdZ%y)N)Cache DynamicCache)layer_type_validation)create_causal_mask!create_sliding_window_causal_mask)BaseModelOutputWithPast)Unpack)TransformersKwargslogging) LlamaConfig)LlamaDecoderLayerLlamaForCausalLM LlamaModelLlamaPreTrainedModel)Qwen2AttentionQwen2RotaryEmbeddingc,eZdZdZdZdZdddddd d d d d dddgdddddddddfdededededededededed ed!ed"e d#edzd$ed%e d&ed'ed(e d)e dzd*ed+e edzf*fd, Z xZ S)- CwmConfiga Configuration for Code World Model (CWM). This is an inherited Llama3-compatible configuration with layer-interleaved sliding-window attention. Configures a `CwmModel`. Designed to yield a configuration mirroring the model in the [facebook/cwm](https://huggingface.co/facebook/cwm) architecture by default. Other models include: - [facebook/cwm-sft](https://huggingface.co/facebook/cwm-sft) - [facebook/cwm-pretrain](https://huggingface.co/facebook/cwm-pretrain) Args: vocab_size (`int`, *optional*, defaults to 128256): Vocabulary size of the CWM model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`CwmModel`] hidden_size (`int`, *optional*, defaults to 6144): Dimension of the hidden representations intermediate_size (`int`, *optional*, defaults to 21504): Dimension of the MLP representations num_hidden_layers (`int`, *optional*, defaults to 64): Number of hidden layers in the Transformer decoder num_attention_heads (`int`, *optional*, defaults to 48): Number of attention heads for each attention layer in the Transformer decoder num_key_value_heads (`int`, *optional*, defaults to 8): This is the number of key_value heads that should be used to implement Grouped Query Attention (GQA). If it is not specified, will default to `num_attention_heads`. head_dim (`int`, *optional*, defaults to 128): The attention head dimension. hidden_act (`str` or `function`, *optional*, defaults to `"silu"`): The non-linear activation function (function or string) in the decoder. max_position_embeddings (`int`, *optional*, defaults to 131072): The maximum sequence length that this model might ever be used with. CWM's attention allows sequence lengths up to 131072 tokens. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. rms_norm_eps (`float`, *optional*, defaults to 1e-05): The epsilon used by the rms normalization layers. use_cache (`bool`, *optional*, defaults to `True`): Whether or not the model should return the last key/values attentions (not used by all models). Only relevant if `config.is_decoder=True`. pad_token_id (`int`, *optional*): Padding token id. eos_token_id (`int` or `list[int]`, *optional*, defaults to `[128001, 128008, 128009]`): The id of the *end-of-sequence* token. Optionally, use a list to set multiple *end-of-sequence* tokens. bos_token_id (`int`, *optional*, defaults to 128000): The id of the *beginning-of-sequence* token. tie_word_embeddings (`bool`, *optional*, defaults to `False`): Whether to tie weight embeddings attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. pretraining_tp (`int`, *optional*, defaults to 1): Tensor parallelism degree used during pretraining. See [this document](https://huggingface.co/docs/transformers/parallelism) and [this issue](https://github.com/pytorch/pytorch/issues/76232). mlp_bias (`bool`, *optional*, defaults to `False`): Whether to use a bias in up_proj, down_proj and gate_proj layers in the MLP layers. rope_parameters (`RopeParameters`, *optional*): Dictionary containing the configuration parameters for the RoPE embeddings. The dictionary should contain a value for `rope_theta` and optionally parameters used for scaling in case you want to use RoPE with longer `max_position_embeddings`. sliding_window (`int`, *optional*, defaults to 8192): Sliding window attention window size. layer_types (`List[str]`, *optional*): List of layer types for each layer. Each element should be either "full_attention" or "sliding_attention". If not specified, will default to alternating pattern based on the provided window pattern. cwm.AiiiT@0siluig{Gz?gh㈵>TN)iii iFg vocab_size hidden_sizeintermediate_sizenum_hidden_layersnum_attention_headsnum_key_value_headshead_dim hidden_actmax_position_embeddingsinitializer_range rms_norm_eps use_cache pad_token_id bos_token_idtie_word_embeddingsattention_dropoutpretraining_tpmlp_biasrope_parameterssliding_window layer_typesc | ddddddd}|%d}t|Dcgc]}||zd k(rd nd }}n t|||r t|nd|_t ||_t |d"id |d |d|d|d|d|d|d|d| d| d| d| d| dt |d|d|ddd|d|d |d!|||`ycc}w)#Nrg0@g@g?rllama3) rope_thetafactorhigh_freq_factorlow_freq_factor original_max_position_embeddings rope_typerfull_attentionsliding_attentionr r!r"r#r$r%r&r'r(r)r*r+r, eos_token_idr-r.attention_biasFr/r2r0r1) rangerintr3listr4super__init__rA)selfr r!r"r#r$r%r&r'r(r)r*r+r,r@r-r.r/r0r1r2r3r4kwargswindow_patterni __class__s _/mnt/e/genesis-system/.venv/lib/python3.12/site-packages/transformers/models/cwm/modular_cwm.pyrGzCwmConfig.__init__is6  ")$'#&48% O  N01'(.&8A&=!DWWK "+/@ A5Cc.1 ,  ! # 0 0  !4  !4   " %< 0 &   & l+ & !4! "!# $0% &,' (*) *- 4  IsB>)__name__ __module__ __qualname____doc__ model_type default_thetarDstrfloatbooldictrErG __classcell__rLs@rMrr%s[>@JM!!&!##%#$ '-#'"#'-"$)#&'+"(,1L L L  L  L ! L !L L L "%L !L L L DjL !L ""#L $!%L &'L ()L *+L ./L 0#Y%1L L rc eZdZy)CwmRotaryEmbeddingNrNrOrPrBrZrMr\r\rZr\c(eZdZdedeffd ZxZS) CwmAttentionconfig layer_idxct|||tjj |j |j |jzd|_tjj |j |j|jzd|_ tjj |j |j|jzd|_ y)NrarbF)bias) rFrGtorchnnLinearr!r$r&q_projr%k_projv_projrHrarbrLs rMrGzCwmAttention.__init__s )<hhoof&8&8&:T:TW[WdWd:dkpoq hhoof&8&8&:T:TW[WdWd:dkpoq hhoof&8&8&:T:TW[WdWd:dkpoq rZrNrOrPrrDrGrXrYs@rMr`r`sryrSrrrZr`c(eZdZdedeffd ZxZS)CwmDecoderLayerrarbctt||||j||_t |||_y)Nrd)rFrGr4attention_typer` self_attnrls rMrGzCwmDecoderLayer.__init__s6 )<$00;%VyIrZrmrYs@rMrorosJyJSJJrZroc eZdZy)CwmPreTrainedModelNr]rBrZrMrtrtr^rZrtc eZdZy)CwmModelOutputWithPastNr]rBrZrMrvrvr^rZrvceZdZeZdeffd Z ddejdzdejdzdejdzde dzdejdzd ejdzd e dzd e e d efd ZxZS)CwmModelrac t||tjj t |j Dcgc]}t||c}|_ycc}w)N) rFrGrfrg ModuleListrCr#rolayersrls rMrGzCwmModel.__init__sI  hh))AFvG_G_A` aI_VY / a  asA!N input_idsattention_mask position_idspast_key_values inputs_embedscache_positionr+rIreturnc |du|duz r td||j|}|r|t|j}|E||j nd} t j |jd|j| z}||jd}t|x} ts:|j|||||d} | j} td i| td i| d} |} |j| |}|j d|jj"D]}|| f| |j$||||d|} |j'| } t)| | S) Nz:You must specify exactly one of input_ids or inputs_embeds)rarr)device)rarr}rrr~)r>r?)r}r~rrposition_embeddings)last_hidden_staterrB) ValueError embed_tokensrraget_seq_lengthrfarangeshaper unsqueeze isinstancerWcopyrr rotary_embr{r#rqnormrv)rHr|r}r~rrrr+rIpast_seen_tokenscausal_mask_mapping mask_kwargssliding_mask_kwargs hidden_statesr decoder_layers rMforwardzCwmModel.forwards  -t"; <YZ Z  *.*;*;I*FM 0*$++>O  !CRC^==?de  ]003Mrs  .8R7&03 H   H %P P f - r>rJ'J -  4 H zH V %  rZ