i%0ddlmZmZGddeZdgZy))PreTrainedConfiglayer_type_validationc, eZdZdZdZdgZddddddddZdgdgfd d gd gfd gd gfd Zd Zd ddddddddddddgdddddddddfd e d!e d"e d#e d$e d%e d&e d'e d(e d)e d*e d+e d,e dzd-e d.e d/e d0e d1e d2e dzd3e d4ee dzf*fd5 ZxZS)6 CwmConfiga Configuration for Code World Model (CWM). This is an inherited Llama3-compatible configuration with layer-interleaved sliding-window attention. Configures a `CwmModel`. Designed to yield a configuration mirroring the model in the [facebook/cwm](https://huggingface.co/facebook/cwm) architecture by default. Other models include: - [facebook/cwm-sft](https://huggingface.co/facebook/cwm-sft) - [facebook/cwm-pretrain](https://huggingface.co/facebook/cwm-pretrain) Args: vocab_size (`int`, *optional*, defaults to 128256): Vocabulary size of the CWM model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`CwmModel`] hidden_size (`int`, *optional*, defaults to 6144): Dimension of the hidden representations intermediate_size (`int`, *optional*, defaults to 21504): Dimension of the MLP representations num_hidden_layers (`int`, *optional*, defaults to 64): Number of hidden layers in the Transformer decoder num_attention_heads (`int`, *optional*, defaults to 48): Number of attention heads for each attention layer in the Transformer decoder num_key_value_heads (`int`, *optional*, defaults to 8): This is the number of key_value heads that should be used to implement Grouped Query Attention (GQA). If it is not specified, will default to `num_attention_heads`. head_dim (`int`, *optional*, defaults to 128): The attention head dimension. hidden_act (`str` or `function`, *optional*, defaults to `"silu"`): The non-linear activation function (function or string) in the decoder. max_position_embeddings (`int`, *optional*, defaults to 131072): The maximum sequence length that this model might ever be used with. CWM's attention allows sequence lengths up to 131072 tokens. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. rms_norm_eps (`float`, *optional*, defaults to 1e-05): The epsilon used by the rms normalization layers. use_cache (`bool`, *optional*, defaults to `True`): Whether or not the model should return the last key/values attentions (not used by all models). Only relevant if `config.is_decoder=True`. pad_token_id (`int`, *optional*): Padding token id. eos_token_id (`int` or `list[int]`, *optional*, defaults to `[128001, 128008, 128009]`): The id of the *end-of-sequence* token. Optionally, use a list to set multiple *end-of-sequence* tokens. bos_token_id (`int`, *optional*, defaults to 128000): The id of the *beginning-of-sequence* token. tie_word_embeddings (`bool`, *optional*, defaults to `False`): Whether to tie weight embeddings attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. pretraining_tp (`int`, *optional*, defaults to 1): Tensor parallelism degree used during pretraining. See [this document](https://huggingface.co/docs/transformers/parallelism) and [this issue](https://github.com/pytorch/pytorch/issues/76232). mlp_bias (`bool`, *optional*, defaults to `False`): Whether to use a bias in up_proj, down_proj and gate_proj layers in the MLP layers. rope_parameters (`RopeParameters`, *optional*): Dictionary containing the configuration parameters for the RoPE embeddings. The dictionary should contain a value for `rope_theta` and optionally parameters used for scaling in case you want to use RoPE with longer `max_position_embeddings`. sliding_window (`int`, *optional*, defaults to 8192): Sliding window attention window size. layer_types (`List[str]`, *optional*): List of layer types for each layer. Each element should be either "full_attention" or "sliding_attention". If not specified, will default to alternating pattern based on the provided window pattern. cwmpast_key_valuescolwiserowwise)zlayers.*.self_attn.q_projzlayers.*.self_attn.k_projzlayers.*.self_attn.v_projzlayers.*.self_attn.o_projzlayers.*.mlp.gate_projzlayers.*.mlp.up_projzlayers.*.mlp.down_proj input_ids inputs_embeds hidden_statesattention_mask) embed_tokenslayersnorm.AiiiT@0siluig{Gz?gh㈵>TN)iii iFg vocab_size hidden_sizeintermediate_sizenum_hidden_layersnum_attention_headsnum_key_value_headshead_dim hidden_actmax_position_embeddingsinitializer_range rms_norm_eps use_cache pad_token_id bos_token_idtie_word_embeddingsattention_dropoutpretraining_tpmlp_biasrope_parameterssliding_window layer_typesc H| ddddddd}|%d}t|Dcgc]}||zd k(rd nd }}n t|||r t|nd|_t ||_||_| |_||_||_ ||_ ||_ ||}||_ ||_ | |_| |_||_| |_||_||_||n|j|jz|_||_||_| |_||_||_t5|ld i|ycc}w) Nrg0@g@g?rllama3) rope_thetafactorhigh_freq_factorlow_freq_factor original_max_position_embeddings rope_typefull_attentionsliding_attention)rangerintr-listr.rr"rrrrrr!r#r$r*r%r)r+r r,r(r&r' eos_token_idsuper__init__)selfrrrrrrr r!r"r#r$r%r&r?r'r(r)r*r+r,r-r.kwargswindow_patterni __class__s e/mnt/e/genesis-system/.venv/lib/python3.12/site-packages/transformers/models/cwm/configuration_cwm.pyrAzCwmConfig.__init__msl6  ")$'#&48% O  N01'(.&8A&=!DWWK "+/@ A5Cc.1 ,$'>$&!2!2#6   &"5 #6 $!2(,"!2  $,$8d>N>NRVRjRj>j .#6 ((( "6"GsD)__name__ __module__ __qualname____doc__ model_typekeys_to_ignore_at_inferencebase_model_tp_planbase_model_pp_plan default_thetar=strfloatbooldictr>rA __classcell__)rFs@rGrrs>@J#4"5&/%.%.%."+ )"+&(9:#%568IJ!"_$56  M!!&!##%#$ '-#'"#'-"$)#&'+"(,1K#K#K# K#  K# ! K#!K#K#K#"%K#!K#K#K#DjK# !K#""#K#$!%K#&'K#()K#*+K#./K#0#Y%1K#K#rN)configuration_utilsrrr__all__r;rVrGrYs$,K_# _#D -rV