i d dl mZmZmZ d dlmZ ddlmZ ddlm Z ddl
m Z e j e
Zddd Zd
\ ZZd \ ZZd Z G d
de ZddgZeZy) ) Tokenizerdecoderspre_tokenizers)BPE )_get_prepend_scheme)TokenizersBackend)loggingztokenizer.modelztokenizer.json)
vocab_filetokenizer_file)z[INST]z[/INST])z<>
z
<>
a You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.
If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.c r e Zd ZdZeZdZddgZeZ d de
e z e z dz de
e z dz f fd
Z
xZS )
LlamaTokenizera>
Construct a Llama tokenizer. Based on byte-level Byte-Pair-Encoding.
This uses notably ByteFallback and no normalization.
```python
>>> from transformers import LlamaTokenizer
>>> tokenizer = LlamaTokenizer.from_pretrained("hf-internal-testing/llama-tokenizer")
>>> tokenizer.encode("Hello this is a test")
[1, 15043, 445, 338, 263, 1243]
```
If you want to change the `bos_token` or the `eos_token`, make sure to specify them when initializing the model, or
call `tokenizer.update_post_processor()` to make sure that the post-processing is correctly done (otherwise the
values of the first token and final token of an encoded sequence will not be correct). For more details, checkout
[post-processors] (https://huggingface.co/docs/tokenizers/api/post-processors) documentation.
This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should
refer to this superclass for more information regarding those methods.
Args:
vocab (`str`, `dict` or `list`, *optional*):
Path to the vocabulary file, a dictionary or a list of tokens.
merges (`str` or `list`, *optional*):
Path to the merges file or a list of merges.
clean_up_tokenization_spaces (`bool`, *optional*, defaults to `False`):
Whether or not to cleanup spaces after decoding, cleanup consists in removing potential artifacts like
extra spaces.
unk_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
bos_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`):
The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
eos_token (`str` or `tokenizers.AddedToken`, *optional*, defaults to `""`):
The end of sequence token.
add_bos_token (`bool`, *optional*, defaults to `True`):
Whether or not to add an `bos_token` at the start of sequences.
add_eos_token (`bool`, *optional*, defaults to `False`):
Whether or not to add an `eos_token` at the end of sequences.
use_default_system_prompt (`bool`, *optional*, defaults to `False`):
Whether or not the default system prompt for Llama should be used
add_prefix_space (`bool`, *optional*):
Whether or not the tokenizer should automatically add a prefix space
left input_idsattention_maskNvocabmergesc
| | nd| _ || _ || _ |(t | dt | dt | di| _ |xs g | _ t t
| j | j ddd | _ d | j _ t j dt | j | d | j _ t j dd t j t j g} | j r| t j" d d
gz
} t j$ | | j _ || _ t+ | X d |||||| d |
y )
NTr )r r fuse_unk
byte_fallbackdropoutu ▁F) replacementprepend_schemesplit )contentr ) clean_up_tokenization_spaces unk_token bos_token eos_tokenuse_default_system_promptadd_prefix_space )r$ legacy_vocabstr_mergesr r
_tokenizer
normalizerr Metaspacer
pre_tokenizerr Replace ByteFallbackFuseStripSequencedecoderr# super__init__)
selfr r r r r! r" r# r& r$ kwargssequence __class__s
h/mnt/e/genesis-system/.venv/lib/python3.12/site-packages/transformers/models/llama/tokenization_llama.pyr5 zLlamaTokenizer.__init__\ s[ 5E4P 0VZ =IIIDK | # dkk$,,UYcg h
&*"(6(@(@ .A$BWBWY].^fk)
%
UC ( ! ! # MMO
!<= =H"*"3"3H"= )B&
)E &? -
) NNFzzzFFN)__name__
__module__ __qualname____doc__VOCAB_FILES_NAMESvocab_files_names padding_sidemodel_input_namesr modelr( dictlistr5
__classcell__)r9 s @r: r r ' sw -^ *L$&67 E +/$(%*"' 3
TzD 4'3
d
T!3
3
r; r LlamaTokenizerFastN)
tokenizersr r r tokenizers.modelsr tokenization_utils_baser tokenization_utils_tokenizersr utilsr
get_loggerr< loggerr@ B_INSTE_INSTB_SYSE_SYSDEFAULT_SYSTEM_PROMPTr __all__rH r% r; r: rV sy ; : ! : >
H %#4HXY $, u^ h
& h
V 1
2 $ r;