謜ij f d Z ddlZddlZddlZddlmZ ddlZddlmZ ddl m
Z
m Z m Z ddl
mZ ddlmZ ej" e Ze G d d
Z G d d Ze G d
de Z G d d Z G d de Z G d de Z G d de Z G d de Z G d dee Z y)zJ
Callbacks to use with the Trainer class and customize the training loop.
N) dataclass)tqdm )IntervalStrategy SaveStrategy
has_length)TrainingArguments)loggingc e Zd ZU dZdZedz ed< dZeed< dZ eed< dZ
eed< dZ eed < dZ eed
< dZ
edz ed < dZeed < dZeed
< dZeed< dZeeeef ed< dZedz ed< dZedz ed< dZedz ed< dZeed< dZeed< dZeed< dZ edz ed< dZ eeeez ez ez f dz ed< dZ ed dz ed< d Z d efd Z e!d efd Z"d Z#d! Z$y)" TrainerStatea
A class containing the [`Trainer`] inner state that will be saved along the model and optimizer when checkpointing
and passed to the [`TrainerCallback`].
In all this class, one step is to be understood as one update step. When using gradient accumulation, one update
step may require several forward and backward passes: if you use `gradient_accumulation_steps=n`, then one update
step requires going through *n* batches.
Args:
epoch (`float`, *optional*):
Only set during training, will represent the epoch the training is at (the decimal part being the
percentage of the current epoch completed).
global_step (`int`, *optional*, defaults to 0):
During training, represents the number of update steps completed.
max_steps (`int`, *optional*, defaults to 0):
The number of update steps to do during the current training.
logging_steps (`int`, *optional*, defaults to 500):
Log every X updates steps
eval_steps (`int`, *optional*):
Run an evaluation every X steps.
save_steps (`int`, *optional*, defaults to 500):
Save checkpoint every X updates steps.
train_batch_size (`int`, *optional*):
The batch size for the training dataloader. Only needed when
`auto_find_batch_size` has been used.
num_input_tokens_seen (`int`, *optional*, defaults to 0):
When tracking the inputs tokens, the number of tokens seen during training (number of input tokens, not the
number of prediction tokens).
total_flos (`float`, *optional*, defaults to 0):
The total number of floating operations done by the model since the beginning of training (stored as floats
to avoid overflow).
log_history (`list[dict[str, float]]`, *optional*):
The list of logs done since the beginning of training.
best_metric (`float`, *optional*):
When tracking the best model, the value of the best metric encountered so far.
best_global_step (`int`, *optional*):
When tracking the best model, the step at which the best metric was encountered.
Used for setting `best_model_checkpoint`.
best_model_checkpoint (`str`, *optional*):
When tracking the best model, the value of the name of the checkpoint for the best model encountered so
far.
is_local_process_zero (`bool`, *optional*, defaults to `True`):
Whether or not this process is the local (e.g., on one machine if training in a distributed fashion on
several machines) main process.
is_world_process_zero (`bool`, *optional*, defaults to `True`):
Whether or not this process is the global main process (when training in a distributed fashion on several
machines, this is only going to be `True` for one process).
is_hyper_param_search (`bool`, *optional*, defaults to `False`):
Whether we are in the process of a hyper parameter search using Trainer.hyperparameter_search. This will
impact the way data will be logged in TensorBoard.
stateful_callbacks (`list[StatefulTrainerCallback]`, *optional*):
Callbacks attached to the `Trainer` that should have their states be saved or restored.
Relevant callbacks should implement a `state` and `from_state` function.
Nepochr global_step max_stepsi
logging_steps
eval_steps
save_stepstrain_batch_sizenum_train_epochsnum_input_tokens_seen
total_flos log_history best_metricbest_global_stepbest_model_checkpointTis_local_process_zerois_world_process_zeroFis_hyper_param_search
trial_name trial_paramsTrainerCallbackstateful_callbacksc | j g | _ | j i | _ y t | j t ry i }| j D ] }t |t st dt
| |j j }||v r?t || t s || g||< || j |j |j ||< || _ y )NzNAll callbacks passed to be saved must inherit `ExportableState`, but received ) r r!
isinstancedictExportableState TypeErrortype __class____name__listappendstate)selfr! callbacknames Y/mnt/e/genesis-system/.venv/lib/python3.12/site-packages/transformers/trainer_callback.py
__post_init__zTrainerState.__post_init__t s # !D " " *&(D #
//
6 "$ 3 3
@!(_>#himnviwhxy ))22-- &&8&>E4Ft4L3M*40&t,33HNN4DE/7~~/?&t,
@ '9D # json_pathc t j t j | dd dz }t |dd 5 }|j | ddd y# 1 sw Y yxY w) zDSave the content of this instance in JSON format inside `json_path`. T)indent sort_keys
wutf-8encodingN)jsondumps dataclassesasdictopenwrite)r- r3 json_stringfs r0 save_to_jsonzTrainerState.save_to_json sT jj!3!3D!9!tTW[[
)S7
3 !q
GGK ! ! !s AA!c t |d 5 }|j }ddd | di t j S # 1 sw Y $xY w)z3Create an instance from the content of `json_path`.r: r; N )rA readr= loads)clsr3 rD texts r0 load_from_jsonzTrainerState.load_from_json sG )g
. !668D &TZZ%&& s AA c dD ]A }t || d }||dk rt j ||z }t | | d| C y)z
Calculates and stores the absolute value for logging,
eval, and save steps based on if it was a proportion
or not.
)r
evalsave_stepsNr )getattrmathceilsetattr)r- argsr step_kind num_stepss r0
compute_stepszTrainerState.compute_steps sY 5 ?I 6&:;I$ q= $ )i*? @I 6 2I> ?r2 c |j ,|j |j |j | _ d| _ |ddlm} || | _ || _ || _ |j | _ |j | _ y)zI
Stores the initial training references needed in `self`
Nr ) hp_params)
hp_name_trialr r transformers.integrationsrZ r r r r )r- trainerr r trialrZ s r0 init_training_referencesz%TrainerState.init_training_references s| ?? &7>>+E &oognn =DO ; )% 0D " 0 %,%B%B%D"%,%B%B%D"r2 )%r)
__module__ __qualname____doc__r
float__annotations__r intr r r r r r r r r r* r$ strr r r r boolr r r r r! r1 rE classmethodrL rX r` rG r2 r0 r r " s` 9v E54< KIs M3 JJ#'cDj' c !"3"J*.Kd3:&'. $K $#'cDj'(,3:,"&4&"&4&"'4' !Jd
!?CL$sC%K# - 445<C9= ./$6=96!c ! 's ' ' ?Er2 r c , e Zd ZdZdefdZed Zy)r% aj
A class for objects that include the ability to have its state
be saved during `Trainer._save_checkpoint` and loaded back in during
`Trainer._load_from_checkpoint`.
These must implement a `state` function that gets called during the respective
Trainer function call. It should only include parameters and attributes needed to
recreate the state at a particular time, to avoid utilizing pickle/maintain standard
file IO writing.
Example:
```python
class EarlyStoppingCallback(TrainerCallback, ExportableState):
def __init__(self, early_stopping_patience: int = 1, early_stopping_threshold: Optional[float] = 0.0):
self.early_stopping_patience = early_stopping_patience
self.early_stopping_threshold = early_stopping_threshold
# early_stopping_patience_counter denotes the number of times validation metrics failed to improve.
self.early_stopping_patience_counter = 0
def state(self) -> dict:
return {
"args": {
"early_stopping_patience": self.early_stopping_patience,
"early_stopping_threshold": self.early_stopping_threshold,
},
"attributes": {
"early_stopping_patience_counter": self.early_stopping_patience_counter,
}
}
```returnc t d )NzInternal class that just calls the list of callbacks in order.c g | _ |D ] }| j | || _ || _ || _ || _ d | _ d | _ t d | j D s#t j d| j z y y )Nc 3 < K | ] }t |t y wN)r# DefaultFlowCallback.0cbs r0 z+CallbackHandler.__init__.. s P2:b"56Ps zThe Trainer will not work properly if you don't have a `DefaultFlowCallback` in its callbacks. You
should add one before training with `trainer.add_callback(DefaultFlowCallback). The current list ofcallbacks is
:) callbacks add_callbackmodelprocessing_class optimizer lr_schedulertrain_dataloadereval_dataloaderanyloggerwarning
callback_list)r- r r r r r r s r0 __init__zCallbackHandler.__init__ s "B b ! "
0 " ( $ # P PP NN$ $$%
Qr2 c P t |t r | n|}t |t r|n |j }|| j D cg c] }|j c}v r)t j d| ddz | j z | j j | y c c}w )NzYou are adding a zH to the callbacks of this Trainer, but there is already one. The currentzlist of callbacks is
:)r# r' r( r r r r r+ )r- r. r cb_classcs r0 r z CallbackHandler.add_callback s %h5XZ8 )(D 98x?Q?Q T^^< < < NN#H:-uv+,$$%
b!
=s B#c t |t r=| j D ]- }t || s| j j | |c S y | j D ]& }||k( s | j j | |c S y r r# r' r remover- r. r s r0 pop_callbackz CallbackHandler.pop_callback ss h %nn
b(+NN))"- I
nn
>NN))"- I
r2 c t |t r;| j D ]+ }t || s| j j | y y | j j | y r r r s r0 remove_callbackz CallbackHandler.remove_callback sR h %nn
b(+NN))"-
NN ! !( +r2 c F dj d | j D S )Nr8 c 3 H K | ] }|j j y wr )r( r) r s r0 r z0CallbackHandler.callback_list.. s H2 ..Hs ")joinr rn s r0 r z CallbackHandler.callback_list s yyH HHHr2 rU r, r c * | j d||| S )Nr
call_eventr- rU r, r s r0 r zCallbackHandler.on_init_end }dE7CCr2 c 8 d|_ | j d||| S )NFr )rx r r s r0 r z CallbackHandler.on_train_begin s ',$ /ugFFr2 c * | j d||| S )Nr r r s r0 r z CallbackHandler.on_train_end ~tUGDDr2 c 8 d|_ | j d||| S )NFr )ry r r s r0 r z CallbackHandler.on_epoch_begin s $)! /ugFFr2 c * | j d||| S )Nr r r s r0 r z CallbackHandler.on_epoch_end r r2 c T d|_ d|_ d|_ | j d||| S )NFr )r| r{ rz r r s r0 r z CallbackHandler.on_step_begin s/ ""' #eWEEr2 c * | j d||| S )Nr r r s r0 r z%CallbackHandler.on_pre_optimizer_step s 6eWMMr2 c * | j d||| S )Nr r r s r0 r z!CallbackHandler.on_optimizer_step s 2D%IIr2 c * | j d||| S )Nr r r s r0 r z CallbackHandler.on_substep_end s /ugFFr2 c * | j d||| S )Nr r r s r0 r zCallbackHandler.on_step_end r r2 c < d|_ | j d|||| S )NFr r )r{ r r- rU r, r r s r0 r zCallbackHandler.on_evaluate
s# "' }dE7GTTr2 c . | j d|||| S )Nr r r r s r0 r zCallbackHandler.on_predict s |T5'7SSr2 c 8 d|_ | j d||| S )NFr )rz r r s r0 r zCallbackHandler.on_save s #y$w??r2 c < d|_ | j d|||| S )NFr )logs)r| r )r- rU r, r r s r0 r zCallbackHandler.on_log s# "xugDIIr2 c * | j d||| S )Nr r r s r0 r z"CallbackHandler.on_prediction_step s 3T5'JJr2 c . | j d|||fi |S )Nr r r s r0 r z CallbackHandler.on_push_begin s teWOOOr2 c | j D ]_ } t || |||f| j | j | j | j
| j | j d|}|^|}a |S )N)r r r r r r )r rQ r r r r r r )r- eventrU r, r r r. results r0 r zCallbackHandler.call_event s !H-WXu- jj!%!6!6 .. ! . .!%!6!6 $ 4 4 F ! ! r2 N) r) ra rb rc r r r r propertyr r r rw r r r r r r r r r r r r r r r r r rG r2 r0 r r s2 H& "
, I ID 1 D, DQ_ DG#4 G\ GTb GE!2 E< ER` EG#4 G\ GTb GE!2 E< ER` EF"3 FL FSa F N*; NL N[i NJ&7 J JWe JG#4 G\ GTb GD 1 D, DQ_ DU 1 U, UQ_ UT 0 T TP^ T@ - @l @^ @J, J\ JN JK'8 K KXf KP"3 PL PSa Pr2 r c 8 e Zd ZdZdededefdZdededefdZy)r zx
A [`TrainerCallback`] that handles the default flow of the training loop for logs, evaluation and checkpoints.
rU r, r c |j dk( r|j rd|_ |j t j
k( r#|j |j z dk( rd|_ |j t j
k( r<|j |j z dk( r |j |j k rd|_
|j t j
k( r2|j dkD r#|j |j z dk( rd|_ |j |j k\ r+d|_ |j t j
k( rd|_ |S )Nr Tr )r logging_first_stepr| logging_strategyr STEPSr
eval_strategyr
eval_delayr{
save_strategyr r rz r rx r s r0 r z DefaultFlowCallback.on_step_end8 s# !d&=&=!%G $4$:$: :u?P?PSXSfSf?fjk?k!%G
"2"8"8 8!!E$4$4495#4#44&*G #
,"4"4 4 1$!!E$4$449"&G /+/G (!!\%7%77&*#r2 c |j t j k( rd|_ |j t j k( r |j
|j k rd|_ |j t j k( rd|_
|S )NT) r r EPOCHr| r r r
r{ r r rz r s r0 r z DefaultFlowCallback.on_epoch_endX sp $4$:$: :!%G !1!7!7 7DOOu{{ z/. Consider increasing `max_str_len` if needed.].4gr )
r r rq r# rg r r rd poprB )
r- rU r, r r r shallow_logsrs rt _s
r0 r zProgressCallback.on_log s & &4+<+<+H L
(1 a%#a&43C3C*C?Axs4K[K[J\ ]H H !O a')*3LO&'LO
( t4A # #C $5 6! ,I &r2 c ` |j r"| j j d | _ y y r )r r r r s r0 r z ProgressCallback.on_train_end s* & & # # % $D 'r2 )d r )
r) ra rb rc rf r r r r r r r r rG r2 r0 r r h s6
'C '
2
*' ' 7&%r2 r c e Zd ZdZddZy)PrinterCallbackz?
A bare [`TrainerCallback`] that just prints the logs.
Nc |j dd }|j rE|7|j D ci c] \ }}|t |t r|dn| }}}t | y y c c}}w )Nr r )r r rq r# rd print) r- rU r, r r r r rs rt s r0 r zPrinterCallback.on_log sd HH\4 ( & & SWS]S]S_`41a*Q*>qgAE`` $K '`s !A%r )r) ra rb rc r rG r2 r0 r r s r2 r c F e Zd ZdZd dededz fdZd Zd Zd Z d e
fd
Z y) EarlyStoppingCallbacka1
A [`TrainerCallback`] that handles early stopping.
Args:
early_stopping_patience (`int`):
Use with `metric_for_best_model` to stop training when the specified metric worsens for
`early_stopping_patience` evaluation calls.
early_stopping_threshold(`float`, *optional*):
Use with TrainingArguments `metric_for_best_model` and `early_stopping_patience` to denote how much the
specified metric must improve to satisfy early stopping conditions. `
This callback depends on [`TrainingArguments`] argument *load_best_model_at_end* functionality to set best_metric
in [`TrainerState`]. Note that if the [`TrainingArguments`] argument *save_steps* differs from *eval_steps*, the
early stopping will not occur until the next save step.
early_stopping_patienceearly_stopping_thresholdNc . || _ || _ d| _ y )Nr r r early_stopping_patience_counter)r- r r s r0 r z EarlyStoppingCallback.__init__ s '>$(@%/0,r2 c |j rt j nt j }|j 8 |||j r-t ||j z
| j kD rd| _ y | xj dz
c_ y )Nr r )greater_is_betternpgreaterlessr absr r )r- rU r, r metric_valueoperators r0 check_metric_valuez(EarlyStoppingCallback.check_metric_value sl !%!7!72::RWW $ \5#4#4 5L5#4#4458U8UU34D 0 0 0A 5 0r2 c |j st j d |j J d |j t
j k7 sJ d y )NzUsing EarlyStoppingCallback without load_best_model_at_end=True. Once training is finished, the best model will not be loaded automatically.zBEarlyStoppingCallback requires metric_for_best_model to be definedzAEarlyStoppingCallback requires IntervalStrategy of steps or epoch)load_best_model_at_endr r metric_for_best_modelr r NOr s r0 r z$EarlyStoppingCallback.on_train_begin sb ** NN^
))5
P
5 !!%5%8%88
O
8r2 c |j }|j d sd| }|j | }|t j d| d y | j |||| | j | j k\ rd|_ y y )Neval_z@early stopping required metric_for_best_model, but did not find z so early stopping is disabledT) r'
startswithgetr r r$ r r rx )r- rU r, r r r metric_to_checkr" s r0 r z!EarlyStoppingCallback.on_evaluate s 44 ))'2 %o%6 7O {{?3 NNRSbRc d
eWlC / /43O3O O+/G ( Pr2 rk c R | j | j dd| j idS )N)r r r r r rn s r0 r, zEarlyStoppingCallback.state s7 ,0+G+G,0,I,I
243W3W
r2 )r g ) r) ra rb rc rf rd r r$ r r r$ r, rG r2 r0 r r s< 1 1SX[_S_ 1 6
0"
t
r2 r ) rc r? r= rR r numpyr tqdm.autor
trainer_utilsr r r
training_argsr utilsr
get_loggerr) r r r% rw r r r r r r rG r2 r0 r5 s ! E E ,
H % WE WE WEt) )X :
_ :
:
zC C LCo CL2/ 2jG% G%T
o
I
O_ I
r2