Path: blob/main/a5/mingpt-demo/mingpt/__pycache__/trainer.cpython-310.pyc
3764 views
o
M��c� � @ s~ d Z ddlZddlZddlmZ ddlZddlZddlmZ ddl m
Z
ddl m Z e�
e�ZG dd� d�ZG dd � d �ZdS )
z�
Simple training loop; Boilerplate that could apply to any arbitrary neural network,
so nothing in this file really has anything to do with GPT specifically.
� N)�tqdm)�LambdaLR)�
DataLoaderc @ s@ e Zd ZdZdZdZdZdZdZdZ dZ
d Z d
Z d Z
d d
� Zd
S )�
TrainerConfig�
�@ ga2U0*�3?)g�������?gffffff�?� �?皙�����?Fg � Z�Ag ��DNBNr c K s" |� � D ]
\}}t| ||� qd S )N)�items�setattr)�self�kwargs�k�v� r �L/Users/yimingwang/Desktop/cs224n/assignment/a5/mingpt-demo/mingpt/trainer.py�__init__# s �zTrainerConfig.__init__)�__name__�
__module__� __qualname__�
max_epochs�
batch_size�
learning_rate�betas�grad_norm_clip� weight_decay�lr_decay�
warmup_tokens� final_tokens� ckpt_path� num_workersr r r r r r s r c @ s$ e Zd Zdd� Zdd� Zdd� ZdS )�Trainerc C sT || _ || _|| _|| _d| _tj�� r(tj�� | _tj �
| j �� | j�| _ d S d S )N�cpu) �model�
train_dataset� test_dataset�config�device�torch�cuda� is_available�current_device�nn� DataParallel�to)r r# r$ r% r& r r r r ) s
�zTrainer.__init__c C sB t | jd�r
| jjn| j}t�d| jj� t�|� � | jj� d S )N�modulez saving %s)
�hasattrr# r/ �logger�infor& r r( �save�
state_dict)r � raw_modelr r r �save_checkpoint5 s zTrainer.save_checkpointc s� �j �j�� t�j d�r�jn�}|�� ��� ����fdd�}td�}d�_t� j�D ]&�|d� �j d ur<|d�}�j d u pD||k }�jj
d urS|rS|}�� � q-d S )Nr/ c s
| dk}�� |� |r�jn�j}t|dd� j� jd�}g }|r*tt|�t|�d�nt|�}|D ]�\}\}}|� �j
�}|� �j
�}t � |�� �||�\} }
|
�
� }
|�|
�� � W d � n1 sdw Y |r���� |
�� t jj���� � j� ��� � jrֈ j|dk�� 7 _�j� jk r�t �j�t t d� j�� } n"t �j� j �t t d� j � j �� } t ddd t � t j!| � �} � j"| }
�j#D ]}|
|d
< q�n� j"}
|�$d �d � d |� d
|
�� d�d|
d��� q0|�st t%�
|��}t&�'d|� |S d S )N�trainT)�shuffle�
pin_memoryr r )�totalr � r g �?r �lrzepoch z iter z
: train loss z.5fz. lr �ez
test loss: %f)(r7 r$ r% r r r r � enumerate�lenr. r' r( �set_grad_enabled�mean�append�item� zero_grad�backwardr, �utils�clip_grad_norm_�
parametersr �stepr �tokens�sumr �float�maxr �math�cos�pir � param_groups�set_description�npr1 r2 )�splitZis_train�data�loaderZlosses�pbar�it�x�y�logits�lossZlr_mult�progressr< � param_group� test_loss�r& �epochr# � optimizerr r r � run_epoch@ sP
� � &
�.� �z Trainer.train.<locals>.run_epoch�infr r7 �test) r# r& r0 r/ �configure_optimizersrL rJ �ranger r% r r6 )r r5 rc Z best_lossr_ Z
good_modelr r` r r7 ; s
6
��z
Trainer.trainN)r r r r r6 r7 r r r r r! ' s r! )�__doc__rN �loggingr �numpyrS r( Z torch.optim�optimZtorch.optim.lr_schedulerr �torch.utils.data.dataloaderr � getLoggerr r1 r r! r r r r �<module> s