Path: blob/main/a5/mingpt-demo/mingpt/__pycache__/model.cpython-310.pyc
1004 views
o
M��c! � @ s� d Z ddlZddlZddlZddlmZ ddlmZ e�e �Z
G dd� d�Z G dd� de �Z G dd � d ej
�ZG d
d � d ej
�ZG d d
� d
ej
�ZdS )a�
GPT model:
- the initial stem consists of a combination of token encoding and a positional encoding
- the meat of it is a uniform sequence of Transformer blocks
- each Transformer is a sequential combination of a 1-hidden-layer MLP block and a self-attention block
- all blocks feed into a central residual pathway similar to resnets
- the final decoder is a linear projection into a vanilla Softmax classifier
� N)�
functionalc @ s$ e Zd ZdZdZdZdZdd� ZdS )� GPTConfigz4 base GPT config, params common to all GPT versions g�������?c K s. || _ || _|�� D ]
\}}t| ||� q
d S �N)�
vocab_size�
block_size�items�setattr)�selfr r �kwargs�k�v� r
�J/Users/yimingwang/Desktop/cs224n/assignment/a5/mingpt-demo/mingpt/model.py�__init__ s
�zGPTConfig.__init__N)�__name__�
__module__� __qualname__�__doc__�
embd_pdrop� resid_pdrop�
attn_pdropr r
r
r
r r s r c @ s e Zd ZdZdZdZdZdS )�
GPT1Configz( GPT-1 like network roughly 125M params � i N)r r r r �n_layer�n_head�n_embdr
r
r
r r s
r c s* e Zd ZdZ� fdd�Zddd�Z� ZS )�CausalSelfAttentionz�
A vanilla multi-head masked self-attention layer with a projection at the end.
It is possible to use torch.nn.MultiheadAttention here but I am including an
explicit implementation here to show that there is nothing too scary here.
c s� t � �� |j|j dksJ �t�|j|j�| _t�|j|j�| _t�|j|j�| _t� |j
�| _ t� |j �| _
t�|j|j�| _| �dt�t�|j|j���dd|j|j�� |j| _d S )Nr �mask� )�superr r r �nn�Linear�key�query�value�Dropoutr � attn_dropr �
resid_drop�proj�register_buffer�torch�tril�onesr �view�r �config�� __class__r
r r , s
� z CausalSelfAttention.__init__Nc C s$ |� � \}}}| �|��||| j|| j ��dd�}| �|��||| j|| j ��dd�}| �|��||| j|| j ��dd�}||�dd� dt�|� d�� } | � | j
d d �d d �d |�d |�f dkt d��} t j
| dd�} | �| �} | | }
|
�dd��� �|||�}
| �| �|
��}
|
S ) Nr � ����������� �?r z-inf)�dim)�sizer" r- r � transposer# r$ �math�sqrt� masked_fillr �float�F�softmaxr& �
contiguousr' r( ) r �xZ
layer_past�B�T�Cr �qr Zatt�yr
r
r �forward= s &&&$2
zCausalSelfAttention.forwardr �r r r r r rF �
__classcell__r
r
r0 r r % s r c s( e Zd ZdZ� fdd�Zdd� Z� ZS )�Blockz! an unassuming Transformer block c sr t � �� t�|j�| _t�|j�| _t|�| _t� t�
|jd|j �t� � t�
d|j |j�t� |j
��| _d S )N� )r r r � LayerNormr �ln1�ln2r �attn�
Sequentialr! �GELUr% r �mlpr. r0 r
r r T s
�zBlock.__init__c C s, || � | �|�� }|| �| �|�� }|S r )rN rL rQ rM )r r@ r
r
r rF ` s z
Block.forwardrG r
r
r0 r rI Q s rI c sB e Zd ZdZ� fdd�Zdd� Zdd� Zdd � Zd
d d �Z� Z S )�GPTzA the full GPT language model, with a context size of block_size c s� t � �� t�� j� j�| _t�t� d� j
� j��| _ t� � j
�| _tj� fdd�t� j�D �� | _t�� j�| _tj� j� jdd�| _� j
| _
| �| j� t�dtdd� | � � D ��� d S ) Nr c s g | ]}t � ��qS r
)rI )�.0�_�r/ r
r �
<listcomp>p � z GPT.__init__.<locals>.<listcomp>F)�biasznumber of parameters: %ec s s � | ]}|� � V qd S r )�numel)rS �pr
r
r � <genexpr>x s � z GPT.__init__.<locals>.<genexpr>) r r r � Embeddingr r �tok_emb� Parameterr* �zerosr �pos_embr% r �droprO �ranger �blocksrK �ln_fr! �head�apply�
_init_weights�logger�info�sum�
parametersr. r0 rU r r h s
"z GPT.__init__c C s | j S r )r )r r
r
r �get_block_sizez s zGPT.get_block_sizec C s� t |tjtjf�r)|jjjddd� t |tj�r%|jd ur'|jj�� d S d S d S t |tj �r>|jj�� |jj�
d� d S d S )N� g{�G�z�?)�mean�stdr5 ) �
isinstancer r! r\ �weight�data�normal_rX �zero_rK �fill_)r �moduler
r
r rg } s � �zGPT._init_weightsc s� t � }t � }tjjf}tjjtjjf}| �� D ]A\}}|�� D ]8\}} |r+d||f n|}
|�d�r8|� |
� q |�d�rHt
||�rH|� |
� q |�d�rWt
||�rW|� |
� q q|� d� dd� | �� D �� ||@ } ||B } t | �dks~J dt | �f ��t � �
� | �dks�J d t � �
� | �f ��� fd
d �tt|��D �|jd �� fd
d �tt|��D �dd �g}
tjj|
|j|jd�}|S )ay
This long function is unfortunately doing something very simple and is being very defensive:
We are separating out all parameters of the model into two buckets: those that will experience
weight decay for regularization and those that won't (biases, and layernorm/embedding weights).
We are then returning the PyTorch optimizer object.
z%s.%srX rq r` c S s i | ]\}}||�qS r
r
)rS �pnrZ r
r
r �
<dictcomp>� s z,GPT.configure_optimizers.<locals>.<dictcomp>r z4parameters %s made it into both decay/no_decay sets!z@parameters %s were not separated into either decay/no_decay set!c � g | ]}� | �qS r
r
�rS rw ��
param_dictr
r rV � rW z,GPT.configure_optimizers.<locals>.<listcomp>)�params� weight_decayc ry r
r
rz r{ r
r rV � rW rm )�lr�betas)�setr* r r! rK r\ �
named_modules�named_parameters�endswith�addrp �len�str�keys�sorted�listr~ �optim�AdamWZ
learning_rater� )r Z train_configZdecayZno_decayZwhitelist_weight_modulesZblacklist_weight_modules�mn�mrw rZ ZfpnZ inter_paramsZ union_paramsZ optim_groups� optimizerr
r{ r �configure_optimizers� s8
��
� �zGPT.configure_optimizersNc
C s� |� � \}}|| jksJ d��| �|�}| jd d �d |�d d �f }| �|| �}| �|�}| �|�}| �|�}d } |d urNt� |�
d|� d��|�
d��} || fS )Nz.Cannot forward, model block size is exhausted.r4 ) r7 r r] r` ra rc rd re r= �
cross_entropyr- )
r �idx�targets�b�tZtoken_embeddingsZposition_embeddingsr@ �logits�lossr
r
r rF � s
z GPT.forwardr )
r r r r r rl rg r� rF rH r
r
r0 r rR e s .rR )r r9 �loggingr* �torch.nnr r r= � getLoggerr rh r r �Moduler rI rR r
r
r
r �<module> s
,