Path: blob/main/models/__pycache__/layers.cpython-38.pyc
153 views
U
1$za�" � @ sx d dl Z d dlZd dlmZ d dlm mZ G dd� dej�ZG dd� dej�Z G dd� de �Z
G dd � d ej�Z dS )
� Nc s* e Zd ZdZd� fdd� Zdd� Z� ZS )�SqueezeEmbeddingzK
Squeeze sequence embedding length to the longest one in the batch
Tc s t t| ��� || _d S �N)�superr �__init__� batch_first)�selfr �� __class__� � D:\CCF-BDCI\AQY\models\layers.pyr s zSqueezeEmbedding.__init__c C s| t �| �d �� }t �|�d �� }|| }|| }t jjjj||�� | jd�}t jjjj || jd�}|d }|| }|S )z�
sequence -> sort -> pad and pack -> unpack ->unsort
:param x: sequence embedding vectors
:param x_len: numpy/tensor list
:return:
� �r r )
�torch�sort�long�nn�utils�rnn�pack_padded_sequence�cpur �pad_packed_sequence)r �x�x_len�
x_sort_idx� x_unsort_idx�x_emb_p�outr
r
r �forward s zSqueezeEmbedding.forward)T)�__name__�
__module__� __qualname__�__doc__r r �
__classcell__r
r
r r r s r c s. e Zd Zd � fdd� Zdd� Zd d
� Z� ZS ) � AttentionNr � dot_productr c s� t t| ��� |dkr || }|dkr*|}|| _|| _|| _|| _t�||| �| _ t�||| �| _
t�|| |�| _ t� |�| _
|dkr�t�t�|d ��| _n,| jdkr�t�t�||��| _n | �dd� | �� dS )a$ Attention Mechanism
:param embed_dim:
:param hidden_dim:
:param out_dim:
:param n_head: num of head (Multi-Head Attention)
:param score_function: scaled_dot_product / mlp (concat) / bi_linear (general dot)
:return (?, q_len, out_dim,)
N�mlp� � bi_linear�weight)r r# r � embed_dim�
hidden_dim�n_head�score_functionr �Linear�w_k�w_q�proj�Dropout�dropout� Parameterr �Tensorr( �register_parameter�reset_parameters)r r) r* �out_dimr+ r, r2 r r
r r ' s&
zAttention.__init__c C s0 dt �| j� }| jd k r,| jj�| |� d S �Ng �?)�math�sqrtr* r( �data�uniform_�r �stdvr
r
r r6 E s
zAttention.reset_parametersc C s" t |j�dkr tj|dd�}t |j�dkr8tj|dd�}|jd }|jd }|jd }| �|��||| j| j�}|�dddd�� � �d|| j�}| �
|��||| j| j�}|�dddd�� � �d|| j�}| j dkr�|�ddd�}t� ||�} n�| j dk�r,|�ddd�}t� ||�}
t�
|
t�| j��} n�| j d k�r�tj|dd��d|dd�} tj|dd��dd|d�} tj| | fdd�}
t�t�|
| j��} n>| j d
k�r�t�|| j�}|�ddd�}t� ||�} ntd ��tj| dd�} t� | |�}tjtj||dd�dd�}| �|�}| �|�}|| fS ) Nr& r )�dimr � �����r$ Zscaled_dot_productr% r' zinvalid score_function)�len�shaper � unsqueezer. �viewr+ r* �permute�
contiguousr/ r, �bmm�divr9 r: �expand�cat�F�tanh�matmulr( � RuntimeError�softmax�splitr0 r2 )r �k�q�mb_sizeZk_len�q_lenZkxZqx�kt�scoreZqktZkxxZqxxZkqZqw�outputr
r
r r J sD
zAttention.forward)NNr r$ r )r r r r r6 r r" r
r
r r r# & s r# c s6 e Zd ZdZd � fdd� Zdd � Z� fd
d �Z� ZS )
�NoQueryAttentionzq is a parameterNr r$ r c s@ t t| ��||||||� || _t�t�||��| _| � � d S r )
r rY r rU r r3 r r4 rS �reset_q)r r) r* r7 r+ r, rU r2 r r
r r z s zNoQueryAttention.__init__c C s& dt �| j� }| jj�| |� d S r8 )r9 r: r) rS r; r< r= r
r
r rZ � s zNoQueryAttention.reset_qc s, |j d }| j�|dd�}tt| ��||�S )Nr rA )rC rS rJ r rY r )r rR �kwargsrT rS r r
r r � s
zNoQueryAttention.forward)NNr r$ r r )r r r r! r rZ r r" r
r
r r rY w s �rY c s& e Zd Zd
� fdd� Zdd � Z� ZS ) � DynamicLSTMr Tr F�LSTMc
s� t t| ��� || _|| _|| _|| _|| _|| _|| _ || _
| | _ | j dkrjt j
|||||||d�| _nJ| j dkr�t j|||||||d�| _n$| j dkr�t j|||||||d�| _dS )u�
使LSTM支持一个batch中含有长度不同的句子，基于pack_padded_sequence方法实现
参考：https://zhuanlan.zhihu.com/p/34418001
r] )�
input_size� hidden_size�
num_layers�biasr r2 �
bidirectional�GRU�RNNN)r r\ r r^ r_ r` ra r r2 rb �only_use_last_hidden_state�rnn_typer r] rd rc )
r r^ r_ r` ra r r2 rb re rf r r
r r � sP
�
�
�zDynamicLSTM.__init__c
C s t �| �d �� }t �|�d �� }|| �d�}|| }t jjjj||| jd�}| j dkrt| �
|d�\}\}}n| �
|d�\}}d}t � |dd�| }t � |dd�}| j r�|S t jjjj
|| jd�} | d } | | } | j dk�rt � |dd�| }t � |dd�}| ||ffS dS )z�
sequence -> sort -> pad and pack ->process using RNN -> unpack ->unsort
:param x: sequence embedding vectors
:param x_len: numpy/tensor list
:return:
r r r
r] Nr )r r r �tor r r r r rf rd � transposere r )
r r r r r r Zout_pack�ht�ctr r
r
r r � s0
� �zDynamicLSTM.forward)r TTr FFr] )r r r r r r" r
r
r r r\ � s � r\ ) r9 r �torch.nnr Ztorch.nn.functional�
functionalrL �Moduler r# rY r\ r
r
r
r �<module> s Q