Book a Demo!
CoCalc Logo Icon
StoreFeaturesDocsShareSupportNewsAboutPoliciesSign UpSign In
yiming-wange
GitHub Repository: yiming-wange/cs224n-2023-solution
 Path: blob/main/a2/__pycache__/word2vec.cpython-39.pyc
1003 views
a

�d0�@s:ddlZddlZddlZddlmZmZmZddlm	Z	m
Z
dd�Zdd�Zdd	�Z
d%dd�Zefd
d�Zefdd�Zdd�Zdd�Zdd�Zdd�Zdd�Zdd�Zedk�r6ejdd�Zejd d!ed"d#d$�e��Zejdkr�e�nLejdkr�e�n:ejdk�re�n&ejdk�r$e�nejd"k�r6e�dS)&�N)�gradcheck_naive�grad_tests_softmax�grad_tests_negsamp)�
normalizeRows�softmaxcCsddt�|�}|S)z�
    Compute the sigmoid function for the input here.
    Arguments:
    x -- A scalar or numpy array.
    Return:
    s -- sigmoid(x)
    �)�np�exp)�x�s�r�:/Users/yimingwang/Desktop/cs224n/assignment/a2/word2vec.py�sigmoids
rc	CsTt||�}t�|�}d||<t�||�}|||}t�|||�}|||fS)a[ Naive Softmax loss & gradient function for word2vec models

    Implement the naive softmax loss and gradients between a center word's 
    embedding and an outside word's embedding. This will be the building block
    for our word2vec models. For those unfamiliar with numpy notation, note 
    that a numpy ndarray with a shape of (x, ) is a one-dimensional array, which
    you can effectively treat as a vector with length x.

    Arguments:
    centerWordVec -- numpy ndarray, center word's embedding
                    in shape (word vector length, )
                    (v_c in the pdf handout)
    outsideWordIdx -- integer, the index of the outside word
                    (o of u_o in the pdf handout)
    outsideVectors -- outside vectors is
                    in shape (num words in vocab, word vector length) 
                    for all words in vocab (tranpose of U in the pdf handout)
    dataset -- needed for negative sampling, unused here.

    Return:
    loss -- naive softmax loss
    gradCenterVec -- the gradient with respect to the center word vector
                     in shape (word vector length, )
                     (dJ / dv_c in the pdf handout)
    gradOutsideVecs -- the gradient with respect to all the outside word vectors
                    in shape (num words in vocab, word vector length) 
                    (dJ / dU)
    r)rr�
zeros_like�log�outer)	�
centerWordVec�outsideWordIdx�outsideVectors�datasetZy_hat�y�loss�
gradCenterVec�gradOutsideVecsrrr
�naiveSoftmaxLossAndGradients(
rcCs>dg|}t|�D]&}|��}||kr0|��}q|||<q|S)z4 Samples K indexes which are not the outsideWordIdx N)�range�sampleTokenIdx)rr�K�negSampleWordIndices�kZnewidxrrr
�getNegativeSamplesOs


r �
cCs�t|||�}|g|}||dd�f}|dd�d9<t||�}t�t�|��}	|d}
|
|}t�|
|�}|dd�d9<t�|�}
tj�|
||�|	||
fS)aI Negative sampling loss function for word2vec models

    Implement the negative sampling loss and gradients for a centerWordVec
    and a outsideWordIdx word vector as a building block for word2vec
    models. K is the number of negative samples to take.

    Note: The same word may be negatively sampled multiple times. For
    example if an outside word is sampled twice, you shall have to
    double count the gradient with respect to this word. Thrice if
    it was sampled three times, and so forth.

    Arguments/Return Specifications: same as naiveSoftmaxLossAndGradient
    Nr�����)	r rr�sumrrr�add�at)rrrrrr�indices�negative�scorer�doutrZgrad_samplerrrr
�negSamplingLossAndGradient[s

r*cs�d}t�|j�}	t�|j�}
�|}||dd�f}�fdd�|D�}
|
D]8}|||||�\}}}||7}|	||7<|
|7}
qJ||	|
fS)a Skip-gram model in word2vec

    Implement the skip-gram model in this function.

    Arguments:
    currentCenterWord -- a string of the current center word
    windowSize -- integer, context window size
    outsideWords -- list of no more than 2*windowSize strings, the outside words
    word2Ind -- a dictionary that maps words to their indices in
              the word vector list
    centerWordVectors -- center word vectors (as rows) is in shape 
                        (num words in vocab, word vector length) 
                        for all words in vocab (V in pdf handout)
    outsideVectors -- outside vectors is in shape 
                        (num words in vocab, word vector length) 
                        for all words in vocab (transpose of U in the pdf handout)
    word2vecLossAndGradient -- the loss and gradient function for
                               a prediction vector given the outsideWordIdx
                               word vectors, could be one of the two
                               loss functions you implemented above.

    Return:
    loss -- the loss function value for the skip-gram model
            (J in the pdf handout)
    gradCenterVecs -- the gradient with respect to the center word vector
                     in shape (num words in vocab, word vector length)
                     (dJ / dv_c in the pdf handout)
    gradOutsideVecs -- the gradient with respect to all the outside word vectors
                    in shape (num words in vocab, word vector length) 
                    (dJ / dU)
    �Ncsg|]}�|�qSrr)�.0�word��word2Indrr
�
<listcomp>��zskipgram.<locals>.<listcomp>)r�zeros�shape)ZcurrentCenterWord�
windowSizeZoutsideWordsr/�centerWordVectorsrr�word2vecLossAndGradientrZgradCenterVecsZgradOutsideVectorsZ
center_idxrr�idxZloss_Z
gradCenterZgradOutsiderr.r
�skipgram�s#
r8c
Cs�d}d}t�|j�}|jd}	|dt|	d��dd�f}
|t|	d�d�dd�f}t|�D]�}t�d|�}
|�|
�\}}|||
|||
|||�\}}}|||7}|dt|	d��dd�f||7<|t|	d�d�dd�f||7<q^||fS)N�2r+r�r)rr2r3�intr�random�randint�getRandomContext)Z
word2vecModelr/�wordVectorsrr4r6Z	batchsizerZgrad�Nr5r�iZwindowSize1Z
centerWord�context�cZginZgoutrrr
�word2vec_sgd_wrapper�s"

�
(*rDcCsptd�td�dksJ�t�tt�dg��t�dg��s<J�t�tt�gd���t�gd���sdJ�td�dS)z Test sigmoid function z === Sanity check for sigmoid ===rg�?)rr:�)g(���d�?g�d�[}/�?gm��|{�?zTests for sigmoid passed!N)�printrr�allclose�arrayrrrr
�test_sigmoid�s$�
rIcCshdd�}dd�}tddi��}||_||_t�d�tj�d�ttj�d	d
��}tgd��}|||fS)zT Helper method for naiveSoftmaxLossAndGradient and negSamplingLossAndGradient tests cSst�dd�S)Nr��r<r=rrrr
�dummySampleTokenIdx�sz,getDummyObjects.<locals>.dummySampleTokenIdxcs2gd���t�dd��fdd�td|�D�fS)N)�a�brC�d�errJcsg|]}�t�dd��qS)rrJrK)r,rA��tokensrr
r0�r1z=getDummyObjects.<locals>.getRandomContext.<locals>.<listcomp>r:)r<r=r)�CrrQr
r>�s�z)getDummyObjects.<locals>.getRandomContext�dummyri�zi1$r!rE))rMr)rNr)rCr:)rOrE)rPrJ)	�typerr>r<�seedrr�randn�dict)rLr>r�
dummy_vectors�dummy_tokensrrr
�getDummyObjects�s
r[cs`t�\��}td���fdd�}t|tj�d�d�tj�d����fdd�}t|�d�dS)	z" Test naiveSoftmaxLossAndGradient z8==== Gradient check for naiveSoftmaxLossAndGradient ====cst|d���\}}}||fS�Nr�r��vecrrr�rrYrr
�temps�
z.test_naiveSoftmaxLossAndGradient.<locals>.temprEz)naiveSoftmaxLossAndGradient gradCenterVeccst�d|��\}}}||fSr\r]r^��	centerVecrrr
ra	s�
z+naiveSoftmaxLossAndGradient gradOutsideVecsN�r[rFrrr<rW�rZrar�rcrrYr
� test_naiveSoftmaxLossAndGradient�s
���rgcs`t�\��}td���fdd�}t|tj�d�d�tj�d����fdd�}t|�d�dS)	z! Test negSamplingLossAndGradient z7==== Gradient check for negSamplingLossAndGradient ====cst|d���\}}}||fSr\�r*r^r`rr
ras�
z-test_negSamplingLossAndGradient.<locals>.temprEz(negSamplingLossAndGradient gradCenterVeccst�d|��\}}}||fSr\rhr^rbrr
ra s�
z*negSamplingLossAndGradient gradOutsideVecsNrdrerrfr
�test_negSamplingLossAndGradients
���ricsjt�\�}�td�t��fdd�|d�tt�|��td�t��fdd�|d�tt�|�t�dS)	z1 Test skip-gram with naiveSoftmaxLossAndGradient zG==== Gradient check for skip-gram with naiveSoftmaxLossAndGradient ====cstt�|�dt�S�N�)rDr8r�r_�rrZrr
�<lambda>-s�ztest_skipgram.<locals>.<lambda>z$naiveSoftmaxLossAndGradient GradientzF==== Gradient check for skip-gram with negSamplingLossAndGradient ====cstt�|�dt�Srj)rDr8r*rlrmrr
rn3s�z#negSamplingLossAndGradient GradientN)r[rFrrr8rr*)rYrrmr
�
test_skipgram(s���rocCst�t�t�t�dS)zV Test the two word2vec implementations, before running on Stanford Sentiment Treebank N)rIrgrirorrrr
�
test_word2vec:srp�__main__zTest your implementations.)�description�function�?�allz,Name of the function you would like to test.)�nargsrU�default�help)r!)�argparse�numpyrr<Zutils.gradcheckrrrZutils.utilsrrrrr r*r8rDrIr[rgrirorp�__name__�ArgumentParser�parser�add_argument�str�
parse_args�argsrsrrrr
�<module>sF4�
/�
;�


�