Path: blob/master/clustering/tfidf/__pycache__/feature_extraction.cpython-35.pyc
2596 views
uz�Y$? � @ s� d d l Z d d l Z d d l m Z d d l m Z m Z d d l m Z d d l
m Z m Z d d l
m Z Gd d � d e � Z Gd d
� d
e e � Z Gd d � d e � Z d S)
� N)� defaultdict)�spdiags�
csr_matrix)� normalize)�
BaseEstimator�TransformerMixin)�ENGLISH_STOP_WORDSc @ s� e Z d Z d Z d d d d d d d � Z d d d
� Z d d d � Z d
d � Z d d � Z d d � Z d d � Z
d d � Z d d � Z d S)�CountVectorizera�
Convert a collection of text documents to a matrix of token counts,
this implementation produces a sparse representation of the counts using
scipy.sparse.csr_matrix.
The number of features will be equal to the vocabulary size found by
analyzing all input documents and removal of stop words
Parameters
----------
analyzer : str {'word'} or callable
Whether the feature should be made of word, if n-grams is specified,
then the words are concatenated with space.
If a callable is passed it is used to extract the sequence of features
out of the raw, unprocessed input.
ngram_range : tuple (min_n, max_n)
The lower and upper boundary of the range of n-values for different
n-grams to be extracted. All values of n such that min_n <= n <= max_n
will be used.
token_pattern : str
Regular expression denoting what constitutes a "token", only used
if ``analyzer == 'word'``. The default regexp select tokens of 2
or more alphanumeric characters (punctuation is completely ignored
and always treated as a token separator).
stop_words : string {'english'}, collection, or None, default None
- If 'english', a built-in stop word list for English is used.
- If a collection, that list or setis assumed to contain stop words, all of which
will be removed from the resulting tokens. Only applies if ``analyzer == 'word'``.
- If None, no stop words will be used.
lowercase : bool, default True
Convert all characters to lowercase before tokenizing.
Attributes
----------
vocabulary_ : dict
A mapping of terms to feature indices.
�word� z \b\w\w+\bNTc C s1 | | _ | | _ | | _ | | _ | | _ d S)N)�analyzer� lowercase�
stop_words� ngram_range�
token_pattern)�selfr r r r r
� r �D/Users/ethen/machine-learning/clustering/tfidf/feature_extraction.py�__init__5 s
zCountVectorizer.__init__c C s | j | � | S)z�
Learn the vocabulary dictionary of all tokens in the raw documents.
Parameters
----------
raw_documents : iterable
An iterable which yields str
Returns
-------
self
)�
fit_transform)r �
raw_documents�yr r r �fit= s
zCountVectorizer.fitc C sC t | t � r t d � � | j | d d �\ } } | | _ | S)a�
Learn the vocabulary dictionary and return document-term matrix.
This is equivalent to calling fit followed by transform, but more
efficiently implemented.
Parameters
----------
raw_documents : iterable
An iterable which yields either str.
Returns
-------
X : scipy sparse matrix, shape [n_samples, n_features]
Count document-term matrix.
zCIterable over raw text documents expected, string objected received� fixed_vocabF)�
isinstance�str�
ValueError� _count_vocab� vocabulary_)r r r �X�
vocabularyr r r r M s z CountVectorizer.fit_transformc
C s� | r | j } n t � } | j | _ | j � } g } g } g } | j d � x� | D]� } i } xa | | � D]S }
y5 | |
} | | k r� d | | <n | | d 7<Wqr t k
r� wr Yqr Xqr W| j | j � � | j | j � � | j t
| � � qY W| st | � } t j
| d t j �} t j
| d t j �} t j
| d t j �} t
| � d t
| � f } t | | | f d | d t j �}
|
| f S)zBCreate sparse feature matrix and vocabulary if fixed_vocab = Falser r �dtype�shape)r r �__len__�default_factory�_build_analyzer�append�KeyError�extend�keys�values�len�dict�np�asarrayZintcr )r r r r Zanalyzer* Zindptr�indices�docZfeature_counter�featureZ feature_idxr" r r r r r i s<
$z CountVectorizer._count_vocabc so t � j � r � j S� j d k rS � j � � � j � � � � � f d d � St d j � j � � � d S)z=Return a callable that handles preprocessing and tokenizationr
c s � j � | � � � S)N)� _word_ngrams)r0 )r r �tokenizer r �<lambda>� s z1CountVectorizer._build_analyzer.<locals>.<lambda>z.{} is not a valid tokenization scheme/analyzerN)�callabler �_build_tokenizer�_get_stop_wordsr �format)r r )r r r3 r r% � s z CountVectorizer._build_analyzerc s? t j | j � � | j r+ � f d d � S� f d d � Sd S)zAReturns a function that splits a string into a sequence of tokensc s � j | j � � S)N)�findall�lower)r0 )r r r r4 � s z2CountVectorizer._build_tokenizer.<locals>.<lambda>c s
� j | � S)N)r9 )r0 )r r r r4 � s N)�re�compiler r
)r r )r r r6 � s z CountVectorizer._build_tokenizerc C sU | j } | d k r t S| d k r) d St | t � rG t d � � n
t | � Sd S)z1Build or fetch the effective stop words frozenset�englishNzStop words not a collection)r r r r r � frozenset)r �stopr r r r7 � s z CountVectorizer._get_stop_wordsc s� � d k r% � f d d � | D� } | j \ } } | d k rD | St | � } t | � } | d k ru | d 7} n g } | j } d j } xe t | t | d | d � � D]C } x: t | | d � D]$ }
| | | |
|
| � � � q� Wq� W| Sd S)zGTokenize document into a sequence of n-grams after stop words filteringNc s" g | ] } | � k r | � q Sr r )�.0�w)r r r �
<listcomp>� s z0CountVectorizer._word_ngrams.<locals>.<listcomp>r � )r �listr+ r& �join�range�min) r �tokensr Zmin_nZmax_nZoriginal_tokensZn_original_tokensZ
tokens_appendZ
space_join�n�ir )r r r2 � s
'&z CountVectorizer._word_ngramsc C s | j | d d �\ } } | S)a�
Transform documents to document-term matrix.
Extract token counts out of raw text documents using the vocabulary
fitted with fit or fit_transform.
Parameters
----------
raw_documents : iterable
An iterable which yields either str.
Returns
-------
X : scipy sparse matrix, [n_samples, n_features]
Document-term matrix.
r T)r )r r r �_r r r � transform� s zCountVectorizer.transform)r r )
�__name__�
__module__� __qualname__�__doc__r r r r r% r6 r7 r2 rL r r r r r
s ) . r c @ sI e Z d Z d Z d d d d d d � Z d d d � Z d
d � Z d S) �TfidfTransformera�
Transform a count matrix to a tf-idf representation.
Parameters
----------
norm : 'l1', 'l2' or None, default 'l2'
Norm used to normalize term vectors. None for no normalization.
smooth_idf : bool, default True
Smooth idf weights by adding one to document frequencies, as if an
extra document was seen containing every term in the collection
exactly once. Prevents zero divisions.
sublinear_tf : bool, default False
Apply sublinear tf scaling, i.e. replace tf with 1 + log(tf).
copy : bool, default True
Whether to copy input data and operate on the copy or perform in-place operations.
�l2TFc C s( | | _ | | _ | | _ | | _ d S)N)�norm�copy�
smooth_idf� sublinear_tf)r rS rU rV rT r r r r � s zTfidfTransformer.__init__Nc
C s� | j \ } } t j | j d | j d �} | t | j � 7} | t | j � 7} t j t | � | � d } t | d d d | d | d d �| _ | S)
z�
Learn the idf vector.
Parameters
----------
X : scipy sparse matrix, shape [n_samples, n_features]
Count document-term matrix.
Z minlengthr g �?Zdiagsr �mrI r8 Zcsr)
r" r- �bincountr/ �intrU �log�floatr � _idf_diag)r r r Z n_samplesZ
n_featuresZdoc_freqZidfr r r r s 'zTfidfTransformer.fitc C s} | j r | j � } | j rB t j | j � | _ | j d 7_ | | j 9} | j d k ry t | d | j d d �} | S)aX
Transform a count matrix to tf-idf representation.
Parameters
----------
X : scipy sparse matrix, [n_samples, n_features]
Count document-term matrix.
Returns
-------
X : scipy sparse matrix, [n_samples, n_features]
Tf-idf weighted document-term matrix.
r NrS rT F)rT rV r- rZ �datar\ rS r )r r r r r rL s
zTfidfTransformer.transform)rM rN rO rP r r rL r r r r rQ � s rQ c s e Z d Z d Z d d d d d d d d d � f d d
� Z d � f d d � Z d � f d
d � Z d � f d d � Z e d d � � Z e j
d d � � Z e d d � � Z e j
d d � � Z e d d � � Z e j
d d � � Z e d d � � Z
e
j
d d � � Z
� S) �TfidfVectorizera[
Convert a collection of raw documents to a matrix of TF-IDF features.
This is equivalent to CountVectorizer followed by TfidfTransformer.
Parameters
----------
analyzer : str {'word'} or callable
Whether the feature should be made of word, if n-grams is specified,
then the words are concatenated with space.
If a callable is passed it is used to extract the sequence of features
out of the raw, unprocessed input.
ngram_range : tuple (min_n, max_n)
The lower and upper boundary of the range of n-values for different
n-grams to be extracted. All values of n such that min_n <= n <= max_n
will be used.
token_pattern : str
Regular expression denoting what constitutes a "token", only used
if ``analyzer == 'word'``. The default regexp select tokens of 2
or more alphanumeric characters (punctuation is completely ignored
and always treated as a token separator).
stop_words : string {'english'}, collection, or None, default None
- If 'english', a built-in stop word list for English is used.
- If a collection, that list or setis assumed to contain stop words, all of which
will be removed from the resulting tokens. Only applies if ``analyzer == 'word'``.
- If None, no stop words will be used.
lowercase : bool, default True
Convert all characters to lowercase before tokenizing.
norm : 'l1', 'l2' or None, default 'l2'
Norm used to normalize term vectors. None for no normalization.
smooth_idf : bool, default True
Smooth idf weights by adding one to document frequencies, as if an
extra document was seen containing every term in the collection
exactly once. Prevents zero divisions.
sublinear_tf : bool, default False
Apply sublinear tf scaling, i.e. replace tf with 1 + log(tf).
copy : bool, default True
Whether to copy input data and operate on the copy or perform in-place operations.
Attributes
----------
vocabulary_ : dict
A mapping of terms to feature indices.
r
r z \b\w\w+\bNTrR Fc
sS t � j d | d | d | d | d | � t d | d | d | d | � | _ d S)
Nr r r r r
rS rU rV rT )�superr rQ �_tfidf)
r r r r r r
rS rU rV rT )� __class__r r r n s
zTfidfVectorizer.__init__c s& t � j | � } | j j | � | S)z�
Learn vocabulary and idf from training set.
Parameters
----------
raw_documents : iterable
An iterable which yields str.
Returns
-------
self
)r_ r r` r )r r r r )ra r r r x s zTfidfVectorizer.fitc s" t � j | � } | j j | � S)a�
Learn vocabulary and idf, return term-document matrix.
This is equivalent to calling fit followed by transform, but more
efficiently implemented.
Parameters
----------
raw_documents : iterable
An iterable which yields str.
Returns
-------
X : scipy sparse matrix, shape [n_samples, n_features]
Tf-idf weighted document-term matrix.
)r_ r r` )r r r r )ra r r r � s z TfidfVectorizer.fit_transformc s" t � j | � } | j j | � S)a)
Transform documents to document-term matrix.
Uses the vocabulary and document frequencies learned by fit or
fit_transform.
Parameters
----------
raw_documents : iterable
An iterable which yields str.
copy : boolean, default True
Whether to copy X and operate on the copy or perform in-place
operations.
Returns
-------
X : scipy sparse matrix, shape [n_samples, n_features]
Tf-idf weighted document-term matrix.
)r_ rL r` )r r rT r )ra r r rL � s zTfidfVectorizer.transformc C s
| j j S)N)r` rS )r r r r rS � s zTfidfVectorizer.normc C s | | j _ d S)N)r` rS )r �valuer r r rS � s c C s
| j j S)N)r` �use_idf)r r r r rc � s zTfidfVectorizer.use_idfc C s | | j _ d S)N)r` rc )r rb r r r rc � s c C s
| j j S)N)r` rU )r r r r rU � s zTfidfVectorizer.smooth_idfc C s | | j _ d S)N)r` rU )r rb r r r rU � s c C s
| j j S)N)r` rV )r r r r rV � s z TfidfVectorizer.sublinear_tfc C s | | j _ d S)N)r` rV )r rb r r r rV � s )r r )rM rN rO rP r r r rL �propertyrS �setterrc rU rV r r )ra r r^ 9 s 3 r^ )r; �numpyr- � collectionsr Z scipy.sparser r Zsklearn.preprocessingr Z sklearn.baser r Z sklearn.feature_extraction.textr r rQ r^ r r r r �<module> s �O