Path: blob/master/Model-4/ocr/datahelpers.pyc
427 views
�
��Zc S @ s3 d Z d d l Z d d l Z d d l Z d d l Z d d l Z d d l m Z d d l m
Z
d d l m Z d d d d
d d d
d d d d d d d d d d d d d d d d d d d d! d" d# d$ d% d& d' d( d) d* d+ d, d- d. d/ d0 d1 d2 d3 d4 d5 d6 d7 d8 d9 d: d; d< d= d> d? d@ dA dB dC dD dE dF dG dH dI dJ dK dL dM dN dO dP dQ dR dS dT dU dV dW dX dY gS Z
g e e e
� � D] Z e ^ q�Z e e e e
� � Z e e e
e � � Z e dZ � Z e d[ � Z d\ e e d] � Z d^ d_ � Z d` d\ d^ da � Z db dh e e de � Z df � Z dg � Z d S(i s4
Helper functions for loading and creating datasets
i����Ni ( t implt( t
letterNorm( t printProgressBart t At Bt Ct Dt Et Ft Gt Ht It Jt Kt Lt Mt Nt Ot Pt Qt Rt St Tt Ut Vt Wt Xt Yt Zt at bt ct dt et ft gt ht it jt kt lt mt nt ot pt qt rt st tt ut vt wt xt yt zs Ás És Ís Ós Ús Ýs ás és ís ós ús ýs Čs čs Ďs ďs Ěs ěs Ňs ňs Řs řs Šs šs Ťs ťs Ůs ůs Žs žc C s | r t | d St | S( Ni ( t chars_to_idx( R t sequence( ( s6 /home/harshit/work/Opensoft/Model-4/ocr/datahelpers.pyt char2idx s c C s | r t | d St | S( Ni ( t idx_to_chars( t idxR9 ( ( s6 /home/harshit/work/Opensoft/Model-4/ocr/datahelpers.pyt idx2char" s s data/words/c C s� d GHg } g } t | � t k r� x� | D]w } | | d d k rI d n d 7} t j | d � } | | 7} | g | D]# } | t | � j d � d ^ qw 7} q* Wnc | | d d k r� d n d 7} t j | d � } g | D]# } | t | � j d � d ^ q� } t j | � } t j t | � d t �} x0 t | � D]" \ }
} t
j | d � | |
<qBW| r�t j t | � d t �} xY t | � D]H \ }
} t | d d
d � �# }
t j t
j |
� � | |
<Wd QXq�Wn | r"t | � t | � k ot | � k n s@t � n t | � t | � k s@t � d
t | � f GH| r�t | d d d � d | d f GH| r�d | d f GHq�n | r�| | | f S| | f S( sP
Load word images with corresponding labels and gaplines (if loadGaplines == True)
Args:
dataloc: image folder location - can be list of multiple locations,
loadGaplines: wheter or not load gaplines positions files
debug: for printing example image
Returns:
(images, labels (, gaplines))
s Loading words...i����t /R s *.jpgt _i t dtypei����t txtR/ Ns -> Number of words:t grayt Examples Word:s Gaplines:( t typet listt globt lent splitt npt arrayt emptyt objectt enumeratet cv2t imreadt opent
simplejsont loadt AssertionErrorR ( t dataloct loadGaplinest debugt imglistt tmpLabelst loct tmpListt namet labelst imagesR& t imgt gaplinest fp( ( s6 /home/harshit/work/Opensoft/Model-4/ocr/datahelpers.pyt
loadWordsData( sB
; 0)7
t czc
C s2 t g | D] } t | � ^ q
� } t j | d t �} g } | d j d } d } x� t | � D]� \ }
} x� t t | � d � D]� } | |
d | � | | | | d � f | | <| d k r� | j t | |
| � � n$ | j t t
j
| |
| � � � | d 7} q� Wqd Wd | f GH| | f S( s; Transform word images with gaplines into individual chars R@ i i Rb s Loaded chars from words:( t sumRG RI RK RL t shapeRM t ranget appendR: t unidecode(
R] R\ R_ t langR) t lengtht imgst newLabelst heightR< R& t gapst pos( ( s6 /home/harshit/work/Opensoft/Model-4/ocr/datahelpers.pyt words2charsa s % 0 $ s data/charclas/c C s( d GHt j d � } g } | d k rMt j | | d � } | j � | d k r\ t } n t } g | D]& } | d d k r� | d n d ^ qi | k s� t � x� t t | � � D]� } t j | | d � } t j g | D] }
t
t j |
d
� � ^ q� � } t j
| | j t | � d � g � } | j | g t | � � q� Wn | d k r�t | � \ } } }
t | | |
| � \ } } | j | � x[ t t | � � D]D } t | t | � � t j
| t
| | � j d d � g � } q�Wn | d } t j | � } d t | � f GH| | f S(
s�
Load chars images with corresponding labels
Args:
charloc: char images FOLDER LOCATION
wordloc: word images with gaplines FOLDER LOCATION
Returns:
(images, labels)
s Loading chars...i i R s /*/t eni����t 0s *.jpgi s -> Number of chars:( i i ( RI t zerosRF t sortt CHARS_ENt CHARS_CZRS Re RG RJ R RN RO t concatenatet reshapet extendRa Ro R ( t charloct wordlocRh R] R\ t dirlistt charsR! R& RW R^ Rj t wordsR_ ( ( s6 /home/harshit/work/Opensoft/Model-4/ocr/datahelpers.pyt
loadCharsDatay s6
?4'!
)
s data/gapdet/large/i< ix c s� d GH� � d d k r d n d 7� t j � d � } | j � | d d k r_ d | d <n d | d d d k r} d n d | d d } d | d d d k r� d n d | d d } | r t j t | � d
t �} t j t | � d
t �} xSt | � D]\ } � t j � d � }
t |
� d k r t |
d � f d
� �} t j g |
D]b } | r�t
j | d � d d � | | � f j � n% t
j | d � d d � | | � f ^ qb� | | <t j g |
D]) }
t
|
t � � j d � d � ^ q�� | | <q q Wn4t j d | d | d f � } g } x� t t | � � D]� } t j | | d � }
t |
� d k rXt j g |
D]. } t
j | d � d d � | | � f ^ q�� } t j | | j t | � | d | d � g � } | j g |
D] } t
| t | | � � ^ q � qXqXW| d } t j | � } | r�d t | � d t g | D] } t | � ^ qq� f GHn d t | � f GH| | f S( s�
Load gap data from location with corresponding labels
Args:
loc: location of folder with words separated into gap data
images have to by named as label_timestamp.jpg, label is 0 or 1
slider: dimensions of of output images
seq: Store images from one word as a sequence
flatten: Flatten the output images
Returns:
(images, labels)
s Loading gap data...i����R> R s */i ix i i R@ s *.jpgt keyc s% t | t � � j d � d d � S( NR? i i����( t intRG RH ( R5 ( RY ( s6 /home/harshit/work/Opensoft/Model-4/ocr/datahelpers.pyt <lambda>� s NR? s& -> Number of words / gaps and letters:s -> Number of gaps and letters:( RF Rs t NoneRI RK RG RL RM t sortedRJ RN RO t flattenR� RH Rr Re Rv Rw Rx Rc ( RY t slidert seqR� R{ t cut_st cut_eR] R\ R& RW t imgListR^ R[ Rj R) ( ( RY s6 /home/harshit/work/Opensoft/Model-4/ocr/datahelpers.pyt loadGapData� sB
01 sM!D3;
6c C s� t g t t | � � D]& } t | d � t | | � k ^ q � sK t � t j j t | d � � } x, t t | � � D] } | | | | | <qz W| S( s�
Shuffle array of numpy arrays such that
each pair a[x][i] and a[y][i] remains the same
Args:
a: array of same length numpy arrays
Returns:
Array a with shuffled numpy arrays
i ( t allRe RG RS RI t randomt permutation( R R& R- ( ( s6 /home/harshit/work/Opensoft/Model-4/ocr/datahelpers.pyt correspondingShuffle� s
K c C s� g } g } xV t | � D]H \ } } | j t | g t | � t t | � � � � | j | � q Wt j | d t j �} t j | d t j �} t j t | � t j | � j d � d d g d t j �} | | | f S( s�
Create a sparse representention of sequences.
Args:
sequences: a list of lists of type dtype where each element is a sequence
Returns:
A tuple with (indices, values, shape)
R@ i i (
RM Rx t zipRG Re RI t asarrayt int64t int32t max( t sequencest indicest valuesR+ R� Rd ( ( s6 /home/harshit/work/Opensoft/Model-4/ocr/datahelpers.pyt sequences_to_sparse� s />( i< ix ( t __doc__t numpyRI RF RQ RN Rg t helpersR t
normalizationR t vizR t CHARSRe RG R& t idxst dictR� R; R8 t FalseR: R= t TrueRa Ro R~ R� R� R� ( ( ( s6 /home/harshit/work/Opensoft/Model-4/ocr/datahelpers.pyt <module> s8 % 9 3<