CoCalc -- BERT_email_classification.ipynb

GitHub Repository: codebasics/deep-learning-keras-tf-tutorial
Path: blob/master/48_tf_serving/BERT_email_classification.ipynb
¹¹⁴¹ views

Kernel: Python 3

BERT tutorial: Classify spam vs no spam emails

In [1]:

import tensorflow as tf
import tensorflow_hub as hub
import tensorflow_text as text

Import the dataset (Dataset is taken from kaggle)

In [2]:

import pandas as pd

df = pd.read_csv("spam.csv")
df.head(5)

Out[2]:

In [3]:

df.groupby('Category').describe()

Out[3]:

In [4]:

df['spam']=df['Category'].apply(lambda x: 1 if x=='spam' else 0)
df.head()

Out[4]:

Split it into training and test data set

In [5]:

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(df['Message'],df['spam'], stratify=df['spam'])

In [6]:

X_train.head(4)

Out[6]:

  1. Tension face 2. Smiling face 3. Waste face ...
  Oh ho. Is this the first time u use these type...
  Dunno cos i was v late n when i reach they ins...
  Mum say we wan to go then go... Then she can s...
Name: Message, dtype: object

Now lets import BERT model and get embeding vectors for few sample statements

In [7]:

bert_preprocess = hub.KerasLayer("https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3")
bert_encoder = hub.KerasLayer("https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/4")

In [8]:

def get_sentence_embeding(sentences):
    preprocessed_text = bert_preprocess(sentences)
    return bert_encoder(preprocessed_text)['pooled_output']

get_sentence_embeding([
    "500$ discount. hurry up", 
    "Bhavin, are you up for a volleybal game tomorrow?"]
)

Out[8]:

<tf.Tensor: shape=(2, 768), dtype=float32, numpy=
array([[-0.8435169 , -0.51327276, -0.8884574 , ..., -0.74748874,
        -0.75314736,  0.91964483],
       [-0.87208366, -0.50543964, -0.94446677, ..., -0.858475  ,
        -0.7174535 ,  0.8808298 ]], dtype=float32)>

Get embeding vectors for few sample words. Compare them using cosine similarity

In [9]:

e = get_sentence_embeding([
    "banana", 
    "grapes",
    "mango",
    "jeff bezos",
    "elon musk",
    "bill gates"
]
)

In [10]:

from sklearn.metrics.pairwise import cosine_similarity
cosine_similarity([e[0]],[e[1]])

Out[10]:

array([[0.9911089]], dtype=float32)

Values near to 1 means they are similar. 0 means they are very different. Above you can use comparing "banana" vs "grapes" you get 0.99 similarity as they both are fruits

In [11]:

cosine_similarity([e[0]],[e[3]])

Out[11]:

array([[0.8470385]], dtype=float32)

Comparing banana with jeff bezos you still get 0.84 but it is not as close as 0.99 that we got with grapes

In [12]:

cosine_similarity([e[3]],[e[4]])

Out[12]:

array([[0.98720354]], dtype=float32)

Jeff bezos and Elon musk are more similar then Jeff bezos and banana as indicated above

Build Model

There are two types of models you can build in tensorflow.

(1) Sequential (2) Functional

So far we have built sequential model. But below we will build functional model. More information on these two is here: https://becominghuman.ai/sequential-vs-functional-model-in-keras-20684f766057

In [13]:

# Bert layers
text_input = tf.keras.layers.Input(shape=(), dtype=tf.string, name='text')
preprocessed_text = bert_preprocess(text_input)
outputs = bert_encoder(preprocessed_text)

# Neural network layers
l = tf.keras.layers.Dropout(0.1, name="dropout")(outputs['pooled_output'])
l = tf.keras.layers.Dense(1, activation='sigmoid', name="output")(l)

# Use inputs and outputs to construct a final model
model = tf.keras.Model(inputs=[text_input], outputs = [l])

https://stackoverflow.com/questions/47605558/importerror-failed-to-import-pydot-you-must-install-pydot-and-graphviz-for-py

In [14]:

model.summary()

Out[14]:

Model: "model"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
text (InputLayer)               [(None,)]            0                                            
__________________________________________________________________________________________________
keras_layer (KerasLayer)        {'input_mask': (None 0           text[0][0]                       
__________________________________________________________________________________________________
keras_layer_1 (KerasLayer)      {'default': (None, 7 109482241   keras_layer[0][0]                
                                                                 keras_layer[0][1]                
                                                                 keras_layer[0][2]                
__________________________________________________________________________________________________
dropout (Dropout)               (None, 768)          0           keras_layer_1[0][13]             
__________________________________________________________________________________________________
output (Dense)                  (None, 1)            769         dropout[0][0]                    
==================================================================================================
Total params: 109,483,010
Trainable params: 769
Non-trainable params: 109,482,241
__________________________________________________________________________________________________

In [15]:

len(X_train)

Out[15]:

4179

In [16]:

model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['accuracy'])

Train the model

In [17]:

model.fit(X_train, y_train, epochs=5)

Out[17]:

Epoch 1/5
131/131 [==============================] - 31s 188ms/step - loss: 0.3172 - accuracy: 0.8686
Epoch 2/5
131/131 [==============================] - 26s 197ms/step - loss: 0.2286 - accuracy: 0.8995
Epoch 3/5
131/131 [==============================] - 26s 199ms/step - loss: 0.1925 - accuracy: 0.9244
Epoch 4/5
131/131 [==============================] - 27s 204ms/step - loss: 0.1703 - accuracy: 0.9354
Epoch 5/5
131/131 [==============================] - 25s 189ms/step - loss: 0.1550 - accuracy: 0.9457

<tensorflow.python.keras.callbacks.History at 0x2add6f58c70>

In [18]:

model.evaluate(X_test, y_test)

Out[18]:

44/44 [==============================] - 9s 187ms/step - loss: 0.1490 - accuracy: 0.9476

[0.1490102857351303, 0.9475951194763184]

Inference

In [19]:

reviews = [
    'Reply to win Â£100 weekly! Where will the 2006 FIFA World Cup be held? Send STOP to 87239 to end service',
    'You are awarded a SiPix Digital Camera! call 09061221061 from landline. Delivery within 28days. T Cs Box177. M221BP. 2yr warranty. 150ppm. 16 . p pÂ£3.99',
    'it to 80488. Your 500 free text messages are valid until 31 December 2005.',
    'Hey Sam, Are you coming for a cricket game tomorrow',
    "Why don't you wait 'til at least wednesday to see if you get your ."
]
model.predict(reviews)

Out[19]:

array([[0.5297462 ],
       [0.58646685],
       [0.40190795],
       [0.03496679],
       [0.01311444]], dtype=float32)

In [22]:

model.save("saved_models/1/")

Out[22]:

WARNING:absl:Found untraced functions such as restored_function_body, restored_function_body, restored_function_body, restored_function_body, restored_function_body while saving (showing 5 of 915). These functions will not be directly callable after loading.

INFO:tensorflow:Assets written to: saved_models/1/assets

INFO:tensorflow:Assets written to: saved_models/1/assets

In [23]:

model.save("saved_models/2/")

Out[23]:

WARNING:absl:Found untraced functions such as restored_function_body, restored_function_body, restored_function_body, restored_function_body, restored_function_body while saving (showing 5 of 915). These functions will not be directly callable after loading.

INFO:tensorflow:Assets written to: saved_models/2/assets

INFO:tensorflow:Assets written to: saved_models/2/assets

In [24]:

model.save("saved_models/3/")

Out[24]:

WARNING:absl:Found untraced functions such as restored_function_body, restored_function_body, restored_function_body, restored_function_body, restored_function_body while saving (showing 5 of 915). These functions will not be directly callable after loading.

INFO:tensorflow:Assets written to: saved_models/3/assets

INFO:tensorflow:Assets written to: saved_models/3/assets