GitHub Repository: rasbt/machine-learning-book
Path: blob/main/ch15/ch15_part3.ipynb
¹²⁴⁵ views

Kernel: Python 3 (ipykernel)

Machine Learning with PyTorch and Scikit-Learn

-- Code Examples

Package version checks

Add folder to path in order to load from the check_packages.py script:

In [1]:

import sys
sys.path.insert(0, '..')

Check recommended package versions:

In [2]:

from python_environment_check import check_packages


d = {
    'torch': '1.8.0',
}
check_packages(d)

Out[2]:

[OK] Your Python version is 3.9.7 | packaged by conda-forge | (default, Sep 29 2021, 19:24:02) 
[Clang 11.1.0 ]
[OK] torch 1.10.2

Chapter 15: Modeling Sequential Data Using Recurrent Neural Networks (part 3/3)

Outline

Implementing RNNs for sequence modeling in PyTorch
- Project two -- character-level language modeling in PyTorch
  - Preprocessing the dataset
  - Evaluation phase -- generating new text passages
Summary

Note that the optional watermark extension is a small IPython notebook plugin that I developed to make the code reproducible. You can just skip the following line(s).

In [3]:

from IPython.display import Image
%matplotlib inline

Project two: character-level language modeling in PyTorch

In [4]:

Image(filename='figures/15_11.png', width=500)

Out[4]:

Preprocessing the dataset

In [5]:

import numpy as np

## Reading and processing text
with open('1268-0.txt', 'r', encoding="utf8") as fp:
    text=fp.read()
    
start_indx = text.find('THE MYSTERIOUS ISLAND')
end_indx = text.find('End of the Project Gutenberg')

text = text[start_indx:end_indx]
char_set = set(text)
print('Total Length:', len(text))
print('Unique Characters:', len(char_set))

Out[5]:

Total Length: 1112350
Unique Characters: 80

In [6]:

Image(filename='figures/15_12.png', width=500)

Out[6]:

In [7]:

chars_sorted = sorted(char_set)
char2int = {ch:i for i,ch in enumerate(chars_sorted)}
char_array = np.array(chars_sorted)

text_encoded = np.array(
    [char2int[ch] for ch in text],
    dtype=np.int32)

print('Text encoded shape: ', text_encoded.shape)

print(text[:15], '     == Encoding ==> ', text_encoded[:15])
print(text_encoded[15:21], ' == Reverse  ==> ', ''.join(char_array[text_encoded[15:21]]))

Out[7]:

Text encoded shape:  (1112350,)
THE MYSTERIOUS       == Encoding ==>  [44 32 29  1 37 48 43 44 29 42 33 39 45 43  1]
[33 43 36 25 38 28]  == Reverse  ==>  ISLAND

In [8]:

for ex in text_encoded[:5]:
    print('{} -> {}'.format(ex, char_array[ex]))

Out[8]:

-> T
-> H
-> E
->  
-> M

In [9]:

Image(filename='figures/15_13.png', width=500)

Out[9]:

In [10]:

Image(filename='figures/15_14.png', width=500)

Out[10]:

In [11]:

seq_length = 40
chunk_size = seq_length + 1

text_chunks = [text_encoded[i:i+chunk_size] 
               for i in range(len(text_encoded)-chunk_size+1)] 

## inspection:
for seq in text_chunks[:1]:
    input_seq = seq[:seq_length]
    target = seq[seq_length] 
    print(input_seq, ' -> ', target)
    print(repr(''.join(char_array[input_seq])), 
          ' -> ', repr(''.join(char_array[target])))

Out[11]:

[44 32 29  1 37 48 43 44 29 42 33 39 45 43  1 33 43 36 25 38 28  1  6  6
  6  0  0  0  0  0 40 67 64 53 70 52 54 53  1 51]  ->  74
'THE MYSTERIOUS ISLAND ***\n\n\n\n\nProduced b'  ->  'y'

In [12]:

import torch
from torch.utils.data import Dataset

class TextDataset(Dataset):
    def __init__(self, text_chunks):
        self.text_chunks = text_chunks

    def __len__(self):
        return len(self.text_chunks)
    
    def __getitem__(self, idx):
        text_chunk = self.text_chunks[idx]
        return text_chunk[:-1].long(), text_chunk[1:].long()
    
seq_dataset = TextDataset(torch.tensor(text_chunks))

Out[12]:

/var/folders/jg/tpqyh1fd5js5wsr1d138k3n40000gn/T/ipykernel_44396/2527503007.py:15: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  /Users/runner/miniforge3/conda-bld/pytorch-recipe_1645462109041/work/torch/csrc/utils/tensor_new.cpp:201.)
  seq_dataset = TextDataset(torch.tensor(text_chunks))

In [13]:

for i, (seq, target) in enumerate(seq_dataset):
    print(' Input (x):', repr(''.join(char_array[seq])))
    print('Target (y):', repr(''.join(char_array[target])))
    print()
    if i == 1:
        break

Out[13]:

 Input (x): 'THE MYSTERIOUS ISLAND ***\n\n\n\n\nProduced b'
Target (y): 'HE MYSTERIOUS ISLAND ***\n\n\n\n\nProduced by'

 Input (x): 'HE MYSTERIOUS ISLAND ***\n\n\n\n\nProduced by'
Target (y): 'E MYSTERIOUS ISLAND ***\n\n\n\n\nProduced by '

In [14]:

device = torch.device("cuda:0")
# device = 'cpu'

In [15]:

from torch.utils.data import DataLoader
 
batch_size = 64

torch.manual_seed(1)
seq_dl = DataLoader(seq_dataset, batch_size=batch_size, shuffle=True, drop_last=True)

Building a character-level RNN model

In [16]:

import torch.nn as nn

class RNN(nn.Module):
    def __init__(self, vocab_size, embed_dim, rnn_hidden_size):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim) 
        self.rnn_hidden_size = rnn_hidden_size
        self.rnn = nn.LSTM(embed_dim, rnn_hidden_size, 
                           batch_first=True)
        self.fc = nn.Linear(rnn_hidden_size, vocab_size)

    def forward(self, x, hidden, cell):
        out = self.embedding(x).unsqueeze(1)
        out, (hidden, cell) = self.rnn(out, (hidden, cell))
        out = self.fc(out).reshape(out.size(0), -1)
        return out, hidden, cell

    def init_hidden(self, batch_size):
        hidden = torch.zeros(1, batch_size, self.rnn_hidden_size)
        cell = torch.zeros(1, batch_size, self.rnn_hidden_size)
        return hidden.to(device), cell.to(device)
    
vocab_size = len(char_array)
embed_dim = 256
rnn_hidden_size = 512

torch.manual_seed(1)
model = RNN(vocab_size, embed_dim, rnn_hidden_size) 
model = model.to(device)
model

Out[16]:

RNN(
  (embedding): Embedding(80, 256)
  (rnn): LSTM(256, 512, batch_first=True)
  (fc): Linear(in_features=512, out_features=80, bias=True)
)

In [17]:

loss_fn = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.005)

num_epochs = 10000 

torch.manual_seed(1)

for epoch in range(num_epochs):
    hidden, cell = model.init_hidden(batch_size)
    seq_batch, target_batch = next(iter(seq_dl))
    seq_batch = seq_batch.to(device)
    target_batch = target_batch.to(device)
    optimizer.zero_grad()
    loss = 0
    for c in range(seq_length):
        pred, hidden, cell = model(seq_batch[:, c], hidden, cell) 
        loss += loss_fn(pred, target_batch[:, c])
    loss.backward()
    optimizer.step()
    loss = loss.item()/seq_length
    if epoch % 500 == 0:
        print(f'Epoch {epoch} loss: {loss:.4f}')

Out[17]:

Epoch 0 loss: 4.3719
Epoch 500 loss: 1.3804
Epoch 1000 loss: 1.2956
Epoch 1500 loss: 1.2816
Epoch 2000 loss: 1.1968
Epoch 2500 loss: 1.2456
Epoch 3000 loss: 1.1763
Epoch 3500 loss: 1.1868
Epoch 4000 loss: 1.1476
Epoch 4500 loss: 1.2118
Epoch 5000 loss: 1.2072
Epoch 5500 loss: 1.1243
Epoch 6000 loss: 1.1637
Epoch 6500 loss: 1.1513
Epoch 7000 loss: 1.1196
Epoch 7500 loss: 1.1439
Epoch 8000 loss: 1.1536
Epoch 8500 loss: 1.1263
Epoch 9000 loss: 1.1597
Epoch 9500 loss: 1.1048

Evaluation phase: generating new text passages

In [18]:

from torch.distributions.categorical import Categorical

torch.manual_seed(1)

logits = torch.tensor([[1.0, 1.0, 1.0]])

print('Probabilities:', nn.functional.softmax(logits, dim=1).numpy()[0])

m = Categorical(logits=logits)
samples = m.sample((10,))
 
print(samples.numpy())

Out[18]:

Probabilities: [0.33333334 0.33333334 0.33333334]
[[0]
 [0]
 [0]
 [0]
 [1]
 [0]
 [1]
 [2]
 [1]
 [1]]

In [19]:

torch.manual_seed(1)

logits = torch.tensor([[1.0, 1.0, 3.0]])

print('Probabilities:', nn.functional.softmax(logits, dim=1).numpy()[0])

m = Categorical(logits=logits)
samples = m.sample((10,))
 
print(samples.numpy())

Out[19]:

Probabilities: [0.10650698 0.10650698 0.78698605]
[[0]
 [2]
 [2]
 [1]
 [2]
 [1]
 [2]
 [2]
 [2]
 [2]]

In [20]:

def sample(model, starting_str, 
           len_generated_text=500, 
           scale_factor=1.0):

    encoded_input = torch.tensor([char2int[s] for s in starting_str])
    encoded_input = torch.reshape(encoded_input, (1, -1))

    generated_str = starting_str

    model.eval()
    hidden, cell = model.init_hidden(1)
    hidden = hidden.to('cpu')
    cell = cell.to('cpu')
    for c in range(len(starting_str)-1):
        _, hidden, cell = model(encoded_input[:, c].view(1), hidden, cell) 
    
    last_char = encoded_input[:, -1]
    for i in range(len_generated_text):
        logits, hidden, cell = model(last_char.view(1), hidden, cell) 
        logits = torch.squeeze(logits, 0)
        scaled_logits = logits * scale_factor
        m = Categorical(logits=scaled_logits)
        last_char = m.sample()
        generated_str += str(char_array[last_char])
        
    return generated_str

torch.manual_seed(1)
model.to('cpu')
print(sample(model, starting_str='The island'))

Out[20]:

The island had been neged to reward from them with denies he giving these gigant of some taking two years from the question has been employed without
consequences in his depth. As
to the notice beate can be vapor when there we must forgot.

“Shat’s likely,” said Pencroft.

“What a
region of Neption!” Cyrus Harding would have
been pushed their return to the shored him, to do in the dockyar and formation of the settlers’ animal which would be better dead of freshest.

Cyrus, was well presented by means of t

Predictability vs. randomness

In [21]:

logits = torch.tensor([[1.0, 1.0, 3.0]])

print('Probabilities before scaling:        ', nn.functional.softmax(logits, dim=1).numpy()[0])

print('Probabilities after scaling with 0.5:', nn.functional.softmax(0.5*logits, dim=1).numpy()[0])

print('Probabilities after scaling with 0.1:', nn.functional.softmax(0.1*logits, dim=1).numpy()[0])

Out[21]:

Probabilities before scaling:         [0.10650698 0.10650698 0.78698605]
Probabilities after scaling with 0.5: [0.21194156 0.21194156 0.57611686]
Probabilities after scaling with 0.1: [0.3104238  0.3104238  0.37915248]

In [22]:

torch.manual_seed(1)
print(sample(model, starting_str='The island', 
             scale_factor=2.0))

Out[22]:

The island on the shore, and thought the island was finished, and the stranger had constituted to this precipitation of the corral, which he suffered in the depths of the cone of a ship was only a long time was a sort of the passage there to the sea, he was ready to
do anything to the south and to the shore, the settlers did not think the convicts were to be on the corral, and the engineer then to hope of the settlers, and shelter at the corral, the colonists had not happened to the settlers had only to t

In [23]:

torch.manual_seed(1)
print(sample(model, starting_str='The island', 
             scale_factor=0.5))

Out[23]:

The island deep ising side; his”not
awoke-prtaugh? Gideon Spilett too?,”

Helofe, and ihos eyes; “re” helm usn’t produtiuso,
at Tajow him, somucn!”

Above Verishy, silene;
ote fat wire, notwiths, and
bank pludes
attackleep, plaions fraitr byll,
you ironaliers thefee
groat, however
parcht mad. Would wideney twontashmonest.

A fugitors plan ascererver; Thousand-shwark orhrankheaded abope had
let us alwayberrusible, withwh.
Blacaneara,moss, Net susnid! 6

I- Poingin’s new kay was “Muldhidge?”
-!

“It phincip

...

Summary

...

Readers may ignore the next cell.

In [24]:

! python ../.convert_notebook_to_script.py --input ch15_part3.ipynb --output ch15_part3.py

Out[24]:

[NbConvertApp] Converting notebook ch15_part3.ipynb to script
[NbConvertApp] Writing 7922 bytes to ch15_part3.py