ch15_part1.ipynb

Kernel: Python 3 (ipykernel)

Machine Learning with PyTorch and Scikit-Learn

-- Code Examples

Package version checks

Add folder to path in order to load from the check_packages.py script:

import sys
sys.path.insert(0, '..')

Check recommended package versions:

from python_environment_check import check_packages


d = {
    'torch': '1.8.0',
}
check_packages(d)

[OK] Your Python version is 3.9.7 | packaged by conda-forge | (default, Sep 29 2021, 19:24:02) 
[Clang 11.1.0 ]
[OK] torch 1.10.1

Chapter 15: Modeling Sequential Data Using Recurrent Neural Networks (Part 1/3)

Outline

• Introducing sequential data

  • Modeling sequential data -- order matters

  • Sequential data versus time series data

  • Representing sequences

  • The different categories of sequence modeling

• RNNs for modeling sequences

  • Understanding the dataflow in RNNs

  • Computing activations in an RNN

  • Hidden recurrence versus output recurrence

  • The challenges of learning long-range interactions

  • Long short-term memory cells

from IPython.display import Image
%matplotlib inline

Introducing sequential data

Modeling sequential data⁠—order matters

Representing sequences

Image(filename='figures/15_01.png', width=500)

The different categories of sequence modeling

Image(filename='figures/15_02.png', width=500)

RNNs for modeling sequences

Understanding the RNN looping mechanism

Image(filename='figures/15_03.png', width=500)

Image(filename='figures/15_04.png', width=500)

Computing activations in an RNN

Image(filename='figures/15_05.png', width=500)

Image(filename='figures/15_06.png', width=700)

Hidden-recurrence vs. output-recurrence

Image(filename='figures/15_07.png', width=500)

import torch
import torch.nn as nn

torch.manual_seed(1)

rnn_layer = nn.RNN(input_size=5, hidden_size=2, num_layers=1, batch_first=True) 

w_xh = rnn_layer.weight_ih_l0
w_hh = rnn_layer.weight_hh_l0
b_xh = rnn_layer.bias_ih_l0
b_hh = rnn_layer.bias_hh_l0

print('W_xh shape:', w_xh.shape)
print('W_hh shape:', w_hh.shape)
print('b_xh shape:', b_xh.shape)
print('b_hh shape:', b_hh.shape)

W_xh shape: torch.Size([2, 5])
W_hh shape: torch.Size([2, 2])
b_xh shape: torch.Size([2])
b_hh shape: torch.Size([2])

x_seq = torch.tensor([[1.0]*5, [2.0]*5, [3.0]*5]).float()

## output of the simple RNN:
output, hn = rnn_layer(torch.reshape(x_seq, (1, 3, 5)))

## manually computing the output:
out_man = []
for t in range(3):
    xt = torch.reshape(x_seq[t], (1, 5))
    print(f'Time step {t} =>')
    print('   Input           :', xt.numpy())
    
    ht = torch.matmul(xt, torch.transpose(w_xh, 0, 1)) + b_xh    
    print('   Hidden          :', ht.detach().numpy())
    
    if t>0:
        prev_h = out_man[t-1]
    else:
        prev_h = torch.zeros((ht.shape))

    ot = ht + torch.matmul(prev_h, torch.transpose(w_hh, 0, 1)) + b_hh
    ot = torch.tanh(ot)
    out_man.append(ot)
    print('   Output (manual) :', ot.detach().numpy())
    print('   RNN output      :', output[:, t].detach().numpy())
    print()

Time step 0 =>
   Input           : [[1. 1. 1. 1. 1.]]
   Hidden          : [[-0.4701929  0.5863904]]
   Output (manual) : [[-0.3519801   0.52525216]]
   RNN output      : [[-0.3519801   0.52525216]]

Time step 1 =>
   Input           : [[2. 2. 2. 2. 2.]]
   Hidden          : [[-0.88883156  1.2364397 ]]
   Output (manual) : [[-0.68424344  0.76074266]]
   RNN output      : [[-0.68424344  0.76074266]]

Time step 2 =>
   Input           : [[3. 3. 3. 3. 3.]]
   Hidden          : [[-1.3074701  1.886489 ]]
   Output (manual) : [[-0.8649416   0.90466356]]
   RNN output      : [[-0.8649416   0.90466356]]

The challenges of learning long-range interactions

Image(filename='figures/15_08.png', width=500)

Long Short-Term Memory cells

Image(filename='figures/15_09.png', width=500)

---

Readers may ignore the next cell.

! python ../.convert_notebook_to_script.py --input ch15_part1.ipynb --output ch15_part1.py

[NbConvertApp] Converting notebook ch15_part1.ipynb to script
[NbConvertApp] Writing 3919 bytes to ch15_part1.py