Path: blob/master/Neural machine translation with attention_latest.ipynb
5780 views
Neural Machine Translation
Welcome to your first programming assignment for this week!
You will build a Neural Machine Translation (NMT) model to translate human readable dates ("25th of June, 2009") into machine readable dates ("2009-06-25"). You will do this using an attention model, one of the most sophisticated sequence to sequence models.
This notebook was produced together with NVIDIA's Deep Learning Institute.
Let's load all the packages you will need for this assignment.
Defaulting to user installation because normal site-packages is not writeable
Collecting faker
Downloading Faker-33.1.0-py3-none-any.whl.metadata (15 kB)
Requirement already satisfied: python-dateutil>=2.4 in /Users/msatidas/Library/Python/3.9/lib/python/site-packages (from faker) (2.8.2)
Requirement already satisfied: typing-extensions in /Users/msatidas/Library/Python/3.9/lib/python/site-packages (from faker) (4.5.0)
Requirement already satisfied: six>=1.5 in /Users/msatidas/Library/Python/3.9/lib/python/site-packages (from python-dateutil>=2.4->faker) (1.16.0)
Downloading Faker-33.1.0-py3-none-any.whl (1.9 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.9/1.9 MB 10.8 MB/s eta 0:00:00 0:00:01
Installing collected packages: faker
Successfully installed faker-33.1.0
[notice] A new release of pip is available: 24.0 -> 24.3.1
[notice] To update, run: /Library/Developer/CommandLineTools/usr/bin/python3 -m pip install --upgrade pip
1 - Translating human readable dates into machine readable dates
The model you will build here could be used to translate from one language to another, such as translating from English to Hindi. However, language translation requires massive datasets and usually takes days of training on GPUs. To give you a place to experiment with these models even without using massive datasets, we will instead use a simpler "date translation" task.
The network will input a date written in a variety of possible formats (e.g. "the 29th of August 1958", "03/30/1968", "24 JUNE 1987") and translate them into standardized, machine readable dates (e.g. "1958-08-29", "1968-03-30", "1987-06-24"). We will have the network learn to output dates in the common machine-readable format YYYY-MM-DD.
1.1 - Dataset
We will train the model on a dataset of 10000 human readable dates and their equivalent, standardized, machine readable dates. Let's run the following cells to load the dataset and print some examples.
You've loaded:
dataset: a list of tuples of (human readable date, machine readable date)human_vocab: a python dictionary mapping all characters used in the human readable dates to an integer-valued indexmachine_vocab: a python dictionary mapping all characters used in machine readable dates to an integer-valued index. These indices are not necessarily consistent withhuman_vocab.inv_machine_vocab: the inverse dictionary ofmachine_vocab, mapping from indices back to characters.
Let's preprocess the data and map the raw text data into the index values. We will also use Tx=30 (which we assume is the maximum length of the human readable date; if we get a longer input, we would have to truncate it) and Ty=10 (since "YYYY-MM-DD" is 10 characters long).
You now have:
X: a processed version of the human readable dates in the training set, where each character is replaced by an index mapped to the character viahuman_vocab. Each date is further padded to values with a special character (< pad >).X.shape = (m, Tx)Y: a processed version of the machine readable dates in the training set, where each character is replaced by the index it is mapped to inmachine_vocab. You should haveY.shape = (m, Ty).Xoh: one-hot version ofX, the "1" entry's index is mapped to the character thanks tohuman_vocab.Xoh.shape = (m, Tx, len(human_vocab))Yoh: one-hot version ofY, the "1" entry's index is mapped to the character thanks tomachine_vocab.Yoh.shape = (m, Tx, len(machine_vocab)). Here,len(machine_vocab) = 11since there are 11 characters ('-' as well as 0-9).
Lets also look at some examples of preprocessed training examples. Feel free to play with index in the cell below to navigate the dataset and see how source/target dates are preprocessed.
2 - Neural machine translation with attention
If you had to translate a book's paragraph from French to English, you would not read the whole paragraph, then close the book and translate. Even during the translation process, you would read/re-read and focus on the parts of the French paragraph corresponding to the parts of the English you are writing down.
The attention mechanism tells a Neural Machine Translation model where it should pay attention to at any step.
2.1 - Attention mechanism
In this part, you will implement the attention mechanism presented in the lecture videos. Here is a figure to remind you how the model works. The diagram on the left shows the attention model. The diagram on the right shows what one "Attention" step does to calculate the attention variables , which are used to compute the context variable for each timestep in the output ().
 |
 |
Here are some properties of the model that you may notice:
There are two separate LSTMs in this model (see diagram on the left). Because the one at the bottom of the picture is a Bi-directional LSTM and comes before the attention mechanism, we will call it pre-attention Bi-LSTM. The LSTM at the top of the diagram comes after the attention mechanism, so we will call it the post-attention LSTM. The pre-attention Bi-LSTM goes through time steps; the post-attention LSTM goes through time steps.
The post-attention LSTM passes from one time step to the next. In the lecture videos, we were using only a basic RNN for the post-activation sequence model, so the state captured by the RNN output activations . But since we are using an LSTM here, the LSTM has both the output activation and the hidden cell state . However, unlike previous text generation examples (such as Dinosaurus in week 1), in this model the post-activation LSTM at time does will not take the specific generated as input; it only takes and as input. We have designed the model this way, because (unlike language generation where adjacent characters are highly correlated) there isn't as strong a dependency between the previous character and the next character in a YYYY-MM-DD date.
We use to represent the concatenation of the activations of both the forward-direction and backward-directions of the pre-attention Bi-LSTM.
The diagram on the right uses a
RepeatVectornode to copy 's value times, and thenConcatenationto concatenate and to compute , which is then passed through a softmax to compute . We'll explain how to useRepeatVectorandConcatenationin Keras below.
Lets implement this model. You will start by implementing two functions: one_step_attention() and model().
1) one_step_attention(): At step , given all the hidden states of the Bi-LSTM () and the previous hidden state of the second LSTM (), one_step_attention() will compute the attention weights () and output the context vector (see Figure 1 (right) for details):
Note that we are denoting the attention in this notebook . In the lecture videos, the context was denoted , but here we are calling it to avoid confusion with the (post-attention) LSTM's internal memory cell variable, which is sometimes also denoted .
2) model(): Implements the entire model. It first runs the input through a Bi-LSTM to get back . Then, it calls one_step_attention() times (for loop). At each iteration of this loop, it gives the computed context vector to the second LSTM, and runs the output of the LSTM through a dense layer with softmax activation to generate a prediction .
Exercise: Implement one_step_attention(). The function model() will call the layers in one_step_attention() using a for-loop, and it is important that all copies have the same weights. I.e., it should not re-initiaiize the weights every time. In other words, all steps should have shared weights. Here's how you can implement layers with shareable weights in Keras:
Define the layer objects (as global variables for examples).
Call these objects when propagating the input.
We have defined the layers you need as global variables. Please run the following cells to create them. Please check the Keras documentation to make sure you understand what these layers are: RepeatVector(), Concatenate(), Dense(), Activation(), Dot().
Init signature: Dense(*args, **kwargs)
Source:
@keras_export("keras.layers.Dense")
class Dense(Layer):
"""Just your regular densely-connected NN layer.
`Dense` implements the operation:
`output = activation(dot(input, kernel) + bias)`
where `activation` is the element-wise activation function
passed as the `activation` argument, `kernel` is a weights matrix
created by the layer, and `bias` is a bias vector created by the layer
(only applicable if `use_bias` is `True`).
Note: If the input to the layer has a rank greater than 2, `Dense`
computes the dot product between the `inputs` and the `kernel` along the
last axis of the `inputs` and axis 0 of the `kernel` (using `tf.tensordot`).
For example, if input has dimensions `(batch_size, d0, d1)`, then we create
a `kernel` with shape `(d1, units)`, and the `kernel` operates along axis 2
of the `input`, on every sub-tensor of shape `(1, 1, d1)` (there are
`batch_size * d0` such sub-tensors). The output in this case will have
shape `(batch_size, d0, units)`.
Args:
units: Positive integer, dimensionality of the output space.
activation: Activation function to use.
If you don't specify anything, no activation is applied
(ie. "linear" activation: `a(x) = x`).
use_bias: Boolean, whether the layer uses a bias vector.
kernel_initializer: Initializer for the `kernel` weights matrix.
bias_initializer: Initializer for the bias vector.
kernel_regularizer: Regularizer function applied to
the `kernel` weights matrix.
bias_regularizer: Regularizer function applied to the bias vector.
activity_regularizer: Regularizer function applied to
the output of the layer (its "activation").
kernel_constraint: Constraint function applied to
the `kernel` weights matrix.
bias_constraint: Constraint function applied to the bias vector.
lora_rank: Optional integer. If set, the layer's forward pass
will implement LoRA (Low-Rank Adaptation)
with the provided rank. LoRA sets the layer's kernel
to non-trainable and replaces it with a delta over the
original kernel, obtained via multiplying two lower-rank
trainable matrices. This can be useful to reduce the
computation cost of fine-tuning large dense layers.
You can also enable LoRA on an existing
`Dense` layer by calling `layer.enable_lora(rank)`.
Input shape:
N-D tensor with shape: `(batch_size, ..., input_dim)`.
The most common situation would be
a 2D input with shape `(batch_size, input_dim)`.
Output shape:
N-D tensor with shape: `(batch_size, ..., units)`.
For instance, for a 2D input with shape `(batch_size, input_dim)`,
the output would have shape `(batch_size, units)`.
"""
def __init__(
self,
units,
activation=None,
use_bias=True,
kernel_initializer="glorot_uniform",
bias_initializer="zeros",
kernel_regularizer=None,
bias_regularizer=None,
activity_regularizer=None,
kernel_constraint=None,
bias_constraint=None,
lora_rank=None,
**kwargs,
):
super().__init__(activity_regularizer=activity_regularizer, **kwargs)
self.units = units
self.activation = activations.get(activation)
self.use_bias = use_bias
self.kernel_initializer = initializers.get(kernel_initializer)
self.bias_initializer = initializers.get(bias_initializer)
self.kernel_regularizer = regularizers.get(kernel_regularizer)
self.bias_regularizer = regularizers.get(bias_regularizer)
self.kernel_constraint = constraints.get(kernel_constraint)
self.bias_constraint = constraints.get(bias_constraint)
self.lora_rank = lora_rank
self.lora_enabled = False
self.input_spec = InputSpec(min_ndim=2)
self.supports_masking = True
def build(self, input_shape):
input_dim = input_shape[-1]
if self.quantization_mode:
self.quantized_build(input_shape, mode=self.quantization_mode)
if self.quantization_mode != "int8":
# If the layer is quantized to int8, `self._kernel` will be added
# in `self._int8_build`. Therefore, we skip it here.
self._kernel = self.add_weight(
name="kernel",
shape=(input_dim, self.units),
initializer=self.kernel_initializer,
regularizer=self.kernel_regularizer,
constraint=self.kernel_constraint,
)
if self.use_bias:
self.bias = self.add_weight(
name="bias",
shape=(self.units,),
initializer=self.bias_initializer,
regularizer=self.bias_regularizer,
constraint=self.bias_constraint,
)
else:
self.bias = None
self.input_spec = InputSpec(min_ndim=2, axes={-1: input_dim})
self.built = True
if self.lora_rank:
self.enable_lora(self.lora_rank)
@property
def kernel(self):
if not self.built:
raise AttributeError(
"You must build the layer before accessing `kernel`."
)
if self.lora_enabled:
return self._kernel + ops.matmul(
self.lora_kernel_a, self.lora_kernel_b
)
return self._kernel
def call(self, inputs, training=None):
x = ops.matmul(inputs, self.kernel)
if self.bias is not None:
x = ops.add(x, self.bias)
if self.activation is not None:
x = self.activation(x)
return x
def compute_output_shape(self, input_shape):
output_shape = list(input_shape)
output_shape[-1] = self.units
return tuple(output_shape)
def enable_lora(
self, rank, a_initializer="he_uniform", b_initializer="zeros"
):
if self.kernel_constraint:
raise ValueError(
"Lora is incompatible with kernel constraints. "
"In order to enable lora on this layer, remove the "
"`kernel_constraint` argument."
)
if not self.built:
raise ValueError(
"Cannot enable lora on a layer that isn't yet built."
)
if self.lora_enabled:
raise ValueError(
"lora is already enabled. "
"This can only be done once per layer."
)
self._tracker.unlock()
self.lora_kernel_a = self.add_weight(
name="lora_kernel_a",
shape=(self.kernel.shape[0], rank),
initializer=initializers.get(a_initializer),
regularizer=self.kernel_regularizer,
)
self.lora_kernel_b = self.add_weight(
name="lora_kernel_b",
shape=(rank, self.kernel.shape[1]),
initializer=initializers.get(b_initializer),
regularizer=self.kernel_regularizer,
)
self._kernel.trainable = False
self._tracker.lock()
self.lora_enabled = True
self.lora_rank = rank
def save_own_variables(self, store):
# Do nothing if the layer isn't yet built
if not self.built:
return
# The keys of the `store` will be saved as determined because the
# default ordering will change after quantization
kernel_value, kernel_scale = self._get_kernel_with_merged_lora()
target_variables = [kernel_value]
if self.use_bias:
target_variables.append(self.bias)
if self.quantization_mode is not None:
if self.quantization_mode == "int8":
target_variables.append(kernel_scale)
elif self.quantization_mode == "float8":
target_variables.append(self.inputs_scale)
target_variables.append(self.inputs_amax_history)
target_variables.append(self.kernel_scale)
target_variables.append(self.kernel_amax_history)
target_variables.append(self.outputs_grad_scale)
target_variables.append(self.outputs_grad_amax_history)
else:
raise self._quantization_mode_error(self.quantization_mode)
for i, variable in enumerate(target_variables):
store[str(i)] = variable
def load_own_variables(self, store):
if not self.lora_enabled:
self._check_load_own_variables(store)
# Do nothing if the layer isn't yet built
if not self.built:
return
# The keys of the `store` will be saved as determined because the
# default ordering will change after quantization
target_variables = [self._kernel]
if self.use_bias:
target_variables.append(self.bias)
if self.quantization_mode is not None:
if self.quantization_mode == "int8":
target_variables.append(self.kernel_scale)
elif self.quantization_mode == "float8":
target_variables.append(self.inputs_scale)
target_variables.append(self.inputs_amax_history)
target_variables.append(self.kernel_scale)
target_variables.append(self.kernel_amax_history)
target_variables.append(self.outputs_grad_scale)
target_variables.append(self.outputs_grad_amax_history)
else:
raise self._quantization_mode_error(self.quantization_mode)
for i, variable in enumerate(target_variables):
variable.assign(store[str(i)])
if self.lora_enabled:
self.lora_kernel_a.assign(ops.zeros(self.lora_kernel_a.shape))
self.lora_kernel_b.assign(ops.zeros(self.lora_kernel_b.shape))
def get_config(self):
base_config = super().get_config()
config = {
"units": self.units,
"activation": activations.serialize(self.activation),
"use_bias": self.use_bias,
"kernel_initializer": initializers.serialize(
self.kernel_initializer
),
"bias_initializer": initializers.serialize(self.bias_initializer),
"kernel_regularizer": regularizers.serialize(
self.kernel_regularizer
),
"bias_regularizer": regularizers.serialize(self.bias_regularizer),
"kernel_constraint": constraints.serialize(self.kernel_constraint),
"bias_constraint": constraints.serialize(self.bias_constraint),
}
if self.lora_rank:
config["lora_rank"] = self.lora_rank
return {**base_config, **config}
def _check_load_own_variables(self, store):
all_vars = self._trainable_variables + self._non_trainable_variables
if len(store.keys()) != len(all_vars):
if len(all_vars) == 0 and not self.built:
raise ValueError(
f"Layer '{self.name}' was never built "
"and thus it doesn't have any variables. "
f"However the weights file lists {len(store.keys())} "
"variables for this layer.\n"
"In most cases, this error indicates that either:\n\n"
"1. The layer is owned by a parent layer that "
"implements a `build()` method, but calling the "
"parent's `build()` method did NOT create the state of "
f"the child layer '{self.name}'. A `build()` method "
"must create ALL state for the layer, including "
"the state of any children layers.\n\n"
"2. You need to implement "
"the `def build_from_config(self, config)` method "
f"on layer '{self.name}', to specify how to rebuild "
"it during loading. "
"In this case, you might also want to implement the "
"method that generates the build config at saving time, "
"`def get_build_config(self)`. "
"The method `build_from_config()` is meant "
"to create the state "
"of the layer (i.e. its variables) upon deserialization.",
)
raise ValueError(
f"Layer '{self.name}' expected {len(all_vars)} variables, "
"but received "
f"{len(store.keys())} variables during loading. "
f"Expected: {[v.name for v in all_vars]}"
)
# Quantization-related (int8 and float8) methods
def quantized_build(self, input_shape, mode):
if mode == "int8":
input_dim = input_shape[-1]
kernel_shape = (input_dim, self.units)
self._int8_build(kernel_shape)
elif mode == "float8":
self._float8_build()
else:
raise self._quantization_mode_error(mode)
def _int8_build(
self,
kernel_shape,
kernel_initializer="zeros",
kernel_scale_initializer="ones",
):
self.inputs_quantizer = quantizers.AbsMaxQuantizer(axis=-1)
self._kernel = self.add_weight(
name="kernel",
shape=kernel_shape,
initializer=kernel_initializer,
dtype="int8",
trainable=False,
)
self.kernel_scale = self.add_weight(
name="kernel_scale",
shape=(self.units,),
initializer=kernel_scale_initializer,
trainable=False,
)
self._is_quantized = True
def _float8_build(self):
from keras.src.dtype_policies import QuantizedFloat8DTypePolicy
# If `self.dtype_policy` is not QuantizedFloat8DTypePolicy, then set
# `amax_history_length` to its default value.
amax_history_length = getattr(
self.dtype_policy,
"amax_history_length",
QuantizedFloat8DTypePolicy.default_amax_history_length,
)
# We set `trainable=True` because we will use the gradients to overwrite
# these variables
scale_kwargs = {
"shape": (),
"initializer": "ones",
"dtype": "float32", # Always be float32
"trainable": True,
"autocast": False,
}
amax_history_kwargs = {
"shape": (amax_history_length,),
"initializer": "zeros",
"dtype": "float32", # Always be float32
"trainable": True,
"autocast": False,
}
self.inputs_scale = self.add_weight(name="inputs_scale", **scale_kwargs)
self.inputs_amax_history = self.add_weight(
name="inputs_amax_history", **amax_history_kwargs
)
self.kernel_scale = self.add_weight(name="kernel_scale", **scale_kwargs)
self.kernel_amax_history = self.add_weight(
name="kernel_amax_history", **amax_history_kwargs
)
self.outputs_grad_scale = self.add_weight(
name="outputs_grad_scale", **scale_kwargs
)
self.outputs_grad_amax_history = self.add_weight(
name="outputs_grad_amax_history", **amax_history_kwargs
)
# We need to set `overwrite_with_gradient=True` to instruct the
# optimizer to directly overwrite these variables with their computed
# gradients during training
self.inputs_scale.overwrite_with_gradient = True
self.inputs_amax_history.overwrite_with_gradient = True
self.kernel_scale.overwrite_with_gradient = True
self.kernel_amax_history.overwrite_with_gradient = True
self.outputs_grad_scale.overwrite_with_gradient = True
self.outputs_grad_amax_history.overwrite_with_gradient = True
self._is_quantized = True
def _int8_call(self, inputs, training=None):
@ops.custom_gradient
def matmul_with_inputs_gradient(inputs, kernel, kernel_scale):
def grad_fn(*args, upstream=None):
if upstream is None:
(upstream,) = args
float_kernel = ops.divide(
ops.cast(kernel, dtype=self.compute_dtype),
kernel_scale,
)
inputs_grad = ops.matmul(upstream, ops.transpose(float_kernel))
return (inputs_grad, None, None)
inputs, inputs_scale = self.inputs_quantizer(inputs)
x = ops.matmul(inputs, kernel)
# De-scale outputs
x = ops.cast(x, self.compute_dtype)
x = ops.divide(x, ops.multiply(inputs_scale, kernel_scale))
return x, grad_fn
x = matmul_with_inputs_gradient(
inputs,
ops.convert_to_tensor(self._kernel),
ops.convert_to_tensor(self.kernel_scale),
)
if self.lora_enabled:
lora_x = ops.matmul(inputs, self.lora_kernel_a)
lora_x = ops.matmul(lora_x, self.lora_kernel_b)
x = ops.add(x, lora_x)
if self.bias is not None:
x = ops.add(x, self.bias)
if self.activation is not None:
x = self.activation(x)
return x
def _float8_call(self, inputs, training=None):
if self.lora_enabled:
raise NotImplementedError(
"Currently, `_float8_call` doesn't support LoRA"
)
@ops.custom_gradient
def quantized_dequantize_inputs(inputs, scale, amax_history):
if training:
new_scale = quantizers.compute_float8_scale(
ops.max(amax_history, axis=0),
scale,
ops.cast(
float(ml_dtypes.finfo("float8_e4m3fn").max), "float32"
),
)
new_amax_history = quantizers.compute_float8_amax_history(
inputs, amax_history
)
else:
new_scale = None
new_amax_history = None
qdq_inputs = quantizers.quantize_and_dequantize(
inputs, scale, "float8_e4m3fn", self.compute_dtype
)
def grad(*args, upstream=None, variables=None):
if upstream is None:
(upstream,) = args
return upstream, new_scale, new_amax_history
return qdq_inputs, grad
@ops.custom_gradient
def quantized_dequantize_outputs(outputs, scale, amax_history):
"""Quantize-dequantize the output gradient but not the output."""
def grad(*args, upstream=None, variables=None):
if upstream is None:
(upstream,) = args
new_scale = quantizers.compute_float8_scale(
ops.max(amax_history, axis=0),
scale,
ops.cast(
float(ml_dtypes.finfo("float8_e5m2").max), "float32"
),
)
qdq_upstream = quantizers.quantize_and_dequantize(
upstream, scale, "float8_e5m2", self.compute_dtype
)
new_amax_history = quantizers.compute_float8_amax_history(
upstream, amax_history
)
return qdq_upstream, new_scale, new_amax_history
return outputs, grad
x = ops.matmul(
quantized_dequantize_inputs(
inputs,
ops.convert_to_tensor(self.inputs_scale),
ops.convert_to_tensor(self.inputs_amax_history),
),
quantized_dequantize_inputs(
ops.convert_to_tensor(self._kernel),
ops.convert_to_tensor(self.kernel_scale),
ops.convert_to_tensor(self.kernel_amax_history),
),
)
# `quantized_dequantize_outputs` is placed immediately after
# `ops.matmul` for the sake of pattern matching in gemm_rewrite. That
# way, the qdq will be adjacent to the corresponding matmul_bprop in the
# bprop.
x = quantized_dequantize_outputs(
x,
ops.convert_to_tensor(self.outputs_grad_scale),
ops.convert_to_tensor(self.outputs_grad_amax_history),
)
if self.bias is not None:
# Under non-mixed precision cases, F32 bias has to be converted to
# BF16 first to get the biasAdd fusion support. ref. PR
# https://github.com/tensorflow/tensorflow/pull/60306
bias = self.bias
if self.dtype_policy.compute_dtype == "float32":
bias_bf16 = ops.cast(bias, "bfloat16")
bias = ops.cast(bias_bf16, bias.dtype)
x = ops.add(x, bias)
if self.activation is not None:
x = self.activation(x)
return x
def quantize(self, mode, type_check=True):
# Prevent quantization of the subclasses
if type_check and (type(self) is not Dense):
raise self._not_implemented_error(self.quantize)
if mode == "int8":
# Quantize `self._kernel` to int8 and compute corresponding scale
kernel_value, kernel_scale = quantizers.abs_max_quantize(
self._kernel, axis=0, to_numpy=True
)
kernel_scale = ops.squeeze(kernel_scale, axis=0)
kernel_shape = tuple(self._kernel.shape)
del self._kernel
# Utilize a lambda expression as an initializer to prevent adding a
# large constant to the computation graph.
self._int8_build(kernel_shape, kernel_value, kernel_scale)
elif mode == "float8":
self._float8_build()
else:
raise self._quantization_mode_error(mode)
# Set new dtype policy
if self.dtype_policy.quantization_mode is None:
policy = dtype_policies.get(f"{mode}_from_{self.dtype_policy.name}")
self.dtype_policy = policy
def _get_kernel_with_merged_lora(self):
if self.dtype_policy.quantization_mode is not None:
kernel_value = self._kernel
kernel_scale = self.kernel_scale
if self.lora_enabled:
# Dequantize & quantize to merge lora weights into int8 kernel
# Note that this is a lossy compression
kernel_value = ops.divide(kernel_value, kernel_scale)
kernel_value = ops.add(
kernel_value,
ops.matmul(self.lora_kernel_a, self.lora_kernel_b),
)
kernel_value, kernel_scale = quantizers.abs_max_quantize(
kernel_value, axis=0, to_numpy=True
)
kernel_scale = ops.squeeze(kernel_scale, axis=0)
return kernel_value, kernel_scale
return self.kernel, None
File: ~/Library/Python/3.9/lib/python/site-packages/keras/src/layers/core/dense.py
Type: type
Subclasses: Now you can use these layers to implement one_step_attention(). In order to propagate a Keras tensor object X through one of these layers, use layer(X) (or layer([X,Y]) if it requires multiple inputs.), e.g. densor(X) will propagate X through the Dense(1) layer defined above.
You will be able to check the expected output of one_step_attention() after you've coded the model() function.
Exercise: Implement model() as explained in figure 2 and the text above. Again, we have defined global layers that will share weights to be used in model().
Now you can use these layers times in a for loop to generate the outputs, and their parameters will not be reinitialized. You will have to carry out the following steps:
Propagate the input into a Bidirectional LSTM
Iterate for :
Call
one_step_attention()on and to get the context vector .Give to the post-attention LSTM cell. Remember pass in the previous hidden-state and cell-states of this LSTM using
initial_state= [previous hidden state, previous cell state]. Get back the new hidden state and the new cell state .Apply a softmax layer to , get the output.
Save the output by adding it to the list of outputs.
Create your Keras model instance, it should have three inputs ("inputs", and ) and output the list of "outputs".
Run the following cell to create your model.
Let's get a summary of the model to check if it matches the expected output.
Expected Output:
Here is the summary you should see
| **Total params:** | 52,960 |
| **Trainable params:** | 52,960 |
| **Non-trainable params:** | 0 |
| **bidirectional_1's output shape ** | (None, 30, 64) |
| **repeat_vector_1's output shape ** | (None, 30, 64) |
| **concatenate_1's output shape ** | (None, 30, 128) |
| **attention_weights's output shape ** | (None, 30, 1) |
| **dot_1's output shape ** | (None, 1, 64) |
| **dense_3's output shape ** | (None, 11) |
The last step is to define all your inputs and outputs to fit the model:
You already have X of shape containing the training examples.
You need to create
s0andc0to initialize yourpost_attention_LSTM_cellwith 0s.Given the
model()you coded, you need the "outputs" to be a list of 11 elements of shape (m, T_y). So that:outputs[i][0], ..., outputs[i][Ty]represent the true labels (characters) corresponding to the training example (X[i]). More generally,outputs[i][j]is the true label of the character in the training example.
Let's now fit the model and run it for one epoch.
Epoch 1/10
100/100 ━━━━━━━━━━━━━━━━━━━━ 4s 40ms/step - dense_2_accuracy: 1.0000 - dense_2_accuracy_1: 1.0000 - dense_2_accuracy_2: 0.9998 - dense_2_accuracy_3: 0.9981 - dense_2_accuracy_4: 1.0000 - dense_2_accuracy_5: 1.0000 - dense_2_accuracy_6: 1.0000 - dense_2_accuracy_7: 1.0000 - dense_2_accuracy_8: 1.0000 - dense_2_accuracy_9: 1.0000 - dense_2_loss: 0.0010 - loss: 0.0244
Epoch 2/10
100/100 ━━━━━━━━━━━━━━━━━━━━ 4s 40ms/step - dense_2_accuracy: 0.9999 - dense_2_accuracy_1: 0.9999 - dense_2_accuracy_2: 0.9995 - dense_2_accuracy_3: 0.9966 - dense_2_accuracy_4: 1.0000 - dense_2_accuracy_5: 1.0000 - dense_2_accuracy_6: 1.0000 - dense_2_accuracy_7: 1.0000 - dense_2_accuracy_8: 1.0000 - dense_2_accuracy_9: 1.0000 - dense_2_loss: 9.0653e-04 - loss: 0.0315
Epoch 3/10
100/100 ━━━━━━━━━━━━━━━━━━━━ 4s 40ms/step - dense_2_accuracy: 0.9998 - dense_2_accuracy_1: 0.9999 - dense_2_accuracy_2: 0.9988 - dense_2_accuracy_3: 0.9973 - dense_2_accuracy_4: 1.0000 - dense_2_accuracy_5: 0.9997 - dense_2_accuracy_6: 0.9996 - dense_2_accuracy_7: 1.0000 - dense_2_accuracy_8: 0.9975 - dense_2_accuracy_9: 0.9990 - dense_2_loss: 0.0038 - loss: 0.0394
Epoch 4/10
100/100 ━━━━━━━━━━━━━━━━━━━━ 4s 41ms/step - dense_2_accuracy: 0.9968 - dense_2_accuracy_1: 0.9975 - dense_2_accuracy_2: 0.9781 - dense_2_accuracy_3: 0.9817 - dense_2_accuracy_4: 0.9997 - dense_2_accuracy_5: 0.9896 - dense_2_accuracy_6: 0.9882 - dense_2_accuracy_7: 0.9997 - dense_2_accuracy_8: 0.9610 - dense_2_accuracy_9: 0.9928 - dense_2_loss: 0.0390 - loss: 0.4622
Epoch 5/10
100/100 ━━━━━━━━━━━━━━━━━━━━ 4s 41ms/step - dense_2_accuracy: 0.9997 - dense_2_accuracy_1: 0.9997 - dense_2_accuracy_2: 0.9995 - dense_2_accuracy_3: 0.9979 - dense_2_accuracy_4: 1.0000 - dense_2_accuracy_5: 0.9998 - dense_2_accuracy_6: 1.0000 - dense_2_accuracy_7: 1.0000 - dense_2_accuracy_8: 0.9999 - dense_2_accuracy_9: 1.0000 - dense_2_loss: 0.0014 - loss: 0.0414
Epoch 6/10
100/100 ━━━━━━━━━━━━━━━━━━━━ 4s 41ms/step - dense_2_accuracy: 0.9999 - dense_2_accuracy_1: 0.9999 - dense_2_accuracy_2: 0.9998 - dense_2_accuracy_3: 0.9979 - dense_2_accuracy_4: 1.0000 - dense_2_accuracy_5: 0.9999 - dense_2_accuracy_6: 1.0000 - dense_2_accuracy_7: 1.0000 - dense_2_accuracy_8: 0.9998 - dense_2_accuracy_9: 1.0000 - dense_2_loss: 8.5868e-04 - loss: 0.0283
Epoch 7/10
100/100 ━━━━━━━━━━━━━━━━━━━━ 4s 44ms/step - dense_2_accuracy: 0.9998 - dense_2_accuracy_1: 0.9998 - dense_2_accuracy_2: 0.9998 - dense_2_accuracy_3: 0.9974 - dense_2_accuracy_4: 1.0000 - dense_2_accuracy_5: 1.0000 - dense_2_accuracy_6: 1.0000 - dense_2_accuracy_7: 1.0000 - dense_2_accuracy_8: 1.0000 - dense_2_accuracy_9: 1.0000 - dense_2_loss: 6.9754e-04 - loss: 0.0281
Epoch 8/10
100/100 ━━━━━━━━━━━━━━━━━━━━ 4s 42ms/step - dense_2_accuracy: 0.9999 - dense_2_accuracy_1: 0.9999 - dense_2_accuracy_2: 0.9997 - dense_2_accuracy_3: 0.9979 - dense_2_accuracy_4: 1.0000 - dense_2_accuracy_5: 1.0000 - dense_2_accuracy_6: 1.0000 - dense_2_accuracy_7: 1.0000 - dense_2_accuracy_8: 1.0000 - dense_2_accuracy_9: 1.0000 - dense_2_loss: 5.5270e-04 - loss: 0.0207
Epoch 9/10
100/100 ━━━━━━━━━━━━━━━━━━━━ 4s 43ms/step - dense_2_accuracy: 0.9999 - dense_2_accuracy_1: 0.9999 - dense_2_accuracy_2: 0.9999 - dense_2_accuracy_3: 0.9973 - dense_2_accuracy_4: 1.0000 - dense_2_accuracy_5: 1.0000 - dense_2_accuracy_6: 1.0000 - dense_2_accuracy_7: 1.0000 - dense_2_accuracy_8: 1.0000 - dense_2_accuracy_9: 1.0000 - dense_2_loss: 4.4554e-04 - loss: 0.0223
Epoch 10/10
100/100 ━━━━━━━━━━━━━━━━━━━━ 4s 44ms/step - dense_2_accuracy: 1.0000 - dense_2_accuracy_1: 1.0000 - dense_2_accuracy_2: 0.9999 - dense_2_accuracy_3: 0.9979 - dense_2_accuracy_4: 1.0000 - dense_2_accuracy_5: 1.0000 - dense_2_accuracy_6: 1.0000 - dense_2_accuracy_7: 1.0000 - dense_2_accuracy_8: 1.0000 - dense_2_accuracy_9: 1.0000 - dense_2_loss: 3.9239e-04 - loss: 0.0164
While training you can see the loss as well as the accuracy on each of the 10 positions of the output. The table below gives you an example of what the accuracies could be if the batch had 2 examples:
We have run this model for longer, and saved the weights. Run the next cell to load our weights. (By training a model for several minutes, you should be able to obtain a model of similar accuracy, but loading our model will save you time.)
0 (30, 37) <class 'numpy.ndarray'> ('29 nov 1992', '1992-11-29')
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step
['1', '9', '9', '2', '-', '1', '1', '-', '2', '9']
1 (30, 37) <class 'numpy.ndarray'> ('24.07.70', '1970-07-24')
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 25ms/step
['1', '9', '7', '0', '-', '0', '7', '-', '2', '4']
2 (30, 37) <class 'numpy.ndarray'> ('5/19/15', '2015-05-19')
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 24ms/step
['2', '0', '1', '5', '-', '0', '5', '-', '1', '9']
3 (30, 37) <class 'numpy.ndarray'> ('wednesday june 4 1986', '1986-06-04')
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step
['1', '9', '8', '6', '-', '0', '6', '-', '0', '4']
4 (30, 37) <class 'numpy.ndarray'> ('thursday april 5 1990', '1990-04-05')
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step
['1', '9', '9', '0', '-', '0', '4', '-', '0', '5']
5 (30, 37) <class 'numpy.ndarray'> ('monday august 25 1980', '1980-08-25')
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step
['1', '9', '8', '0', '-', '0', '8', '-', '2', '5']
6 (30, 37) <class 'numpy.ndarray'> ('thursday february 15 2001', '2001-02-15')
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step
['2', '0', '0', '1', '-', '0', '2', '-', '1', '5']
7 (30, 37) <class 'numpy.ndarray'> ('22 nov 1978', '1978-11-22')
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step
['1', '9', '7', '8', '-', '1', '1', '-', '2', '2']
8 (30, 37) <class 'numpy.ndarray'> ('31 oct 1976', '1976-10-31')
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step
['1', '9', '7', '6', '-', '1', '0', '-', '3', '1']
9 (30, 37) <class 'numpy.ndarray'> ('friday october 22 1993', '1993-10-22')
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step
['1', '9', '9', '3', '-', '1', '0', '-', '2', '2']
10 (30, 37) <class 'numpy.ndarray'> ('tuesday november 28 2000', '2000-11-28')
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step
['2', '0', '0', '0', '-', '1', '1', '-', '2', '8']
You can now see the results on new examples.
(30, 37)
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 26ms/step
source: 22rd maY 1994
output: 1994-05-22
(30, 37)
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step
source: 5 April 09
output: 2009-04-05
(30, 37)
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step
source: 21th of August 2016
output: 2016-08-21
(30, 37)
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step
source: Tue 10 Jul 2007
output: 2007-07-10
(30, 37)
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step
source: Saturday May 9 2018
output: 2018-05-09
(30, 37)
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 13ms/step1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step
source: March 3 2001
output: 2001-03-03
(30, 37)
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 21ms/step
source: March 3rd 2001
output: 2001-03-03
(30, 37)
1/1 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step
source: 1 March 2001
output: 2001-03-01
You can also change these examples to test with your own examples. The next part will give you a better sense on what the attention mechanism is doing--i.e., what part of the input the network is paying attention to when generating a particular output character.
3 - Visualizing Attention (Optional / Ungraded)
Since the problem has a fixed output length of 10, it is also possible to carry out this task using 10 different softmax units to generate the 10 characters of the output. But one advantage of the attention model is that each part of the output (say the month) knows it needs to depend only on a small part of the input (the characters in the input giving the month). We can visualize what part of the output is looking at what part of the input.
Consider the task of translating "Saturday 9 May 2018" to "2018-05-09". If we visualize the computed we get this:
Notice how the output ignores the "Saturday" portion of the input. None of the output timesteps are paying much attention to that portion of the input. We see also that 9 has been translated as 09 and May has been correctly translated into 05, with the output paying attention to the parts of the input it needs to to make the translation. The year mostly requires it to pay attention to the input's "18" in order to generate "2018."
3.1 - Getting the activations from the network
Lets now visualize the attention values in your network. We'll propagate an example through the network, then visualize the values of .
To figure out where the attention values are located, let's start by printing a summary of the model .
Navigate through the output of model.summary() above. You can see that the layer named attention_weights outputs the alphas of shape (m, 30, 1) before dot_2 computes the context vector for every time step . Lets get the activations from this layer.
The function attention_map() pulls out the attention values from your model and plots them.
On the generated plot you can observe the values of the attention weights for each character of the predicted output. Examine this plot and check that where the network is paying attention makes sense to you.
In the date translation application, you will observe that most of the time attention helps predict the year, and hasn't much impact on predicting the day/month.
Congratulations!
You have come to the end of this assignment
Here's what you should remember from this notebook:
Machine translation models can be used to map from one sequence to another. They are useful not just for translating human languages (like French->English) but also for tasks like date format translation.
An attention mechanism allows a network to focus on the most relevant parts of the input when producing a specific part of the output.
A network using an attention mechanism can translate from inputs of length to outputs of length , where and can be different.
You can visualize attention weights to see what the network is paying attention to while generating each output.
Congratulations on finishing this assignment! You are now able to implement an attention model and use it to learn complex mappings from one sequence to another.