experiment.ipynb

Kernel: Python 3 (ipykernel)

Fuzzy Tiling Activations

Here we train a transformer that uses Fuzzy Tiling Activation in the Feed-Forward Network. We use it for a language model and train it on Tiny Shakespeare dataset for demonstration. However, this is probably not the ideal task for FTA, and we believe FTA is more suitable for modeling data with continuous variables.

Install the packages

!pip install labml-nn --quiet

Imports

import torch
import torch.nn as nn

from labml import experiment
from labml.configs import option
from labml_nn.activations.fta.experiment import Configs

Create an experiment

experiment.create(name="fta", writers={'screen'})

Configurations

conf = Configs()

Set experiment configurations and assign a configurations dictionary to override configurations

experiment.configs(conf, {
    'tokenizer': 'character',
    'prompt_separator': '',
    'prompt': 'It is ',
    'text': 'tiny_shakespeare',

    'seq_len': 256,
    'epochs': 32,
    'batch_size': 16,
    'inner_iterations': 10,

    'optimizer.optimizer': 'Adam',
    'optimizer.learning_rate': 3e-4,
})

Set PyTorch models for loading and saving

experiment.add_pytorch_models({'model': conf.model})

Start the experiment and run the training loop.

# Start the experiment
with experiment.start():
    conf.run()