Real-time collaboration for Jupyter Notebooks, Linux Terminals, LaTeX, VS Code, R IDE, and more,
all in one place. Commercial Alternative to JupyterHub.
Real-time collaboration for Jupyter Notebooks, Linux Terminals, LaTeX, VS Code, R IDE, and more,
all in one place. Commercial Alternative to JupyterHub.
Path: blob/main/examples/patch_tsmixer.ipynb
Views: 2535
PatchTSMixer in HuggingFace - Getting Started
PatchTSMixer is a lightweight time-series modeling approach based on the MLP-Mixer architecture. It is proposed in TSMixer: Lightweight MLP-Mixer Model for Multivariate Time Series Forecasting by IBM Research authors Vijay Ekambaram, Arindam Jati, Nam Nguyen, Phanwadee Sinthong and Jayant Kalagnanam.
For effective mindshare and to promote open-sourcing - IBM Research joins hands with the HuggingFace team to release this model in the Transformers library.
In the Hugging Face implementation, we provide PatchTSMixer’s capabilities to effortlessly facilitate lightweight mixing across patches, channels, and hidden features for effective multivariate time-series modeling. It also supports various attention mechanisms starting from simple gated attention to more complex self-attention blocks that can be customized accordingly. The model can be pretrained and subsequently used for various downstream tasks such as forecasting, classification, and regression.
PatchTSMixer outperforms state-of-the-art MLP and Transformer models in forecasting by a considerable margin of 8-60%. It also outperforms the latest strong benchmarks of Patch-Transformer models (by 1-2%) with a significant reduction in memory and runtime (2-3X). For more details, refer to the paper.
In this blog, we will demonstrate examples of getting started with PatchTSMixer. We will first demonstrate the forecasting capability of PatchTSMixer on the Electricity dataset. We will then demonstrate the transfer learning capability of PatchTSMixer by using the model trained on Electricity to do zero-shot forecasting on the ETTH2 dataset.
Installation
This demo requires Hugging Face Transformers for the model and the IBM tsfm package for auxiliary data pre-processing. Both can be installed by following the steps below.
Install IBM Time Series Foundation Model Repository
tsfm.
Install Hugging Face
Transformers
Test it with the following commands in a
pythonterminal.
Part 1: Forecasting on Electricity dataset
Set seed
Load and prepare datasets
In the next cell, please adjust the following parameters to suit your application:
dataset_path: path to local .csv file, or web address to a csv file for the data of interest. Data is loaded with pandas, so anything supported bypd.read_csvis supported: (https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html).timestamp_column: column name containing timestamp information, useNoneif there is no such column.id_columns: List of column names specifying the IDs of different time series. If no ID column exists, use[].forecast_columns: List of columns to be modeled.context_length: The amount of historical data used as input to the model. Windows of the input time series data with length equal tocontext_lengthwill be extracted from the input dataframe. In the case of a multi-time series dataset, the context windows will be created so that they are contained within a single time series (i.e., a single ID).forecast_horizon: Number of timestamps to forecast in the future.train_start_index,train_end_index: the start and end indices in the loaded data which delineate the training data.valid_start_index,valid_end_index: the start and end indices in the loaded data which delineate the validation data.test_start_index,test_end_index: the start and end indices in the loaded data which delineate the test data.num_workers: Number of CPU workers in the PyTorch dataloader.batch_size: Batch size. The data is first loaded into a Pandas dataframe and split into training, validation, and test parts. Then the Pandas dataframes are converted to the appropriate PyTorch dataset required for training.
Configure the PatchTSMixer model
Next, we instantiate a randomly initialized PatchTSMixer model with a configuration. The settings below control the different hyperparameters related to the architecture.
num_input_channels: the number of input channels (or dimensions) in the time series data. This is automatically set to the number for forecast columns.context_length: As described above, the amount of historical data used as input to the model.prediction_length: This is same as the forecast horizon as described above.patch_length: The patch length for thePatchTSMixermodel. It is recommended to choose a value that evenly dividescontext_length.patch_stride: The stride used when extracting patches from the context window.d_model: Hidden feature dimension of the model.num_layers: The number of model layers.dropout: Dropout probability for all fully connected layers in the encoder.head_dropout: Dropout probability used in the head of the model.mode: PatchTSMixer operating mode. "common_channel"/"mix_channel". Common-channel works in channel-independent mode. For pretraining, use "common_channel".scaling: Per-widow standard scaling. Recommended value: "std".
For full details on the parameters, we refer to the documentation.
We recommend that you only adjust the values in the next cell.
Train model
Next, we can leverage the Hugging Face Trainer class to train the model based on the direct forecasting strategy. We first define the TrainingArguments which lists various hyperparameters regarding training such as the number of epochs, learning rate, and so on.
Evaluate the model on the test set.
Note that the training and evaluation loss for PatchTSMixer is the Mean Squared Error (MSE) loss. Hence, we do not separately compute the MSE metric in any of the following evaluation experiments.
We get MSE score of 0.128 which is the SOTA result on the Electricity data.
Save model
Part 2: Transfer Learning from Electricity to ETTH2
In this section, we will demonstrate the transfer learning capability of the PatchTSMixer model. We use the model pre-trained on the Electricity dataset to do zero-shot forecasting on the ETTH2 dataset.
By Transfer Learning, we mean that we first pretrain the model for a forecasting task on a source dataset (which we did above on the Electricity dataset). Then, we will use the pretrained model for zero-shot forecasting on a target dataset. By zero-shot, we mean that we test the performance in the target domain without any additional training. We hope that the model gained enough knowledge from pretraining which can be transferred to a different dataset.
Subsequently, we will do linear probing and (then) finetuning of the pretrained model on the train split of the target data, and will validate the forecasting performance on the test split of the target data. In this example, the source dataset is the Electricity dataset and the target dataset is ETTH2.
Transfer Learning on ETTh2 data
All evaluations are on the test part of the ETTh2 data:
Step 1: Directly evaluate the electricity-pretrained model. This is the zero-shot performance. Step 2: Evaluate after doing linear probing. Step 3: Evaluate after doing full finetuning.
Load ETTh2 dataset
Below, we load the ETTh2 dataset as a Pandas dataframe. Next, we create 3 splits for training, validation and testing. We then leverage the TimeSeriesPreprocessor class to prepare each split for the model.
Zero-shot forecasting on ETTh2
As we are going to test forecasting performance out-of-the-box, we load the model which we pretrained above.
As can be seen, we get a mean-squared error (MSE) of 0.3 zero-shot which is near to the state-of-the-art result.
Next, let's see how we can do by performing linear probing, which involves training a linear classifier on top of a frozen pre-trained model. Linear probing is often done to test the performance of features of a pretrained model.
Linear probing on ETTh2
We can do a quick linear probing on the train part of the target data to see any possible test performance improvement.
As can be seen, by training a simple linear layer on top of the frozen backbone, the MSE decreased from 0.3 to 0.271 achieving state-of-the-art results.
Finally, let's see if we get any more improvements by doing a full finetune of the model on the target dataset.
Full finetuning on ETTh2
We can do a full model finetune (instead of probing the last linear layer as shown above) on the train part of the target data to see a possible test performance improvement. The code looks similar to the linear probing task above, except that we are not freezing any parameters.
In this case, there is not much improvement by doing full finetuning. Let's save the model anyway.
Summary
In this blog, we presented a step-by-step guide on leveraging PatchTSMixer for tasks related to forecasting and transfer learning. We intend to facilitate the seamless integration of the PatchTSMixer HF model for your forecasting use cases. We trust that this content serves as a useful resource to expedite your adoption of PatchTSMixer. Thank you for tuning in to our blog, and we hope you find this information beneficial for your projects.