Real-time collaboration for Jupyter Notebooks, Linux Terminals, LaTeX, VS Code, R IDE, and more,
all in one place. Commercial Alternative to JupyterHub.
Real-time collaboration for Jupyter Notebooks, Linux Terminals, LaTeX, VS Code, R IDE, and more,
all in one place. Commercial Alternative to JupyterHub.
Path: blob/main/examples/text_classification-tf.ipynb
Views: 2535
If you're opening this Notebook on colab, you will probably need to install the most recent versions of 🤗 Transformers and 🤗 Datasets. We will also need scipy and scikit-learn for some of the metrics. Uncomment the following cell and run it.
If you're opening this notebook locally, make sure your environment has an install from the latest version of those libraries.
To be able to share your model with the community and generate results like the one shown in the picture below via the inference API, there are a few more steps to follow:
First you have to create an access token on the Hugging Face website (sign up here if you haven't already!) then uncomment the following cell and input your token.
Then you need to install Git-LFS and setup Git if you haven't already. Uncomment the following instructions and adapt with your name and email:
Make sure your version of Transformers is at least 4.16.0 since the functionality was introduced in that version:
You can find a script version of this notebook to fine-tune your model in a distributed fashion using multiple GPUs or TPUs here.
We also quickly upload some telemetry - this tells us which examples and software versions are getting used so we know where to prioritize our maintenance efforts. We don't collect (or care about) any personally identifiable information, but if you'd prefer not to be counted, feel free to skip this step or delete this cell entirely.
Fine-tuning a model on a text classification task
In this notebook, we will see how to fine-tune one of the 🤗 Transformers model to a text classification task of the GLUE Benchmark.
The GLUE Benchmark is a group of nine classification tasks on sentences or pairs of sentences which are:
CoLA (Corpus of Linguistic Acceptability) Determine if a sentence is grammatically correct or not.is a dataset containing sentences labeled grammatically correct or not.
MNLI (Multi-Genre Natural Language Inference) Determine if a sentence entails, contradicts or is unrelated to a given hypothesis. (This dataset has two versions, one with the validation and test set coming from the same distribution, another called mismatched where the validation and test use out-of-domain data.)
MRPC (Microsoft Research Paraphrase Corpus) Determine if two sentences are paraphrases from one another or not.
QNLI (Question-answering Natural Language Inference) Determine if the answer to a question is in the second sentence or not. (This dataset is built from the SQuAD dataset.)
QQP (Quora Question Pairs2) Determine if two questions are semantically equivalent or not.
RTE (Recognizing Textual Entailment) Determine if a sentence entails a given hypothesis or not.
SST-2 (Stanford Sentiment Treebank) Determine if the sentence has a positive or negative sentiment.
STS-B (Semantic Textual Similarity Benchmark) Determine the similarity of two sentences with a score from 1 to 5.
WNLI (Winograd Natural Language Inference) Determine if a sentence with an anonymous pronoun and a sentence with this pronoun replaced are entailed or not. (This dataset is built from the Winograd Schema Challenge dataset.)
We will see how to easily load the dataset for each one of those tasks and use Keras to fine-tune a model on it. Each task is named by its acronym, with mnli-mm standing for the mismatched version of MNLI (a task with the same training set as mnli but different validation and test sets):
This notebook is built to run on any of the tasks in the list above, with any model checkpoint from the Model Hub as long as that model has a version with a classification head. Depending on your model and the GPU you are using, you might need to adjust the batch size to avoid out-of-memory errors. Set these three parameters, then the rest of the notebook should run smoothly:
Loading the dataset
We will use the 🤗 Datasets library to download the data and the 🤗 Evaluate library to get the metric we need to use for evaluation (to compare our model to the benchmark). This can be easily done with the load_dataset function from datasets and and the load function from evaluate.
With the exception of mnli-mm, we can directly pass our task name to those functions. load_dataset will cache the dataset to avoid downloading it again the next time you run this cell.
The dataset object itself is DatasetDict, which contains one key for the training, validation and test set (with more keys for the mismatched validation and test set in the special case of mnli).
To access an actual element, you need to select a split first, then give an index:
To get a sense of what the data looks like, the following function will show some examples picked randomly in the dataset.
The metric is an instance of datasets.Metric:
You can call its compute method with your predictions and labels directly and it will return a dictionary with the metric(s) value:
Note that load has loaded the proper metric associated to your task, which is:
for CoLA: Matthews Correlation Coefficient
for MNLI (matched or mismatched): Accuracy
for MRPC: Accuracy and F1 score
for QNLI: Accuracy
for QQP: Accuracy and F1 score
for RTE: Accuracy
for SST-2: Accuracy
for STS-B: Pearson Correlation Coefficient and Spearman's_Rank_Correlation_Coefficient
for WNLI: Accuracy
so the metric object only computes the one(s) needed for your task.
Preprocessing the data
Before we can feed those texts to our model, we need to preprocess them. This is done by a 🤗 Transformers Tokenizer which will (as the name indicates) tokenize the inputs (including converting the tokens to their corresponding IDs in the pretrained vocabulary) and put it in a format the model expects, as well as generate the other inputs that model requires.
To do all of this, we instantiate our tokenizer with the AutoTokenizer.from_pretrained method, which will ensure:
we get a tokenizer that corresponds to the model architecture we want to use,
we download the vocabulary used when pretraining this specific checkpoint.
That vocabulary will be cached, so it's not downloaded again the next time we run the cell.
You can directly call this tokenizer on one sentence or a pair of sentences:
Depending on the model you selected, you will see different keys in the dictionary returned by the cell above. They don't matter much for what we're doing here (just know they are required by the model we will instantiate later), you can learn more about them in this tutorial if you're interested.
To preprocess our dataset, we will thus need the names of the columns containing the sentence(s). The following dictionary keeps track of the correspondence task to column names:
We can double check it does work on our current dataset:
We can them write the function that will preprocess our samples. We just feed them to the tokenizer with the arguments truncation=True and padding='longest. This will ensure that an input longer that what the model selected can handle will be truncated to the maximum length accepted by the model, and all inputs will be padded to the maximum input length to give us a single input array. A more performant method that reduces the number of padding tokens is to write a generator or tf.data.Dataset to only pad each batch to the maximum length in that batch, but most GLUE tasks are relatively quick on modern GPUs either way.
This function works with one or several examples. In the case of several examples, the tokenizer will return a list of lists for each key:
To apply this function on all the sentences (or pairs of sentences) in our dataset, we just use the map method of our dataset object we created earlier. This will apply the function on all the elements of all the splits in dataset, so our training, validation and testing data will be preprocessed in one single command.
Even better, the results are automatically cached by the 🤗 Datasets library to avoid spending time on this step the next time you run your notebook. The 🤗 Datasets library is normally smart enough to detect when the function you pass to map has changed (and thus requires to not use the cache data). For instance, it will properly detect if you change the task in the first cell and rerun the notebook. 🤗 Datasets warns you when it uses cached files, you can pass load_from_cache_file=False in the call to map to not use the cached files and force the preprocessing to be applied again.
Note that we passed batched=True to encode the texts by batches together. This is to leverage the full benefit of the fast tokenizer we loaded earlier, which will use multi-threading to treat the texts in a batch concurrently.
Fine-tuning the model
Now that our data is ready, we can download the pretrained model and fine-tune it. Since all our tasks are about sentence classification, we use the TFAutoModelForSequenceClassification class. Like with the tokenizer, the from_pretrained method will download and cache the model for us. The only thing we have to specify is the number of labels for our problem (which is always 2, except for STS-B which is a regression problem and MNLI where we have 3 labels).
The warning is telling us we are throwing away some weights (the vocab_transform and vocab_layer_norm layers) and randomly initializing some other (the pre_classifier and classifier layers). This is absolutely normal in this case, because we are removing the head used to pretrain the model on a masked language modeling objective and replacing it with a new head for which we don't have pretrained weights, so the library warns us we should fine-tune this model before using it for inference, which is exactly what we are going to do.
Next, we convert our datasets to tf.data.Dataset, which Keras understands natively. There are two ways to do this - we can use the slightly more low-level Dataset.to_tf_dataset() method, or we can use Model.prepare_tf_dataset(). The main difference between these two is that the Model method can inspect the model to determine which column names it can use as input, which means you don't need to specify them yourself. Unless our samples are all the same length, we will also need to pass a tokenizer or collate_fn so that the tf.data.Dataset knows how to pad and combine samples into a batch.
Next, we need to set up our optimizer and compile() our model. The create_optimizer function in the Transformers library creates a very useful AdamW optimizer with weight and learning rate decay. This performs very well for training most transformer networks - we recommend using it as your default unless you have a good reason not to! Note, however, that because it decays the learning rate over the course of training, it needs to know how many batches it will see during training.
Note that all models in transformers can pick a sensible loss function by default. To use this loss, simply do not pass a loss argument to compile(). Although the losses for GLUE tasks are usually just simple cross-entropy, this can be very helpful in models when the loss is intricate and contains multiple terms.
In some of our other examples, we use jit_compile to compile the model with XLA. In this case, we should be careful about that - because our inputs have variable sequence lengths, we may end up having to do a new XLA compilation for each possible length, because XLA compilation expects a static input shape! For small datasets, this will probably result in spending more time on XLA compilation than actually training, which isn't very helpful.
If you really want to use XLA without these problems (for example, if you're training on TPU), you can create a tokenizer with padding="max_length". This will pad all of your samples to the same length, ensuring that a single XLA compilation will suffice for your entire dataset. Note that depending on the nature of your dataset, this may result in a lot of wasted computation on padding tokens!
The last thing to define is how to compute the metrics from the predictions. We need to define a function for this, which will just use the metric we loaded earlier. The only preprocessing we have to do is to take the argmax of our predicted logits (our just squeeze the last axis in the case of STS-B).
In addition, let's wrap this metric computation function in a KerasMetricCallback. This callback will compute the metric on the validation set each epoch, including printing it and logging it for other callbacks like TensorBoard and EarlyStopping.
Why do it this way, though, and not just use a straightforward Keras Metric object? This is a good question - on this task, several of the metrics such as Accuracy are very straightforward, and it would probably make more sense to just use a Keras metric for those instead. However, we want to demonstrate the use of KerasMetricCallback here, because it can handle any arbitrary Python function for the metric computation. This turns out to be very important for other NLP tasks like summarization and translation, where standard metrics like BLEU and ROUGE are much more complex to compute, and often involve decoding tokens generated by the model to strings and comparing their similarity to target sentences. If you want to stop training once ROUGE scores on the validation set start to decline, then KerasMetricCallback is essential.
That said, if you're only interested in tasks like text classification with straightforward metrics, then by all means remove the KerasMetricCallback and use a Keras Accuracy metric instead!
With that out of the way, how do we actually use KerasMetricCallback? It's straightfoward: We simply define a function that computes metrics given a tuple of numpy arrays of predictions and labels, then we pass that, along with the validation set to compute metrics on, to the callback:
We can now finetune our model by just calling the fit method. Be sure to pass the TF datasets, and not the original datasets! We can also add a callback to sync up our model with the Hub - this allows us to resume training from other machines and even test the model's inference quality midway through training! Make sure to change the username if you do. If you don't want to do this, simply remove the callbacks argument in the call to fit().
To see how your model fared you can compare it to the GLUE Benchmark leaderboard.
If you used the callback above, you can now share this model with all your friends, family or favorite pets: they can all load it with the identifier "your-username/the-name-you-picked" so for instance:
Inference
Training a model is fun, but once it's trained you usually want to use it to get predictions on new data. Let's take a look at how to do that. Firstly, we'll load our trained model from the hub - this lets us resume the code from here without needing to rerun all the training above every time.
Now, let's make up some sentences and see if the model can classify them properly! The first sentence is valid English, but the second one makes a grammatical mistake.
To feed them into our model, we'll need to tokenize them and then get our model's predictions:
What do those label values mean? Let's use the id2label property set on our model to make them a little more comprehensible:
Looks right to me!
Pipeline API
An alternative way to quickly perform inference with any model on the hub is to use the Pipeline API, which abstracts away all the steps we did manually above. It will perform the preprocessing, forward pass and postprocessing all in a single object.
Let's showcase this for our trained model:
And that's it - the code above is all you need to get classifications from your model in future! Note how the id2label property we set during training is automatically read by our pipeline to assign sensible names to the output classes.