GitHub Repository: ibm/watson-machine-learning-samples
Path: blob/master/cpd4.8/notebooks/python_sdk/deployments/foundation_models/Use watsonx, and Google `flan-t5-xxl` to analyze car rental customer satisfaction from text.ipynb
⁶⁴⁰⁸ views

Kernel: Python 3 (ipykernel)

Use watsonx, and Google `flan-t5-xxl` to analyze car rental customer satisfaction from text

Disclaimers

Use only Projects and Spaces that are available in watsonx context.

Notebook content

This notebook contains the steps and code to demonstrate support of text sentiment analysis in watsonx. It introduces commands for data retrieval, model testing and scoring.

Some familiarity with Python is helpful. This notebook uses Python 3.10.

Learning goal

The goal of this notebook is to demonstrate how to use flan-t5-xxl model to analyze customer satisfaction from text.

This notebook contains the following parts:

Set up the environment

Before you use the sample code in this notebook, you must perform the following setup tasks:

Contact with your Cloud Pack for Data administrator and ask him for your account credentials

Install and import the `datasets` and dependecies

In [ ]:

!pip install wget | tail -n 1
!pip install datasets | tail -n 1
!pip install scikit-learn | tail -n 1
!pip install "ibm-watson-machine-learning>=1.0.320" | tail -n 1

Connection to WML

Authenticate the Watson Machine Learning service on IBM Cloud Pack for Data. You need to provide platform url, your username and api_key.

In [ ]:

username = 'PASTE YOUR USERNAME HERE'
api_key = 'PASTE YOUR API_KEY HERE'
url = 'PASTE THE PLATFORM URL HERE'

In [4]:

wml_credentials = {
    "username": username,
    "apikey": api_key,
    "url": url,
    "instance_id": 'openshift',
    "version": '4.8'
}

Alternatively you can use username and password to authenticate WML services.

wml_credentials = {
    "username": ***,
    "password": ***,
    "url": ***,
    "instance_id": 'openshift',
    "version": '4.8'
}

Defining the project id

The Foundation Model requires project id that provides the context for the call. We will obtain the id from the project in which this notebook runs. Otherwise, please provide the project id.

In [5]:

import os

try:
    project_id = os.environ["PROJECT_ID"]
except KeyError:
    project_id = input("Please enter your project_id (hit enter): ")

Data loading

Download the car_rental_training_data dataset. The dataset provides insight about customers opinions on car rental. It has a label that consists of values: unsatisfied, satisfied.

In [6]:

import wget
import pandas as pd

filename = 'car_rental_training_data.csv'
url = 'https://raw.githubusercontent.com/IBM/watson-machine-learning-samples/master/cloud/data/cars-4-you/car_rental_training_data.csv'

if not os.path.isfile(filename): 
    wget.download(url, out=filename)

data = pd.read_csv("car_rental_training_data.csv", sep=';')
comments = list(data.Customer_Service)
satisfaction = list(data.Satisfaction)

Examine donwloaded data.

In [7]:

data.head()

Out[7]:

Define label map.

In [8]:

label_map= {0: "unsatisfied",
            1: "satisfied"}

Inspect data labels distribution.

In [9]:

pd.value_counts(data['Satisfaction'])

Out[9]:

1    274
0    212
Name: Satisfaction, dtype: int64

Prepare train and test sets.

In [10]:

from sklearn.model_selection import train_test_split

data_train, data_test, y_train, y_test = train_test_split(data.Customer_Service, 
                                                    data.Satisfaction,
                                                    test_size=0.3,
                                                    random_state=33, 
                                                    stratify=data.Satisfaction)
data_train = pd.DataFrame(data_train)
data_test = pd.DataFrame(data_test)

data_train["satisfaction"] = list(map(label_map.get, y_train))
data_test["satisfaction"] = list(map(label_map.get, y_test))

Foundation Models on `watsonx.ai`

List available models

All avaliable models are presented under ModelTypes class. For more information refer to documentation.

In [11]:

from ibm_watson_machine_learning.foundation_models.utils.enums import ModelTypes

print([model.name for model in ModelTypes])

Out[11]:

['FLAN_T5_XXL', 'FLAN_UL2', 'MT0_XXL', 'GPT_NEOX', 'MPT_7B_INSTRUCT2', 'STARCODER', 'LLAMA_2_70B_CHAT']

You need to specify model_id that will be used for inferencing:

In [12]:

model_id = ModelTypes.FLAN_T5_XXL

Defining the model parameters

You might need to adjust model parameters for different models or tasks, to do so please refer to documentation.

In [13]:

from ibm_watson_machine_learning.metanames import GenTextParamsMetaNames as GenParams

parameters = {
    GenParams.DECODING_METHOD: "greedy"
}

Initialize the model

Initialize the Model class with previous set params.

In [14]:

from ibm_watson_machine_learning.foundation_models import Model

model = Model(
    model_id=model_id, 
    params=parameters, 
    credentials=wml_credentials,
    project_id=project_id)

Model's details

In [15]:

model.get_details()

Out[15]:

{'model_id': 'google/flan-t5-xxl',
 'label': 'flan-t5-xxl-11b',
 'provider': 'Google',
 'source': 'Hugging Face',
 'short_description': 'flan-t5-xxl is an 11 billion parameter model based on the Flan-T5 family.',
 'long_description': 'flan-t5-xxl (11B) is an 11 billion parameter model based on the Flan-T5 family. It is a pretrained T5 - an encoder-decoder model pre-trained on a mixture of supervised / unsupervised tasks converted into a text-to-text format, and fine-tuned on the Fine-tuned LAnguage Net (FLAN) with instructions for better zero-shot and few-shot performance.',
 'task_ids': ['question_answering',
  'summarization',
  'classification',
  'generation',
  'extraction'],
 'tasks': [],
 'min_shot_size': 0,
 'tier': 'class_2',
 'number_params': '11b'}

Analyze the sentiment

Define instructions for the model.

In [16]:

instruction = "Classify the satisfaction expressed in this sentence using: satisfied, unsatisfied.\n"

Prepare model inputs - build zero-shot examples from the test set.

In [17]:

import json

zero_shot_inputs = [{"input": text} for text in data_test.Customer_Service.values]
print(json.dumps(zero_shot_inputs[:5], indent=2))

Out[17]:

[
  {
    "input": "Provide more convenient car pickup from the airport parking."
  },
  {
    "input": "They could really try work harder."
  },
  {
    "input": "the rep was friendly but it was so loud in there that I could not hear what she was saying. I HATE having to walk across a big lot with all of my bags in search of my car which is always in the furthest corner."
  },
  {
    "input": "The agents were not friendly when I checked in initially, that was annoying because I had just spent 3 hours on a plane and wanted to be greeted with a better attitude."
  },
  {
    "input": "It was not as bad as it usually is."
  }
]

Prepare model inputs - build few-shot examples. To build a few-shot example few instances of training data phrases are passed together with the reference sentiment and then appended with a test data phrase.

In this notebook, training phrases are stratified over all possible sentiments for each test case.

In [18]:

few_shot_inputs = []
singleoutput= []

for test_phrase in data_test.Customer_Service.values:
    for train_phrase, sentiment in data_train.groupby('satisfaction', group_keys=False).apply(lambda x: x.sample(2)).values:
        singleoutput.append(f"\tsentence:\t{train_phrase}\n\tsatisfaction: {sentiment}\n")
    singleoutput.append(f"\tsentence:\t{test_phrase}\n\tsatisfaction:")
    few_shot_inputs.append("".join(singleoutput))
    singleoutput = []

Inspect an exemplary few-shot prompt.

In [19]:

print(json.dumps(print(few_shot_inputs[0]), indent=2))

Out[19]:

	sentence:	They had the class of car that I ordered.
	satisfaction: satisfied
	sentence:	They were competent.
	satisfaction: satisfied
	sentence:	They did not have the car I wanted.  upgraded me to a car I did not like and did not want.
	satisfaction: unsatisfied
	sentence:	Was average, nothing out of the ordinary.  Took a long time to get the car.
	satisfaction: unsatisfied
	sentence:	Provide more convenient car pickup from the airport parking.
	satisfaction:
null

Analyze the satisfaction using Google `flan-t5-xxl` model.

Analyze the sentiment for a sample of zero-shot inputs from the test set.

In [20]:

results = []
for inp in zero_shot_inputs[:5]:
    results.append(model.generate(" ".join([instruction, inp['input']]))["results"][0])

Explore model output.

In [21]:

print(json.dumps(results, indent=2))

Out[21]:

[
  {
    "generated_text": "unsatisfied",
    "generated_token_count": 6,
    "input_token_count": 29,
    "stop_reason": "EOS_TOKEN"
  },
  {
    "generated_text": "unsatisfied",
    "generated_token_count": 6,
    "input_token_count": 26,
    "stop_reason": "EOS_TOKEN"
  },
  {
    "generated_text": "unsatisfied",
    "generated_token_count": 6,
    "input_token_count": 71,
    "stop_reason": "EOS_TOKEN"
  },
  {
    "generated_text": "unsatisfied",
    "generated_token_count": 6,
    "input_token_count": 57,
    "stop_reason": "EOS_TOKEN"
  },
  {
    "generated_text": "satisfied",
    "generated_token_count": 2,
    "input_token_count": 29,
    "stop_reason": "EOS_TOKEN"
  }
]

Score the model

Note: To run the Score section for model scoring on the whole car rental customer satisfaction dataset please transform following markdown cells to code cells. Have in mind that scoring model on the whole test set can take significant amount of time.

Get the true labels.

y_true = [label for label in data_test.satisfaction[:5]]

Get the sentiment labels returned by the flan-t5-xxl model.

y_pred = [res["generated_text"] for res in results]

Calculate the accuracy score.

from sklearn.metrics import accuracy_score

print(accuracy_score(y_pred, y_true))

HINT: Sentiments generated using few-shot input prompts might provide better performance in terms of accuracy then the zero-shot ones. Following cells present test scores for zero-shot prompts received for the flan-t5-xxl model on the whole test set from this notebook.

The zero-shot test accuracy score:

0.9178082191780822

Summary and next steps

You successfully completed this notebook!.

You learned how to analyze car rental customer satisfaction with Google's flan-t5-xxl on watsonx.

Check out our Online Documentation for more samples, tutorials, documentation, how-tos, and blog posts.

Authors

Mateusz Szewczyk, Software Engineer at Watson Machine Learning.

Use watsonx, and Google `flan-t5-xxl` to analyze car rental customer satisfaction from text

Disclaimers

Notebook content

Learning goal

Contents

Set up the environment

Install and import the `datasets` and dependecies

Connection to WML

Defining the project id

Data loading

Foundation Models on `watsonx.ai`

List available models

Defining the model parameters

Initialize the model

Model's details

Analyze the sentiment

Analyze the satisfaction using Google `flan-t5-xxl` model.

Score the model

Summary and next steps

Authors

Product

Resources

Company

Use watsonx, and Google flan-t5-xxl to analyze car rental customer satisfaction from text

Disclaimers

Notebook content

Learning goal

Contents

Set up the environment

Install and import the datasets and dependecies

Connection to WML

Defining the project id

Data loading

Foundation Models on watsonx.ai

List available models

Defining the model parameters

Initialize the model

Model's details

Analyze the sentiment

Analyze the satisfaction using Google flan-t5-xxl model.

Score the model

Summary and next steps

Authors

Use watsonx, and Google `flan-t5-xxl` to analyze car rental customer satisfaction from text

Install and import the `datasets` and dependecies

Foundation Models on `watsonx.ai`

Analyze the satisfaction using Google `flan-t5-xxl` model.