GitHub Repository: ibm/watson-machine-learning-samples
Path: blob/master/cloud/notebooks/rest_api/deployments/foundation_models/Use watsonx, and Google `flan-t5-xxl` to analyze car rental customer satisfaction from text.ipynb
⁶⁴⁰⁵ views

Kernel: Python 3 (ipykernel)

Use watsonx, and Google `flan-t5-xxl` to analyze car rental customer satisfaction from text

Disclaimers

Use only Projects and Spaces that are available in watsonx context.

Notebook content

This notebook contains the steps and code to demonstrate support of text sentiment analysis in watsonx. It introduces commands for data retrieval, model testing and scoring.

Some familiarity with Python is helpful. This notebook uses Python 3.11.

Learning goal

The goal of this notebook is to demonstrate how to use flan-t5-xxl model to analyze customer satisfaction from text.

This notebook contains the following parts:

Set up the environment

Before you use the sample code in this notebook, you must perform the following setup tasks:

Create a watsonx.ai Runtime Service instance (a free plan is offered and information about how to create the instance can be found here).

Install and import the `datasets` and dependecies

In [ ]:

!pip install datasets | tail -n 1
!pip install requests | tail -n 1
!pip install wget | tail -n 1
!pip install ibm-cloud-sdk-core | tail -n 1
!pip install "scikit-learn==1.3.2" | tail -n 1

In [2]:

import os, getpass, wget, json
import requests
from ibm_cloud_sdk_core import IAMTokenManager
from pandas import value_counts, read_csv, DataFrame
from sklearn.model_selection import train_test_split

Inferencing class

This cell defines a class that makes a REST API call to the watsonx Foundation Model inferencing API that we will use to generate output from the provided input. The class takes the access token created in the previous step, and uses it to make a REST API call with input, model id and model parameters. The response from the API call is returned as the cell output.

Action: Provide watsonx.ai Runtime url to work with watsonx.ai.

In [3]:

endpoint_url = input("Please enter your watsonx.ai Runtime endpoint url (hit enter): ")

Define a Prompt class for prompts generation.

In [4]:

class Prompt:
    def __init__(self, access_token, project_id):
        self.access_token = access_token
        self.project_id = project_id

    def generate(self, input, model_id, parameters):
        wml_url = f"{endpoint_url}/ml/v1/text/generation?version=2024-03-19"
        Headers = {
            "Authorization": "Bearer " + self.access_token,
            "Content-Type": "application/json",
            "Accept": "application/json"
        }
        data = {
            "model_id": model_id,
            "input": input,
            "parameters": parameters,
            "project_id": self.project_id
        }
        response = requests.post(wml_url, json=data, headers=Headers)
        if response.status_code == 200:
            return response.json()["results"][0]
        else:
            return response.text

watsonx API connection

This cell defines the credentials required to work with watsonx API for Foundation Model inferencing.

Action: Provide the IBM Cloud user API key. For details, see documentation.

In [5]:

access_token = IAMTokenManager(
    apikey = getpass.getpass("Please enter your watsonx.ai api key (hit enter): "),
    url = "https://iam.cloud.ibm.com/identity/token"
).get_token()

Defining the project id

The API requires project id that provides the context for the call. We will obtain the id from the project in which this notebook runs. Otherwise, please provide the project id.

In [6]:

try:
    project_id = os.environ["PROJECT_ID"]
except KeyError:
    project_id = input("Please enter your project_id (hit enter): ")

Data loading

Download the car_rental_training_data dataset. The dataset provides insight about customers opinions on car rental. It has a label that consists of values: unsatisfied, satisfied.

In [7]:

filename = 'car_rental_training_data.csv'

url = 'https://raw.githubusercontent.com/IBM/watson-machine-learning-samples/master/cloud/data/cars-4-you/car_rental_training_data.csv'
if not os.path.isfile(filename): wget.download(url, out=filename)

In [8]:

data = read_csv("car_rental_training_data.csv", sep=';')

Examine donwloaded data.

In [9]:

data.head()

Out[9]:

Define label map.

In [10]:

label_map= {0: "unsatisfied",
            1: "satisfied"}

Inspect data labels distribution.

In [11]:

value_counts(data['Satisfaction'])

Out[11]:

1    274
0    212
Name: Satisfaction, dtype: int64

Prepare train and test sets.

In [12]:

data_train, data_test, y_train, y_test = train_test_split(data.Customer_Service, 
                                                    data.Satisfaction,
                                                    test_size=0.3,
                                                    random_state=33, 
                                                    stratify=data.Satisfaction)
data_train = DataFrame(data_train)
data_test = DataFrame(data_test)

data_train["satisfaction"] = list(map(label_map.get, y_train))
data_test["satisfaction"] = list(map(label_map.get, y_test))

Foundation Models on watsonx

List available models

In [14]:

models_json = requests.get(endpoint_url + '/ml/v1/foundation_model_specs?version=2024-03-19&limit=50',
                           headers={
                                    'Authorization': f'Bearer {access_token}',
                                    'Content-Type': 'application/json',
                                    'Accept': 'application/json'
                            }).json()
models_ids = [m['model_id'] for m in models_json['resources']]
models_ids

Out[14]:

['bigcode/starcoder',
 'bigscience/mt0-xxl',
 'codellama/codellama-34b-instruct-hf',
 'eleutherai/gpt-neox-20b',
 'google/flan-t5-xl',
 'google/flan-t5-xxl',
 'google/flan-ul2',
 'ibm-mistralai/mixtral-8x7b-instruct-v01-q',
 'ibm/granite-13b-chat-v1',
 'ibm/granite-13b-chat-v2',
 'ibm/granite-13b-instruct-v1',
 'ibm/granite-13b-instruct-v2',
 'ibm/granite-20b-multilingual',
 'ibm/mpt-7b-instruct2',
 'meta-llama/llama-2-13b-chat',
 'meta-llama/llama-2-70b-chat']

You need to specify model_id that will be used for inferencing:

In [15]:

model_id = "google/flan-t5-xxl"

Analyze the sentiment

Define instructions for the model.

In [16]:

instruction = "Classify the satisfaction expressed in this sentence using: satisfied, unsatisfied.\n"

Prepare model inputs - build zero-shot examples from the test set.

In [17]:

zero_shot_inputs = [{"input": text} for text in data_test.Customer_Service.values]
print(json.dumps(zero_shot_inputs[:5], indent=2))

Out[17]:

[
  {
    "input": "Provide more convenient car pickup from the airport parking."
  },
  {
    "input": "They could really try work harder."
  },
  {
    "input": "the rep was friendly but it was so loud in there that I could not hear what she was saying. I HATE having to walk across a big lot with all of my bags in search of my car which is always in the furthest corner."
  },
  {
    "input": "The agents were not friendly when I checked in initially, that was annoying because I had just spent 3 hours on a plane and wanted to be greeted with a better attitude."
  },
  {
    "input": "It was not as bad as it usually is."
  }
]

Prepare model inputs - build few-shot examples. To build a few-shot example few instances of training data phrases are passed together with the reference sentiment and then appended with a test data phrase.

In this notebook, training phrases are stratified over all possible sentiments for each test case.

In [18]:

few_shot_inputs = []
singleoutput= []

for test_phrase in data_test.Customer_Service.values:
    for train_phrase, sentiment in data_train.groupby('satisfaction', group_keys=False).apply(lambda x: x.sample(2)).values:
        singleoutput.append(f"\tsentence:\t{train_phrase}\n\tsatisfaction: {sentiment}\n")
    singleoutput.append(f"\tsentence:\t{test_phrase}\n\tsatisfaction:")
    few_shot_inputs.append("".join(singleoutput))
    singleoutput = []

Inspect an exemplary few-shot prompt.

In [19]:

print(json.dumps(print(few_shot_inputs[0]), indent=2))

Out[19]:

	sentence:	Friendly and very well informed
	satisfaction: satisfied
	sentence:	Everyone was very friendly. They even drove us to the airport since we were running late for our flight.
	satisfaction: satisfied
	sentence:	Please lower the prices.
	satisfaction: unsatisfied
	sentence:	I haven't actually spoken with anyone from a car rental organization for quite a while.  When I did (probably about three years ago), I believe they were polite enough. However, I always hate to wait in lines when we have a lot of luggage.
	satisfaction: unsatisfied
	sentence:	Provide more convenient car pickup from the airport parking.
	satisfaction:
null

Defining the model parameters

We need to provide a set of model parameters that will influence the result:

In [20]:

parameters = {
    "decoding_method": "greedy"
}

Analyze the satisfaction using Google `flan-t5-xxl` model.

Note: You might need to adjust model parameters for different models or tasks, to do so please refer to documentation.

Initialize the Prompt class.

Hint: Your authentication token might expire, if so please regenerate the access_token reinitialize the Prompt class.

In [21]:

prompt = Prompt(access_token, project_id)

Analyze the sentiment for a sample of zero-shot inputs from the test set.

In [22]:

results = []
for inp in zero_shot_inputs[:5]:
    results.append(prompt.generate(" ".join([instruction, inp['input']]), model_id, parameters))

Explore model output.

In [23]:

print(json.dumps(results, indent=2))

Out[23]:

[
  {
    "generated_text": "unsatisfied",
    "generated_token_count": 6,
    "input_token_count": 29,
    "stop_reason": "eos_token"
  },
  {
    "generated_text": "unsatisfied",
    "generated_token_count": 6,
    "input_token_count": 26,
    "stop_reason": "eos_token"
  },
  {
    "generated_text": "unsatisfied",
    "generated_token_count": 6,
    "input_token_count": 71,
    "stop_reason": "eos_token"
  },
  {
    "generated_text": "unsatisfied",
    "generated_token_count": 6,
    "input_token_count": 57,
    "stop_reason": "eos_token"
  },
  {
    "generated_text": "satisfied",
    "generated_token_count": 2,
    "input_token_count": 29,
    "stop_reason": "eos_token"
  }
]

Score the model

Note: To run the Score section for model scoring on the whole car rental customer satisfaction dataset please transform following markdown cells to code cells. Have in mind that scoring model on the whole test set can take significant amount of time.

Get the true labels.

y_true = [label for label in data_test.satisfaction[:5]]

Get the sentiment labels returned by the flan-t5-xxl model.

y_pred = [res["generated_text"] for res in results]

Calculate the accuracy score.

from sklearn.metrics import accuracy_score

print(accuracy_score(y_pred, y_true))

HINT: Sentiments generated using few-shot input prompts might provide better performance in terms of accuracy then the zero-shot ones. Following cells present test scores for zero-shot prompts received for the flan-t5-xxl model on the whole test set from this notebook.

The zero-shot test accuracy score:

0.9178082191780822

Summary and next steps

You successfully completed this notebook!

You learned how to analyze car rental customer satisfaction with Google's flan-t5-xxl on watsonx.

Check out our Online Documentation for more samples, tutorials, documentation, how-tos, and blog posts.

Authors

Mateusz Szewczyk, Software Engineer at watsonx.ai.

Use watsonx, and Google `flan-t5-xxl` to analyze car rental customer satisfaction from text

Disclaimers

Notebook content

Learning goal

Contents

Set up the environment

Install and import the `datasets` and dependecies

Inferencing class

watsonx API connection

Defining the project id

Data loading

Foundation Models on watsonx

List available models

Analyze the sentiment

Defining the model parameters

Analyze the satisfaction using Google `flan-t5-xxl` model.

Score the model

Summary and next steps

Authors

Product

Resources

Company

Use watsonx, and Google flan-t5-xxl to analyze car rental customer satisfaction from text

Disclaimers

Notebook content

Learning goal

Contents

Set up the environment

Install and import the datasets and dependecies

Inferencing class

watsonx API connection

Defining the project id

Data loading

Foundation Models on watsonx

List available models

Analyze the sentiment

Defining the model parameters

Analyze the satisfaction using Google flan-t5-xxl model.

Score the model

Summary and next steps

Authors

Use watsonx, and Google `flan-t5-xxl` to analyze car rental customer satisfaction from text

Install and import the `datasets` and dependecies

Analyze the satisfaction using Google `flan-t5-xxl` model.