GitHub Repository: ibm/watson-machine-learning-samples
Path: blob/master/cloud/notebooks/rest_api/deployments/foundation_models/Use watsonx, and Google `flan-ul2` to classify Consumer Financial Protection Bureau (CFPB) document.ipynb
⁶⁴⁰⁵ views

Kernel: Python 3 (ipykernel)

Use watsonx, and Google `google/flan-ul2` to classify consumer financial protection bureau document

Disclaimers

Use only Projects and Spaces that are available in watsonx context.

Notebook content

This notebook contains the steps and code to demonstrate support of complaint classification in watsonx. It introduces commands for data retrieval and model testing.

Some familiarity with Python is helpful. This notebook uses Python 3.11.

Learning goal

The goal of this notebook is to demonstrate how to use google/flan-ul2 model to Consumer Financial Protection Bureau (CFPB) document.

Use case & dataset

The Consumer Financial Protection Bureau (CFPB) is a federal U.S. agency that acts as a mediator when disputes arise between financial institutions and consumers. Via a web form, consumers can send the agency a narrative of their dispute. A foundation model would make the classification of complaints and their routing to the appropriate teams more efficient than manually tagged complaints. Dataset has narrative and 5 product categories as follow:

credit reporting
debt collection
mortgages and loans
credit cards
retail banking

This notebook contains the following parts:

Set up the environment

Before you use the sample code in this notebook, you must perform the following setup tasks:

Create a watsonx.ai Runtime Service instance (a free plan is offered and information about how to create the instance can be found here).

Install and import the `datasets` and dependecies

you need to install below required dependencies to be able to continue

In [ ]:

!pip install datasets | tail -n 1
!pip install requests | tail -n 1
!pip install wget | tail -n 1
!pip install pandas | tail -n 1
!pip install ibm-cloud-sdk-core | tail -n 1
!pip install "scikit-learn==1.3.2" | tail -n 1

In [2]:

import os, getpass, wget
import requests
from ibm_cloud_sdk_core import IAMTokenManager
from pandas import value_counts, read_csv, DataFrame
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

Inferencing class

This cell defines a class that makes a REST API call to the watsonx Foundation Model inferencing API that we will use to generate output from the provided input. The class takes the access token created in the previous step, and uses it to make a REST API call with input, model id and model parameters. The response from the API call is returned as the cell output.

Action: Provide watsonx.ai Runtime url to work with watsonx.ai.

In [3]:

endpoint_url = input("Please enter your watsonx.ai Runtime endpoint url (hit enter): ")

Define a Prompt class for prompts generation.

In [4]:

class Prompt:
    def __init__(self, access_token, project_id):
        self.access_token = access_token
        self.project_id = project_id

    def generate(self, input, model_id, parameters):
        wml_url = f"{endpoint_url}/ml/v1/text/generation?version=2024-03-19"
        Headers = {
            "Authorization": "Bearer " + self.access_token,
            "Content-Type": "application/json",
            "Accept": "application/json"
        }
        data = {
            "model_id": model_id,
            "input": input,
            "parameters": parameters,
            "project_id": self.project_id
        }
        response = requests.post(wml_url, json=data, headers=Headers)
        if response.status_code == 200:
            return response.json()["results"][0]
        else:
            return response.text

watsonx API connection

This cell defines the credentials required to work with watsonx API for Foundation Model inferencing.

Action: Provide the IBM Cloud user API key. For details, see documentation.

In [5]:

access_token = IAMTokenManager(
    apikey = getpass.getpass("Please enter your watsonx.ai api key (hit enter): "),
    url = "https://iam.cloud.ibm.com/identity/token"
).get_token()

Defining the project id

The API requires project id that provides the context for the call. We will obtain the id from the project in which this notebook runs:

In [6]:

try:
    project_id = os.environ["PROJECT_ID"]
except KeyError:
    project_id = input("Please enter your project_id (hit enter): ")

Data loading

Download the financial documents dataset.

In [7]:

filename = 'Data_Finance.csv'
url = 'https://raw.githubusercontent.com/IBM/watson-machine-learning-samples/master/cloud/data/cfpb_complaints/cfpb_compliants.csv'
if not os.path.isfile(filename): wget.download(url, out=filename)

Read the data.

In [8]:

data= read_csv("Data_Finance.csv")
data=data[['narrative','product']]
data.head()

Out[8]:

Inspect the product class distribution.

In [9]:

data['product'].value_counts().sort_values()

Out[9]:

retail_banking         3863
credit_card            5031
mortgages_and_loans    5822
credit_reporting       6000
debt_collection        6642
Name: product, dtype: int64

Split data to train and test

In [10]:

data_train, data_test, y_train, y_test = train_test_split(data['narrative'], 
                                                    data['product'],
                                                    test_size=0.3,
                                                    random_state=33, 
                                                    stratify=data['product'])
data_train = DataFrame(data_train)
data_test = DataFrame(data_test)

Foundation Models on watsonx

List available models

In [11]:

models_json = requests.get(endpoint_url + '/ml/v1/foundation_model_specs?version=2024-03-19&limit=50',
                           headers={
                                    'Authorization': f'Bearer {access_token}',
                                    'Content-Type': 'application/json',
                                    'Accept': 'application/json'
                            }).json()
models_ids = [m['model_id'] for m in models_json['resources']]
models_ids

Out[11]:

['bigcode/starcoder',
 'bigscience/mt0-xxl',
 'codellama/codellama-34b-instruct-hf',
 'eleutherai/gpt-neox-20b',
 'google/flan-t5-xl',
 'google/flan-t5-xxl',
 'google/flan-ul2',
 'ibm-mistralai/mixtral-8x7b-instruct-v01-q',
 'ibm/granite-13b-chat-v1',
 'ibm/granite-13b-chat-v2',
 'ibm/granite-13b-instruct-v1',
 'ibm/granite-13b-instruct-v2',
 'ibm/granite-20b-multilingual',
 'ibm/mpt-7b-instruct2',
 'meta-llama/llama-2-13b-chat',
 'meta-llama/llama-2-70b-chat']

You need to specify model_id that will be used for inferencing:

In [12]:

model_id = "google/flan-ul2"

Classify customer complaints

Define instructions for the model.

In [13]:

instruction="""Determine the class of product expressed in the sentense.
Use either 'credit_card', 'debt_collection','mortgages_and_loans',retail_banking, or 'credit_reporting'.
Use the provided examples as a template. """

Prepare model inputs for zero-shot example - use below zero_shot_inputs.

In [14]:

zero_shot_inputs = [{"input": text} for text in data_test['narrative']]
for i in range(5):
    print(f"The sentence example {i+1} is:\n {zero_shot_inputs[i]['input']}\n")

Out[14]:

The sentence example 1 is:
 receiving current copy credit report discovered entry identified inquiry qualified deletion report

The sentence example 2 is:
 got scam told send via cash apps able sent via cash apps adamant need send complete total via paid total amount reading since going remove called need put downpayment material need church helping remove need put downpayment return money within sent money since limit consumer protection program able send noticed scammed filed report return money back denied dispute something need get done scammer cash apps need better proctection protect innocent customer like

The sentence example 3 is:
 service member elected servicemembers civil relief act usaa prior departure reduced auto insurance due vehicle use gone austere environment limited communication mail forwarded location member deactivate cell phone gone turn back upon return mail sent arriving late month later due remote location logistical challenge usaa visa credit card well year issue credit limit card balance reward point due equating cashed apparently charge appeared card departed unaware receive communication lender could reach assuming tried usaa closed credit card reported day non payment credit bureau upon return immediately paid amount attempted numerous time unsuccessfully resolve issue usaa mentioned account voluntarily closed point would available day would allow credit instance negative reporting credit report brought score scra respectfully request removal negative credit reporting allow cash payment reward point balance credit reporting loss available credit would good credit limit restored well negative reporting credit cost year come inability obtain lower mortgage loan rate credit card etc due relocate week plan purchase home many thanks relief assistance could provide sincerely military member

The sentence example 4 is:
 contacted u bank fraud department said couldnt find account worry need make sure case fraud happening suspect might part

The sentence example 5 is:
 pandemic hoping youre safe sound listen shocked reviewed credit report today found late payment

Prepare model inputs for few-shot examples - use below few_shot_inputs.

In [15]:

data_train_and_labels=data_train.copy()
data_train_and_labels['product']=y_train

In [16]:

few_shot_example=[]
few_shot_examples=[]
for phrase,product in data_train_and_labels.groupby('product').apply(lambda x: x.sample(2)).values:
    few_shot_example.append(f"\tsentence:\t{phrase}\n\tproduct: {product}\n")
few_shot_examples=[''.join(few_shot_example)]

In [17]:

few_shot_inputs_ = [{"input": text} for text in data_test['narrative'].values]
for i in range(5):
    print(f"The sentence example {i+1} is:\n {few_shot_inputs_[i]['input']}\n")

Out[17]:

The sentence example 1 is:
 receiving current copy credit report discovered entry identified inquiry qualified deletion report

The sentence example 2 is:
 got scam told send via cash apps able sent via cash apps adamant need send complete total via paid total amount reading since going remove called need put downpayment material need church helping remove need put downpayment return money within sent money since limit consumer protection program able send noticed scammed filed report return money back denied dispute something need get done scammer cash apps need better proctection protect innocent customer like

The sentence example 3 is:
 service member elected servicemembers civil relief act usaa prior departure reduced auto insurance due vehicle use gone austere environment limited communication mail forwarded location member deactivate cell phone gone turn back upon return mail sent arriving late month later due remote location logistical challenge usaa visa credit card well year issue credit limit card balance reward point due equating cashed apparently charge appeared card departed unaware receive communication lender could reach assuming tried usaa closed credit card reported day non payment credit bureau upon return immediately paid amount attempted numerous time unsuccessfully resolve issue usaa mentioned account voluntarily closed point would available day would allow credit instance negative reporting credit report brought score scra respectfully request removal negative credit reporting allow cash payment reward point balance credit reporting loss available credit would good credit limit restored well negative reporting credit cost year come inability obtain lower mortgage loan rate credit card etc due relocate week plan purchase home many thanks relief assistance could provide sincerely military member

The sentence example 4 is:
 contacted u bank fraud department said couldnt find account worry need make sure case fraud happening suspect might part

The sentence example 5 is:
 pandemic hoping youre safe sound listen shocked reviewed credit report today found late payment

Defining the model parameters

We need to provide a set of model parameters that will influence the result:Based on decoding strategy that ww have for the models, the parameters can change.

There are two decoding strategies: greedy, sampling.

We usually use greedy for classification, Summarization, Extraction and Q&A.

We usually use sampling for content generation.

In [18]:

parameters = {
         "decoding_method": "greedy",
         "random_seed": 33,
         "repetition_penalty":1,
         "min_new_tokens": 1,
         "max_new_tokens": 5
}

Generate the classification of Consumer Financial Protection Bureau (CFPB) document using `google/flan-ul2` model.

Note: You might need to adjust model parameters for different models or tasks, to do so please refer to documentation.

Initialize the Prompt class.

Hint: Your authentication token might expire, if so please regenerate the access_token reinitialize the Prompt class.

In [19]:

prompt = Prompt(access_token, project_id)

Get the product classes.

In [20]:

results = []
for inp in few_shot_inputs_[:5]:
    results.append(prompt.generate(" ".join([instruction+few_shot_examples[0], inp['input']]), model_id, parameters))

Explore model output.

In [21]:

results

Out[21]:

[{'generated_text': 'credit_reporting',
  'generated_token_count': 5,
  'input_token_count': 1575,
  'stop_reason': 'max_tokens'},
 {'generated_text': 'product: credit_card',
  'generated_token_count': 5,
  'input_token_count': 1648,
  'stop_reason': 'max_tokens'},
 {'generated_text': 'credit_card',
  'generated_token_count': 4,
  'input_token_count': 1758,
  'stop_reason': 'eos_token'},
 {'generated_text': 'retail_banking',
  'generated_token_count': 5,
  'input_token_count': 1585,
  'stop_reason': 'eos_token'},
 {'generated_text': 'credit_reporting',
  'generated_token_count': 5,
  'input_token_count': 1580,
  'stop_reason': 'max_tokens'}]

Score the model

Note: To run the Score section for model scoring on the whole financial phrasebank dataset, please transform following markdown cells to code cells. Have in mind that scoring model on the whole test set can take significant amount of time.

Get the true labels.

y_true = y_test.values[:5]
print(y_true)

Get the prediction labels.

y_pred = [result['generated_text'] for result in results]
y_pred

Calculate the accuracy score.

print(accuracy_score(y_pred, y_true))

Summary and next steps

You successfully completed this notebook!

You learned how to classify Consumer Financial Protection Bureau documents with Google's google-flan-ul2 on watsonx.

Check out our Online Documentation for more samples, tutorials, documentation, how-tos, and blog posts.

Author:

Kahila Mokhtari

Mateusz Szewczyk, Software Engineer at watsonx.ai.

Use watsonx, and Google `google/flan-ul2` to classify consumer financial protection bureau document

Disclaimers

Notebook content

Learning goal

Use case & dataset

Contents

Set up the environment

Install and import the `datasets` and dependecies

Inferencing class

watsonx API connection

Defining the project id

Data loading

Foundation Models on watsonx

List available models

Classify customer complaints

Defining the model parameters

Generate the classification of Consumer Financial Protection Bureau (CFPB) document using `google/flan-ul2` model.

Score the model

Summary and next steps

Author:

Product

Resources

Company

Use watsonx, and Google google/flan-ul2 to classify consumer financial protection bureau document

Disclaimers

Notebook content

Learning goal

Use case & dataset

Contents

Set up the environment

Install and import the datasets and dependecies

Inferencing class

watsonx API connection

Defining the project id

Data loading

Foundation Models on watsonx

List available models

Classify customer complaints

Defining the model parameters

Generate the classification of Consumer Financial Protection Bureau (CFPB) document using google/flan-ul2 model.

Score the model

Summary and next steps

Author:

Use watsonx, and Google `google/flan-ul2` to classify consumer financial protection bureau document

Install and import the `datasets` and dependecies

Generate the classification of Consumer Financial Protection Bureau (CFPB) document using `google/flan-ul2` model.