Use watsonx to classify Consumer Financial Protection Bureau (CFPB) document.ipynb

Kernel: .venv_watsonx_ai_samples_py_312

Use watsonx to classify consumer financial protection bureau document

Disclaimers

• Use only Projects and Spaces that are available in watsonx context.

Notebook content

This notebook contains the steps and code to demonstrate support of complaint classification in watsonx. It introduces commands for data retrieval and model testing.

Some familiarity with Python is helpful. This notebook uses Python 3.12.

Learning goal

The goal of this notebook is to demonstrate how to use watsonx.ai model to Consumer Financial Protection Bureau (CFPB) document.

Use case & dataset

The Consumer Financial Protection Bureau (CFPB) is a federal U.S. agency that acts as a mediator when disputes arise between financial institutions and consumers. Via a web form, consumers can send the agency a narrative of their dispute. A foundation model would make the classification of complaints and their routing to the appropriate teams more efficient than manually tagged complaints. Dataset has narrative and 5 product categories as follow:

• credit reporting

• debt collection

• mortgages and loans

• credit cards

• retail banking

Contents

This notebook contains the following parts:

• Setup

• Data loading

• Foundation Models on watsonx

• Model testing

• Score

• Summary

Set up the environment

Before you use the sample code in this notebook, you must perform the following setup tasks:

• Create a watsonx.ai Runtime Service instance (a free plan is offered and information about how to create the instance can be found here).

Install and import the datasets and dependecies

you need to install below required dependencies to be able to continue

%pip install wget | tail -n 1
%pip install httpx | tail -n 1
%pip install pandas | tail -n 1
%pip install datasets | tail -n 1
%pip install ibm-cloud-sdk-core | tail -n 1
%pip install "scikit-learn==1.6.1" | tail -n 1

import getpass
import json
import os

import httpx
import pandas as pd
import wget
from ibm_cloud_sdk_core import IAMTokenManager
from sklearn.model_selection import train_test_split

Inferencing class

This cell defines a class that makes a REST API call to the watsonx Foundation Model inferencing API that we will use to generate output from the provided input. The class takes the access token created in the previous step, and uses it to make a REST API call with input, model id and model parameters. The response from the API call is returned as the cell output.

Action: Provide watsonx.ai Runtime url to work with watsonx.ai.

endpoint_url = input("Please enter your watsonx.ai Runtime endpoint url (hit enter): ")

Define a PromptClient class for prompts generation.

class PromptClient:
    def __init__(self, access_token: str, project_id: str, endpoint_url: str):
        self.project_id = project_id
        self.url = f"{endpoint_url.rstrip('/')}/ml/v1/text/chat"
        self.headers = {
            "Authorization": f"Bearer {access_token}",
            "Content-Type": "application/json",
        }

    def chat(self, model_id: str, messages: list[str], **params):
        payload = {
            "model_id": model_id,
            "messages": messages,
            "project_id": self.project_id,
            **params,
        }

        response = httpx.post(
            self.url,
            params={"version": "2024-03-19"},
            json=payload,
            headers=self.headers,
            timeout=30,
        )
        if response.status_code == 200:
            return response.json()
        else:
            raise RuntimeError(response.text)

watsonx API connection

This cell defines the credentials required to work with watsonx API for Foundation Model inferencing.

Action: Provide the IBM Cloud user API key. For details, see documentation.

access_token = IAMTokenManager(
    apikey=getpass.getpass("Please enter your watsonx.ai api key (hit enter): "),
    url="https://iam.cloud.ibm.com/identity/token",
).get_token()

Defining the project id

The API requires project id that provides the context for the call. We will obtain the id from the project in which this notebook runs:

try:
    project_id = os.environ["PROJECT_ID"]
except KeyError:
    project_id = input("Please enter your project_id (hit enter): ")

Data loading

Download the financial documents dataset.

filename = "Data_Finance.csv"

url = "https://raw.githubusercontent.com/IBM/watsonx-ai-samples/master/cloud/data/cfpb_complaints/cfpb_compliants.csv"

if not os.path.isfile(filename):
    wget.download(url, out=filename)

Read the data.

data = pd.read_csv(filename)
data = data[["narrative", "product"]]
data.head()

Inspect the product class distribution.

data["product"].value_counts().sort_values()

product
retail_banking         3863
credit_card            5031
mortgages_and_loans    5822
credit_reporting       6000
debt_collection        6642
Name: count, dtype: int64

Split data to train and test

data_train, data_test, y_train, y_test = train_test_split(
    data["narrative"],
    data["product"],
    test_size=0.3,
    random_state=33,
    stratify=data["product"],
)
data_train = pd.DataFrame(data_train)
data_test = pd.DataFrame(data_test)

Foundation Models on watsonx

List available chat models

models_json = httpx.get(
    endpoint_url + "/ml/v1/foundation_model_specs",
    headers={
        "Authorization": f"Bearer {access_token}",
        "Content-Type": "application/json",
        "Accept": "application/json",
    },
    params={
        "limit": 50,
        "version": "2024-03-19",
        "filters": "function_text_chat,!lifecycle_withdrawn:and",
    },
).json()

models_ids = [m["model_id"] for m in models_json["resources"]]
models_ids

['ibm/granite-3-2-8b-instruct',
 'ibm/granite-3-3-8b-instruct',
 'ibm/granite-3-3-8b-instruct-np',
 'ibm/granite-3-8b-instruct',
 'ibm/granite-4-h-small',
 'ibm/granite-guardian-3-8b',
 'meta-llama/llama-3-2-11b-vision-instruct',
 'meta-llama/llama-3-2-90b-vision-instruct',
 'meta-llama/llama-3-3-70b-instruct',
 'meta-llama/llama-3-405b-instruct',
 'meta-llama/llama-4-maverick-17b-128e-instruct-fp8',
 'meta-llama/llama-guard-3-11b-vision',
 'mistral-large-2512',
 'mistralai/mistral-medium-2505',
 'mistralai/mistral-small-3-1-24b-instruct-2503',
 'openai/gpt-oss-120b']

You need to specify model_id that will be used for inferencing:

model_id = "meta-llama/llama-3-3-70b-instruct"

Classify customer complaints

Prepare model inputs for zero-shot example - use below zero_shot_inputs.

zero_shot_inputs = [text for text in data_test["narrative"]]
for i in range(5):
    print(f"The sentence example {i+1} is:\n {zero_shot_inputs[i]}\n")

The sentence example 1 is:
 receiving current copy credit report discovered entry identified inquiry qualified deletion report

The sentence example 2 is:
 got scam told send via cash apps able sent via cash apps adamant need send complete total via paid total amount reading since going remove called need put downpayment material need church helping remove need put downpayment return money within sent money since limit consumer protection program able send noticed scammed filed report return money back denied dispute something need get done scammer cash apps need better proctection protect innocent customer like

The sentence example 3 is:
 service member elected servicemembers civil relief act usaa prior departure reduced auto insurance due vehicle use gone austere environment limited communication mail forwarded location member deactivate cell phone gone turn back upon return mail sent arriving late month later due remote location logistical challenge usaa visa credit card well year issue credit limit card balance reward point due equating cashed apparently charge appeared card departed unaware receive communication lender could reach assuming tried usaa closed credit card reported day non payment credit bureau upon return immediately paid amount attempted numerous time unsuccessfully resolve issue usaa mentioned account voluntarily closed point would available day would allow credit instance negative reporting credit report brought score scra respectfully request removal negative credit reporting allow cash payment reward point balance credit reporting loss available credit would good credit limit restored well negative reporting credit cost year come inability obtain lower mortgage loan rate credit card etc due relocate week plan purchase home many thanks relief assistance could provide sincerely military member

The sentence example 4 is:
 contacted u bank fraud department said couldnt find account worry need make sure case fraud happening suspect might part

The sentence example 5 is:
 pandemic hoping youre safe sound listen shocked reviewed credit report today found late payment

Prepare model inputs for few-shot examples - use below few_shot_inputs.

data_train_and_labels = data_train.copy()
data_train_and_labels["product"] = y_train

few_shot_example = []

sampled = data_train_and_labels.groupby("product", group_keys=False).sample(n=2)

for _, row in sampled.iterrows():
    phrase = row["narrative"]
    product = row["product"]
    few_shot_example.append(f"\tsentence:\t{phrase}\n\tproduct: {product}\n")

few_shot_examples = ["".join(few_shot_example)]

few_shot_inputs_ = [text for text in data_test["narrative"].values]
for i in range(5):
    print(f"The sentence example {i+1} is:\n {few_shot_inputs_[i]}\n")

The sentence example 1 is:
 receiving current copy credit report discovered entry identified inquiry qualified deletion report

The sentence example 2 is:
 got scam told send via cash apps able sent via cash apps adamant need send complete total via paid total amount reading since going remove called need put downpayment material need church helping remove need put downpayment return money within sent money since limit consumer protection program able send noticed scammed filed report return money back denied dispute something need get done scammer cash apps need better proctection protect innocent customer like

The sentence example 3 is:
 service member elected servicemembers civil relief act usaa prior departure reduced auto insurance due vehicle use gone austere environment limited communication mail forwarded location member deactivate cell phone gone turn back upon return mail sent arriving late month later due remote location logistical challenge usaa visa credit card well year issue credit limit card balance reward point due equating cashed apparently charge appeared card departed unaware receive communication lender could reach assuming tried usaa closed credit card reported day non payment credit bureau upon return immediately paid amount attempted numerous time unsuccessfully resolve issue usaa mentioned account voluntarily closed point would available day would allow credit instance negative reporting credit report brought score scra respectfully request removal negative credit reporting allow cash payment reward point balance credit reporting loss available credit would good credit limit restored well negative reporting credit cost year come inability obtain lower mortgage loan rate credit card etc due relocate week plan purchase home many thanks relief assistance could provide sincerely military member

The sentence example 4 is:
 contacted u bank fraud department said couldnt find account worry need make sure case fraud happening suspect might part

The sentence example 5 is:
 pandemic hoping youre safe sound listen shocked reviewed credit report today found late payment

Generate the classification of Consumer Financial Protection Bureau (CFPB) document using watsonx.ai model.

Note: You might need to adjust model parameters for different models or tasks, to do so please refer to documentation.

Initialize the PromptClient class.

Hint: Your authentication token might expire, if so please regenerate the access_token and reinitialize the PromptClient class.

prompt_client = PromptClient(access_token, project_id, endpoint_url)

Define instructions for the model and get the product classes.

instruction = """Determine the class of product expressed in the sentense.
Use either 'credit_card', 'debt_collection', 'mortgages_and_loans', 'retail_banking', or 'credit_reporting'.
Use the provided examples as a template. """

results = []

for inp in zero_shot_inputs[:5]:
    results.append(
        prompt_client.chat(
            messages=[
                {"role": "system", "content": instruction + few_shot_examples[0]},
                {"role": "user", "content": inp},
            ],
            model_id=model_id,
            guided_choice=[
                "credit_card",
                "unsatisdebt_collectionfied",
                "mortgages_and_loans",
                "retail_banking",
                "credit_reporting",
            ],
        )
    )

Explore model output.

print(json.dumps(results, indent=2))

[
  {
    "id": "chatcmpl-be347cf2e04a831365abea163bfaf138---99b7b8f0-e460-4a79-8315-c9a5d879de3e",
    "object": "chat.completion",
    "model_id": "meta-llama/llama-3-3-70b-instruct",
    "model": "meta-llama/llama-3-3-70b-instruct",
    "choices": [
      {
        "index": 0,
        "message": {
          "role": "assistant",
          "content": "credit_reporting"
        },
        "finish_reason": "stop"
      }
    ],
    "created": 1768914409,
    "model_version": "3.3.0",
    "created_at": "2026-01-20T13:06:50.431Z",
    "usage": {
      "completion_tokens": 3,
      "prompt_tokens": 1835,
      "total_tokens": 1838
    },
    "system": {
      "warnings": [
        {
          "message": "This model is a Non-IBM Product governed by a third-party license that may impose use restrictions and other obligations. By using this model you agree to its terms as identified in the following URL.",
          "id": "disclaimer_warning",
          "more_info": "https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-models.html?context=wx"
        },
        {
          "message": "The value of 'max_tokens' for this model was set to value 1024",
          "id": "unspecified_max_token",
          "additional_properties": {
            "limit": 0,
            "new_value": 1024,
            "parameter": "max_tokens",
            "value": 0
          }
        }
      ]
    }
  },
  {
    "id": "chatcmpl-76ffa3019989d542f511715a92a13435---1c64c2c1-abbc-4bd1-99b1-8640496b1c44",
    "object": "chat.completion",
    "model_id": "meta-llama/llama-3-3-70b-instruct",
    "model": "meta-llama/llama-3-3-70b-instruct",
    "choices": [
      {
        "index": 0,
        "message": {
          "role": "assistant",
          "content": "retail_banking"
        },
        "finish_reason": "stop"
      }
    ],
    "created": 1768914410,
    "model_version": "3.3.0",
    "created_at": "2026-01-20T13:06:51.697Z",
    "usage": {
      "completion_tokens": 5,
      "prompt_tokens": 1900,
      "total_tokens": 1905
    },
    "system": {
      "warnings": [
        {
          "message": "This model is a Non-IBM Product governed by a third-party license that may impose use restrictions and other obligations. By using this model you agree to its terms as identified in the following URL.",
          "id": "disclaimer_warning",
          "more_info": "https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-models.html?context=wx"
        },
        {
          "message": "The value of 'max_tokens' for this model was set to value 1024",
          "id": "unspecified_max_token",
          "additional_properties": {
            "limit": 0,
            "new_value": 1024,
            "parameter": "max_tokens",
            "value": 0
          }
        }
      ]
    }
  },
  {
    "id": "chatcmpl-e117bf4b1c130c22cc1d18d6a93fa9c1---a079a6ff-6c6f-438a-be84-285a69b5e704",
    "object": "chat.completion",
    "model_id": "meta-llama/llama-3-3-70b-instruct",
    "model": "meta-llama/llama-3-3-70b-instruct",
    "choices": [
      {
        "index": 0,
        "message": {
          "role": "assistant",
          "content": "credit_card"
        },
        "finish_reason": "stop"
      }
    ],
    "created": 1768914412,
    "model_version": "3.3.0",
    "created_at": "2026-01-20T13:06:53.353Z",
    "usage": {
      "completion_tokens": 3,
      "prompt_tokens": 1999,
      "total_tokens": 2002
    },
    "system": {
      "warnings": [
        {
          "message": "This model is a Non-IBM Product governed by a third-party license that may impose use restrictions and other obligations. By using this model you agree to its terms as identified in the following URL.",
          "id": "disclaimer_warning",
          "more_info": "https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-models.html?context=wx"
        },
        {
          "message": "The value of 'max_tokens' for this model was set to value 1024",
          "id": "unspecified_max_token",
          "additional_properties": {
            "limit": 0,
            "new_value": 1024,
            "parameter": "max_tokens",
            "value": 0
          }
        }
      ]
    }
  },
  {
    "id": "chatcmpl-2dc400af2a28177ff7d92d021c2210ce---e1b9eb9f-193c-4b49-a52a-04a299fdbb83",
    "object": "chat.completion",
    "model_id": "meta-llama/llama-3-3-70b-instruct",
    "model": "meta-llama/llama-3-3-70b-instruct",
    "choices": [
      {
        "index": 0,
        "message": {
          "role": "assistant",
          "content": "retail_banking"
        },
        "finish_reason": "stop"
      }
    ],
    "created": 1768914413,
    "model_version": "3.3.0",
    "created_at": "2026-01-20T13:06:55.363Z",
    "usage": {
      "completion_tokens": 5,
      "prompt_tokens": 1843,
      "total_tokens": 1848
    },
    "system": {
      "warnings": [
        {
          "message": "This model is a Non-IBM Product governed by a third-party license that may impose use restrictions and other obligations. By using this model you agree to its terms as identified in the following URL.",
          "id": "disclaimer_warning",
          "more_info": "https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-models.html?context=wx"
        },
        {
          "message": "The value of 'max_tokens' for this model was set to value 1024",
          "id": "unspecified_max_token",
          "additional_properties": {
            "limit": 0,
            "new_value": 1024,
            "parameter": "max_tokens",
            "value": 0
          }
        }
      ]
    }
  },
  {
    "id": "chatcmpl-49c5da71da08b71a0ab947c5fbbe6814---8e81f820-2463-417b-b0a2-dd2a68c5e687",
    "object": "chat.completion",
    "model_id": "meta-llama/llama-3-3-70b-instruct",
    "model": "meta-llama/llama-3-3-70b-instruct",
    "choices": [
      {
        "index": 0,
        "message": {
          "role": "assistant",
          "content": "credit_reporting"
        },
        "finish_reason": "stop"
      }
    ],
    "created": 1768914415,
    "model_version": "3.3.0",
    "created_at": "2026-01-20T13:06:59.520Z",
    "usage": {
      "completion_tokens": 3,
      "prompt_tokens": 1838,
      "total_tokens": 1841
    },
    "system": {
      "warnings": [
        {
          "message": "This model is a Non-IBM Product governed by a third-party license that may impose use restrictions and other obligations. By using this model you agree to its terms as identified in the following URL.",
          "id": "disclaimer_warning",
          "more_info": "https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-models.html?context=wx"
        },
        {
          "message": "The value of 'max_tokens' for this model was set to value 1024",
          "id": "unspecified_max_token",
          "additional_properties": {
            "limit": 0,
            "new_value": 1024,
            "parameter": "max_tokens",
            "value": 0
          }
        }
      ]
    }
  }
]

Score the model

Note: To run the Score section for model scoring on the whole financial phrasebank dataset, please transform following markdown cells to code cells. Have in mind that scoring model on the whole test set can take significant amount of time.

Get the true labels.

y_true = y_test.values[:5]
print(y_true)

Get the prediction labels.

y_pred = [res["choices"][0]["message"]["content"] for res in results]
y_pred

Calculate the accuracy score.

from sklearn.metrics import accuracy_score

print(accuracy_score(y_pred, y_true))

Summary and next steps

You successfully completed this notebook!

You learned how to classify Consumer Financial Protection Bureau documents with LLM's on watsonx.

Check out our Online Documentation for more samples, tutorials, documentation, how-tos, and blog posts.

Author:

Kahila Mokhtari

Mateusz Szewczyk, Software Engineer at watsonx.ai.

Copyright © 2026 IBM. This notebook and its source code are released under the terms of the MIT License.