GitHub Repository: ibm/watson-machine-learning-samples
Path: blob/master/cloud/notebooks/python_sdk/deployments/foundation_models/Use watsonx, and `mistral-small-3-1-24b-instruct-2503` to find sentiments of legal documents.ipynb
⁹⁴⁷⁹ views

Kernel: clear_env

Use watsonx, and `mistral-small-3-1-24b-instruct-2503` to analyze sentiments of legal documents

Disclaimers

Use only Projects and Spaces that are available in watsonx context.

Notebook content

This notebook contains the steps and code to demonstrate support of sentiment analysis in watsonx. It introduces commands for data retrieval and model testing.

Some familiarity with Python is helpful. This notebook uses Python 3.11.

Learning goal

The goal of this notebook is to demonstrate how to use mistralai/mistral-small-3-1-24b-instruct-2503 model to analyze sentiments of legal documents.

Use case & dataset

One of the key use cases of legal sentiment analysis is in assisting legal professionals in predicting case outcomes. By analyzing the sentiment expressed in previous court decisions and related documents, sentiment analysis algorithms can identify patterns and correlations between the sentiment and the final verdict. This can help lawyers and judges in assessing the strength of legal arguments, evaluating the potential impact of public opinion on the case, and making more accurate predictions about the likely outcome of ongoing cases. The dataset consists of two colums; the phrases and the sentiments.

This notebook contains the following parts:

Set up the environment

Before you use the sample code in this notebook, you must perform the following setup tasks:

Create a watsonx.ai Runtime Service instance (a free plan is offered and information about how to create the instance can be found here).

Install and import dependencies

Note: ibm-watsonx-ai documentation can be found here.

In [1]:

%pip install wget | tail -n 1
%pip install "scikit-learn==1.3.2" | tail -n 1
%pip install -U ibm-watsonx-ai | tail -n 1

Out[1]:

Successfully installed wget-3.2
Successfully installed joblib-1.5.2 numpy-1.26.4 scikit-learn-1.3.2 scipy-1.16.3 threadpoolctl-3.6.0
Successfully installed anyio-4.11.0 cachetools-6.2.1 certifi-2025.10.5 charset_normalizer-3.4.4 h11-0.16.0 httpcore-1.0.9 httpx-0.28.1 ibm-cos-sdk-2.14.3 ibm-cos-sdk-core-2.14.3 ibm-cos-sdk-s3transfer-2.14.3 ibm-watsonx-ai-1.4.4 idna-3.11 jmespath-1.0.1 lomond-0.3.3 pandas-2.2.3 pytz-2025.2 requests-2.32.5 sniffio-1.3.1 tabulate-0.9.0 tzdata-2025.2 urllib3-2.5.0

Defining the watsonx.ai credentials

This cell defines the watsonx.ai credentials required to work with watsonx Foundation Model inferencing.

Action: Provide the IBM Cloud user API key. For details, see documentation.

In [2]:

import getpass

from ibm_watsonx_ai import Credentials

credentials = Credentials(
    url="https://us-south.ml.cloud.ibm.com",
    api_key=getpass.getpass("Please enter your watsonx.ai api key (hit enter): "),
)

Defining the project id

The Foundation Model requires project id that provides the context for the call. We will obtain the id from the project in which this notebook runs. Otherwise, please provide the project id.

In [3]:

import os

try:
    project_id = os.environ["PROJECT_ID"]
except KeyError:
    project_id = input("Please enter your project_id (hit enter): ")

In [4]:

from ibm_watsonx_ai import APIClient

api_client = APIClient(credentials=credentials, project_id=project_id)

Data loading

Download the legal documents dataset.

In [5]:

import wget

filename = "Legal_Sentences.csv"
url = "https://raw.githubusercontent.com/kmokht1/Datasets/main/Legal_Sentences.csv"

if not os.path.isfile(filename):
    wget.download(url, out=filename)

Read the data.

In [6]:

import pandas as pd

data = pd.read_csv("Legal_Sentences.csv", index_col=0)
data = data[["Phrase", "Sentiment"]]
data.head()

Out[6]:

Prepare dataset label map.

In [7]:

label_map = {-1: "negative", 0: "neutral", 1: "positive"}

Inspect data sample.

In [8]:

data.value_counts(["Sentiment"])

Out[8]:

Sentiment
-1           282
 1           172
 0           122
Name: count, dtype: int64

Split the data into training and test sets.

In [9]:

from sklearn.model_selection import train_test_split

data_train, data_test, y_train, y_test = train_test_split(
    data["Phrase"],
    data["Sentiment"],
    test_size=0.3,
    random_state=33,
    stratify=data["Sentiment"],
)
data_train = pd.DataFrame(data_train)
data_test = pd.DataFrame(data_test)

Foundation Models on `watsonx.ai`

List available models

All avaliable models are presented under TextModels class. For more information refer to documentation.

In [10]:

print([model.name for model in api_client.foundation_models.TextModels])

Out[10]:

['DEFENCE_GRANITE', 'GRANITE_3_2_8B_INSTRUCT', 'GRANITE_3_2B_INSTRUCT', 'GRANITE_3_3_8B_INSTRUCT', 'GRANITE_3_8B_INSTRUCT', 'GRANITE_4_H_SMALL', 'GRANITE_8B_CODE_INSTRUCT', 'GRANITE_GUARDIAN_3_8B', 'GRANITE_VISION_3_2_2B', 'LLAMA_3_2_11B_VISION_INSTRUCT', 'LLAMA_3_2_90B_VISION_INSTRUCT', 'LLAMA_3_3_70B_INSTRUCT', 'LLAMA_3_405B_INSTRUCT', 'LLAMA_4_MAVERICK_17B_128E_INSTRUCT_FP8', 'LLAMA_GUARD_3_11B_VISION', 'MISTRAL_MEDIUM_2505', 'MISTRAL_SMALL_3_1_24B_INSTRUCT_2503', 'GPT_OSS_120B', 'ALLAM_1_13B_INSTRUCT']

You need to specify model_id that will be used for inferencing:

In [11]:

model_id = "mistralai/mistral-small-3-1-24b-instruct-2503"

Defining the model parameters

You might need to adjust model parameters for different models or tasks, to do so please refer to documentation.

In [12]:

from ibm_watsonx_ai.metanames import GenTextParamsMetaNames as GenParams

parameters = {
    GenParams.DECODING_METHOD: "greedy",
    GenParams.MIN_NEW_TOKENS: 1,
    GenParams.MAX_NEW_TOKENS: 2,
    GenParams.RANDOM_SEED: 33,
    GenParams.REPETITION_PENALTY: 1,
    GenParams.STOP_SEQUENCES: ["-1", "0", "1"],
}

Initialize the model

Initialize the ModelInference class with previous set params.

In [13]:

from ibm_watsonx_ai.foundation_models import ModelInference

model = ModelInference(
    model_id=model_id, params=parameters, credentials=credentials, project_id=project_id
)

Model's details

In [14]:

model.get_details()

Out[14]:

{'model_id': 'mistralai/mistral-small-3-1-24b-instruct-2503',
 'label': 'mistral-small-3-1-24b-instruct-2503',
 'provider': 'Mistral AI',
 'source': 'Hugging Face',
 'indemnity': 'NON_IBM',
 'functions': [{'id': 'autoai_rag'},
  {'id': 'image_chat'},
  {'id': 'text_chat'},
  {'id': 'text_generation'}],
 'short_description': 'This model is an instruction-finetuned version of: Mistral-Small-3.1-24B-Base-2503.',
 'long_description': 'Mistral Small 3 (2501), Mistral Small 3.1 (2503) adds state-of-the-art vision understanding and enhances long context capabilities up to 128k tokens without compromising text performance. With 24 billion parameters, this model achieves top-tier capabilities in both text and vision tasks.',
 'terms_url': 'https://www.apache.org/licenses/LICENSE-2.0',
 'input_tier': 'class_c1',
 'output_tier': 'class_17',
 'number_params': '24b',
 'min_shot_size': 1,
 'task_ids': ['question_answering',
  'summarization',
  'retrieval_augmented_generation',
  'retrieval_augmented_generation',
  'classification',
  'generation',
  'code',
  'extraction',
  'function_calling'],
 'tasks': [{'id': 'question_answering'},
  {'id': 'summarization'},
  {'id': 'retrieval_augmented_generation'},
  {'id': 'classification'},
  {'id': 'generation'},
  {'id': 'code'},
  {'id': 'extraction'},
  {'id': 'function_calling'}],
 'model_limits': {'max_sequence_length': 128000, 'max_output_tokens': 16384},
 'limits': {'3f6acf43-ede8-413a-ac69-f8af3bb0cbfe': {'max_output_tokens': 16384},
  'a3d2f92f-06f9-48d0-b2e6-a7ba2b4e0577': {'max_output_tokens': 16384},
  'd18d88b9-be7a-46ec-be1e-aff14904f1e9': {'max_output_tokens': 16384}},
 'lifecycle': [{'id': 'available', 'start_date': '2025-04-22'}],
 'versions': [{'version': '1.0.0', 'available_date': '2025-04-22'}]}

Find legal documents sentiments

Define instructions for the model.

In [15]:

instruction = """Determine the sentiment of the Input sentense. Response use one of the following sentiments 1 for positive, -1 for negative or 0 for neutral\n\n"""

Prepare model inputs for zero-shot example - use below zero_shot_inputs.

In [16]:

zero_shot_inputs = [{"input": text} for text in data_test["Phrase"]]
for i in range(2):
    print(f"The sentence example {i+1} is:\n\t {zero_shot_inputs[i]['input']}\n")

Out[16]:

The sentence example 1 is:
	 The Court rejects the CCAs conclusion that Moore failed to make the requisite showings with respect to intellectual functioning

The sentence example 2 is:
	 He argues on appeal that had Defendants written truthful reports, or testified truthfully in deposition

Prepare model inputs for few-shot examples - use below few_shot_inputs.

In [17]:

data_train_and_labels = data_train.copy()
data_train_and_labels["Sentiment"] = y_train

In [18]:

few_shot_example = []
few_shot_examples = []
for phrase, sentiment in (
    data_train_and_labels.groupby("Sentiment").apply(lambda x: x.sample(2)).values
):
    few_shot_example.append(f"Input: {phrase}\nOutput: {sentiment}\n\n")
few_shot_examples = ["".join(few_shot_example)]

In [19]:

few_shot_inputs_ = [
    {"input": f"Input: {text}\n"} for text in data_test["Phrase"].values
]
for i in range(4):
    print(f"The sentence example {i+1} is:\n {few_shot_inputs_[i]['input']}")
    print(f"\tSentiment: {y_test.values[i]}\n")

Out[19]:

The sentence example 1 is:
 Input: The Court rejects the CCAs conclusion that Moore failed to make the requisite showings with respect to intellectual functioning

	Sentiment: -1

The sentence example 2 is:
 Input: He argues on appeal that had Defendants written truthful reports, or testified truthfully in deposition

	Sentiment: 0

The sentence example 3 is:
 Input: obstruction statutes to include a proceeding requirement

	Sentiment: -1

The sentence example 4 is:
 Input: The North Carolina statute impermissibly restricts lawful speech

	Sentiment: -1

Generate the sentiments of legal documents using `mistral-small-3-1-24b-instruct-2503` model.

Get the docs summaries.

In [20]:

results = []
for inp in few_shot_inputs_[:4]:
    results.append(
        model.generate(
            "".join([instruction + few_shot_examples[0], inp["input"], "Output:"])
        )["results"][0]
    )

Explore model output.

In [21]:

import json

print(json.dumps(results, indent=2))

Out[21]:

[
  {
    "generated_text": " -1",
    "generated_token_count": 2,
    "input_token_count": 183,
    "stop_reason": "stop_sequence",
    "seed": 33
  },
  {
    "generated_text": " 0",
    "generated_token_count": 2,
    "input_token_count": 181,
    "stop_reason": "stop_sequence",
    "seed": 33
  },
  {
    "generated_text": " 0",
    "generated_token_count": 2,
    "input_token_count": 168,
    "stop_reason": "stop_sequence",
    "seed": 33
  },
  {
    "generated_text": " 1",
    "generated_token_count": 2,
    "input_token_count": 173,
    "stop_reason": "stop_sequence",
    "seed": 33
  }
]

Score the model

Note: To run the Score section for model scoring on the whole financial phrasebank dataset, please transform following markdown cells to code cells. Have in mind that scoring model on the whole test set can take significant amount of time.

Get the true labels.

y_true = [label_map[label] for label in y_test.values[:4]]
y_true

Get the prediction labels.

y_pred = [label_map[int(result['generated_text'].strip())] for result in results]
y_pred

Calculate the accuracy score.

from sklearn.metrics import accuracy_score

print(accuracy_score(y_pred, y_true))

Summary and next steps

You successfully completed this notebook!

You learned how to find sentiments of legal documents with mistral-small-3-1-24b-instruct-2503 on watsonx.

Check out our Online Documentation for more samples, tutorials, documentation, how-tos, and blog posts.

Authors:

Mateusz Szewczyk, Software Engineer at watsonx.ai.

Use watsonx, and `mistral-small-3-1-24b-instruct-2503` to analyze sentiments of legal documents

Disclaimers

Notebook content

Learning goal

Use case & dataset

Contents

Set up the environment

Install and import dependencies

Defining the watsonx.ai credentials

Defining the project id

Data loading

Foundation Models on `watsonx.ai`

List available models

Defining the model parameters

Initialize the model

Model's details

Find legal documents sentiments

Generate the sentiments of legal documents using `mistral-small-3-1-24b-instruct-2503` model.

Score the model

Summary and next steps

Authors:

Product

Resources

Company

Use watsonx, and mistral-small-3-1-24b-instruct-2503 to analyze sentiments of legal documents

Disclaimers

Notebook content

Learning goal

Use case & dataset

Contents

Set up the environment

Install and import dependencies

Defining the watsonx.ai credentials

Defining the project id

Data loading

Foundation Models on watsonx.ai

List available models

Defining the model parameters

Initialize the model

Model's details

Find legal documents sentiments

Generate the sentiments of legal documents using mistral-small-3-1-24b-instruct-2503 model.

Score the model

Summary and next steps

Authors:

Use watsonx, and `mistral-small-3-1-24b-instruct-2503` to analyze sentiments of legal documents

Foundation Models on `watsonx.ai`

Generate the sentiments of legal documents using `mistral-small-3-1-24b-instruct-2503` model.