GitHub Repository: ibm/watson-machine-learning-samples
Path: blob/master/cloud/notebooks/python_sdk/deployments/foundation_models/Use watsonx, and `mixtral-8x7b-instruct-v01` to find sentiments of legal documents.ipynb
⁶⁴⁰⁵ views

Kernel: Python 3 (ipykernel)

Use watsonx, and `mixtral-8x7b-instruct-v01` to analyze sentiments of legal documents

Disclaimers

Use only Projects and Spaces that are available in watsonx context.

Notebook content

This notebook contains the steps and code to demonstrate support of sentiment analysis in watsonx. It introduces commands for data retrieval and model testing.

Some familiarity with Python is helpful. This notebook uses Python 3.11.

Learning goal

The goal of this notebook is to demonstrate how to use mistralai/mixtral-8x7b-instruct-v01 model to analyze sentiments of legal documents.

Use case & dataset

One of the key use cases of legal sentiment analysis is in assisting legal professionals in predicting case outcomes. By analyzing the sentiment expressed in previous court decisions and related documents, sentiment analysis algorithms can identify patterns and correlations between the sentiment and the final verdict. This can help lawyers and judges in assessing the strength of legal arguments, evaluating the potential impact of public opinion on the case, and making more accurate predictions about the likely outcome of ongoing cases. The dataset consists of two colums; the phrases and the sentiments.

This notebook contains the following parts:

Set up the environment

Before you use the sample code in this notebook, you must perform the following setup tasks:

Create a watsonx.ai Runtime Service instance (a free plan is offered and information about how to create the instance can be found here).

Install and import the `datasets` and dependecies

In [ ]:

!pip install wget | tail -n 1
!pip install "scikit-learn==1.3.2" | tail -n 1
!pip install -U ibm-watsonx-ai | tail -n 1

Defining the watsonx.ai credentials

This cell defines the watsonx.ai credentials required to work with watsonx Foundation Model inferencing.

Action: Provide the IBM Cloud user API key. For details, see documentation.

In [1]:

import getpass
from ibm_watsonx_ai import Credentials

credentials = Credentials(
    url="https://us-south.ml.cloud.ibm.com",
    api_key=getpass.getpass("Please enter your watsonx.ai api key (hit enter): "),
)

Defining the project id

The Foundation Model requires project id that provides the context for the call. We will obtain the id from the project in which this notebook runs. Otherwise, please provide the project id.

In [2]:

import os

try:
    project_id = os.environ["PROJECT_ID"]
except KeyError:
    project_id = input("Please enter your project_id (hit enter): ")

Data loading

Download the legal documents dataset.

In [3]:

import wget

filename = 'Legal_Sentences.csv'
url = 'https://raw.githubusercontent.com/kmokht1/Datasets/main/Legal_Sentences.csv'

if not os.path.isfile(filename): 
    wget.download(url, out=filename)

Read the data.

In [4]:

import pandas as pd

data = pd.read_csv("Legal_Sentences.csv", index_col=0 )
data = data[['Phrase','Sentiment']]
data.head()

Out[4]:

Prepare dataset label map.

In [5]:

label_map = {
    -1: "negative",
    0: "neutral",
    1: "positive"
}

Inspect data sample.

In [6]:

data.value_counts(['Sentiment'])

Out[6]:

Sentiment
-1           282
 1           172
 0           122
dtype: int64

Split the data into training and test sets.

In [7]:

from sklearn.model_selection import train_test_split

data_train, data_test, y_train, y_test = train_test_split(data['Phrase'], 
                                                    data['Sentiment'],
                                                    test_size=0.3,
                                                    random_state=33, 
                                                    stratify=data['Sentiment'])
data_train = pd.DataFrame(data_train)
data_test = pd.DataFrame(data_test)

Foundation Models on `watsonx.ai`

List available models

All avaliable models are presented under ModelTypes class. For more information refer to documentation.

In [8]:

from ibm_watsonx_ai.foundation_models.utils.enums import ModelTypes

print([model.name for model in ModelTypes])

Out[8]:

['FLAN_T5_XXL', 'FLAN_UL2', 'MT0_XXL', 'GPT_NEOX', 'MPT_7B_INSTRUCT2', 'STARCODER', 'LLAMA_2_70B_CHAT', 'LLAMA_2_13B_CHAT', 'GRANITE_13B_INSTRUCT', 'GRANITE_13B_CHAT', 'FLAN_T5_XL', 'GRANITE_13B_CHAT_V2', 'GRANITE_13B_INSTRUCT_V2', 'ELYZA_JAPANESE_LLAMA_2_7B_INSTRUCT', 'MIXTRAL_8X7B_INSTRUCT_V01_Q', 'CODELLAMA_34B_INSTRUCT_HF', 'GRANITE_20B_MULTILINGUAL']

You need to specify model_id that will be used for inferencing:

In [9]:

model_id = "mistralai/mixtral-8x7b-instruct-v01"

Defining the model parameters

You might need to adjust model parameters for different models or tasks, to do so please refer to documentation.

In [10]:

from ibm_watsonx_ai.metanames import GenTextParamsMetaNames as GenParams

parameters = {
    GenParams.DECODING_METHOD: "greedy",
    GenParams.MIN_NEW_TOKENS: 1,
    GenParams.MAX_NEW_TOKENS: 2,
    GenParams.RANDOM_SEED: 33,
    GenParams.REPETITION_PENALTY: 1,
    GenParams.STOP_SEQUENCES: ["-1", "0", "1"]
}

Initialize the model

Initialize the ModelInference class with previous set params.

In [11]:

from ibm_watsonx_ai.foundation_models import ModelInference

model = ModelInference(
    model_id=model_id, 
    params=parameters, 
    credentials=credentials,
    project_id=project_id)

Model's details

In [12]:

model.get_details()

Out[12]:

{'model_id': 'mistralai/mixtral-8x7b-instruct-v01',
 'label': 'mixtral-8x7b-instruct-v01',
 'provider': 'Mistral AI',
 'source': 'Hugging Face',
 'functions': [],
 'short_description': 'The Mixtral-8x7B Large Language Model (LLM) is a pretrained generative Sparse Mixture of Experts.',
 'long_description': "This model is made with AutoGPTQ, which mainly leverages the quantization technique to 'compress' the model weights from FP16 to 4-bit INT and performs 'decompression' on-the-fly before computation (in FP16). As a result, the GPU memory, and the data transferring between GPU memory and GPU compute engine, compared to the original FP16 model, is greatly reduced. The major quantization parameters used in the process are listed below.",
 'tier': 'class_1',
 'number_params': '46.7b',
 'min_shot_size': 1,
 'task_ids': ['summarization',
  'retrieval_augmented_generation',
  'classification',
  'generation',
  'code',
  'extraction'],
 'tasks': [{'id': 'summarization', 'ratings': {'quality': 4}},
  {'id': 'retrieval_augmented_generation', 'ratings': {'quality': 3}},
  {'id': 'classification', 'ratings': {'quality': 4}},
  {'id': 'generation'},
  {'id': 'code'},
  {'id': 'extraction', 'ratings': {'quality': 4}}],
 'model_limits': {'max_sequence_length': 32768},
 'limits': {'lite': {'call_time': '5m0s', 'max_output_tokens': 16384},
  'v2-professional': {'call_time': '10m0s', 'max_output_tokens': 16384},
  'v2-standard': {'call_time': '10m0s', 'max_output_tokens': 16384}},
 'lifecycle': [{'id': 'available', 'start_date': '2024-04-04'}]}

Find legal documents sentiments

Define instructions for the model.

In [13]:

instruction="""Determine the sentiment of the Input sentense. Response use one of the following sentiments 1 for positive, -1 for negative or 0 for neutral\n\n"""

Prepare model inputs for zero-shot example - use below zero_shot_inputs.

In [14]:

zero_shot_inputs = [{"input": text} for text in data_test['Phrase']]
for i in range(2):
    print(f"The sentence example {i+1} is:\n\t {zero_shot_inputs[i]['input']}\n")

Out[14]:

The sentence example 1 is:
	 The Court rejects the CCAs conclusion that Moore failed to make the requisite showings with respect to intellectual functioning

The sentence example 2 is:
	 He argues on appeal that had Defendants written truthful reports, or testified truthfully in deposition

Prepare model inputs for few-shot examples - use below few_shot_inputs.

In [15]:

data_train_and_labels=data_train.copy()
data_train_and_labels['Sentiment']=y_train

In [16]:

few_shot_example=[]
few_shot_examples=[]
for phrase,sentiment in data_train_and_labels.groupby('Sentiment').apply(lambda x: x.sample(2)).values:
    few_shot_example.append(f"Input: {phrase}\nOutput: {sentiment}\n\n")
few_shot_examples=[''.join(few_shot_example)]

In [17]:

few_shot_inputs_ = [{"input": f"Input: {text}\n"} for text in data_test['Phrase'].values]
for i in range(4):
    print(f"The sentence example {i+1} is:\n {few_shot_inputs_[i]['input']}")
    print(f"\tSentiment: {y_test.values[i]}\n")

Out[17]:

The sentence example 1 is:
 Input: The Court rejects the CCAs conclusion that Moore failed to make the requisite showings with respect to intellectual functioning

	Sentiment: -1

The sentence example 2 is:
 Input: He argues on appeal that had Defendants written truthful reports, or testified truthfully in deposition

	Sentiment: 0

The sentence example 3 is:
 Input: obstruction statutes to include a proceeding requirement

	Sentiment: -1

The sentence example 4 is:
 Input: The North Carolina statute impermissibly restricts lawful speech

	Sentiment: -1

Generate the sentiments of legal documents using `mixtral-8x7b-instruct-v01` model.

Get the docs summaries.

In [19]:

results = []
for inp in few_shot_inputs_[:4]:
    results.append(model.generate("".join([instruction+few_shot_examples[0], inp['input'], "Output:"]))["results"][0])

Explore model output.

In [20]:

import json

print(json.dumps(results, indent=2))

Out[20]:

[
  {
    "generated_text": " 1",
    "generated_token_count": 2,
    "input_token_count": 227,
    "stop_reason": "stop_sequence",
    "seed": 33
  },
  {
    "generated_text": " 0",
    "generated_token_count": 2,
    "input_token_count": 226,
    "stop_reason": "stop_sequence",
    "seed": 33
  },
  {
    "generated_text": " 0",
    "generated_token_count": 2,
    "input_token_count": 213,
    "stop_reason": "stop_sequence",
    "seed": 33
  },
  {
    "generated_text": " -1",
    "generated_token_count": 2,
    "input_token_count": 216,
    "stop_reason": "stop_sequence",
    "seed": 33
  }
]

Score the model

Note: To run the Score section for model scoring on the whole financial phrasebank dataset, please transform following markdown cells to code cells. Have in mind that scoring model on the whole test set can take significant amount of time.

Get the true labels.

y_true = [label_map[label] for label in y_test.values[:4]]
y_true

Get the prediction labels.

y_pred = [label_map[int(result['generated_text'].strip())] for result in results]
y_pred

Calculate the accuracy score.

from sklearn.metrics import accuracy_score

print(accuracy_score(y_pred, y_true))

Summary and next steps

You successfully completed this notebook!

You learned how to find sentiments of legal documents with mixtral-8x7b-instruct-v01 on watsonx.

Check out our Online Documentation for more samples, tutorials, documentation, how-tos, and blog posts.

Authors:

Mateusz Szewczyk, Software Engineer at watsonx.ai.

Use watsonx, and `mixtral-8x7b-instruct-v01` to analyze sentiments of legal documents

Disclaimers

Notebook content

Learning goal

Use case & dataset

Contents

Set up the environment

Install and import the `datasets` and dependecies

Defining the watsonx.ai credentials

Defining the project id

Data loading

Foundation Models on `watsonx.ai`

List available models

Defining the model parameters

Initialize the model

Model's details

Find legal documents sentiments

Generate the sentiments of legal documents using `mixtral-8x7b-instruct-v01` model.

Score the model

Summary and next steps

Authors:

Product

Resources

Company

Use watsonx, and mixtral-8x7b-instruct-v01 to analyze sentiments of legal documents

Disclaimers

Notebook content

Learning goal

Use case & dataset

Contents

Set up the environment

Install and import the datasets and dependecies

Defining the watsonx.ai credentials

Defining the project id

Data loading

Foundation Models on watsonx.ai

List available models

Defining the model parameters

Initialize the model

Model's details

Find legal documents sentiments

Generate the sentiments of legal documents using mixtral-8x7b-instruct-v01 model.

Score the model

Summary and next steps

Authors:

Use watsonx, and `mixtral-8x7b-instruct-v01` to analyze sentiments of legal documents

Install and import the `datasets` and dependecies

Foundation Models on `watsonx.ai`

Generate the sentiments of legal documents using `mixtral-8x7b-instruct-v01` model.