CoCalc -- 3 Transformers for NLP.ipynb

GitHub Repository: suyashi29/python-su
Path: blob/master/GenAI Transformers Basics/3 Transformers for NLP.ipynb
³⁰⁷⁴ views

Kernel: Python 3 (ipykernel)

Transformers" in the context of artificial intelligence are a type of computer model designed to understand and generate human language. They're really good at tasks like translating languages, answering questions, and generating text.

Transformers rely on a mechanism called "self-attention" to weigh the importance of different words in a sentence when processing language data. This mechanism allows them to capture long-range dependencies and relationships between words more effectively than previous models. As a result, Transformers have achieved state-of-the-art performance in many NLP tasks, including language translation, text summarization, question answering, and sentiment analysis.
In the context of Gen AI, Transformers represent a foundational technology that enables machines to understand and generate human-like text, facilitating more advanced and natural interactions between AI systems and humans. They are a key component in advancing AI capabilities towards more sophisticated language understanding and generation tasks.

The key components of Transformer models include:

1. Self-Attention Mechanism: This is the core component of Transformers. Self-attention allows the model to weigh the importance of different words in a sentence when processing language data. It enables capturing contextual relationships between words in a sequence, facilitating better understanding of the input.
1. Multi-Head Attention: In Transformers, self-attention is typically used in multiple "heads" or parallel attention mechanisms. Each head allows the model to focus on different parts of the input, enabling it to capture different types of relationships simultaneously.
1. Positional Encoding: Since Transformer models do not inherently understand the sequential order of input tokens like recurrent neural networks (RNNs), positional encoding is added to the input embeddings to provide information about the position of each token in the sequence.
1. Feedforward Neural Networks: Transformers include feedforward neural networks as part of their architecture. These networks are applied independently to each token's representation after self-attention and positional encoding, allowing the model to capture non-linear relationships between features.
1. Encoder and Decoder Layers: Transformer architectures often consist of encoder and decoder layers. The encoder processes the input sequence, while the decoder generates the output sequence in tasks like sequence-to-sequence translation. Each layer in the encoder and decoder typically includes self-attention and feedforward neural network sub-layers.
1. Residual Connections and Layer Normalization: To facilitate training deep networks, Transformers use residual connections around each sub-layer followed by layer normalization. These techniques help alleviate the vanishing gradient problem and improve the flow of information through the network.
1. Masking: In tasks like language translation, where the entire input sequence is available during training, masking is applied to prevent the model from attending to future tokens when predicting the output sequence.

These components work together to enable Transformers to achieve state-of-the-art performance in various natural language processing tasks.

In [ ]:

"Life is good"

["Life","is","Good"] - Token of words

[1 0  0 
0  1  0
0  0  1] * Weight Random matrix

Word Embedding

Word embedding is a technique used to represent words as vectors (arrays of numbers). These vectors capture semantic relationships between words.

In [1]:

## Word Emdedding using Python 
import numpy as np

# Define the vocabulary
vocab = {"Suyashi": 0, "is": 1, "Happy": 2}


# One-hot encode each word
def one_hot_encoding(word_index, vocab_size):
    one_hot = np.zeros(vocab_size)
    one_hot[word_index] = 1
    return one_hot

# Define the weight matrix (random initialization for simplicity)
weight_matrix = np.random.rand(len(vocab), 3)

# Multiply one-hot encoded words with the weight matrix
word_embeddings = {}
for word, index in vocab.items():
    one_hot_encoded = one_hot_encoding(index, len(vocab))
    embedding = np.dot(one_hot_encoded, weight_matrix)
    word_embeddings[word] = embedding

# Display word embeddings
for word, embedding in word_embeddings.items():
    print(f"{word}: {embedding}")

Out[1]:

Suyashi: [0.51893542 0.53861632 0.74286038]
is: [0.69534138 0.39759632 0.06065792]
Happy: [0.99723853 0.41859917 0.95514426]

let's implement a basic position encoding scheme to add positional information to the word embeddings:

Positional encoding is a crucial concept in the Transformer architecture, enabling the model to capture the order of words in a sequence since Transformers lack inherent sequential information. Let's break down the concept using a simple example with the sentence "suyashi is happy".

Understanding Positional Encoding In a Transformer model, each word is embedded into a high-dimensional space using an embedding matrix. However, the position of each word in the sequence is not captured by these embeddings. Positional encoding addresses this by adding a unique positional information to each word embedding.
Positional Encoding Formula The most common method for positional encoding in Transformers involves using sine and cosine functions of different frequencies. For a position 𝑝 𝑜 𝑠 pos and a dimension 𝑖 i of the encoding, the positional encoding is defined as:

3. Example: "suyashi is happy"

Let's consider a simple example where we encode a small part of the sequence using positional encoding. Assume the sentence "suyashi is happy" has four positions (0, 1, 2, 3), and we use a small dimension d=4 for simplicity.

Resulting Positional Encodings

Combining these, we get the positional encodings for each position

If the word embeddings for "suyashi", "is", "happy" are vectors, the positional encoding vectors would be added to these embeddings. This addition ensures that the model is aware of the position of each word in the sentence.

In practice, these operations are done over higher-dimensional spaces and with many more positions, but the fundamental idea remains the same. The positional encodings help the Transformer model understand the order and position of words within a sequence

In [2]:

import numpy as np

# Position Encoding
def position_encoding(sentence_length, embedding_dim):
    position_encodings = np.zeros((sentence_length, embedding_dim))
    for pos in range(sentence_length):
        for i in range(embedding_dim):
            if i % 2 == 0:
                position_encodings[pos, i] = np.sin(pos / (10000 ** (i / embedding_dim)))
            else:
                position_encodings[pos, i] = np.cos(pos / (10000 ** ((i - 1) / embedding_dim)))
    return position_encodings

# Assuming we have word embeddings for "Suyashi", "is", "Happy" as follows
word_embeddings = {
    "Suyashi": np.array([0.1, 0.2, 0.3]),
    "is": np.array([0.2, 0.3, 0.1]),
    "Happy": np.array([0.3, 0.1, 0.2])
}

# Get the position encodings
sentence_length = 3
embedding_dim = 3
pos_encodings = position_encoding(sentence_length, embedding_dim)

# Add position encodings to word embeddings
for i, word in enumerate(word_embeddings):
    word_embeddings[word] += pos_encodings[i % sentence_length]

# Display word embeddings with position encodings
print("Word embeddings with position encodings:")
for word, embedding in word_embeddings.items():
    print(f"{word}: {embedding}")

Out[2]:

Word embeddings with position encodings:
Suyashi: [0.1 1.2 0.3]
is: [1.04147098 0.84030231 0.10215443]
Happy: [ 1.20929743 -0.31614684  0.20430886]

In [ ]:

import numpy as np
A = np.array([ [4, 10, 11], [21, 22, 23], [31, 32, 33] ])
B = np.ones((3,3))

print("\nMultiplying two arrays: ")
print(A * B)

print("Matrix Multiplication")
print(np.dot (A,B))

In [ ]:

B = np.ones((3,3))
B

We define the position encoding function and word embeddings as we did previously.
We concatenate word embeddings with position encodings to create input vectors.
The self_attention function calculates attention scores using dot product and scales them by the square root of the dimensionality of the embeddings.
Softmax is applied to obtain attention weights.
The attention weights are then used to compute the attended inputs by applying them to the input vectors.
Finally, we display the attended inputs and attention weights.

In [3]:

import numpy as np

# Position Encoding
def position_encoding(sentence_length, embedding_dim):
    position_encodings = np.zeros((sentence_length, embedding_dim))
    for pos in range(sentence_length):
        for i in range(embedding_dim):
            if i % 2 == 0:
                position_encodings[pos, i] = np.sin(pos / (10000 ** (i / embedding_dim)))
            else:
                position_encodings[pos, i] = np.cos(pos / (10000 ** ((i - 1) / embedding_dim)))
    return position_encodings

# Word embeddings for "Suyashi", "is", "Happy"
word_embeddings = {
    "Suyashi": np.array([0.1, 0.2, 0.3]),
    "is": np.array([0.2, 0.3, 0.1]),
    "Happy": np.array([0.3, 0.1, 0.2])
}

# Get the position encodings
sentence_length = 3
embedding_dim = 3
pos_encodings = position_encoding(sentence_length, embedding_dim)

# Concatenate word embeddings with position encodings
inputs = np.array([word_embeddings[word] + pos_encodings[i] for i, word in enumerate(word_embeddings)])

# Self-attention mechanism
def self_attention(inputs):
    # Calculate attention scores
    attention_scores = np.dot(inputs, inputs.T) / np.sqrt(inputs.shape[-1])
    
    # Apply softmax to obtain attention weights
    attention_weights = np.exp(attention_scores) / np.sum(np.exp(attention_scores), axis=-1, keepdims=True)
    
    # Apply attention weights to inputs
    attended_inputs = np.dot(attention_weights, inputs)
    
    return attended_inputs, attention_weights

# Apply self-attention
attended_inputs, attention_weights = self_attention(inputs)

# Display results
print("Attended Inputs:")
print(attended_inputs)
print("\nAttention Weights:")
print(attention_weights)

Out[3]:

Attended Inputs:
[[0.63448685 0.81047639 0.21099442]
 [0.80976342 0.62970605 0.18847823]
 [0.96158921 0.34185482 0.18548879]]

Attention Weights:
[[0.46252785 0.36781824 0.16965391]
 [0.29492673 0.4312344  0.27383886]
 [0.17117121 0.34457284 0.48425595]]

Simplified implementation using NumPy to demonstrate self-attention

steps:

X represents the input token embeddings.
W_q, W_k, and W_v represent the query, key, and value projection matrices respectively.
We compute the dot products of query and key matrices to get the attention scores.
Softmax is applied to the attention scores to get the attention weights.
Finally, we compute the weighted sum of value vectors to get the output.

In [4]:

import numpy as np

# Define input tokens (embedding vectors)
X = np.array([
    [0.1, 0.2, 0.3],  # Embedding vector for "Ashi"
    [0.4, 0.5, 0.6],  # Embedding vector for "is"
    [0.7, 0.8, 0.9],  # Embedding vector for "beautiful"
])

In [5]:


# Define query, key, and value matrices (linear projections)
W_q = np.random.rand(3, 3)  # Random query matrix
W_k = np.random.rand(3, 3)  # Random key matrix
W_v = np.random.rand(3, 3)  # Random value matrix

W_k

Out[5]:

array([[0.48504083, 0.50896575, 0.30045929],
       [0.09040745, 0.93158125, 0.24975536],
       [0.08285484, 0.24772642, 0.71724311]])

In [8]:

# Compute dot products of query and key matrices
Q = np.dot(X, W_q.T)  # Query vectors
K = np.dot(X, W_k.T)  # Key vectors
Q
K

Out[8]:

array([[0.24043502, 0.2702836 , 0.2730037 ],
       [0.62877478, 0.65180682, 0.58735101],
       [1.01711454, 1.03333004, 0.90169833]])

In [9]:

# Compute attention scores
attention_scores = np.dot(Q, K.T) / np.sqrt(3)  # Divided by sqrt(d_k)

# Apply softmax to get attention weights
attention_weights = np.exp(attention_scores) / np.sum(np.exp(attention_scores), axis=1, keepdims=True)

# Compute weighted sum of value vectors to get output
V = np.dot(X, W_v.T)  # Value vectors
output = np.dot(attention_weights, V)

print("Attention Weights:")
print(attention_weights)
print("\nOutput:")
print(output)

Out[9]:

Attention Weights:
[[0.2302623  0.32132819 0.44840951]
 [0.11893887 0.26966432 0.61139681]
 [0.05478679 0.20181306 0.74340015]]

Output:
[[0.66318874 0.55080783 1.13119021]
 [0.76553312 0.62036954 1.29871099]
 [0.83871803 0.67011204 1.41850253]]

Simplified Python code demonstrating how self-attention might be applied in a neural machine translation scenario using the transformers library

This code uses the BERT model from the transformers library to tokenize the input text, compute its hidden states, and extract the self-attention weights. These weights indicate how much each token attends to every other token in each layer of the model. However, note that BERT is not specifically trained for machine translation, so this is just an illustration of self-attention in a language model context.

Steps

Tokenization: The input text "Ashi is beautiful." is tokenized into its constituent tokens using the BERT tokenizer. Each token is represented by an integer ID. Let's denote the tokenized input as 𝑋 X.
Model Computation: The tokenized input 𝑋
X is fed into the BERT model, which consists of multiple layers of self-attention and feedforward neural networks. The BERT model processes the input tokens and produces hidden states for each token. Let's denote the hidden states as 𝐻
Self-Attention: During each layer of the BERT model, self-attention is applied to the input tokens. The self-attention mechanism computes attention scores between each token and every other token in the sequence. These attention scores are calculated using the formula:
Self-Attention Weights: The self-attention weights represent the importance of each token attending to every other token in the sequence. These weights are computed for each layer of the model. In the code, the mean of the attention weights across the sequence dimension is calculated for each layer and printed out.

In [1]:

import torch
from transformers import BertModel, BertTokenizer, BertConfig

# Load pre-trained BERT model and tokenizer
model_name = 'bert-base-multilingual-cased'
tokenizer = BertTokenizer.from_pretrained(model_name)
model = BertModel.from_pretrained(model_name)

# Input text
input_text = "Ashi is beautiful."

# Tokenize input text
input_ids = tokenizer.encode(input_text, add_special_tokens=True, return_tensors="pt")

# Get BERT model's output
outputs = model(input_ids)

# Check if the model supports attention weights
config = BertConfig.from_pretrained(model_name)
if config.output_attentions:
    # Extract hidden states
    hidden_states = outputs.last_hidden_state

    # Self-attention
    self_attention_weights = outputs.attentions

    # Print self-attention weights
    print("Self-attention weights:")
    for layer, attn_weights in enumerate(self_attention_weights):
        print(f"Layer {layer+1}: {attn_weights.mean(dim=1)}")

# Decoding input text
decoded_output = tokenizer.decode(input_ids[0], skip_special_tokens=True)
print("Decoded output from positional encoder:", decoded_output)

# Get the model's final output
final_output = outputs[0]

# Print the final output
print("Final output:", final_output)

Out[1]:

Some weights of the model checkpoint at bert-base-multilingual-cased were not used when initializing BertModel: ['cls.predictions.transform.dense.weight', 'cls.seq_relationship.bias', 'cls.predictions.bias', 'cls.predictions.transform.dense.bias', 'cls.seq_relationship.weight', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.LayerNorm.weight']
- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

Decoded output from positional encoder: Ashi is beautiful.
Final output: tensor([[[-1.3180e-01,  3.0580e-01,  4.7159e-01,  ...,  1.5933e-01,
          -2.4468e-01,  1.0558e-01],
         [-7.7932e-01, -6.8627e-02,  1.1448e+00,  ...,  2.0894e-01,
           4.1032e-01, -2.7293e-01],
         [-1.7906e-01,  6.4059e-02,  1.2786e+00,  ..., -3.1068e-01,
           1.0254e-01, -2.6945e-01],
         ...,
         [ 8.9791e-04,  4.3956e-01,  5.1864e-01,  ...,  5.6646e-01,
          -5.9460e-01, -3.7842e-02],
         [ 5.6526e-02,  3.6165e-01,  7.9546e-01,  ...,  2.0986e-01,
          -7.0414e-01, -8.5388e-02],
         [-2.2144e-01,  1.5282e-01,  1.0640e+00,  ...,  1.2131e-01,
          -2.6769e-01, -2.4737e-01]]], grad_fn=<NativeLayerNormBackward>)

Example Language Translation

In [2]:

import torch
from transformers import MarianMTModel, MarianTokenizer

# Load pre-trained MarianMT model and tokenizer for English to Hindi translation
model_name = 'Helsinki-NLP/opus-mt-en-hi'
tokenizer = MarianTokenizer.from_pretrained(model_name)
hindi_tran = MarianMTModel.from_pretrained(model_name)

# Input text
input_text = "Life is beautiful."
print("Input=",input_text)

# Tokenize input text
input_ids = tokenizer(input_text, return_tensors="pt")

# Perform translation
translated_output = hindi_tran.generate(**input_ids)

# Decode the translated output
translated_text = tokenizer.decode(translated_output[0], skip_special_tokens=True)

# Print the translated output
print("Translated Output (Hindi):")
print(translated_text)

Out[2]:

Input= Life is beautiful.
Translated Output (Hindi):
जीवन सुंदर है.

C:\Users\suyashi144893\Anaconda3\lib\site-packages\torch\_tensor.py:575: UserWarning: floor_divide is deprecated, and will be removed in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values.
To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). (Triggered internally at  ..\aten\src\ATen\native\BinaryOps.cpp:467.)
  return torch.floor_divide(self, other)

In [3]:

from transformers import MarianMTModel, MarianTokenizer

# Load the pre-trained translation model and tokenizer for English to German
model_name = "Helsinki-NLP/opus-mt-en-de"
tokenizer = MarianTokenizer.from_pretrained(model_name)
model = MarianMTModel.from_pretrained(model_name)

# Input text in English
input_text = input("Enter the Text: ")

# Tokenize the input text
inputs = tokenizer(input_text, return_tensors="pt")

# Perform translation
outputs = model.generate(**inputs)

# Decode the translated text
decoded_text = tokenizer.decode(outputs[0], skip_special_tokens=True)

# Print the decoded text
print("Decoded text in German:", decoded_text)

Out[3]:

Enter the Text: hello
Decoded text in German: Guten Tag.

pip install wordcloud --trusted-host pypi.org --trusted-host files.pythonhosted.org pip install pipeline

Hindi to Tamil

In [7]:

from transformers import MarianMTModel, MarianTokenizer
import torch

# Define the model and tokenizer for English to Hindi
model_name_en_to_hi = 'Helsinki-NLP/opus-mt-en-hi'
model_en_to_hi = MarianMTModel.from_pretrained(model_name_en_to_hi)
tokenizer_en_to_hi = MarianTokenizer.from_pretrained(model_name_en_to_hi)

def translate_en_to_hi(text):
    inputs = tokenizer_en_to_hi(text, return_tensors='pt', padding=True, truncation=True)
    with torch.no_grad():
        translated_tokens = model_en_to_hi.generate(**inputs)
    return tokenizer_en_to_hi.batch_decode(translated_tokens, skip_special_tokens=True)[0]

english_text = "Hello, how are you?"
hindi_text = translate_en_to_hi(english_text)
print(f'Hindi Translation: {hindi_text}')

Out[7]:

Hindi Translation: हैलो, तुम कैसे हो?

In [10]:

pip install googletrans==4.0.0-rc1

Out[10]:

Collecting googletrans==4.0.0-rc1Note: you may need to restart the kernel to use updated packages.

  WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)'))': /packages/fa/0d/a5fe8fb53dbf401f8a34cef9439c4c5b8f5037e20123b3e731397808d839/googletrans-4.0.0rc1.tar.gz
  WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)'))': /packages/fa/0d/a5fe8fb53dbf401f8a34cef9439c4c5b8f5037e20123b3e731397808d839/googletrans-4.0.0rc1.tar.gz
  WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)'))': /packages/fa/0d/a5fe8fb53dbf401f8a34cef9439c4c5b8f5037e20123b3e731397808d839/googletrans-4.0.0rc1.tar.gz
  WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)'))': /packages/fa/0d/a5fe8fb53dbf401f8a34cef9439c4c5b8f5037e20123b3e731397808d839/googletrans-4.0.0rc1.tar.gz
  WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)'))': /packages/fa/0d/a5fe8fb53dbf401f8a34cef9439c4c5b8f5037e20123b3e731397808d839/googletrans-4.0.0rc1.tar.gz
ERROR: Could not install packages due to an OSError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Max retries exceeded with url: /packages/fa/0d/a5fe8fb53dbf401f8a34cef9439c4c5b8f5037e20123b3e731397808d839/googletrans-4.0.0rc1.tar.gz (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)')))


[notice] A new release of pip is available: 24.0 -> 24.2
[notice] To update, run: python.exe -m pip install --upgrade pip

In [9]:

from googletrans import Translator

translator = Translator()

def translate_hi_to_tam(hindi_text):
    translated = translator.translate(hindi_text, src='hi', dest='ta')
    return translated.text

tamil_translation = translate_hi_to_tam(hindi_text)
print(f'Tamil Translation: {tamil_translation}')

Out[9]:

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Input In [9], in <cell line: 1>()
----> 1 from googletrans import Translator
      3 translator = Translator()
      5 def translate_hi_to_tam(hindi_text):
ModuleNotFoundError: No module named 'googletrans'

In [11]:

pip install wordcloud --trusted-host pypi.org --trusted-host files.pythonhosted.org pip install googletrans==4.0.0-rc1

Out[11]:

Requirement already satisfied: wordcloud in c:\users\suyashi144893\anaconda3\lib\site-packages (1.9.3)
Requirement already satisfied: pip in c:\users\suyashi144893\anaconda3\lib\site-packages (24.0)
Requirement already satisfied: install in c:\users\suyashi144893\anaconda3\lib\site-packages (1.3.5)
Collecting googletrans==4.0.0-rc1
  Downloading googletrans-4.0.0rc1.tar.gz (20 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting httpx==0.13.3 (from googletrans==4.0.0-rc1)
  Downloading httpx-0.13.3-py3-none-any.whl.metadata (25 kB)
Requirement already satisfied: certifi in c:\users\suyashi144893\anaconda3\lib\site-packages (from httpx==0.13.3->googletrans==4.0.0-rc1) (2021.10.8)
Collecting hstspreload (from httpx==0.13.3->googletrans==4.0.0-rc1)
  Downloading hstspreload-2024.8.1-py3-none-any.whl.metadata (2.1 kB)
Requirement already satisfied: sniffio in c:\users\suyashi144893\anaconda3\lib\site-packages (from httpx==0.13.3->googletrans==4.0.0-rc1) (1.3.1)
Collecting chardet==3.* (from httpx==0.13.3->googletrans==4.0.0-rc1)
  Downloading chardet-3.0.4-py2.py3-none-any.whl.metadata (3.2 kB)
Collecting idna==2.* (from httpx==0.13.3->googletrans==4.0.0-rc1)
  Downloading idna-2.10-py2.py3-none-any.whl.metadata (9.1 kB)
Collecting rfc3986<2,>=1.3 (from httpx==0.13.3->googletrans==4.0.0-rc1)
  Downloading rfc3986-1.5.0-py2.py3-none-any.whl.metadata (6.5 kB)
Collecting httpcore==0.9.* (from httpx==0.13.3->googletrans==4.0.0-rc1)
  Downloading httpcore-0.9.1-py3-none-any.whl.metadata (4.6 kB)
Collecting h11<0.10,>=0.8 (from httpcore==0.9.*->httpx==0.13.3->googletrans==4.0.0-rc1)
  Downloading h11-0.9.0-py2.py3-none-any.whl.metadata (8.1 kB)
Collecting h2==3.* (from httpcore==0.9.*->httpx==0.13.3->googletrans==4.0.0-rc1)
  Downloading h2-3.2.0-py2.py3-none-any.whl.metadata (32 kB)
Collecting hyperframe<6,>=5.2.0 (from h2==3.*->httpcore==0.9.*->httpx==0.13.3->googletrans==4.0.0-rc1)
  Downloading hyperframe-5.2.0-py2.py3-none-any.whl.metadata (7.2 kB)
Collecting hpack<4,>=3.0 (from h2==3.*->httpcore==0.9.*->httpx==0.13.3->googletrans==4.0.0-rc1)
  Downloading hpack-3.0.0-py2.py3-none-any.whl.metadata (7.0 kB)
Requirement already satisfied: numpy>=1.6.1 in c:\users\suyashi144893\anaconda3\lib\site-packages (from wordcloud) (1.24.3)
Requirement already satisfied: pillow in c:\users\suyashi144893\anaconda3\lib\site-packages (from wordcloud) (9.5.0)
Requirement already satisfied: matplotlib in c:\users\suyashi144893\anaconda3\lib\site-packages (from wordcloud) (3.5.1)
Requirement already satisfied: cycler>=0.10 in c:\users\suyashi144893\anaconda3\lib\site-packages (from matplotlib->wordcloud) (0.11.0)
Requirement already satisfied: fonttools>=4.22.0 in c:\users\suyashi144893\anaconda3\lib\site-packages (from matplotlib->wordcloud) (4.25.0)
Requirement already satisfied: kiwisolver>=1.0.1 in c:\users\suyashi144893\anaconda3\lib\site-packages (from matplotlib->wordcloud) (1.3.2)
Requirement already satisfied: packaging>=20.0 in c:\users\suyashi144893\anaconda3\lib\site-packages (from matplotlib->wordcloud) (21.3)
Requirement already satisfied: pyparsing>=2.2.1 in c:\users\suyashi144893\anaconda3\lib\site-packages (from matplotlib->wordcloud) (3.0.4)
Requirement already satisfied: python-dateutil>=2.7 in c:\users\suyashi144893\anaconda3\lib\site-packages (from matplotlib->wordcloud) (2.8.2)
Requirement already satisfied: six>=1.5 in c:\users\suyashi144893\anaconda3\lib\site-packages (from python-dateutil>=2.7->matplotlib->wordcloud) (1.16.0)
Downloading httpx-0.13.3-py3-none-any.whl (55 kB)
   ---------------------------------------- 55.1/55.1 kB 2.8 MB/s eta 0:00:00
Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)
   ---------------------------------------- 133.4/133.4 kB 3.8 MB/s eta 0:00:00
Downloading httpcore-0.9.1-py3-none-any.whl (42 kB)
   ---------------------------------------- 42.6/42.6 kB 2.2 MB/s eta 0:00:00
Downloading idna-2.10-py2.py3-none-any.whl (58 kB)
   ---------------------------------------- 58.8/58.8 kB ? eta 0:00:00
Downloading h2-3.2.0-py2.py3-none-any.whl (65 kB)
   ---------------------------------------- 65.0/65.0 kB 3.4 MB/s eta 0:00:00
Downloading rfc3986-1.5.0-py2.py3-none-any.whl (31 kB)
Downloading hstspreload-2024.8.1-py3-none-any.whl (1.2 MB)
   ---------------------------------------- 1.2/1.2 MB 6.8 MB/s eta 0:00:00
Downloading h11-0.9.0-py2.py3-none-any.whl (53 kB)
   ---------------------------------------- 53.6/53.6 kB 2.9 MB/s eta 0:00:00
Downloading hpack-3.0.0-py2.py3-none-any.whl (38 kB)
Downloading hyperframe-5.2.0-py2.py3-none-any.whl (12 kB)
Building wheels for collected packages: googletrans
  Building wheel for googletrans (setup.py): started
  Building wheel for googletrans (setup.py): finished with status 'done'
  Created wheel for googletrans: filename=googletrans-4.0.0rc1-py3-none-any.whl size=17411 sha256=8a9b91d93092b9788c781d906f702edb545f19c449e033468e4f7b9232f0b620
  Stored in directory: c:\users\suyashi144893\appdata\local\pip\cache\wheels\60\b3\27\d8aff3e2d5c2d0d97a117cdf0d5f13cd121e2c2b5fb49b55a0
Successfully built googletrans
Installing collected packages: rfc3986, hyperframe, hpack, h11, chardet, idna, hstspreload, h2, httpcore, httpx, googletrans
  Attempting uninstall: h11
    Found existing installation: h11 0.14.0
    Uninstalling h11-0.14.0:
      Successfully uninstalled h11-0.14.0
  Attempting uninstall: chardet
    Found existing installation: chardet 4.0.0
    Uninstalling chardet-4.0.0:
      Successfully uninstalled chardet-4.0.0
  Attempting uninstall: idna
    Found existing installation: idna 3.3
    Uninstalling idna-3.3:
      Successfully uninstalled idna-3.3
Successfully installed chardet-3.0.4 googletrans-4.0.0rc1 h11-0.9.0 h2-3.2.0 hpack-3.0.0 hstspreload-2024.8.1 httpcore-0.9.1 httpx-0.13.3 hyperframe-5.2.0 idna-2.10 rfc3986-1.5.0
Note: you may need to restart the kernel to use updated packages.

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
spyder 5.1.5 requires pyqt5<5.13, which is not installed.
spyder 5.1.5 requires pyqtwebengine<5.13, which is not installed.
jupyter-server 1.13.5 requires pywinpty<2; os_name == "nt", but you have pywinpty 2.0.2 which is incompatible.
transformers 4.9.2 requires huggingface-hub==0.0.12, but you have huggingface-hub 0.23.2 which is incompatible.

[notice] A new release of pip is available: 24.0 -> 24.2
[notice] To update, run: python.exe -m pip install --upgrade pip

import sys print("Python version:", sys.version)

Sentiment Analysis

We are using distilbert-base-uncased-finetuned-sst-2-english model, which is fine-tuned for sentiment analysis on the Stanford Sentiment Treebank (SST-2) dataset.

In [13]:

from transformers import pipeline

# Explicitly specify the model name
model_name = "distilbert-base-uncased-finetuned-sst-2-english"

# Load the sentiment analysis pipeline with the specified model
sentiment_pipeline = pipeline("sentiment-analysis", model=model_name, tokenizer=model_name)

# Function to perform sentiment analysis
def analyze_sentiment(text):
    results = sentiment_pipeline(text)
    return results

# Example usage
if __name__ == "__main__":
    text = input("Enter the text for sentiment analysis: ")
    results = analyze_sentiment(text)
    for result in results:
        print(f"Label: {result['label']}, Score: {result['score']:.4f}")

Out[13]:

Enter the text for sentiment analysis: I am ok with you idea
Label: POSITIVE, Score: 0.9998

In [ ]:

import torch
from transformers import BertTokenizer, BertForSequenceClassification
import numpy as np

# Load pre-trained BERT model and tokenizer
model_name = "bert-base-uncased"
tokenizer = BertTokenizer.from_pretrained(model_name)
model = BertForSequenceClassification.from_pretrained(model_name)
model.eval()

# Define labels
labels = ['Negative', 'Positive']

# Function to perform text classification
def classify_text(text):
    # Tokenize input text
    inputs = tokenizer(text, return_tensors="pt", truncation=True, padding=True)
    # Classify text using BERT model
    with torch.no_grad():
        outputs = model(**inputs)
    # Get predicted probabilities
    probabilities = torch.softmax(outputs.logits, dim=1).squeeze().numpy()
    # Get predicted label index
    predicted_label_index = np.argmax(probabilities)
    # Get predicted label
    predicted_label = labels[predicted_label_index]
    return predicted_label, probabilities

# User input for text
user_text = input("Enter the text you want to classify: ")

# Perform text classification
predicted_label, probabilities = classify_text(user_text)

# Print results
print("Predicted label:", predicted_label)
print("Probabilities:", probabilities)