GitHub Repository: ibm/watson-machine-learning-samples
Path: blob/master/cloud/notebooks/python_sdk/deployments/ai_services/Use watsonx, and `llama-3-2-11b-vision-instruct` to run as an AI service.ipynb
⁶⁴⁰⁵ views

Kernel: Python 3

Use watsonx, and `meta-llama/llama-3-2-11b-vision-instruct` to run as an AI service

Disclaimers

Use only Projects and Spaces that are available in watsonx context.

Notebook content

This notebook provides a detailed demonstration of the steps and code required to showcase support for watsonx.ai AI service.

Some familiarity with Python is helpful. This notebook uses Python 3.11.

Learning goal

The learning goal for your notebook is to leverage AI services to generate accurate and contextually relevant responses based on a given image and a related question.

This notebook contains the following parts:

Set up the environment

Before you use the sample code in this notebook, you must perform the following setup tasks:

Create a watsonx.ai Runtime Service instance (a free plan is offered and information about how to create the instance can be found here).

Install and import the `datasets` and dependencies

In [1]:

%pip install -U "ibm_watsonx_ai>=1.3.6" | tail -n 1

Out[1]:

Successfully installed anyio-4.9.0 certifi-2025.1.31 charset-normalizer-3.4.1 h11-0.14.0 httpcore-1.0.8 httpx-0.28.1 ibm-cos-sdk-2.14.0 ibm-cos-sdk-core-2.14.0 ibm-cos-sdk-s3transfer-2.14.0 ibm_watsonx_ai-1.3.11 idna-3.10 jmespath-1.0.1 lomond-0.3.3 numpy-2.2.5 pandas-2.2.3 pytz-2025.2 requests-2.32.2 sniffio-1.3.1 tabulate-0.9.0 tzdata-2025.2 urllib3-2.4.0

Define the watsonx.ai credentials

Use the code cell below to define the watsonx.ai credentials that are required to work with watsonx Foundation Model inferencing.

Action: Provide the IBM Cloud user API key. For details, see Managing user API keys.

In [2]:

import getpass
from ibm_watsonx_ai import Credentials

credentials = Credentials(
    url="https://us-south.ml.cloud.ibm.com",
    api_key=getpass.getpass("Enter your watsonx.ai api key and hit enter: "),
)

Working with spaces

You need to create a space that will be used for your work. If you do not have a space, you can use Deployment Spaces Dashboard to create one.

Click New Deployment Space
Create an empty space
Select Cloud Object Storage
Select watsonx.ai Runtime instance and press Create
Go to Manage tab
Copy Space GUID and paste it below

Tip: You can also use SDK to prepare the space for your work. More information can be found here.

Action: assign space ID below

In [3]:

import os

try:
    space_id = os.environ["SPACE_ID"]
except KeyError:
    space_id = input("Please enter your space_id (hit enter): ")

Create an instance of APIClient with authentication details.

In [4]:

from ibm_watsonx_ai import APIClient

api_client = APIClient(credentials=credentials, space_id=space_id)

Specify the model_id of the model you will use for the chat with image.

In [5]:

model_id = "meta-llama/llama-3-2-11b-vision-instruct"

Create AI service

Prepare function which will be deployed using AI service.

Please specify the default parameters that will be passed to the function.

In [6]:

def deployable_ai_service(context, url=credentials["url"], model_id=model_id, params={"temperature": 1}, **kwargs):

    import requests
    import base64
    from ibm_watsonx_ai import APIClient, Credentials
    from ibm_watsonx_ai.foundation_models import ModelInference

    api_client = APIClient(
        credentials=Credentials(url=url, token=context.generate_token()),
        space_id=context.get_space_id(),
    )

    model = ModelInference(
        model_id=model_id,
        api_client=api_client,
        params=params,
    )

    def generate(context) -> dict:
        api_client.set_token(context.get_token())

        payload = context.get_json()
        question = payload["question"]
        image_url = payload["image_url"]

        response = requests.get(image_url)
        response.raise_for_status()
        base64_image = base64.b64encode(response.content).decode("utf-8")

        messages = [
            {
                "role": "user",
                "content": [
                {
                    "type": "text",
                    "text": question
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "data:image/jpeg;base64," + base64_image,
                        "detail": "auto"
                    }
                }
                ]
            }
        ]

        response = model.chat(messages=messages)

        return {"body": response}

    def generate_stream(context):
        api_client.set_token(context.get_token())

        payload = context.get_json()
        question = payload["question"]
        image_url = payload["image_url"]

        response = requests.get(image_url)
        response.raise_for_status()
        base64_image = base64.b64encode(response.content).decode("utf-8")

        messages = [
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": question
                    },
                    {
                        "type": "image_url",
                        "image_url": {
                            "url": "data:image/jpeg;base64," + base64_image,
                            "detail": "auto"
                        }
                    }
                ]
            }
        ]

        yield from model.chat_stream(messages)

    return generate, generate_stream

Testing AI service's function locally

You can test AI service's function locally. Initialise RuntimeContext firstly.

In [7]:

from ibm_watsonx_ai.deployments import RuntimeContext

context = RuntimeContext(api_client=api_client)

In [8]:

local_function = deployable_ai_service(context=context)

Please retrieve an image and display it. This example is based on the IBM logo.

In [9]:

import requests
from IPython.display import Image

image_url = "https://raw.github.com/IBM/watson-machine-learning-samples/master/cloud/data/logo/ibm_logo.jpg"

response = requests.get(image_url)

Image(url=image_url, width=600)

Out[9]:

Prepare request json payload for local invoke.

In [10]:

context.request_payload_json = {"question": "Describe the image", "image_url": image_url}

Execute the generate function locally.

In [11]:

resp = local_function[0](context)
resp

Out[11]:

{'body': {'id': 'chatcmpl-fd71df10-514d-46bf-b716-f9841c764321---89261ca18b0c50363ec5340d39cac781',
  'model_id': 'meta-llama/llama-3-2-11b-vision-instruct',
  'model': 'meta-llama/llama-3-2-11b-vision-instruct',
  'choices': [{'index': 0,
    'message': {'role': 'assistant',
     'content': 'The image presents the IBM logo, a well-known and iconic symbol of the technology company. The logo is prominently displayed in the center of the image, with the word "IBM" written in blue stripes against a white background.\n\n* **Logo:**\n\t+ The logo features the word "IBM" in blue stripes.\n\t+ The stripes are arranged in a horizontal manner, with each letter consisting of multiple horizontal lines.\n\t+ The lines vary in width, creating a sense of depth and dimensionality.\n* **Color Scheme:**\n\t+ The logo\'s color scheme is primarily blue, with white serving as the background.\n\t+ The blue tone used is a deep, rich shade that provides good contrast against the white background.\n* **Background:**\n\t+ The background of the image is a clean and simple white.\n\t+ This allows the logo to take center stage and draws attention to its design and typography.\n\nOverall, the IBM logo is a distinctive and recognizable symbol of the company\'s brand identity. Its simple yet effective design has become synonymous with innovation and technology, and it continues to be an important part of the company\'s visual identity.'},
    'finish_reason': 'stop'}],
  'created': 1745402649,
  'model_version': '3.2.0',
  'created_at': '2025-04-23T10:04:13.523Z',
  'usage': {'completion_tokens': 231,
   'prompt_tokens': 6418,
   'total_tokens': 6649},
  'system': {'warnings': [{'message': 'This model is a Non-IBM Product governed by a third-party license that may impose use restrictions and other obligations. By using this model you agree to its terms as identified in the following URL.',
     'id': 'disclaimer_warning',
     'more_info': 'https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-models.html?context=wx'},
    {'message': "The value of 'max_tokens' for this model was set to value 1024",
     'id': 'unspecified_max_token',
     'additional_properties': {'limit': 0,
      'new_value': 1024,
      'parameter': 'max_tokens',
      'value': 0}}]}}}

Execute the generate_stream function locally.

In [12]:

response = local_function[1](context)

In [13]:

for chunk in response:
    if chunk["choices"]:
        print(chunk["choices"][0]["delta"].get("content", ""), end="", flush=True)

Out[13]:

The image presents the IBM logo, a registered trademark of the International Business Machines Corporation.

Deploy AI service

Store AI service with previous created custom software specifications.

In [14]:

sw_spec_id = api_client.software_specifications.get_id_by_name("runtime-24.1-py3.11")
sw_spec_id

Out[14]:

'45f12dfe-aa78-5b8d-9f38-0ee223c47309'

In [15]:

meta_props = {
    api_client.repository.AIServiceMetaNames.NAME: "AI service with SDK",
    api_client.repository.AIServiceMetaNames.SOFTWARE_SPEC_ID: sw_spec_id
}
stored_ai_service_details = api_client.repository.store_ai_service(deployable_ai_service, meta_props)

In [16]:

ai_service_id = api_client.repository.get_ai_service_id(stored_ai_service_details)
ai_service_id

Out[16]:

'755be541-49e7-49a0-90fc-fe4ba510c6d1'

Create online deployment of AI service.

In [17]:

meta_props = {
    api_client.deployments.ConfigurationMetaNames.NAME: "AI service with SDK",
    api_client.deployments.ConfigurationMetaNames.ONLINE: {},
}

deployment_details = api_client.deployments.create(ai_service_id, meta_props)

Out[17]:

######################################################################################

Synchronous deployment creation for id: '755be541-49e7-49a0-90fc-fe4ba510c6d1' started

######################################################################################


initializing
Note: online_url and serving_urls are deprecated and will be removed in a future release. Use inference instead.
...
ready


-----------------------------------------------------------------------------------------------
Successfully finished deployment creation, deployment_id='3e7aac9b-5a8b-4b1a-8c3c-0afd6971edaa'
-----------------------------------------------------------------------------------------------

Obtain the deployment_id of the previously created deployment.

In [18]:

deployment_id = api_client.deployments.get_id(deployment_details)

Example of Executing an AI service.

Execute generate method.

In [19]:

question = "Describe the image"

deployments_results = api_client.deployments.run_ai_service(
    deployment_id, {"question": question, "image_url": image_url}
)

In [20]:

import json

print(json.dumps(deployments_results, indent=2))

Out[20]:

{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "The image features the IBM logo, a blue graphic of the company's initials \"IBM\" against a white background. The logo is composed of 13 horizontal lines and 2 vertical lines, with the first and last lines being thicker than the others.\n\n*   **Logo Design:**\n    *   The IBM logo is a distinctive and recognizable design that has been used by the company for many years.\n    *   The use of horizontal and vertical lines creates a sense of balance and harmony, reflecting the company's commitment to innovation and stability.\n*   **Color Scheme:**\n    *   The blue color of the logo is a nod to the company's heritage and tradition of using blue in their branding.\n    *   The choice of blue also evokes feelings of trust, reliability, and professionalism, which are all key values associated with the IBM brand.\n*   **Typography:**\n    *   The sans-serif font used for the logo is clean and modern, conveying a sense of simplicity and ease of use.\n    *   The bold lines and sharp edges of the font give the logo a dynamic and energetic feel, suggesting a company that is always moving forward.\n*   **Trademark Symbol:**\n    *   The registered trademark symbol (R) that appears next to the logo indicates that the logo is a protected trademark and cannot be used without permission.\n    *   This symbol reinforces the idea that the IBM brand is valuable and worthy of protection.\n\nOverall, the IBM logo is a powerful symbol of the company's history, values, and commitment to innovation. Its distinctive design and clean typography make it instantly recognizable, while the use of blue conveys a sense of trust and professionalism.",
        "role": "assistant"
      }
    }
  ],
  "created": 1745402691,
  "created_at": "2025-04-23T10:04:57.497Z",
  "id": "chatcmpl-43724fe5-90b5-4ea1-991e-a41a988a2004---1eab7ee4010482d7267cf3fd7a8c5520",
  "model": "meta-llama/llama-3-2-11b-vision-instruct",
  "model_id": "meta-llama/llama-3-2-11b-vision-instruct",
  "model_version": "3.2.0",
  "system": {
    "warnings": [
      {
        "id": "disclaimer_warning",
        "message": "This model is a Non-IBM Product governed by a third-party license that may impose use restrictions and other obligations. By using this model you agree to its terms as identified in the following URL.",
        "more_info": "https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-models.html?context=wx"
      },
      {
        "additional_properties": {
          "limit": 0,
          "new_value": 1024,
          "parameter": "max_tokens",
          "value": 0
        },
        "id": "unspecified_max_token",
        "message": "The value of 'max_tokens' for this model was set to value 1024"
      }
    ]
  },
  "usage": {
    "completion_tokens": 338,
    "prompt_tokens": 6418,
    "total_tokens": 6756
  }
}

Execute generate_stream method.

In [21]:

question = "Describe the image"

deployments_results = api_client.deployments.run_ai_service_stream(
    deployment_id, {"question": question, "image_url": image_url}
)

In [22]:

import json

for chunk in deployments_results:
    chunk_json = json.loads(chunk)
    if chunk_json["choices"]:
        print(chunk_json["choices"][0]["delta"].get("content", ""), end="", flush=True)

Out[22]:

The image presents the IBM logo, which is a stylized representation of the company's name in blue lettering on a white background.

* The letters "IBM" are written in a stylized font, with each letter featuring a unique design.
	+ The "I" is composed of three horizontal lines.
	+ The "B" consists of two curved lines that intersect in the middle.
	+ The "M" is formed by a combination of vertical and horizontal lines.
* The logo also features a registered trademark symbol (®) to the right of the letters.
* The entire logo is set against a plain white background, providing a clean and simple visual representation of the brand identity.

Overall, the IBM logo is a distinctive and recognizable symbol that represents the company's values and mission. The use of a stylized font and a registered trademark symbol emphasizes the brand's commitment to innovation and quality.

Summary and next steps

You successfully completed this notebook!

You learned how to create and deploy AI service using ibm_watsonx_ai SDK.

Check out our Online Documentation for more samples, tutorials, documentation, how-tos, and blog posts.

Author

Mateusz Szewczyk, Software Engineer at watsonx.ai.

Use watsonx, and `meta-llama/llama-3-2-11b-vision-instruct` to run as an AI service

Disclaimers

Notebook content

Learning goal

Table of Contents

Set up the environment

Install and import the `datasets` and dependencies

Define the watsonx.ai credentials

Working with spaces

Create AI service

Testing AI service's function locally

Deploy AI service

Example of Executing an AI service.

Summary and next steps

Author

Product

Resources

Company

Use watsonx, and meta-llama/llama-3-2-11b-vision-instruct to run as an AI service

Disclaimers

Notebook content

Learning goal

Table of Contents

Set up the environment

Install and import the datasets and dependencies

Define the watsonx.ai credentials

Working with spaces

Create AI service

Testing AI service's function locally

Deploy AI service

Example of Executing an AI service.

Summary and next steps

Author

Use watsonx, and `meta-llama/llama-3-2-11b-vision-instruct` to run as an AI service

Install and import the `datasets` and dependencies