GitHub Repository: IBM/watson-machine-learning-samples
Path: blob/master/cloud/notebooks/python_sdk/deployments/ai_services/Use watsonx, and `llama-3-2-11b-vision-instruct` to run as an AI service.ipynb
⁵²¹⁴ views

Kernel: .venv_watsonx_ai_samples_py_312

Use watsonx, and `meta-llama/llama-3-2-11b-vision-instruct` to run as an AI service

Disclaimers

Use only Projects and Spaces that are available in watsonx context.

Notebook content

This notebook provides a detailed demonstration of the steps and code required to showcase support for watsonx.ai AI service.

Some familiarity with Python is helpful. This notebook uses Python 3.12.

Learning goal

The learning goal for your notebook is to leverage AI services to generate accurate and contextually relevant responses based on a given image and a related question.

This notebook contains the following parts:

Set up the environment

Before you use the sample code in this notebook, you must perform the following setup tasks:

Create a watsonx.ai Runtime Service instance (a free plan is offered and information about how to create the instance can be found here).

Install and import the `ibm-watsonx-ai` and dependencies

Note: ibm-watsonx-ai documentation can be found here.

In [1]:

%pip install -U "ibm_watsonx_ai>=1.3.6" | tail -n 1

Out[1]:

Successfully installed anyio-4.12.1 cachetools-6.2.4 certifi-2026.1.4 charset_normalizer-3.4.4 h11-0.16.0 httpcore-1.0.9 httpx-0.28.1 ibm-cos-sdk-2.14.3 ibm-cos-sdk-core-2.14.3 ibm-cos-sdk-s3transfer-2.14.3 ibm_watsonx_ai-1.5.0 idna-3.11 jmespath-1.0.1 lomond-0.3.3 numpy-2.4.1 pandas-2.2.3 pytz-2025.2 requests-2.32.5 tabulate-0.9.0 typing_extensions-4.15.0 tzdata-2025.3 urllib3-2.6.3

Define the watsonx.ai credentials

Use the code cell below to define the watsonx.ai credentials that are required to work with watsonx Foundation Model inferencing.

Action: Provide the IBM Cloud user API key. For details, see Managing user API keys.

In [2]:

import getpass

from ibm_watsonx_ai import Credentials

credentials = Credentials(
    url="https://us-south.ml.cloud.ibm.com",
    api_key=getpass.getpass("Enter your watsonx.ai api key and hit enter: "),
)

Working with spaces

You need to create a space that will be used for your work. If you do not have a space, you can use Deployment Spaces Dashboard to create one.

Click New Deployment Space
Create an empty space
Select Cloud Object Storage
Select watsonx.ai Runtime instance and press Create
Go to Manage tab
Copy Space GUID and paste it below

Tip: You can also use SDK to prepare the space for your work. More information can be found here.

Action: assign space ID below

In [3]:

import os

try:
    space_id = os.environ["SPACE_ID"]
except KeyError:
    space_id = input("Please enter your space_id (hit enter): ")

Create an instance of APIClient with authentication details.

In [4]:

from ibm_watsonx_ai import APIClient

api_client = APIClient(credentials=credentials, space_id=space_id)

Specify the model_id of the model you will use for the chat with image.

In [5]:

model_id = "meta-llama/llama-3-2-11b-vision-instruct"

Create AI service

Prepare function which will be deployed using AI service.

Please specify the default parameters that will be passed to the function.

In [6]:

def deployable_ai_service(
    context, url=credentials["url"], model_id=model_id, params={"temperature": 1}
):
    import base64

    import requests
    from ibm_watsonx_ai import APIClient, Credentials
    from ibm_watsonx_ai.foundation_models import ModelInference

    api_client = APIClient(
        credentials=Credentials(url=url, token=context.generate_token()),
        space_id=context.get_space_id(),
    )

    model = ModelInference(
        model_id=model_id,
        api_client=api_client,
        params=params,
    )

    def generate(context) -> dict:
        api_client.set_token(context.get_token())

        payload = context.get_json()
        question = payload["question"]
        image_url = payload["image_url"]

        response = requests.get(image_url)
        response.raise_for_status()
        base64_image = base64.b64encode(response.content).decode("utf-8")

        messages = [
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": question,
                    },
                    {
                        "type": "image_url",
                        "image_url": {
                            "url": "data:image/jpeg;base64," + base64_image,
                            "detail": "auto",
                        },
                    },
                ],
            }
        ]

        response = model.chat(messages=messages)

        return {"body": response}

    def generate_stream(context):
        api_client.set_token(context.get_token())

        payload = context.get_json()
        question = payload["question"]
        image_url = payload["image_url"]

        response = requests.get(image_url)
        response.raise_for_status()
        base64_image = base64.b64encode(response.content).decode("utf-8")

        messages = [
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": question,
                    },
                    {
                        "type": "image_url",
                        "image_url": {
                            "url": "data:image/jpeg;base64," + base64_image,
                            "detail": "auto",
                        },
                    },
                ],
            }
        ]

        yield from model.chat_stream(messages)

    return generate, generate_stream

Testing AI service's function locally

You can test AI service's function locally. Initialise RuntimeContext firstly.

In [7]:

from ibm_watsonx_ai.deployments import RuntimeContext

context = RuntimeContext(api_client=api_client)

In [8]:

local_function = deployable_ai_service(context=context)

Please retrieve an image and display it. This example is based on the IBM logo.

In [9]:

import requests
from IPython.display import Image

image_url = (
    "https://raw.github.com/IBM/watsonx-ai-samples/master/cloud/data/logo/ibm_logo.jpg"
)

response = requests.get(image_url)

Image(url=image_url, width=600)

Out[9]:

Prepare request json payload for local invoke.

In [10]:

context.request_payload_json = {
    "question": "Describe the image",
    "image_url": image_url,
}

Execute the generate function locally.

In [11]:

resp = local_function[0](context)
resp

Out[11]:

{'body': {'id': 'chatcmpl-4b3587fb90f161761d8c473a9b81863e---e50e85b7-57c9-44a7-b9f1-3c3173f3af1c',
  'object': 'chat.completion',
  'model_id': 'meta-llama/llama-3-2-11b-vision-instruct',
  'model': 'meta-llama/llama-3-2-11b-vision-instruct',
  'choices': [{'index': 0,
    'message': {'role': 'assistant',
     'content': 'The image presents the IBM logo, a distinctive and iconic symbol of the technology company. The logo is centered in the image and features the letters "IBM" in a stylized font, with each letter comprising horizontal lines of varying lengths. The lines are arranged in a way that creates a sense of movement and energy, conveying the company\'s innovative and dynamic approach to technology.\n\nHere is a detailed description of the image:\n\n* **Logo**\n\t+ Font: Stylized, with horizontal lines of varying lengths\n\t+ Color: Blue\n\t+ Arrangement: Centered in the image\n* **Background**\n\t+ Color: White\n\t+ Texture: Smooth, solid color\n* **Additional Elements**\n\t+ Registered trademark symbol (®) in the bottom-right corner of the image\n\nIn summary, the image showcases the IBM logo in a clean and simple design, with a focus on the stylized font and the use of blue color. The logo is centered in the image, with a white background and a registered trademark symbol in the bottom-right corner.'},
    'finish_reason': 'stop'}],
  'created': 1768401797,
  'model_version': '3.2.0',
  'created_at': '2026-01-14T14:43:19.832Z',
  'usage': {'completion_tokens': 215,
   'prompt_tokens': 6418,
   'total_tokens': 6633},
  'system': {'warnings': [{'message': 'This model is a Non-IBM Product governed by a third-party license that may impose use restrictions and other obligations. By using this model you agree to its terms as identified in the following URL.',
     'id': 'disclaimer_warning',
     'more_info': 'https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-models.html?context=wx'},
    {'message': "The value of 'max_tokens' for this model was set to value 1024",
     'id': 'unspecified_max_token',
     'additional_properties': {'limit': 0,
      'new_value': 1024,
      'parameter': 'max_tokens',
      'value': 0}}]}}}

Execute the generate_stream function locally.

In [12]:

response = local_function[1](context)

In [13]:

for chunk in response:
    if chunk["choices"]:
        print(chunk["choices"][0]["delta"].get("content", ""), end="", flush=True)

Out[13]:

The image is a blue logo for IBM, featuring the company name in a stylized font with horizontal lines and rectangles. The logo is set against a white background.

Here are the key features of the image:

* **Logo:**
	+ The logo is in a blue color.
	+ It features the company name "IBM" in a stylized font.
	+ The font has horizontal lines and rectangles.
* **Background:**
	+ The background is a solid white color.
* **Watermark:**
	+ There is a watermark of a registered trademark symbol (R) in the bottom-right corner of the image.

Overall, the image presents a clean and simple design for the IBM logo, with a focus on the company name and its distinctive font style.

Deploy AI service

Store AI service with previous created custom software specifications.

In [14]:

sw_spec_id = api_client.software_specifications.get_id_by_name("genai-A25-py3.12")
sw_spec_id

Out[14]:

'60ddf4d9-65ac-562d-aa8d-9f26da5dfc76'

In [15]:

meta_props = {
    api_client.repository.AIServiceMetaNames.NAME: "AI service with SDK",
    api_client.repository.AIServiceMetaNames.SOFTWARE_SPEC_ID: sw_spec_id,
}
stored_ai_service_details = api_client.repository.store_ai_service(
    deployable_ai_service, meta_props
)

In [16]:

ai_service_id = api_client.repository.get_ai_service_id(stored_ai_service_details)
ai_service_id

Out[16]:

'8678a424-0fa7-4025-8894-9ce109e3803a'

Create online deployment of AI service.

In [17]:

meta_props = {
    api_client.deployments.ConfigurationMetaNames.NAME: "AI service with SDK",
    api_client.deployments.ConfigurationMetaNames.ONLINE: {},
}

deployment_details = api_client.deployments.create(ai_service_id, meta_props)

Out[17]:

######################################################################################

Synchronous deployment creation for id: '8678a424-0fa7-4025-8894-9ce109e3803a' started

######################################################################################


initializing
Note: online_url and serving_urls are deprecated and will be removed in a future release. Use inference instead.
............
ready


-----------------------------------------------------------------------------------------------
Successfully finished deployment creation, deployment_id='45b7454d-76c3-41d2-b506-c0a495815341'
-----------------------------------------------------------------------------------------------

Obtain the deployment_id of the previously created deployment.

In [18]:

deployment_id = api_client.deployments.get_id(deployment_details)

Example of Executing an AI service.

Execute generate method.

In [19]:

question = "Describe the image"

deployments_results = api_client.deployments.run_ai_service(
    deployment_id, {"question": question, "image_url": image_url}
)

In [20]:

import json

print(json.dumps(deployments_results, indent=2))

Out[20]:

{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "The image presents the IBM logo, a prominent and recognizable symbol of the multinational technology company IBM (International Business Machines Corporation). The logo is displayed in a blue color scheme on a white background.\n\n*   **Logo Design**\n    *   The logo features the letters \"IBM\" in large, blue, block capital letters.\n    *   Each letter is composed of horizontal lines of varying lengths, creating a visually striking effect.\n    *   The lines are evenly spaced and aligned, giving the logo a sense of balance and harmony.\n    *   The font used for the logo is sans-serif, which contributes to its modern and sleek appearance.\n*   **Color Scheme**\n    *   The logo is presented in a bold blue color, which is one of IBM's primary brand colors.\n    *   The blue hue is a deep, rich shade that evokes feelings of trust, stability, and professionalism.\n    *   The contrast between the blue logo and the white background creates a clear and legible visual effect.\n*   **Background**\n    *   The background of the image is a plain white color, which helps to highlight the logo and create a clean and simple visual composition.\n    *   The white background also provides a neutral backdrop that allows the viewer to focus on the logo itself.\n*   **Registration Mark**\n    *   In the bottom-right corner of the image, there is a small registration mark (\u00ae) symbol.\n    *   The registration mark indicates that the IBM logo is a registered trademark, signifying that it is a protected brand identity owned by the company.\n\nIn summary, the IBM logo is a well-designed and iconic symbol that effectively represents the company's brand identity. The use of blue as the primary color, combined with the clean and simple design of the logo, creates a professional and trustworthy image that is instantly recognizable.",
        "role": "assistant"
      }
    }
  ],
  "created": 1768401941,
  "created_at": "2026-01-14T14:45:46.896Z",
  "id": "chatcmpl-d0bab4c1b78b9f53da20cff3b3aaf0df---e0dec40d-61b5-431c-90c8-93fc1c514800",
  "model": "meta-llama/llama-3-2-11b-vision-instruct",
  "model_id": "meta-llama/llama-3-2-11b-vision-instruct",
  "model_version": "3.2.0",
  "object": "chat.completion",
  "system": {
    "warnings": [
      {
        "id": "disclaimer_warning",
        "message": "This model is a Non-IBM Product governed by a third-party license that may impose use restrictions and other obligations. By using this model you agree to its terms as identified in the following URL.",
        "more_info": "https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/fm-models.html?context=wx"
      },
      {
        "additional_properties": {
          "limit": 0,
          "new_value": 1024,
          "parameter": "max_tokens",
          "value": 0
        },
        "id": "unspecified_max_token",
        "message": "The value of 'max_tokens' for this model was set to value 1024"
      }
    ]
  },
  "usage": {
    "completion_tokens": 371,
    "prompt_tokens": 6418,
    "total_tokens": 6789
  }
}

Execute generate_stream method.

In [21]:

question = "Describe the image"

deployments_results = api_client.deployments.run_ai_service_stream(
    deployment_id, {"question": question, "image_url": image_url}
)

In [22]:

import json

for chunk in deployments_results:
    chunk_json = json.loads(chunk)
    if chunk_json["choices"]:
        print(chunk_json["choices"][0]["delta"].get("content", ""), end="", flush=True)

Out[22]:

The image displays the IBM logo, which is a stylized representation of the company's name in blue text against a white background. The logo features the letters "IBM" in large, blue, blocky text, with each letter containing horizontal lines that vary in length and width. The lines are arranged in a way that creates a sense of movement and fluidity, giving the logo a dynamic feel.

**Key Features:**

* **Logo Design:** The logo is designed to be simple yet distinctive, making it easily recognizable as the IBM brand.
* **Color Scheme:** The use of blue for the text creates a sense of trust, reliability, and professionalism, which are all values associated with the IBM brand.
* **Background:** The white background provides a clean and neutral backdrop that allows the logo to take center stage.
* **Trademark Symbol:** A small trademark symbol (R) is located in the bottom-right corner of the image, indicating that the logo is a registered trademark of IBM.

**Overall Impression:**

The image effectively communicates the IBM brand identity through its simple yet distinctive design. The use of blue and the dynamic arrangement of horizontal lines create a sense of movement and energy, conveying the company's commitment to innovation and progress. The trademark symbol adds an extra layer of authenticity and protection for the brand.

Summary and next steps

You successfully completed this notebook!

You learned how to create and deploy AI service using ibm_watsonx_ai SDK.

Check out our Online Documentation for more samples, tutorials, documentation, how-tos, and blog posts.

Author

Mateusz Szewczyk, Software Engineer at watsonx.ai.

Use watsonx, and `meta-llama/llama-3-2-11b-vision-instruct` to run as an AI service

Disclaimers

Notebook content

Learning goal

Table of Contents

Set up the environment

Install and import the `ibm-watsonx-ai` and dependencies

Define the watsonx.ai credentials

Working with spaces

Create AI service

Testing AI service's function locally

Deploy AI service

Example of Executing an AI service.

Summary and next steps

Author

Product

Resources

Company

Use watsonx, and meta-llama/llama-3-2-11b-vision-instruct to run as an AI service

Disclaimers

Notebook content

Learning goal

Table of Contents

Set up the environment

Install and import the ibm-watsonx-ai and dependencies

Define the watsonx.ai credentials

Working with spaces

Create AI service

Testing AI service's function locally

Deploy AI service

Example of Executing an AI service.

Summary and next steps

Author

Use watsonx, and `meta-llama/llama-3-2-11b-vision-instruct` to run as an AI service

Install and import the `ibm-watsonx-ai` and dependencies