GitHub Repository: ibm/watson-machine-learning-samples
Path: blob/master/cpd5.2/notebooks/python_sdk/experiments/autoai_rag/Use AutoAI RAG and Chroma to create a pattern about IBM.ipynb
⁶⁴¹² views

Kernel: watsonx-ai-samples-py-312

Use AutoAI RAG and Chroma to create a pattern and get information from `ibm-watsonx-ai` SDK documentation

Disclaimers

Use only Projects and Spaces that are available in the watsonx context.

Notebook content

This notebook contains the steps and code to demonstrate the usage of IBM AutoAI RAG. The AutoAI RAG experiment conducted in this notebook uses data scraped from the ibm-watsonx-ai SDK documentation.

Some familiarity with Python is helpful. This notebook uses Python 3.12.

Learning goal

The learning goals of this notebook are:

Create an AutoAI RAG job that will find the best RAG pattern based on provided data

This notebook contains the following parts:

Set up the environment

Before you use the sample code in this notebook, you must perform the following setup task:

Contact your Cloud Pak for Data administrator and ask them for your account credentials

Install dependencies

Note: ibm-watsonx-ai documentation can be found here.

In [1]:

%pip install -U "ibm-watsonx-ai[rag]>=1.2.4" | tail -n 1

Out[1]:

Successfully installed Pillow-11.2.1 SQLAlchemy-2.0.41 XlsxWriter-3.2.3 aiohappyeyeballs-2.6.1 aiohttp-3.12.4 aiosignal-1.3.2 annotated-types-0.7.0 anyio-4.9.0 asgiref-3.8.1 attrs-25.3.0 backoff-2.2.1 bcrypt-4.3.0 beautifulsoup4-4.12.3 build-1.2.2.post1 cachetools-5.5.2 certifi-2025.4.26 charset-normalizer-3.4.2 chroma-hnswlib-0.7.6 chromadb-0.5.23 click-8.2.1 coloredlogs-15.0.1 dataclasses-json-0.6.7 deprecated-1.2.18 distro-1.9.0 durationpy-0.10 elastic-transport-8.17.1 elasticsearch-8.18.1 fastapi-0.115.12 filelock-3.18.0 flatbuffers-25.2.10 frozenlist-1.6.0 fsspec-2025.5.1 google-auth-2.40.2 googleapis-common-protos-1.70.0 grpcio-1.67.1 h11-0.16.0 hf-xet-1.1.2 httpcore-1.0.9 httptools-0.6.4 httpx-0.28.1 httpx-sse-0.4.0 huggingface-hub-0.32.3 humanfriendly-10.0 ibm-cos-sdk-2.14.1 ibm-cos-sdk-core-2.14.1 ibm-cos-sdk-s3transfer-2.14.1 ibm-watsonx-ai-1.3.23 idna-3.10 importlib-metadata-8.6.1 importlib-resources-6.5.2 jmespath-1.0.1 jsonpatch-1.33 jsonpointer-3.0.0 kubernetes-32.0.1 langchain-0.3.25 langchain-chroma-0.1.4 langchain-community-0.3.24 langchain-core-0.3.63 langchain-elasticsearch-0.3.0 langchain-ibm-0.3.11 langchain-milvus-0.1.8 langchain-text-splitters-0.3.8 langsmith-0.3.43 lomond-0.3.3 lxml-5.4.0 markdown-3.4.1 markdown-it-py-3.0.0 marshmallow-3.26.1 mdurl-0.1.2 milvus-lite-2.4.12 mmh3-5.1.0 mpmath-1.3.0 multidict-6.4.4 mypy-extensions-1.1.0 numpy-1.26.4 oauthlib-3.2.2 onnxruntime-1.22.0 opentelemetry-api-1.33.1 opentelemetry-exporter-otlp-proto-common-1.33.1 opentelemetry-exporter-otlp-proto-grpc-1.33.1 opentelemetry-instrumentation-0.54b1 opentelemetry-instrumentation-asgi-0.54b1 opentelemetry-instrumentation-fastapi-0.54b1 opentelemetry-proto-1.33.1 opentelemetry-sdk-1.33.1 opentelemetry-semantic-conventions-0.54b1 opentelemetry-util-http-0.54b1 orjson-3.10.18 overrides-7.7.0 packaging-24.2 pandas-2.2.3 posthog-4.2.0 propcache-0.3.1 protobuf-5.29.5 pyYAML-6.0.2 pyasn1-0.6.1 pyasn1-modules-0.4.2 pydantic-2.11.5 pydantic-core-2.33.2 pydantic-settings-2.9.1 pymilvus-2.5.6 pypdf-5.5.0 pypika-0.48.9 pyproject_hooks-1.2.0 python-docx-1.1.2 python-dotenv-1.1.0 python-pptx-1.0.2 pytz-2025.2 requests-2.32.2 requests-oauthlib-2.0.0 requests-toolbelt-1.0.0 rich-14.0.0 rsa-4.9.1 setuptools-80.9.0 shellingham-1.5.4 simsimd-6.2.1 sniffio-1.3.1 soupsieve-2.7 starlette-0.46.2 sympy-1.14.0 tabulate-0.9.0 tenacity-9.1.2 tokenizers-0.20.3 tqdm-4.67.1 typer-0.16.0 typing-extensions-4.13.2 typing-inspect-0.9.0 typing-inspection-0.4.1 tzdata-2025.2 ujson-5.10.0 urllib3-2.4.0 uvicorn-0.34.2 uvloop-0.21.0 watchfiles-1.0.5 websocket-client-1.8.0 websockets-15.0.1 wrapt-1.17.2 yarl-1.20.0 zipp-3.22.0 zstandard-0.23.0

Define credentials

Authenticate the watsonx.ai Runtime service on IBM Cloud Pak for Data. You need to provide the admin's username and the platform url.

In [2]:

username = "PASTE YOUR USERNAME HERE"
url = "PASTE THE PLATFORM URL HERE"

Use the admin's api_key to authenticate watsonx.ai Runtime services:

In [ ]:

import getpass
from ibm_watsonx_ai import Credentials

credentials = Credentials(
    username=username,
    api_key=getpass.getpass("Enter your watsonx.ai API key and hit enter: "),
    url=url,
    instance_id="openshift",
    version="5.2",
)

Alternatively you can use the admin's password:

In [3]:

import getpass
from ibm_watsonx_ai import Credentials

if "credentials" not in locals() or not credentials.api_key:
    credentials = Credentials(
        username=username,
        password=getpass.getpass("Enter your watsonx.ai password and hit enter: "),
        url=url,
        instance_id="openshift",
        version="5.2",
    )

Create `APIClient` instance

In [4]:

from ibm_watsonx_ai import APIClient

client = APIClient(credentials)

Working with projects

First, you need to create a project for your work. If you do not have a project already, create one by following these steps:

Open IBM Cloud Pak for Data
From the menu, click View all projects
Create a new project
Go to the Manage tab
Copy the project_id

Action: Assign the project ID below

In [5]:

import os

try:
    project_id = os.environ["PROJECT_ID"]
except KeyError:
    project_id = input("Please enter your project_id (hit enter): ")

To be able to interact with all resources available in watsonx.ai, you need to set the project which you will be using.

In [6]:

client.set.default_project(project_id)

Out[6]:

'SUCCESS'

RAG Optimizer definition

Define a connection to the training data

Upload the training data to the project as a data asset and then define a connection to the file. This example uses the ModelInference description from the ibm_watsonx_ai documentation.

In [7]:

from langchain_community.document_loaders import WebBaseLoader

url = "https://ibm.github.io/watsonx-ai-python-sdk/fm_model_inference.html"

docs = WebBaseLoader(url).load()
model_inference_content = docs[0].page_content

Out[7]:

USER_AGENT environment variable not set, consider setting it to identify your requests.

Upload the training data to the project as a data asset.

In [8]:

import os

document_filename = "ModelInference.txt"

if not os.path.isfile(document_filename):
    with open(document_filename, "w") as file:
        file.write(model_inference_content)

document_asset_details = client.data_assets.create(
    name=document_filename, file_path=document_filename
)

document_asset_id = client.data_assets.get_id(document_asset_details)
document_asset_id

Out[8]:

Creating data asset...
SUCCESS

'f7c0615d-176a-4af8-879b-bdae1f84bfbe'

Define a connection to the training data.

In [9]:

from ibm_watsonx_ai.helpers import DataConnection

input_data_references = [DataConnection(data_asset_id=document_asset_id)]

Define a connection to the test data

Upload a json file that you want to use as a benchmark to the project as a data asset and then define a connection to the file. This example uses content from the ibm_watsonx_ai SDK documentation.

In [10]:

benchmarking_data_IBM_page_content = [
    {
        "question": "What is path to ModelInference class?",
        "correct_answer": "ibm_watsonx_ai.foundation_models.inference.ModelInference",
        "correct_answer_document_ids": ["ModelInference.txt"],
    },
    {
        "question": "What is method for get model inference details?",
        "correct_answer": "get_details()",
        "correct_answer_document_ids": ["ModelInference.txt"],
    },
]

Upload the benchmark testing data to the project as a data asset with json extension.

In [11]:

import json

test_filename = "benchmarking_data_ModelInference.json"

if not os.path.isfile(test_filename):
    with open(test_filename, "w") as json_file:
        json.dump(benchmarking_data_IBM_page_content, json_file, indent=4)

test_asset_details = client.data_assets.create(
    name=test_filename, file_path=test_filename
)

test_asset_id = client.data_assets.get_id(test_asset_details)
test_asset_id

Out[11]:

Creating data asset...
SUCCESS

'fde6e891-8e6e-4f84-8637-0cb6fecd2142'

Define a connection to the benchmark testing data.

In [12]:

test_data_references = [DataConnection(data_asset_id=test_asset_id)]

Configure the RAG Optimizer

Provide the input information for the AutoAI RAG optimizer:

name - experiment name
description - experiment description
max_number_of_rag_patterns - maximum number of RAG patterns to create
optimization_metrics - target optimization metrics

In [13]:

from ibm_watsonx_ai.experiment import AutoAI

experiment = AutoAI(credentials, project_id=project_id)

rag_optimizer = experiment.rag_optimizer(
    name="AutoAI RAG run - ModelInference documentation",
    description="AutoAI RAG Optimizer on ibm_watsonx_ai ModelInference documentation",
    max_number_of_rag_patterns=4,
    optimization_metrics=[AutoAI.RAGMetrics.ANSWER_CORRECTNESS],
)

To retrieve the configuration parameters, use get_params().

In [14]:

rag_optimizer.get_params()

Out[14]:

{'name': 'AutoAI RAG run - ModelInference documentation',
 'description': 'AutoAI RAG Optimizer on ibm_watsonx_ai ModelInference documentation',
 'max_number_of_rag_patterns': 4,
 'optimization_metrics': ['answer_correctness']}

Run the RAG Experiment

Call the run() method to trigger the AutoAI RAG experiment. Choose one of two modes:

To use the interactive mode (synchronous job), specify background_mode=False
To use the background mode (asynchronous job), specify background_mode=True

In [15]:

run_details = rag_optimizer.run(
    input_data_references=input_data_references,
    test_data_references=test_data_references,
    background_mode=False,
)

Out[15]:

##############################################

Running 'fc391c80-4ceb-4961-b6ed-5063cab855f5'

##############################################


pending..............
running..
completed
Training of 'fc391c80-4ceb-4961-b6ed-5063cab855f5' finished successfully.

To monitor the AutoAI RAG jobs in background mode, use the get_run_status() method.

In [16]:

rag_optimizer.get_run_status()

Out[16]:

'completed'

Compare and test RAG Patterns

You can list the trained patterns and information on evaluation metrics in the form of a Pandas DataFrame by calling the summary() method. Use the DataFrame to compare all discovered patterns and select the one you want for further testing.

In [17]:

summary = rag_optimizer.summary()
summary

Out[17]:

Additionally, you can pass the scoring parameter to the summary method to filter RAG patterns, starting with the best.

In [18]:

summary = rag_optimizer.summary(scoring="faithfulness")

Get the selected pattern

Get the RAGPattern object from the RAG Optimizer experiment. By default, the RAGPattern of the best pattern is returned.

In [19]:

best_pattern_name = summary.index.values[0]
print("Best pattern is:", best_pattern_name)

best_pattern = rag_optimizer.get_pattern(pattern_name="Pattern1")
best_pattern

Out[19]:

Best pattern is: Pattern2

<ibm_watsonx_ai.foundation_models.extensions.rag.pattern.pattern.RAGPattern at 0x144a3f0e0>

To retrieve the pattern details, use the get_pattern_details method.

In [20]:

rag_optimizer.get_pattern_details(pattern_name="Pattern2")

Out[20]:

{'composition_steps': ['model_selection',
  'chunking',
  'embeddings',
  'retrieval',
  'generation'],
 'duration_seconds': 4,
 'location': {'evaluation_results': '/projects/d940d2db-e37e-4c6a-b646-beb97e76250c/assets/auto_ml/auto_ml.fd93b72d-b43c-4e0a-badc-7ffb466342e2/wml_data/fc391c80-4ceb-4961-b6ed-5063cab855f5/Pattern2/evaluation_results.json',
  'indexing_notebook': '/projects/d940d2db-e37e-4c6a-b646-beb97e76250c/assets/auto_ml/auto_ml.fd93b72d-b43c-4e0a-badc-7ffb466342e2/wml_data/fc391c80-4ceb-4961-b6ed-5063cab855f5/Pattern2/indexing_inference_notebook.ipynb',
  'inference_notebook': '/projects/d940d2db-e37e-4c6a-b646-beb97e76250c/assets/auto_ml/auto_ml.fd93b72d-b43c-4e0a-badc-7ffb466342e2/wml_data/fc391c80-4ceb-4961-b6ed-5063cab855f5/Pattern2/indexing_inference_notebook.ipynb',
  'inference_service_code': '/projects/d940d2db-e37e-4c6a-b646-beb97e76250c/assets/auto_ml/auto_ml.fd93b72d-b43c-4e0a-badc-7ffb466342e2/wml_data/fc391c80-4ceb-4961-b6ed-5063cab855f5/Pattern2/inference_ai_service.gz',
  'inference_service_metadata': '/projects/d940d2db-e37e-4c6a-b646-beb97e76250c/assets/auto_ml/auto_ml.fd93b72d-b43c-4e0a-badc-7ffb466342e2/wml_data/fc391c80-4ceb-4961-b6ed-5063cab855f5/Pattern2/inference_service_metadata.json'},
 'name': 'Pattern2',
 'settings': {'chunking': {'chunk_overlap': 256,
   'chunk_size': 1024,
   'method': 'recursive'},
  'embeddings': {'model_id': 'ibm/slate-125m-english-rtrvr',
   'truncate_input_tokens': 512,
   'truncate_strategy': 'left'},
  'generation': {'context_template_text': '[Document]\n{document}\n[End]',
   'model_id': 'ibm/granite-13b-instruct-v2',
   'parameters': {'decoding_method': 'greedy',
    'max_new_tokens': 1000,
    'max_sequence_length': 8192,
    'min_new_tokens': 1},
   'prompt_template_text': '<|system|>\nYou are Granite Chat, an AI language model developed by IBM. You are a cautious assistant. You carefully follow instructions. You are helpful and harmless and you follow ethical guidelines and promote positive behavior.<|user|>\nYou are an AI language model designed to function as a specialized Retrieval Augmented Generation (RAG) assistant. When generating responses, prioritize correctness, i.e., ensure that your response is grounded in context and user query. Always make sure that your response is relevant to the question. \nAnswer Length: detailed\n{reference_documents}\nRespond exclusively in the language of the question, regardless of any other language used in the provided context. Ensure that your entire response is in the same language as the question.\n{question} \n\n<|assistant|>',
   'word_to_token_ratio': 2.491},
  'retrieval': {'method': 'window', 'number_of_chunks': 5, 'window_size': 3},
  'vector_store': {'datasource_type': 'chroma',
   'distance_metric': 'cosine',
   'index_name': 'autoai_rag_fc391c80_20250530105635',
   'operation': 'upsert',
   'schema': {'fields': [{'description': 'text field',
      'name': 'text',
      'role': 'text',
      'type': 'string'},
     {'description': 'document name field',
      'name': 'document_id',
      'role': 'document_name',
      'type': 'string'},
     {'description': 'chunk starting token position in the source document',
      'name': 'start_index',
      'role': 'start_index',
      'type': 'number'},
     {'description': 'chunk number per document',
      'name': 'sequence_number',
      'role': 'sequence_number',
      'type': 'number'},
     {'description': 'vector embeddings',
      'name': 'vector',
      'role': 'vector_embeddings',
      'type': 'array'}],
    'id': 'autoai_rag_1.0',
    'name': 'Document schema using open-source loaders',
    'type': 'struct'}}},
 'settings_importance': {'chunking': [{'importance': 0.11111111,
    'parameter': 'chunk_size'},
   {'importance': 0.11111111, 'parameter': 'chunk_overlap'}],
  'embeddings': [{'importance': 0.11111111, 'parameter': 'embedding_model'}],
  'generation': [{'importance': 0.11111111, 'parameter': 'foundation_model'}],
  'retrieval': [{'importance': 0.22222222, 'parameter': 'retrieval_method'},
   {'importance': 0.11111111, 'parameter': 'window_size'},
   {'importance': 0.11111111, 'parameter': 'number_of_chunks'}]}}

Create the index/collection

Build a solution using the best pattern with additional document indexing.

To check the index_name that you are working on, use the best_pattern method.

In [21]:

best_pattern.vector_store._index_name

Out[21]:

'autoai_rag_fc391c80_20250530105632'

In [22]:

urls = [
    "https://ibm.github.io/watsonx-ai-python-sdk/fm_embeddings.html",
    "https://ibm.github.io/watsonx-ai-python-sdk/fm_custom_models.html",
    "https://ibm.github.io/watsonx-ai-python-sdk/fm_text_extraction.html",
]
docs_list = WebBaseLoader(urls).load()
doc_splits = best_pattern.chunker.split_documents(docs_list)

In [23]:

best_pattern.indexing_function(doc_splits)

Out[23]:

['a82844b130d06b51e44648fdc74143602a552fd34d5419397e9f7225f40f36ce',
 '39e675e63899d66333b5cb49bdae94ff07abc18bc4889aa1ba4fc0e2285d52fa',
 'ceb9173fe07e2aabc1b9ed70b21c619153da6b1f4b374e487cd23e3a27a98dac',
 'fa01210dd08d98b49345f215d03182a0f4297f7d7060efeae4f0b2cbab2ebc90',
 '27256afee0899a720bdfaa8c630958a98a8b79d7ac102627aef353c35a74755b',
 'ec8e2661748745a6bedb0ec0589043332a82a585f4750bd1c77041d2dcf103c8',
 '67afb8c0898213fbe54593ffe8c3a139164f86cf07fa02e023f88f850eac840e',
 'bac740d77e22d5d5dc3538f2a35df16cea82677e1ecb6fd499aa92aa1904865a',
 'e006e431a0c03fddf36f5cd99acdc178dfc2262a85646c226e5b726a7955b3f0',
 'cb1251665b793077c4687e1c852a75fc47b77c448a599ade924217e27427094b',
 'a582fb7649e34bdcc43deaf244ec78e4fa5ca42e91c157aed7217a4f106eb9e1',
 '82082085128a22276f4eafd78305d7e9b340f52d97b8f62e2337daa9bebdd51f',
 '79fbdaf9cafafd53ca1b1ac0bfce108d87c82abf34cd11b10847bb205abecf8f',
 'a9c34ffa073ef2bca97787749af7ae4aeb7ce19859cc244833b77c41f0f38027',
 'b3d4068f131da72cc054a2c1ffb7b3895284866dafbc1e361852b080a62c4769',
 'a655284bd8cfbcc07edca37da68d0ae212050fe6f6ea739f111765c810abf599',
 'd95e116a28befe6aa56de183c5c4be877c7995e0397431bbd85b8ff1e48d8eb1',
 '06d6422a78a08f9edcf10a7f9c4545a181fa8dc8a943be81027e61130a6d3658',
 '14ba324283b13fb92f4cf6f8026445812208be6dbeabd576f15020cfcddb5fc4',
 '66aa8083d1ad3730f41c6d4b3f5459b7d0c674bb3fe0db66e6f663dc4ccb5a7e',
 '63074ccae3deb8ec725f6b7c8a893367b29f716531c9f0835dd4daaf625e4092',
 '54013f1a56052deaa9fee7058438c4ba923125f4597e3cd5bab7f5d84423330a',
 '407d5ad252378be5b7fe545752b53a2d21bb3b4af44b4f7c150d9b232051f69c',
 '036fecbdee70c831ccf360a285fec4698be6859ac8cb87206c78f7051890626a',
 'de7c8756c92d70268861ce74f06d7c41c9c51c06870cc702847f715d90734ebb',
 '5f0e691ff7b9c2ba216b0de789881adff45f968c6f96966237b6a878c0dab33c',
 '4fc8009e71748c1fe2a07a099abaa5786915b9c8adf8ac75151803671e45623a',
 'd2dcdd06d0b378e4b5c55adb6b5118f30630f62f4f6f1a567c6e68906caa6ee2',
 'ea9b0a4f80f4798fcf8349de57856d478d8f05bda4be2bf346f88091444f36e8',
 'e0b6d8f05eb0bb973b908c0ece270d45b7c42531a155603ab646f25ec324cc63',
 '3c42575b207752c5697db7bd10b9dbe1579649930b47bd4400a3b865862494f8',
 '56e56edb5f84e16259b682fb2f0ca32cbdea50dc038f87087b1264606a7ea051',
 '7efbf4999ac6e9cbd6f887041e911872ac6165ab5ffb68a91341a049ee1dc117',
 'bea4b651095f5eb46d89937ad50722c13a1f707aa6d302706400acd92404826e',
 'd6bfb76cbbaacc462af5728db93665e588c7bb7d0f9c60bdb6c34594251d4548',
 '8135abdd65a0a65a1cdf63528a5feb3a74fe0863cd3ec4bbae9a1ac8482d6bb2',
 'acb04e2f5e2f15605e7aa4823d379d264451756810c2ac7a3e43d1bf0da8dee4',
 'f289d41a3b79f82bb1185d271bbb7d9bb08c08ae03e41c416fac3dde41fc305e',
 '054038516ac1e65f8d6159fb84115246f164ae574fcb5d9916c0556b82afacd5',
 '4d72978490bd4613215d5480352d759f9408fe264dcd0db41245064b8e22fe08',
 'c0c6faa5120d4e6cce54dd4831b52d091a6ea69ec4d1d38313966faeab449808',
 '8de49d2aa7ba7133dcb9f46b1ddada5a167b2d646ac4eb69dd433064856b0200',
 '523e241b60aca3275209298e017a3f532761a2ad33a61de0af73a43d09e2a807',
 '2df613b9a6f2e23fad164d4b5c5130178363335052ad3f59d36818dd4370a512',
 'b35f4caafd027b6ecd8df78c0418d3cfeb7b31509047e4e6bca86b1e79caa1f5',
 '985b14c8a719589d8655e4295231e09a9322967011a3656f2058f4e62f2f7907',
 'd5ab112209c7c26be7838cd3ca6a81dc9a226a772023f0e31470908fbf015f71',
 '7c850674a9b72fb3c55fe30ff4741a316e4d974fe5b9a67bda087db8b09e71d2',
 '657ccf137057f2c789c3aafa642d83bcf374967e93697cf83f38009bbba8a832',
 'f28f5875b517d543d86e72c40159ae25d72a5841e847f583dacc0c452a540a53',
 '935559da98db7bdf4f553cd2a84a9f10d99221336a6a18a633c02ff39f063dab',
 '22c91f71039950578f482f04d7a6d681dbbfb0243bd492185a6abc164d41c3a7',
 '4feec736840279eb5821049b2e133cbe16718eef09ea65058f646ea72f60aaa6',
 '11ef4fd55dfd97e433877edf9330f553fe6e26f651fd0e9addbe2ba894c641a7',
 '0d4653778290f48e5d0763075f6c0f025f458deb38b8c2670cb7b37866431307',
 '795d52d97efc11446249dd93faeed91b5d65c65858944dbbf85f072b279758af',
 'f48d2f377eec9b032202e5b53ed2a28ad2748b09a4eb34f15a37956084107a63',
 '2a3798d27f648b283b24c9583fff9d8c85935c38efd1506afdf926a690494fbb',
 '002784dd58426a0a1828a338f166dc0cfc78b109eef390b9db23d95df52a0bb6',
 'dff925400d25a70841818de753a07f155dd2d22250be2ed9a9dc006c5a1a1a81',
 '1ba3e2a6925d37ebf6a3dbd6cfb8776ecd1871cc45b9f408ece338d13fc1f576',
 '0858093b241759340cb73ab2e685084f70abeb21bb73dec7baa73e5949300d19',
 '96a1c0ecc84c8f1ff73ccbc8e8f9867e796603650b9a843cbb0a4138215e0a03',
 '138b981731d93210f29db1288a8e88646aeb945adbb2804b443065d37e8b5d1f',
 '7f87333c8e9cb19ab29d6907c39e488bdce99bb9e8e6cf7e64f3eca4b58d5fe4',
 'bdc676a9bec2ec6d0a4bd8c0e1b7b714cadddbe333fad3fa10f9b07da1d20b18',
 '8358bb72eee5be9a03966dec125591c2d8f9d1050ecd8358ea64f5c4207a1645',
 '4c19d68f8a1a4b8c987a3c5a9e829b73e9298775c3739981b962fb9cbfabee70',
 '5f81cc16674f19c9c705383b63c4b7b02fac8c8d1e961d2e76a22fecb5090ae5',
 'b531da0c4f55a313ec7f69f71c33935403a8ace2f63241a51bf095bbcc0b9bba',
 '307c4bdc4fc959fe90c10d9e8902004d63cbee7e596f4d4fc61a20bdcab59492',
 'c8054fe86f45453764703b8b6f2c49e7dbcc75c3eec79cdaa34fdad74d01c0d0',
 'b2bb660eea727fd1610f21e29e53e8f23e12dadfc720d52f3eca347c49a7c7f3',
 'c377697262314d7a4a34996971a97a0d1fc48f70ee84d4ee6d92d4d1e798fdc9',
 '00c83cdc87536f7c6a77c0b356118591a79a287ceea7349dabe30af09122c22c',
 '7b0f6c46d5944c08bb15de405e742b59584f63261fb150e54470b5aec62aa13c',
 'c0254ade7d05b580ab85329e739ce70ae7b646eeaf3820e32c4fa9a548d55d7d',
 '12cf1b76a8c420196d380791b9b35f0052cdc68ad0f6d6adb3397741a64101c0',
 '4388cac339ffe1c1153f971f66e5b242cab36ade69b62f8140a66c15e29b8a59',
 '28ea3e1badfcf1263732faaa74e581d1818736b86fa5e58149773e1da742dbd2',
 '8c82a40cda17329e61f5cc3caa2528eb73748364113c044e5914ed61405872bf',
 '637ea922aea262e7de1438c86a7700f0fc4b480778ec9ccbe24ffff2a78f6eb5',
 'bca950a82f20aec4b45d4ac3e4f0c290c54b0a40d2d5d9dfd5ee6944ffe1c5ef',
 '81ccb1f79114d614a9af5b7f02712132cd8f0bcea4ecbc4fbeca2bf6419476f0',
 '76f7b60eb4dfafb76468c6581e06cfd9e19a97749f25aec40224e72f38b6f413',
 '2e64256b8d0d079790e1f7cd2b76da41994ef339ae313f674347b45838f5b5d0',
 '34f287ffa8928ea2da4fa2e90b7560b7cb472fcd0e56689b00dbaa480a0ff80d',
 '91f8f7f5154f42a76ab7ad6296bf8663698c84dd143cd90748ff98949aa4e079',
 'ea1b8b784e0a96e47c2e5c5e97216d06f989d627fcef233abb98c462aeb86425',
 '5fa3df7ae174166ea74b4777490e3742d4b6424965f4845c4561e343a125fb85',
 '286f7611fdf4b3bddbca3c01788c9b29a2be1a666a220677e00165f64ed3541b',
 '4613cb6c776499e3b2e51de9239360fffacd5b9eaaffa4addff19e53fa660239',
 'f573aac0e8acde727f7ff5baeb0acf524eff44f60fb3658777ca8eb8e120e982',
 '5cc42ec61f14e4006e707e9416a7972fbf2e39c74d10278d3b22dfaa9e494bd4',
 'ced78afa16b34750ff05170caf69afbd0f83bb4c243bd25d021bc5959c37ba87',
 'd608615aa102070d9478ce422c2dd6d1718ec3f5d7cc2b62b7890b279aa44fd4',
 '5c4b5417e2407af74c25970a5c8076d10c0d073c6087cec04aabae9f9d65b494',
 'e5aa5e3950e5b2738637ded602721cab46ec52951b3112a374f87724a8ed9af2',
 '74c2294dd1cc3b7b00eb4103a075851d665dcafa272acc982125c9219edc316d',
 '99bf1069d215f5b6150b341265d25d93bbd82e3202d92a70e5bb7ca1a25d3312',
 '6c7ffc358576a0905e3d19438bdabcdf331dff5c8d8d6b549405ba64ac19f51c',
 '4f5e2fef97c3a50bc7771f956b5e898de00a0b10490c4d9f83d3c02368467914',
 '861318a633bd7d9f3b2b5d0a5fa0480089a1893a1b708d5538799ae1a38d2ba4']

Query the RAGPattern locally to test it.

In [24]:

questions = ["How to add Task Credentials?"]

payload = {
    client.deployments.ScoringMetaNames.INPUT_DATA: [
        {"values": questions, "access_token": client.token}
    ]
}

best_pattern.inference_function()(payload)

Out[24]:

{'predictions': [{'fields': ['answer', 'reference_documents'],
   'values': [[' Add Task Credentials¶\n\nWarning\nIf not already added, Task Credentials are required on IBM watsonx.ai for IBM Cloud to make a deployment.\n\nWith task credentials, you can deploy a custom foundation model and avoid token expiration issues.\nFor more details, see Adding task credentials.\nTo list available task credentials, use the list method:\nclient.task_credentials.list()\n[End]\n[Document]\nNote\nWhen the credentials parameter is passed, one of these parameters is required: [project_id, space_id].\n\nHint\nYou can copy the project_id from the Project’s Manage tab (Project -> Manage -> General -> Details).\n\nExample:\nfrom ibm_watsonx_ai import Credentials\nfrom ibm_watsonx_ai.foundation_models import Embeddings\nfrom ibm_watsonx_ai.metanames import EmbedTextParamsMetaNames as EmbedParams\nfrom ibm_watsonx_ai.foundation_models.utils.enums import EmbeddingTypes\n[End]\nIf the list is empty, you can create new task credentials with the store method:\nclient.task_credentials.store()\n\n\n',
     [{'page_content': 'If the list is empty, you can create new task credentials with the store method:\nclient.task_credentials.store()\n\n\nTo get the status of available task credentials, use the get_details method:\nclient.task_credentials.get_details()',
       'metadata': {'document_id': '338243661372903145',
        'language': 'en',
        'sequence_number': 8,
        'source': 'https://ibm.github.io/watsonx-ai-python-sdk/fm_custom_models.html',
        'start_index': 0,
        'title': 'Custom models - IBM watsonx.ai'}},
      {'page_content': 'client = APIClient(credentials)\nclient.set.default_project(project_id=project_id)\n# or client.set.default_space(space_id=space_id)\n\n\n\n\nAdd Task Credentials¶\n\nWarning\nIf not already added, Task Credentials are required on IBM watsonx.ai for IBM Cloud to make a deployment.\n\nWith task credentials, you can deploy a custom foundation model and avoid token expiration issues.\nFor more details, see Adding task credentials.\nTo list available task credentials, use the list method:\nclient.task_credentials.list()',
       'metadata': {'document_id': '338243661372903145',
        'language': 'en',
        'sequence_number': 7,
        'source': 'https://ibm.github.io/watsonx-ai-python-sdk/fm_custom_models.html',
        'start_index': 0,
        'title': 'Custom models - IBM watsonx.ai'}},
      {'page_content': 'Note\nWhen the credentials parameter is passed, one of these parameters is required: [project_id, space_id].\n\n\nHint\nYou can copy the project_id from the Project’s Manage tab (Project -> Manage -> General -> Details).\n\nExample:\n from ibm_watsonx_ai import Credentials\n from ibm_watsonx_ai.foundation_models import Embeddings\n from ibm_watsonx_ai.metanames import EmbedTextParamsMetaNames as EmbedParams\n from ibm_watsonx_ai.foundation_models.utils.enums import EmbeddingTypes',
       'metadata': {'document_id': '3720827352634741441',
        'language': 'en',
        'sequence_number': 13,
        'source': 'https://ibm.github.io/watsonx-ai-python-sdk/fm_embeddings.html',
        'start_index': 0,
        'title': 'Embeddings - IBM watsonx.ai'}}]]]}]}

Historical runs

In this section, you will learn how to work with historical RAG Optimizer jobs (runs).

To list historical runs, use the list() method and provide the 'rag_optimizer' filter.

In [25]:

experiment.runs(filter="rag_optimizer").list()

Out[25]:

In [26]:

run_id = run_details["metadata"]["id"]
run_id

Out[26]:

'fc391c80-4ceb-4961-b6ed-5063cab855f5'

Get the executed optimizer's configuration parameters

In [27]:

experiment.runs.get_rag_params(run_id=run_id)

Out[27]:

{'name': 'AutoAI RAG run - ModelInference documentation',
 'description': 'AutoAI RAG Optimizer on ibm_watsonx_ai ModelInference documentation',
 'max_number_of_rag_patterns': 4,
 'optimization_metrics': ['answer_correctness']}

Get the historical `rag_optimizer` instance and training details

In [28]:

historical_opt = experiment.runs.get_rag_optimizer(run_id)

List trained patterns for the selected optimizer

In [29]:

historical_opt.summary()

Out[29]:

Clean up

To delete the current experiment, use the cancel_run(hard_delete=True) method.

Warning: Be careful: once you delete an experiment, you will no longer be able to refer to it.

In [30]:

rag_optimizer.cancel_run(hard_delete=True)

Out[30]:

'SUCCESS'

To clean up all of the created assets:

experiments
trainings
pipelines
model definitions
models
functions
deployments

follow the steps in this sample notebook.

Summary and next steps

You successfully completed this notebook!

You learned how to use ibm-watsonx-ai to run AutoAI RAG experiments.

Check out our Online Documentation for more samples, tutorials, documentation, how-tos, and blog posts.

Authors

Mateusz Szewczyk, Software Engineer at watsonx.ai

Use AutoAI RAG and Chroma to create a pattern and get information from `ibm-watsonx-ai` SDK documentation

Disclaimers

Notebook content

Learning goal

Table of Contents

Set up the environment

Install dependencies

Define credentials

Create `APIClient` instance

Working with projects

RAG Optimizer definition

Define a connection to the training data

Define a connection to the test data

Configure the RAG Optimizer

Run the RAG Experiment

Compare and test RAG Patterns

Get the selected pattern

Create the index/collection

Historical runs

Get the executed optimizer's configuration parameters

Get the historical `rag_optimizer` instance and training details

List trained patterns for the selected optimizer

Clean up

Summary and next steps

Authors

Product

Resources

Company

Use AutoAI RAG and Chroma to create a pattern and get information from ibm-watsonx-ai SDK documentation

Disclaimers

Notebook content

Learning goal

Table of Contents

Set up the environment

Install dependencies

Define credentials

Create APIClient instance

Working with projects

RAG Optimizer definition

Define a connection to the training data

Define a connection to the test data

Configure the RAG Optimizer

Run the RAG Experiment

Compare and test RAG Patterns

Get the selected pattern

Create the index/collection

Historical runs

Get the executed optimizer's configuration parameters

Get the historical rag_optimizer instance and training details

List trained patterns for the selected optimizer

Clean up

Summary and next steps

Authors

Use AutoAI RAG and Chroma to create a pattern and get information from `ibm-watsonx-ai` SDK documentation

Create `APIClient` instance

Get the historical `rag_optimizer` instance and training details