Book a Demo!
CoCalc Logo Icon
StoreFeaturesDocsShareSupportNewsAboutPoliciesSign UpSign In
ibm
GitHub Repository: ibm/watson-machine-learning-samples
 Path: blob/master/cpd5.1/notebooks/python_sdk/deployments/custom_library/Use custom software spec to create statsmodels function.ipynb
6408 views
Kernel: Python 3 (ipykernel)

Use custom software_spec to create statsmodels function describing data with ibm-watsonx-ai

This notebook demonstrates how to deploy in Watson Machine Learning service a python function with statsmodel which requires to create custom software specification using conda yaml file with all required libraries. Some familiarity with bash is helpful. This notebook uses Python 3.11 with statsmodel.

Learning goals

The learning goals of this notebook are:

  • Working with the Watson Machine Learning instance

  • Creating custom software specification

  • Online deployment of python function

  • Scoring data using deployed function

Contents

This notebook contains the following parts:

  1. Setup

  2. Function creation

  3. Function upload

  4. Web service creation

  5. Scoring

  6. Clean up

  7. Summary and next steps

1. Set up the environment

Before you use the sample code in this notebook, you must perform the following setup tasks:

  • Contact with your Cloud Pak for Data administrator and ask them for your account credentials

Install and import the ibm-watsonx-ai and dependecies

Note: ibm-watsonx-ai documentation can be found here.

!pip install wget | tail -n 1
!pip install -U ibm-watsonx-ai | tail -n 1

Connection to WML

Authenticate the Watson Machine Learning service on IBM Cloud Pak for Data. You need to provide platform url, your username and api_key.

username = 'PASTE YOUR USERNAME HERE'
api_key = 'PASTE YOUR API_KEY HERE'
url = 'PASTE THE PLATFORM URL HERE'

from ibm_watsonx_ai import Credentials

credentials = Credentials(
    username=username,
    api_key=api_key,
    url=url,
    instance_id="openshift",
    version="5.1"
)

Alternatively you can use username and password to authenticate WML services.

credentials = Credentials(
    username=***,
    password=***,
    url=***,
    instance_id="openshift",
    version="5.1"
)


from ibm_watsonx_ai import APIClient

client = APIClient(credentials)

Working with spaces

First of all, you need to create a space that will be used for your work. If you do not have space already created, you can use {PLATFORM_URL}/ml-runtime/spaces?context=icp4data to create one.

  • Click New Deployment Space

  • Create an empty space

  • Go to space Settings tab

  • Copy space_id and paste it below

Tip: You can also use SDK to prepare the space for your work. More information can be found here.

Action: Assign space ID below

space_id = 'PASTE YOUR SPACE ID HERE'

You can use list method to print all existing spaces.

client.spaces.list(limit=10)

To be able to interact with all resources available in Watson Machine Learning, you need to set space which you will be using.

client.set.default_space(space_id)
'SUCCESS'

2. Create function

In this section you will learn how to create deployable function with statsmodels module calculating describition of a given data. Hint: To install statsmodels execute !pip install statsmodels.

Create deploayable callable which uses stsmodels library

def deployable_callable():
    """
    Deployable python function with score
    function implemented.
    """
    try:
        from statsmodels.stats.descriptivestats import describe
    except ModuleNotFoundError as e:
        print(f"statsmodels not installed: {str(e)}")
        
    def score(payload):
        """
        Score method.
        """
        try:
            data = payload['input_data'][0]['values']
            return {
                'predictions': [
                    {'values': str(describe(data))}
                ]
            }
        except Exception as e:
            return {'predictions': [{'values': [repr(e)]}]}
        
    return score

Test callable locally

Hint: To install numpy execute !pip install numpy.

import numpy as np

data = np.random.randn(10, 10)
data_description = deployable_callable()({
    "input_data": [{
        "values" : data
    }]
})

print(data_description["predictions"][0]["values"])
0 1 2 3 4 \ nobs 10.000000 10.000000 10.000000 10.000000 10.000000 missing 0.000000 0.000000 0.000000 0.000000 0.000000 mean 0.905005 -0.182868 0.212348 -0.196478 -0.066239 std_err 0.298759 0.434762 0.329075 0.411152 0.338328 upper_ci 1.490562 0.669249 0.857322 0.609365 0.596871 lower_ci 0.319448 -1.034986 -0.432627 -1.002320 -0.729350 std 0.944759 1.374838 1.040625 1.300175 1.069887 iqr 1.096161 1.229678 1.127913 1.153995 1.466211 iqr_normal 0.812586 0.911562 0.836123 0.855458 1.086904 mad 0.708938 1.064078 0.810868 0.959040 0.869382 mad_normal 0.888522 1.333624 1.016272 1.201979 1.089609 coef_var 1.043927 -7.518190 4.900571 -6.617419 -16.151875 range 3.008860 4.038466 3.723421 4.562791 3.387978 max 2.393400 2.451232 1.952777 1.776727 1.562780 min -0.615461 -1.587233 -1.770645 -2.786064 -1.825198 skew 0.166580 1.063480 -0.231102 -0.418840 0.000360 kurtosis 2.224792 2.683268 2.750487 2.883175 2.023159 jarque_bera 0.296643 1.926782 0.114954 0.298065 0.397591 jarque_bera_pval 0.862154 0.381597 0.944144 0.861541 0.819718 mode -0.615461 -1.587233 -1.770645 -2.786064 -1.825198 mode_freq 0.100000 0.100000 0.100000 0.100000 0.100000 median 0.799308 -0.655696 0.302684 0.021386 -0.086088 1% -0.555286 -1.551177 -1.661206 -2.652608 -1.755113 5% -0.314589 -1.406952 -1.223453 -2.118784 -1.474771 10% -0.013718 -1.226671 -0.676261 -1.451504 -1.124343 25% 0.274223 -1.100459 -0.388312 -0.783669 -0.828157 50% 0.799308 -0.655696 0.302684 0.021386 -0.086088 75% 1.370384 0.129219 0.739601 0.370326 0.638054 90% 2.248126 2.055993 1.228070 1.325070 1.270827 95% 2.320763 2.253613 1.590423 1.550898 1.416803 99% 2.378872 2.411708 1.880306 1.731561 1.533584 5 6 7 8 9 nobs 10.000000 10.000000 10.000000 10.000000 10.000000 missing 0.000000 0.000000 0.000000 0.000000 0.000000 mean 0.138192 0.304227 -0.085176 -0.619984 0.093436 std_err 0.456912 0.355763 0.385024 0.408175 0.308862 upper_ci 1.033724 1.001509 0.669457 0.180025 0.698794 lower_ci -0.757339 -0.393055 -0.839809 -1.419993 -0.511922 std 1.444884 1.125020 1.217553 1.290764 0.976706 iqr 1.790504 1.818180 1.144880 1.193322 1.018550 iqr_normal 1.327303 1.347819 0.848701 0.884611 0.755052 mad 1.149885 0.896867 0.884064 0.898634 0.719349 mad_normal 1.441167 1.124056 1.108009 1.126271 0.901571 coef_var 10.455595 3.697966 -14.294534 -2.081930 10.453222 range 4.833053 2.950856 4.019097 4.568185 3.121643 max 2.137646 1.643836 1.785339 2.251724 1.184207 min -2.695407 -1.307020 -2.233759 -2.316461 -1.937436 skew -0.615778 -0.225584 -0.160082 0.847988 -0.757840 kurtosis 2.539002 1.719803 2.396965 3.638611 2.878239 jarque_bera 0.720521 0.767691 0.194232 1.368399 0.963379 jarque_bera_pval 0.697494 0.681237 0.907451 0.504494 0.617739 mode -2.695407 -1.307020 -2.233759 -2.316461 -1.937436 mode_freq 0.100000 0.100000 0.100000 0.100000 0.100000 median 0.523338 0.306400 0.021761 -0.621855 0.069720 1% -2.576302 -1.304159 -2.157896 -2.296658 -1.841602 5% -2.099879 -1.292714 -1.854446 -2.217446 -1.458267 10% -1.504351 -1.278407 -1.475133 -2.118431 -0.979099 25% -0.676454 -0.503206 -0.762092 -1.312521 -0.203885 50% 0.523338 0.306400 0.021761 -0.621855 0.069720 75% 1.114050 1.314974 0.382788 -0.119200 0.814665 90% 1.316257 1.627767 1.482916 0.285203 1.182370 95% 1.726952 1.635801 1.634127 1.268464 1.183288 99% 2.055507 1.642229 1.755096 2.055072 1.184023

3. Upload python function

In this section you will learn how to upload the python function to the Cloud.

Custom software_specification

Create new software specification based on default Python 3.11 environment extended by autoai-libs package.

config_yml =\
"""name: python311
channels:
  - empty
  - nodefaults
dependencies:
- pip:
  - statsmodels

prefix: /opt/anaconda3/envs/python311
"""

with open("config.yaml", "w", encoding="utf-8") as f:
    f.write(config_yml)

base_sw_spec_id = client.software_specifications.get_id_by_name("runtime-24.1-py3.11")

!cat config.yaml
name: python311 channels: - empty - nodefaults dependencies: - pip: - statsmodels prefix: /opt/anaconda3/envs/python311

config.yaml file describes details of package extention. Now you need to store new package extention with APIClient.

meta_prop_pkg_extn = {
    client.package_extensions.ConfigurationMetaNames.NAME: "statsmodels env",
    client.package_extensions.ConfigurationMetaNames.DESCRIPTION: "Environment with statsmodels",
    client.package_extensions.ConfigurationMetaNames.TYPE: "conda_yml"
}

pkg_extn_details = client.package_extensions.store(meta_props=meta_prop_pkg_extn, file_path="config.yaml")
pkg_extn_id = client.package_extensions.get_id(pkg_extn_details)
pkg_extn_url = client.package_extensions.get_href(pkg_extn_details)
Creating package extensions SUCCESS

Create new software specification and add created package extention to it.

meta_prop_sw_spec = {
    client.software_specifications.ConfigurationMetaNames.NAME: "statsmodels software_spec",
    client.software_specifications.ConfigurationMetaNames.DESCRIPTION: "Software specification for statsmodels",
    client.software_specifications.ConfigurationMetaNames.BASE_SOFTWARE_SPECIFICATION: {"guid": base_sw_spec_id}
}

sw_spec_details = client.software_specifications.store(meta_props=meta_prop_sw_spec)
sw_spec_id = client.software_specifications.get_id(sw_spec_details)

client.software_specifications.add_package_extension(sw_spec_id, pkg_extn_id)
SUCCESS
'SUCCESS'

Get the details of created software specification

client.software_specifications.get_details(sw_spec_id)

Store the function

meta_props = {
    client.repository.FunctionMetaNames.NAME: "statsmodels function",
    client.repository.FunctionMetaNames.SOFTWARE_SPEC_ID: sw_spec_id
}

function_details = client.repository.store_function(meta_props=meta_props, function=deployable_callable)
function_id = client.repository.get_function_id(function_details)

Get function details

client.repository.get_details(function_id)

Note: You can see that function is successfully stored in Watson Machine Learning Service.

client.repository.list_functions()

4. Create online deployment

You can use commands bellow to create online deployment for stored function (web service).

Create online deployment of a python function

metadata = {
    client.deployments.ConfigurationMetaNames.NAME: "Deployment of statsmodels function",
    client.deployments.ConfigurationMetaNames.ONLINE: {}
}

function_deployment = client.deployments.create(function_id, meta_props=metadata)
###################################################################################### Synchronous deployment creation for id: 'fe302e1b-17d9-470d-9884-5d5d515e14ea' started ###################################################################################### initializing Note: online_url is deprecated and will be removed in a future release. Use serving_urls instead. ..................... ready ----------------------------------------------------------------------------------------------- Successfully finished deployment creation, deployment_id='6828cd36-5548-4e67-b1b3-d67547af8630' ----------------------------------------------------------------------------------------------- 
client.deployments.list()

Get deployment id.

deployment_id = client.deployments.get_id(function_deployment)
print(deployment_id)
6828cd36-5548-4e67-b1b3-d67547af8630

5. Scoring

You can send new scoring records to web-service deployment using score method.

scoring_payload = {
    "input_data": [{
        'values': data
    }]
}

predictions = client.deployments.score(deployment_id, scoring_payload)
print(data_description["predictions"][0]["values"])
0 1 2 3 4 \ nobs 10.000000 10.000000 10.000000 10.000000 10.000000 missing 0.000000 0.000000 0.000000 0.000000 0.000000 mean 0.905005 -0.182868 0.212348 -0.196478 -0.066239 std_err 0.298759 0.434762 0.329075 0.411152 0.338328 upper_ci 1.490562 0.669249 0.857322 0.609365 0.596871 lower_ci 0.319448 -1.034986 -0.432627 -1.002320 -0.729350 std 0.944759 1.374838 1.040625 1.300175 1.069887 iqr 1.096161 1.229678 1.127913 1.153995 1.466211 iqr_normal 0.812586 0.911562 0.836123 0.855458 1.086904 mad 0.708938 1.064078 0.810868 0.959040 0.869382 mad_normal 0.888522 1.333624 1.016272 1.201979 1.089609 coef_var 1.043927 -7.518190 4.900571 -6.617419 -16.151875 range 3.008860 4.038466 3.723421 4.562791 3.387978 max 2.393400 2.451232 1.952777 1.776727 1.562780 min -0.615461 -1.587233 -1.770645 -2.786064 -1.825198 skew 0.166580 1.063480 -0.231102 -0.418840 0.000360 kurtosis 2.224792 2.683268 2.750487 2.883175 2.023159 jarque_bera 0.296643 1.926782 0.114954 0.298065 0.397591 jarque_bera_pval 0.862154 0.381597 0.944144 0.861541 0.819718 mode -0.615461 -1.587233 -1.770645 -2.786064 -1.825198 mode_freq 0.100000 0.100000 0.100000 0.100000 0.100000 median 0.799308 -0.655696 0.302684 0.021386 -0.086088 1% -0.555286 -1.551177 -1.661206 -2.652608 -1.755113 5% -0.314589 -1.406952 -1.223453 -2.118784 -1.474771 10% -0.013718 -1.226671 -0.676261 -1.451504 -1.124343 25% 0.274223 -1.100459 -0.388312 -0.783669 -0.828157 50% 0.799308 -0.655696 0.302684 0.021386 -0.086088 75% 1.370384 0.129219 0.739601 0.370326 0.638054 90% 2.248126 2.055993 1.228070 1.325070 1.270827 95% 2.320763 2.253613 1.590423 1.550898 1.416803 99% 2.378872 2.411708 1.880306 1.731561 1.533584 5 6 7 8 9 nobs 10.000000 10.000000 10.000000 10.000000 10.000000 missing 0.000000 0.000000 0.000000 0.000000 0.000000 mean 0.138192 0.304227 -0.085176 -0.619984 0.093436 std_err 0.456912 0.355763 0.385024 0.408175 0.308862 upper_ci 1.033724 1.001509 0.669457 0.180025 0.698794 lower_ci -0.757339 -0.393055 -0.839809 -1.419993 -0.511922 std 1.444884 1.125020 1.217553 1.290764 0.976706 iqr 1.790504 1.818180 1.144880 1.193322 1.018550 iqr_normal 1.327303 1.347819 0.848701 0.884611 0.755052 mad 1.149885 0.896867 0.884064 0.898634 0.719349 mad_normal 1.441167 1.124056 1.108009 1.126271 0.901571 coef_var 10.455595 3.697966 -14.294534 -2.081930 10.453222 range 4.833053 2.950856 4.019097 4.568185 3.121643 max 2.137646 1.643836 1.785339 2.251724 1.184207 min -2.695407 -1.307020 -2.233759 -2.316461 -1.937436 skew -0.615778 -0.225584 -0.160082 0.847988 -0.757840 kurtosis 2.539002 1.719803 2.396965 3.638611 2.878239 jarque_bera 0.720521 0.767691 0.194232 1.368399 0.963379 jarque_bera_pval 0.697494 0.681237 0.907451 0.504494 0.617739 mode -2.695407 -1.307020 -2.233759 -2.316461 -1.937436 mode_freq 0.100000 0.100000 0.100000 0.100000 0.100000 median 0.523338 0.306400 0.021761 -0.621855 0.069720 1% -2.576302 -1.304159 -2.157896 -2.296658 -1.841602 5% -2.099879 -1.292714 -1.854446 -2.217446 -1.458267 10% -1.504351 -1.278407 -1.475133 -2.118431 -0.979099 25% -0.676454 -0.503206 -0.762092 -1.312521 -0.203885 50% 0.523338 0.306400 0.021761 -0.621855 0.069720 75% 1.114050 1.314974 0.382788 -0.119200 0.814665 90% 1.316257 1.627767 1.482916 0.285203 1.182370 95% 1.726952 1.635801 1.634127 1.268464 1.183288 99% 2.055507 1.642229 1.755096 2.055072 1.184023

6. Clean up

If you want to clean up all created assets:

  • experiments

  • trainings

  • pipelines

  • model definitions

  • models

  • functions

  • deployments

please follow up this sample notebook.

7. Summary and next steps

You successfully completed this notebook! You learned how to use Watson Machine Learning for function deployment and scoring with custom software_spec.

Check out our Online Documentation for more samples, tutorials, documentation, how-tos, and blog posts.

Author

Jan Sołtysik Intern in Watson Machine Learning.

Copyright © 2020-2025 IBM. This notebook and its source code are released under the terms of the MIT License.