Use custom software spec to create statsmodels function.ipynb

Kernel: Python 3 (ipykernel)

Use custom software_spec to create statsmodels function describing data with ibm-watson-machine-learning

This notebook demonstrates how to deploy in Watson Machine Learning service a python function with statsmodel which requires to create custom software specification using conda yaml file with all required libraries. Some familiarity with bash is helpful. This notebook uses Python 3.10 with statsmodel.

Learning goals

The learning goals of this notebook are:

• Working with the Watson Machine Learning instance

• Creating custom software specification

• Online deployment of python function

• Scoring data using deployed function

Contents

This notebook contains the following parts:

1. Setup

2. Function creation

3. Function upload

4. Web service creation

5. Scoring

6. Clean up

7. Summary and next steps

1. Set up the environment

Before you use the sample code in this notebook, you must perform the following setup tasks:

• Contact with your Cloud Pack for Data administrator and ask him for your account credentials

Connection to WML

Authenticate the Watson Machine Learning service on IBM Cloud Pack for Data. You need to provide platform url, your username and api_key.

username = 'PASTE YOUR USERNAME HERE'
api_key = 'PASTE YOUR API_KEY HERE'
url = 'PASTE THE PLATFORM URL HERE'

wml_credentials = {
    "username": username,
    "apikey": api_key,
    "url": url,
    "instance_id": 'openshift',
    "version": '4.8'
}

Alternatively you can use username and password to authenticate WML services.

wml_credentials = {
    "username": ***,
    "password": ***,
    "url": ***,
    "instance_id": 'openshift',
    "version": '4.8'
}

Install and import the ibm-watson-machine-learning package

Note: ibm-watson-machine-learning documentation can be found here.

!pip install -U ibm-watson-machine-learning

from ibm_watson_machine_learning import APIClient

client = APIClient(wml_credentials)

Working with spaces

First of all, you need to create a space that will be used for your work. If you do not have space already created, you can use {PLATFORM_URL}/ml-runtime/spaces?context=icp4data to create one.

• Click New Deployment Space

• Create an empty space

• Go to space Settings tab

• Copy space_id and paste it below

Tip: You can also use SDK to prepare the space for your work. More information can be found here.

Action: Assign space ID below

space_id = 'PASTE YOUR SPACE ID HERE'

You can use list method to print all existing spaces.

client.spaces.list(limit=10)

To be able to interact with all resources available in Watson Machine Learning, you need to set space which you will be using.

client.set.default_space(space_id)

'SUCCESS'

2. Create function

In this section you will learn how to create deployable function with statsmodels module calculating describition of a given data. Hint: To install statsmodels execute !pip install statsmodels.

Create deploayable callable which uses stsmodels library

def deployable_callable():
    """
    Deployable python function with score
    function implemented.
    """
    try:
        from statsmodels.stats.descriptivestats import describe
    except ModuleNotFoundError as e:
        print(f"statsmodels not installed: {str(e)}")
        
    def score(payload):
        """
        Score method.
        """
        try:
            data = payload['input_data'][0]['values']
            return {
                'predictions': [
                    {'values': str(describe(data))}
                ]
            }
        except Exception as e:
            return {'predictions': [{'values': [repr(e)]}]}
        
    return score

Test callable locally

Hint: To install numpy execute !pip install numpy.

import numpy as np

data = np.random.randn(10, 10)
data_description = deployable_callable()({
    "input_data": [{
        "values" : data
    }]
})

print(data_description["predictions"][0]["values"])

                          0          1          2          3          4  \
nobs              10.000000  10.000000  10.000000  10.000000  10.000000   
missing            0.000000   0.000000   0.000000   0.000000   0.000000   
mean              -0.087662  -0.162800   0.021847  -0.484764  -0.151347   
std_err            0.086628   0.082336   0.088078   0.154758   0.060011   
upper_ci           0.082126  -0.001424   0.194477  -0.181444  -0.033728   
lower_ci          -0.257450  -0.324175  -0.150783  -0.788085  -0.268967   
std                0.866281   0.823361   0.880783   1.547581   0.600109   
iqr                1.220332   0.774040   1.173032   1.866595   0.455900   
iqr_normal         0.904633   0.573797   0.869570   1.383709   0.337959   
mad                0.696300   0.556041   0.700227   1.257709   0.403634   
mad_normal         0.872682   0.696895   0.877605   1.576305   0.505881   
coef_var          -9.882045  -5.057510  40.315579  -3.192440  -3.965111   
range              2.514768   2.889136   2.903603   4.598605   2.228215   
max                1.458946   0.788044   1.426277   1.386525   0.671509   
min               -1.055823  -2.101093  -1.477326  -3.212081  -1.556706   
skew               0.621923  -1.182308  -0.110031  -0.582020  -1.165771   
kurtosis           2.083665   4.191979   2.178047   2.099710   4.236195   
jarque_bera        0.994509   2.921762   0.301681   0.902296   2.901779   
jarque_bera_pval   0.608198   0.232032   0.859985   0.636897   0.234362   
mode              -1.055823  -2.101093  -1.477326  -3.212081  -1.556706   
mode_freq          0.100000   0.100000   0.100000   0.100000   0.100000   
median            -0.259626  -0.203613   0.127268  -0.060631  -0.028983   

                          5          6          7          8          9  
nobs              10.000000  10.000000  10.000000  10.000000  10.000000  
missing            0.000000   0.000000   0.000000   0.000000   0.000000  
mean               0.273284   0.205051   0.023256   0.102842  -0.209529  
std_err            0.061042   0.087203   0.126508   0.103825   0.072225  
upper_ci           0.392923   0.375966   0.271207   0.306335  -0.067971  
lower_ci           0.153644   0.034137  -0.224696  -0.100650  -0.351088  
std                0.610416   0.872028   1.265081   1.038247   0.722251  
iqr                0.780093   1.144017   1.897466   1.087920   1.181144  
iqr_normal         0.578284   0.848061   1.406594   0.806476   0.875583  
mad                0.476876   0.702078   0.962314   0.802775   0.623214  
mad_normal         0.597675   0.879924   1.206082   1.006129   0.781083  
coef_var           2.233637   4.252729  54.398852  10.095509  -3.447017  
range              2.012750   2.399845   3.639744   3.386568   1.965249  
max                1.364230   1.624449   1.719675   1.524333   0.677055  
min               -0.648519  -0.775396  -1.920069  -1.862235  -1.288194  
skew               0.305785   0.606322  -0.150434  -0.368515  -0.382710  
kurtosis           2.245390   1.963650   1.837083   2.477825   1.604283  
jarque_bera        0.393106   1.060220   0.601208   0.339950   1.055789  
jarque_bera_pval   0.821558   0.588540   0.740371   0.843686   0.589846  
mode              -0.648519  -0.775396  -1.920069  -1.862235  -1.288194  
mode_freq          0.100000   0.100000   0.100000   0.100000   0.100000  
median             0.228023  -0.050927   0.073169   0.306686   0.022939  

3. Upload python function

In this section you will learn how to upload the python function to the Cloud.

Custom software_specification

Create new software specification based on default Python 3.10 environment extended by autoai-libs package.

config_yml =\
"""name: python38
channels:
  - empty
  - nodefaults
dependencies:
- pip:
  - statsmodels

prefix: /opt/anaconda3/envs/python38
"""

with open("config.yaml", "w", encoding="utf-8") as f:
    f.write(config_yml)

base_sw_spec_uid = client.software_specifications.get_uid_by_name("runtime-23.1-py3.10")

!cat config.yaml

name: python37
channels:
  - defaults
dependencies:
  - pip:
    - statsmodels

prefix: /opt/anaconda3/envs/python37

config.yaml file describes details of package extention. Now you need to store new package extention with APIClient.

meta_prop_pkg_extn = {
    client.package_extensions.ConfigurationMetaNames.NAME: "statsmodels env",
    client.package_extensions.ConfigurationMetaNames.DESCRIPTION: "Environment with statsmodels",
    client.package_extensions.ConfigurationMetaNames.TYPE: "conda_yml"
}

pkg_extn_details = client.package_extensions.store(meta_props=meta_prop_pkg_extn, file_path="config.yaml")
pkg_extn_uid = client.package_extensions.get_uid(pkg_extn_details)
pkg_extn_url = client.package_extensions.get_href(pkg_extn_details)

Creating package extensions
SUCCESS

Create new software specification and add created package extention to it.

meta_prop_sw_spec = {
    client.software_specifications.ConfigurationMetaNames.NAME: "statsmodels software_spec",
    client.software_specifications.ConfigurationMetaNames.DESCRIPTION: "Software specification for statsmodels",
    client.software_specifications.ConfigurationMetaNames.BASE_SOFTWARE_SPECIFICATION: {"guid": base_sw_spec_uid}
}

sw_spec_details = client.software_specifications.store(meta_props=meta_prop_sw_spec)
sw_spec_uid = client.software_specifications.get_uid(sw_spec_details)

client.software_specifications.add_package_extension(sw_spec_uid, pkg_extn_uid)

SUCCESS

'SUCCESS'

Get the details of created software specification

client.software_specifications.get_details(sw_spec_uid)

Store the function

meta_props = {
    client.repository.FunctionMetaNames.NAME: "statsmodels function",
    client.repository.FunctionMetaNames.SOFTWARE_SPEC_UID: sw_spec_uid
}

function_details = client.repository.store_function(meta_props=meta_props, function=deployable_callable)
function_uid = client.repository.get_function_uid(function_details)

Get function details

client.repository.get_details(function_uid)

{'entity': {'software_spec': {'id': '62123911-898c-49be-87ae-e056067645d8',
   'name': 'statsmodels software_spec'},
  'type': 'python'},
 'metadata': {'created_at': '2021-02-03T13:02:50.513Z',
  'id': '77cc037c-6aab-4eac-9e45-c3331e010132',
  'modified_at': '2021-02-03T13:02:53.304Z',
  'name': 'statsmodels function',
  'owner': 'IBMid-55000091VC',
  'space_id': 'd70a423e-bab5-4b24-943a-3b0b29ad7527'},
 'system': {'warnings': []}}

Note: You can see that function is successfully stored in Watson Machine Learning Service.

client.repository.list_functions()

4. Create online deployment

You can use commands bellow to create online deployment for stored function (web service).

Create online deployment of a python function

metadata = {
    client.deployments.ConfigurationMetaNames.NAME: "Deployment of statsmodels function",
    client.deployments.ConfigurationMetaNames.ONLINE: {}
}

function_deployment = client.deployments.create(function_uid, meta_props=metadata)

#######################################################################################

Synchronous deployment creation for uid: '77cc037c-6aab-4eac-9e45-c3331e010132' started

#######################################################################################


initializing....................................................................................................................................................
ready


------------------------------------------------------------------------------------------------
Successfully finished deployment creation, deployment_uid='340eb7f7-4a08-4c9c-8371-f6cd907b7a3e'
------------------------------------------------------------------------------------------------

client.deployments.list()

Get deployment id.

deployment_id = client.deployments.get_uid(function_deployment)
print(deployment_id)

340eb7f7-4a08-4c9c-8371-f6cd907b7a3e

5. Scoring

You can send new scoring records to web-service deployment using score method.

scoring_payload = {
    "input_data": [{
        'values': data
    }]
}

predictions = client.deployments.score(deployment_id, scoring_payload)
print(data_description["predictions"][0]["values"])

                          0          1          2          3          4  \
nobs              10.000000  10.000000  10.000000  10.000000  10.000000   
missing            0.000000   0.000000   0.000000   0.000000   0.000000   
mean              -0.087662  -0.162800   0.021847  -0.484764  -0.151347   
std_err            0.086628   0.082336   0.088078   0.154758   0.060011   
upper_ci           0.082126  -0.001424   0.194477  -0.181444  -0.033728   
lower_ci          -0.257450  -0.324175  -0.150783  -0.788085  -0.268967   
std                0.866281   0.823361   0.880783   1.547581   0.600109   
iqr                1.220332   0.774040   1.173032   1.866595   0.455900   
iqr_normal         0.904633   0.573797   0.869570   1.383709   0.337959   
mad                0.696300   0.556041   0.700227   1.257709   0.403634   
mad_normal         0.872682   0.696895   0.877605   1.576305   0.505881   
coef_var          -9.882045  -5.057510  40.315579  -3.192440  -3.965111   
range              2.514768   2.889136   2.903603   4.598605   2.228215   
max                1.458946   0.788044   1.426277   1.386525   0.671509   
min               -1.055823  -2.101093  -1.477326  -3.212081  -1.556706   
skew               0.621923  -1.182308  -0.110031  -0.582020  -1.165771   
kurtosis           2.083665   4.191979   2.178047   2.099710   4.236195   
jarque_bera        0.994509   2.921762   0.301681   0.902296   2.901779   
jarque_bera_pval   0.608198   0.232032   0.859985   0.636897   0.234362   
mode              -1.055823  -2.101093  -1.477326  -3.212081  -1.556706   
mode_freq          0.100000   0.100000   0.100000   0.100000   0.100000   
median            -0.259626  -0.203613   0.127268  -0.060631  -0.028983   

                          5          6          7          8          9  
nobs              10.000000  10.000000  10.000000  10.000000  10.000000  
missing            0.000000   0.000000   0.000000   0.000000   0.000000  
mean               0.273284   0.205051   0.023256   0.102842  -0.209529  
std_err            0.061042   0.087203   0.126508   0.103825   0.072225  
upper_ci           0.392923   0.375966   0.271207   0.306335  -0.067971  
lower_ci           0.153644   0.034137  -0.224696  -0.100650  -0.351088  
std                0.610416   0.872028   1.265081   1.038247   0.722251  
iqr                0.780093   1.144017   1.897466   1.087920   1.181144  
iqr_normal         0.578284   0.848061   1.406594   0.806476   0.875583  
mad                0.476876   0.702078   0.962314   0.802775   0.623214  
mad_normal         0.597675   0.879924   1.206082   1.006129   0.781083  
coef_var           2.233637   4.252729  54.398852  10.095509  -3.447017  
range              2.012750   2.399845   3.639744   3.386568   1.965249  
max                1.364230   1.624449   1.719675   1.524333   0.677055  
min               -0.648519  -0.775396  -1.920069  -1.862235  -1.288194  
skew               0.305785   0.606322  -0.150434  -0.368515  -0.382710  
kurtosis           2.245390   1.963650   1.837083   2.477825   1.604283  
jarque_bera        0.393106   1.060220   0.601208   0.339950   1.055789  
jarque_bera_pval   0.821558   0.588540   0.740371   0.843686   0.589846  
mode              -0.648519  -0.775396  -1.920069  -1.862235  -1.288194  
mode_freq          0.100000   0.100000   0.100000   0.100000   0.100000  
median             0.228023  -0.050927   0.073169   0.306686   0.022939  

6. Clean up

If you want to clean up all created assets:

• experiments

• trainings

• pipelines

• model definitions

• models

• functions

• deployments

please follow up this sample notebook.

7. Summary and next steps

You successfully completed this notebook! You learned how to use Watson Machine Learning for function deployment and scoring with custom software_spec. Check out our Online Documentation for more samples, tutorials, documentation, how-tos, and blog posts.

Author

Jan Sołtysik Intern in Watson Machine Learning.

Copyright © 2020-2025 IBM. This notebook and its source code are released under the terms of the MIT License.