GitHub Repository: ibm/watson-machine-learning-samples
Path: blob/master/cpd4.5/notebooks/python_sdk/deployments/autoai/Use AutoAI to predict credit risk.ipynb
⁶⁴⁰⁵ views

Kernel: Python 3 (ipykernel)

Use AutoAI to predict credit risk with `ibm-watson-machine-learning`

This notebook demonstrates how to deploy in Watson Machine Learning service an AutoAI model created in Generated Scikit-learn Notebook which is composed during autoai experiments (in order to learn more about AutoAI experiments go to experiments/autoai).

Some familiarity with bash is helpful. This notebook uses Python 3.9.

Learning goals

The learning goals of this notebook are:

Working with the Watson Machine Learning instance
Online deployment of AutoAI model
Scoring data using deployed model

This notebook contains the following parts:

1. Set up the environment

Before you use the sample code in this notebook, you must perform the following setup tasks:

Contact with your Cloud Pack for Data administrator and ask him for your account credentials

Connection to WML

Authenticate the Watson Machine Learning service on IBM Cloud Pack for Data. You need to provide platform url, your username and api_key.

In [ ]:

username = 'PASTE YOUR USERNAME HERE'
api_key = 'PASTE YOUR API_KEY HERE'
url = 'PASTE THE PLATFORM URL HERE'

In [2]:

wml_credentials = {
    "username": username,
    "apikey": api_key,
    "url": url,
    "instance_id": 'openshift',
    "version": '4.5'
}

Alternatively you can use username and password to authenticate WML services.

wml_credentials = {
    "username": ***,
    "password": ***,
    "url": ***,
    "instance_id": 'openshift',
    "version": '4.5'
}

Install and import the `ibm-watson-machine-learning` package

Note: ibm-watson-machine-learning documentation can be found here.

In [ ]:

!pip install -U ibm-watson-machine-learning

In [2]:

from ibm_watson_machine_learning import APIClient

client = APIClient(wml_credentials)

Working with spaces

First of all, you need to create a space that will be used for your work. If you do not have space already created, you can use {PLATFORM_URL}/ml-runtime/spaces?context=icp4data to create one.

Click New Deployment Space
Create an empty space
Go to space Settings tab
Copy space_id and paste it below

Tip: You can also use SDK to prepare the space for your work. More information can be found here.

Action: Assign space ID below

In [ ]:

space_id = 'PASTE YOUR SPACE ID HERE'

You can use list method to print all existing spaces.

In [ ]:

client.spaces.list(limit=10)

To be able to interact with all resources available in Watson Machine Learning, you need to set space which you will be using.

In [3]:

client.set.default_space(space_id)

Out[3]:

'SUCCESS'

2. Upload model

In this section you will learn how to upload the model.

Download the data as an pandas DataFrame and AutoAI saved as scikit pipeline model using `wget`.

Hint: To install required packages exacute command !pip install pandas wget numpy.

We can exract model from executed AutoAI experiment using ibm-watson-machine-learning with following command: experiment.optimizer(...).get_pipeline(astype='sklearn').

In [4]:

import os, wget
import pandas as pd
import numpy as np

filename = 'german_credit_data_biased_training.csv'
url = 'https://raw.githubusercontent.com/IBM/watson-machine-learning-samples/master/cpd4.0/data/credit_risk/german_credit_data_biased_training.csv'
if not os.path.isfile(filename):
    wget.download(url)

model_name = "model.pickle"
url = 'https://raw.githubusercontent.com/IBM/watson-machine-learning-samples/master/cpd4.0/models/autoai/credit-risk/model.pickle'
if not os.path.isfile(model_name):
    wget.download(url)

credit_risk_df = pd.read_csv(filename)
X = credit_risk_df.drop(['Risk'], axis=1)
y = credit_risk_df['Risk']

credit_risk_df.head()

Out[4]:

Custom software_specification

Create new software specification based on default Python 3.9 environment extended by autoai-libs package.

In [5]:

base_sw_spec_uid = client.software_specifications.get_uid_by_name("runtime-22.1-py3.9")

In [6]:

url = 'https://raw.githubusercontent.com/IBM/watson-machine-learning-samples/master/cpd4.0/configs/config.yaml'
if not os.path.isfile('config.yaml'):
    wget.download(url)

In [ ]:

!cat config.yaml

config.yaml file describes details of package extention. Now you need to store new package extention with APIClient.

In [14]:

meta_prop_pkg_extn = {
    client.package_extensions.ConfigurationMetaNames.NAME: "scikt with autoai-libs",
    client.package_extensions.ConfigurationMetaNames.DESCRIPTION: "Extension for autoai-libs",
    client.package_extensions.ConfigurationMetaNames.TYPE: "conda_yml"
}

pkg_extn_details = client.package_extensions.store(meta_props=meta_prop_pkg_extn, file_path="config.yaml")
pkg_extn_uid = client.package_extensions.get_uid(pkg_extn_details)
pkg_extn_url = client.package_extensions.get_href(pkg_extn_details)

Out[14]:

Creating package extensions
SUCCESS

Create new software specification and add created package extention to it.

In [15]:

meta_prop_sw_spec = {
    client.software_specifications.ConfigurationMetaNames.NAME: "Mitigated AutoAI bases on scikit spec",
    client.software_specifications.ConfigurationMetaNames.DESCRIPTION: "Software specification for scikt with autoai-libs",
    client.software_specifications.ConfigurationMetaNames.BASE_SOFTWARE_SPECIFICATION: {"guid": base_sw_spec_uid}
}

sw_spec_details = client.software_specifications.store(meta_props=meta_prop_sw_spec)
sw_spec_uid = client.software_specifications.get_uid(sw_spec_details)

client.software_specifications.add_package_extension(sw_spec_uid, pkg_extn_uid)

Out[15]:

SUCCESS

'SUCCESS'

Get the details of created software specification

In [ ]:

client.software_specifications.get_details(sw_spec_uid)

Load the AutoAI model saved as `scikit-learn` pipeline.

Depending on estimator type in autoai model pipeline may consist models from following frameworks:

xgboost
lightgbm
scikit-learn

In [33]:

from joblib import load

pipeline = load(model_name)

Store the model

In [34]:

model_props = {
    client.repository.ModelMetaNames.NAME: "AutoAI model",
    client.repository.ModelMetaNames.TYPE: 'scikit-learn_1.0',
    client.repository.ModelMetaNames.SOFTWARE_SPEC_UID: sw_spec_uid
    
}
feature_vector = X.columns

In [35]:

published_model = client.repository.store_model(
    model=pipeline, 
    meta_props=model_props,
    training_data=X.values,
    training_target=y.values,
    feature_names=feature_vector,
    label_column_names=['Risk']
)

In [36]:

published_model_uid = client.repository.get_model_id(published_model)

Get model details

In [37]:

client.repository.get_details(published_model_uid)

Out[37]:

{'entity': {'label_column': 'Risk',
  'software_spec': {'id': '0267d74c-3239-4f94-bf72-e4f103b5ae8f',
   'name': 'Mitigated AutoAI bases on scikit spec'},
  'training_data_references': [{'connection': {'access_key_id': 'not_applicable',
     'endpoint_url': 'not_applicable',
     'secret_access_key': 'not_applicable'},
    'id': '1',
    'location': {},
    'schema': {'fields': [{'name': 'CheckingStatus', 'type': 'str'},
      {'name': 'LoanDuration', 'type': 'int'},
      {'name': 'CreditHistory', 'type': 'str'},
      {'name': 'LoanPurpose', 'type': 'str'},
      {'name': 'LoanAmount', 'type': 'int'},
      {'name': 'ExistingSavings', 'type': 'str'},
      {'name': 'EmploymentDuration', 'type': 'str'},
      {'name': 'InstallmentPercent', 'type': 'int'},
      {'name': 'Sex', 'type': 'str'},
      {'name': 'OthersOnLoan', 'type': 'str'},
      {'name': 'CurrentResidenceDuration', 'type': 'int'},
      {'name': 'OwnsProperty', 'type': 'str'},
      {'name': 'Age', 'type': 'int'},
      {'name': 'InstallmentPlans', 'type': 'str'},
      {'name': 'Housing', 'type': 'str'},
      {'name': 'ExistingCreditsCount', 'type': 'int'},
      {'name': 'Job', 'type': 'str'},
      {'name': 'Dependents', 'type': 'int'},
      {'name': 'Telephone', 'type': 'str'},
      {'name': 'ForeignWorker', 'type': 'str'}],
     'id': '1',
     'type': 'ndarray'},
    'type': 'fs'}],
  'type': 'scikit-learn_0.23'},
 'metadata': {'created_at': '2021-07-19T11:06:11.675Z',
  'id': '0a57c7c1-5e3b-4262-893a-e555b07c2f7e',
  'modified_at': '2021-07-19T11:06:13.198Z',
  'name': 'AutoAI model',
  'owner': '1000330999',
  'space_id': '97cebfee-a9da-4c04-822d-f36540c1070d'},
 'system': {'warnings': []}}

Note: You can see that model is successfully stored in Watson Machine Learning Service.

In [ ]:

client.repository.list_models()

3. Create online deployment

You can use commands bellow to create online deployment for stored model (web service).

In [38]:

metadata = {
    client.deployments.ConfigurationMetaNames.NAME: "Deployment of AutoAI model.",
    client.deployments.ConfigurationMetaNames.ONLINE: {}
}

created_deployment = client.deployments.create(published_model_uid, meta_props=metadata)

Out[38]:

#######################################################################################

Synchronous deployment creation for uid: '0a57c7c1-5e3b-4262-893a-e555b07c2f7e' started

#######################################################################################


initializing
Note: online_url is deprecated and will be removed in a future release. Use serving_urls instead.
.........
ready


------------------------------------------------------------------------------------------------
Successfully finished deployment creation, deployment_uid='b89ad484-b557-40f3-bb1c-11e12cd68efa'
------------------------------------------------------------------------------------------------

Get deployment id.

In [39]:

deployment_id = client.deployments.get_uid(created_deployment)
print(deployment_id)

Out[39]:

b89ad484-b557-40f3-bb1c-11e12cd68efa

4. Scoring

You can send new scoring records to web-service deployment using score method.

In [40]:

values = X.values

scoring_payload = {
    "input_data": [{
        'values': values[:5]
    }]
}

In [41]:

predictions = client.deployments.score(deployment_id, scoring_payload)
predictions

Out[41]:

{'predictions': [{'fields': ['prediction', 'probability'],
   'values': [['No Risk', [0.7177145481109619, 0.2822854220867157]],
    ['No Risk', [0.7641035318374634, 0.23589648306369781]],
    ['No Risk', [0.720166802406311, 0.27983322739601135]],
    ['No Risk', [0.7394402623176575, 0.26055973768234253]],
    ['No Risk', [0.5265587568283081, 0.4734412431716919]]]}]}

5. Clean up

If you want to clean up all created assets:

experiments
trainings
pipelines
model definitions
models
functions
deployments

see the steps in this sample notebook.

6. Summary and next steps

You successfully completed this notebook! You learned how to use Watson Machine Learning for AutoA model deployment and scoring. Check out our Online Documentation for more samples, tutorials, documentation, how-tos, and blog posts.

Author

Jan Sołtysik Intern in Watson Machine Learning.

Use AutoAI to predict credit risk with `ibm-watson-machine-learning`

Learning goals

Contents

1. Set up the environment

Connection to WML

Install and import the `ibm-watson-machine-learning` package

Working with spaces

2. Upload model

Download the data as an pandas DataFrame and AutoAI saved as scikit pipeline model using `wget`.

Custom software_specification

Create new software specification and add created package extention to it.

Get the details of created software specification

Load the AutoAI model saved as `scikit-learn` pipeline.

Store the model

Get model details

3. Create online deployment

4. Scoring

5. Clean up

6. Summary and next steps

Author

Product

Resources

Company

Use AutoAI to predict credit risk with ibm-watson-machine-learning

Learning goals

Contents

1. Set up the environment

Connection to WML

Install and import the ibm-watson-machine-learning package

Working with spaces

2. Upload model

Download the data as an pandas DataFrame and AutoAI saved as scikit pipeline model using wget.

Custom software_specification

Create new software specification and add created package extention to it.

Get the details of created software specification

Load the AutoAI model saved as scikit-learn pipeline.

Store the model

Get model details

3. Create online deployment

4. Scoring

5. Clean up

6. Summary and next steps

Author

Use AutoAI to predict credit risk with `ibm-watson-machine-learning`

Install and import the `ibm-watson-machine-learning` package

Download the data as an pandas DataFrame and AutoAI saved as scikit pipeline model using `wget`.

Load the AutoAI model saved as `scikit-learn` pipeline.