Machine Learning artifacts management.ipynb

Kernel: Python 3

Machine Learning artifacts management

This notebook contains steps and code to demonstrate how to manage and clean up Watson Machine Learning instance. This notebook contains steps and code to work with Watson Machine Learning API. This notebook introduces API calls for listing artifacts, getting artifacts details and deleting them.

Some familiarity with Python and REST API is helpful. This notebook uses Python 3.

Learning goals

The learning goals of this notebook are:

• List Watson Machine Learning artifacts.

• Get artifacts details.

• Delete artifacts.

Contents

This notebook contains the following parts:

1. Setup

2. Manage pipelines

3. Manage model definitions

4. Manage models

5. Manage functions

6. Manage experiments

7. Manage trainings

8. Manage deployments

9. Summary and next steps

1. Set up the environment

Before you use the sample code in this notebook, you must fill following cell variables.

You can find your COS credentials in COS instance dashboard under the Service credentials tab. Go to the Endpoint tab in the COS instance's dashboard to get the endpoint information.

Your Cloud API key can be generated by going to the Users section of the Cloud console. From that page, click your name, scroll down to the API Keys section, and click Create an IBM Cloud API key. Give your key a name and click Create, then copy the created key and paste it below.

NOTE: You can also get service specific apikey by going to the Service IDs section of the Cloud Console. From that page, click Create, then copy the created key and paste it below.

API_KEY=""
WML_ENDPOINT_URL=""
WML_INSTANCE_CRN="fill out only if you want to create new space"
WML_INSTANCE_NAME="fill out only if you want to create new space"

COS_CRN="fill out only if you want to create new space"

space_id="fill out only if you have already created a space"

DATAPLATFORM_URL="https://api.dataplatform.cloud.ibm.com"   
AUTH_ENDPOINT="https://iam.cloud.ibm.com/oidc/token"

In order to work with REST API in python you need to import requests package.

import requests

Generate WML authorization token for further REST API calls

Request params preparation.

token_creation_params = {
    "grant_type": "urn:ibm:params:oauth:grant-type:apikey",
    "apikey": API_KEY
}

Token generation.

response = requests.post(AUTH_ENDPOINT, params=token_creation_params)
token = response.json()['access_token']
print(token)

Define requests header for futher REST API calls.

header = {
        'Authorization': f'Bearer {token}',
        'Content-Type': 'application/json',
        'Accept': 'application/json'
}

Space creation

Tip: If you do not have space already created, please convert below three Raw NBConvert cells to code and run them.

Prepare payload json for space creation.

space_json={
  "name": "my_space2",
  "description": "my_description2",
  "storage": {
    "type": "bmcos_object_storage",
    "resource_crn": COS_CRN,
  },
  "compute": [
    {
      "name": WML_INSTANCE_NAME,
      "crn": WML_INSTANCE_CRN
    }
  ]
}

Create new space.

space_details = requests.post(
    url = f"{DATAPLATFORM_URL}/v2/spaces",
    headers = header,
    json = space_json
)

Get created space id.

space_id = space_details.json()['metadata']['id']
print(space_id)

Space creation is asynchronous. This means that you need to check space creation status after creation call. Make sure that your newly created space is active.

Get space details.

space_details = requests.get(
    url = f"{DATAPLATFORM_URL}/v2/spaces/{space_id}?version=2020-08-01",
    headers = header
)

space_details.json()

Managing spaces.

If you want to get all spaces details you can run following REST API call. If you want to get part of spaces please change the limit variable.

limit=2
spaces_details = requests.get(
    url = f"{DATAPLATFORM_URL}/v2/spaces?limit={limit}",
    headers = header
)

Print spaces details json.

spaces_details.json()

If you want to list existing spaces names and ids run next cell.

for space in spaces_details.json()['resources']:
    print(f"{space['entity']['name']} \t Id: {space['metadata']['id']}")

If you want to delete one of existing spaces you can change next cell Format to code and use following REST API call.

space_delete = requests.delete(
    url = f"{DATAPLATFORM_URL}/v2/spaces/{space_id}?version=2020-08-01",
    headers = header
)

2. Manage pipelines

If you want to get all pipelines details you can run following REST API call. If you want to get part of pipelines please change the limit variable.

limit=2
pipelines_details = requests.get(
    url = f"{WML_ENDPOINT_URL}/ml/v4/pipelines?version=2020-08-01&space_id={space_id}&limit={limit}",
    headers = header
)

If you want to list existing pipelines names and ids run next cell.

for pipeline in pipelines_details.json()['resources']:
    print(f"{pipeline['metadata']['name']} \t Id: {pipeline['metadata']['id']}")

Get pipeline 0 id.

pipeline_id = pipelines_details.json()['resources'][0]['metadata']['id']
print(pipeline_id)

If you want to get pipeline details you must provide pipeline_id and run following REST API call.

pipeline_details = requests.get(
    url = f"{WML_ENDPOINT_URL}/ml/v4/pipelines/{pipeline_id}?version=2020-08-01&space_id={space_id}",
    headers = header
)

Print pipeline details.

pipelines_details.json()

You can delete pipeline by next cell API CALL.

pipeline_delete = requests.delete(
    url = f"{WML_ENDPOINT_URL}/ml/v4/pipelines/{pipeline_id}?version=2020-08-01&space_id={space_id}",
    headers = header
)
if pipeline_delete.status_code in [200,202,204]:
    print('SUCCES')
else:
    print(f'Failed to delete pipeline: \n {pipeline_delete.json()}')

If you want to delete more pipelines run following cell.

for pipeline in pipelines_details.json()['resources']:
    pipeline_delete = requests.delete(
    url = f"{WML_ENDPOINT_URL}/ml/v4/pipelines/{pipeline['metadata']['id']}?version=2020-08-01&space_id={space_id}",
    headers = header
    )
    if pipeline_delete.status_code in [200,202,204]:
        print('SUCCES')
    else:
        print(f'Failed to delete pipeline: \n {pipeline_delete.json()}')

3. Manage model definitions

If you want to get all model definitions details you can run following REST API call. If you want to get part of model definitions please change the limit variable.

limit=2
model_definitions_details = requests.get(
    url = f"{WML_ENDPOINT_URL}/ml/v4/model_definitions?version=2020-08-01&space_id={space_id}&limit={limit}",
    headers = header
)

If you want to list existing model definitions names and ids run next cell.

for model_definition in model_definitions_details.json()['resources']:
    print(f"{model_definition['metadata']['name']} \t Id: {model_definition['metadata']['id']}")

Get model definition 0 id.

model_definition_id = model_definitions_details.json()['resources'][0]['metadata']['id']
print(model_definition_id)

If you want to get model definition details you must provide model_definition_id and run following REST API call.

model_definition_details = requests.get(
    url = f"{WML_ENDPOINT_URL}/ml/v4/model_definitions/{model_definition_id}?version=2020-08-01&space_id={space_id}",
    headers = header
)

Print model definition details.

model_definition_details.json()

You can delete model definition by next cell API CALL.

model_definition_delete = requests.delete(
    url = f"{WML_ENDPOINT_URL}/ml/v4/model_definitions/{model_definition_id}?version=2020-08-01&space_id={space_id}",
    headers = header
)
if model_definition_delete.status_code in [200,202,204]:
    print('SUCCES')
else:
    print(f'Failed to delete model definition: \n {model_definition_delete.json()}')

If you want to delete more model definitions run following cell.

for model_definition in model_definitions_details.json()['resources']:
    model_definition_delete = requests.delete(
    url = f"{WML_ENDPOINT_URL}/ml/v4/model_definitions/{model_definition['metadata']['id']}?version=2020-08-01&space_id={space_id}",
    headers = header
    )
    if model_definition_delete.status_code in [200,202,204]:
        print('SUCCES')
    else:
        print(f'Failed to delete model definition: \n {model_definition_delete.json()}')

4. Manage models

If you want to get all models details you can run following REST API call. If you want to get part of models please change the limit variable.

limit=2
models_details = requests.get(
    url = f"{WML_ENDPOINT_URL}/ml/v4/models?version=2020-08-01&space_id={space_id}&limit={limit}",
    headers = header
)

In order to filter models by software specification you can paste software specification id to the software_spec field.

limit=2
software_spec="63dc4cf1-252f-424b-b52d-5cdd9814987f"
models_details = requests.get(
    url = f"{WML_ENDPOINT_URL}/ml/v4/models?version=2020-08-01&space_id={space_id}&limit={limit}&software_spec={software_spec}",
    headers = header
)

If you want to list existing models names and ids run next cell.

for model in models_details.json()['resources']:
    print(f"{model['metadata']['name']} \t Id: {model['metadata']['id']}")

Get model 0 id.

model_id = models_details.json()['resources'][0]['metadata']['id']
print(model_id)

If you want to get model details you must provide model_id and run following REST API call.

model_details = requests.get(
    url = f"{WML_ENDPOINT_URL}/ml/v4/models/{model_id}?version=2020-08-01&space_id={space_id}",
    headers = header
)

Print model details.

model_details.json()

If you want to get model revision you must provide model_id and run following REST API call.

model_revisons = requests.get(
    url = f"{WML_ENDPOINT_URL}/ml/v4/models/{model_id}/revisions?version=2020-08-01&space_id={space_id}",
    headers = header
)

Print model revision.

model_revisons.json()

You can delete model by next cell API CALL.

model_delete = requests.delete(
    url = f"{WML_ENDPOINT_URL}/ml/v4/models/{model_id}?version=2020-08-01&space_id={space_id}",
    headers = header
)
if model_delete.status_code in [200,202,204]:
    print('SUCCES')
else:
    print(f'Failed to delete model: \n {model_delete.json()}')

If you want to delete more models run following cell.

for model in models_details.json()['resources']:
    model_delete = requests.delete(
    url = f"{WML_ENDPOINT_URL}/ml/v4/models/{model['metadata']['id']}?version=2020-08-01&space_id={space_id}",
    headers = header
    )
    if model_delete.status_code in [200,202,204]:
        print('SUCCES')
    else:
        print(f'Failed to delete model: \n {model_delete.json()}')

5. Manage functions

If you want to get all functions details you can run following REST API call. If you want to get part of functions please change the limit variable.

limit=2
functions_details = requests.get(
    url = f"{WML_ENDPOINT_URL}/ml/v4/functions?version=2020-08-01&space_id={space_id}&limit={limit}",
    headers = header
)

If you want to list existing functions names and ids run next cell.

for function in functions_details.json()['resources']:
    print(f"{function['metadata']['name']} \t Id: {function['metadata']['id']}")

Get function 0 id.

function_id = functions_details.json()['resources'][0]['metadata']['id']
print(function_id)

If you want to get function details you must provide function_id and run following REST API call.

function_details = requests.get(
    url = f"{WML_ENDPOINT_URL}/ml/v4/functions/{function_id}?version=2020-08-01&space_id={space_id}",
    headers = header
)

Print function details.

function_details.json()

You can delete function by next cell API CALL.

function_delete = requests.delete(
    url = f"{WML_ENDPOINT_URL}/ml/v4/functions/{function_id}?version=2020-08-01&space_id={space_id}",
    headers = header
)
if function_delete.status_code in [200,202,204]:
    print('SUCCES')
else:
    print(f'Failed to delete function: \n {function_delete.json()}')

If you want to delete more functions run following cell.

for function in functions_details.json()['resources']:
    function_delete = requests.delete(
    url = f"{WML_ENDPOINT_URL}/ml/v4/functions/{function['metadata']['id']}?version=2020-08-01&space_id={space_id}",
    headers = header
    )
    if function_delete.status_code in [200,202,204]:
        print('SUCCES')
    else:
        print(f'Failed to delete function: \n {function_delete.json()}')

6. Manage experiments

If you want to get all experiments details you can run following REST API call. If you want to get part of experiments please change the limit variable.

limit=2
experiments_details = requests.get(
    url = f"{WML_ENDPOINT_URL}/ml/v4/experiments?version=2020-08-01&space_id={space_id}",
    headers = header
)

If you want to list existing experiments names and ids run next cell.

for experiment in experiments_details.json()['resources']:
    print(f"{experiment['metadata']['name']} \t Id: {experiment['metadata']['id']}")

Get experiment 0 id.

experiment_id = experiments_details.json()['resources'][0]['metadata']['id']
print(experiment_id)

If you want to get experiment details you must provide experiment_id and run following REST API call.

experiment_details = requests.get(
    url = f"{WML_ENDPOINT_URL}/ml/v4/experiments/{experiment_id}?version=2020-08-01&space_id={space_id}",
    headers = header
)

Print experiment details.

experiment_details.json()

You can delete experiment by next cell API CALL.

experiment_delete = requests.delete(
    url = f"{WML_ENDPOINT_URL}/ml/v4/experiments/{experiment_id}?version=2020-08-01&space_id={space_id}",
    headers = header
)
if experiment_delete.status_code in [200,202,204]:
    print('SUCCES')
else:
    print(f'Failed to delete experiment: \n {experiment_delete.json()}')

If you want to delete more experiments change run following cell.

for experiment in experiments_details.json()['resources']:
    experiment_delete = requests.delete(
    url = f"{WML_ENDPOINT_URL}/ml/v4/experiments/{experiment['metadata']['id']}?version=2020-08-01&space_id={space_id}",
    headers = header
    )
    if experiment_delete.status_code in [200,202,204]:
        print('SUCCES')
    else:
        print(f'Failed to delete experiment: \n {experiment_delete.json()}')

7. Manage trainings

If you want to get all trainings details you can run following REST API call. If you want to get part of trainings please change the limit variable.

limit=2
trainings_details = requests.get(
    url = f"{WML_ENDPOINT_URL}/ml/v4/trainings?version=2020-08-01&space_id={space_id}&limit={limit}",
    headers = header
)

In order to filter trainings by the training type (e.g. pipeline, experiment) please change training_type variable and run next cell.

limit=2
training_type='pipeline'
trainings_details = requests.get(
    url = f"{WML_ENDPOINT_URL}/ml/v4/trainings?version=2020-08-01&space_id={space_id}&limit={limit}&type={training_type}",
    headers = header
)

If you want to list existing trainings tags and ids run next cell.

for training in trainings_details.json()['resources']:
    print(f"{training['metadata']['tags']} \t Id: {training['metadata']['id']}")

Get training 0 id.

training_id = trainings_details.json()['resources'][0]['metadata']['id']
print(training_id)

If you want to get training details you must provide training_id and run following REST API call.

training_details = requests.get(
    url = f"{WML_ENDPOINT_URL}/ml/v4/trainings/{training_id}?version=2020-08-01&space_id={space_id}",
    headers = header
)

Print training details.

training_details.json()

You can delete training by next cell API CALL.

Note: DELETE CALL has parameter hard_delete, please change it as fallows:

• 'true' - to delete the completed or canceled training runs.

• 'false' - to cancel the currently running training run.

hard_delete = 'true'
training_delete = requests.delete(
    url = f"{WML_ENDPOINT_URL}/ml/v4/trainings/{training_id}?version=2020-08-01&space_id={space_id}&hard_delete={hard_delete}",
    headers = header
)
if training_delete.status_code in [200,202,204]:
    print('SUCCES')
else:
    print(f'Failed to delete training: \n {training_delete.json()}')

8. Manage deployments

If you want to get all deployments details you can run following REST API call. If you want to get part of deployments please change the limit variable.

limit=2
deployments_details = requests.get(
    url = f"{WML_ENDPOINT_URL}/ml/v4/deployments?version=2020-08-01&space_id={space_id}&limit={limit}",
    headers = header
)

If you want to list existing deployments names and ids run next cell.

for deployment in deployments_details.json()['resources']:
    print(f"{deployment['metadata']['name']} \t Id: {deployment['metadata']['id']}")

Get deployment 0 id.

deployment_id = deployments_details.json()['resources'][0]['metadata']['id']
print(deployment_id)

If you want to get deployment details you must provide deployment_id and run following REST API call.

deployment_details = requests.get(
    url = f"{WML_ENDPOINT_URL}/ml/v4/deployments/{deployment_id}?version=2020-08-01&space_id={space_id}",
    headers = header
)

Print deployment details.

deployment_details.json()

You can delete deployment by next cell API CALL.

deployment_delete = requests.delete(
    url = f"{WML_ENDPOINT_URL}/ml/v4/deployments/{deployment_id}?version=2020-08-01&space_id={space_id}",
    headers = header
)
if deployment_delete.status_code in [200,202,204]:
    print('SUCCES')
else:
    print(f'Failed to delete deployment: \n {deployment_delete.json()}')

If you want to delete more deployments run following cell.

for deployment in deployments_details.json()['resources']:
    deployment_delete = requests.delete(
    url = f"{WML_ENDPOINT_URL}/ml/v4/deployments/{deployment['metadata']['id']}?version=2020-08-01&space_id={space_id}",
    headers = header
    )
    if deployment_delete.status_code in [200,202,204]:
        print('SUCCES')
    else:
        print(f'Failed to delete deployment: \n {deployment_delete.json()}')

9. Summary and next steps

You successfully completed this notebook! You learned how to use REST API client for Watson Machine Learning instance management and clean up. Check out our Online Documentation for more samples, tutorials, documentation, how-tos, and blog posts.

Authors

Szymon Kucharczyk, Software Engineer at IBM.

Copyright © 2020, 2021, 2022 IBM. This notebook and its source code are released under the terms of the MIT License.