Book a Demo!
CoCalc Logo Icon
StoreFeaturesDocsShareSupportNewsAboutPoliciesSign UpSign In
ibm
GitHub Repository: ibm/watson-machine-learning-samples
 Path: blob/master/cpd3.5/notebooks/rest_api/curl/experiments/autoai/Use AutoAI and batch deployment to predict credit risk.ipynb
6405 views
Kernel: Python 3

Use AutoAI and batch deployment to predict credit risk with Watson Machine Learning REST API

This notebook contains steps and code to demonstrate support of AutoAI experiments in Watson Machine Learning Service. It introduces commands for getting data, training experiments, persisting pipelines, publishing models, deploying models and scoring.

Some familiarity with cURL is helpful. This notebook uses cURL examples.

Learning goals

The learning goals of this notebook are:

  • Working with Watson Machine Learning experiments to train AutoAI models.

  • Downloading computed models to local storage.

  • Batch deployment and scoring of trained model.

Contents

This notebook contains the following parts:

  1. Setup

  2. Experiment definition

  3. Experiment Run

  4. Historical runs

  5. Deploy and Score

  6. Cleaning

  7. Summary and next steps

1. Set up the environment

Before you use the sample code in this notebook, you must perform the following setup tasks:

  • Contact with your Cloud Pack for Data administrator and ask him for your account credentials

Connection to WML

Authenticate the Watson Machine Learning service on IBM Cloud Pack for Data. You need to provide platform url, your username and password.

%env USERNAME=
%env PASSWORD=
%env DATAPLATFORM_URL=

%env SPACE_ID=
env: USERNAME= env: PASSWORD= env: DATAPLATFORM_URL= env: SPACE_ID=

Getting WML authorization token for further cURL calls

%%bash --out token

curl -sk -X GET \
    --user $USERNAME:$PASSWORD \
    --header "Accept: application/json" \
    "$DATAPLATFORM_URL/v1/preauth/validateAuth" \
    | cut -d '"' -f 44

%env TOKEN=$token

Space creation

Tip: If you do not have space already created, please convert below three cells to code and run them.

First of all, you need to create a space that will be used in all of your further cURL calls. If you do not have space already created, below is the cURL call to create one.

<a href="https://cpd-spaces-api.eu-gb.cf.appdomain.cloud/#/Spaces/spaces_create" target="_blank" rel="noopener no referrer">Space creation

%%bash --out space_id

curl -sk -X POST \
    --header "Authorization: Bearer $TOKEN" \
    --header "Content-Type: application/json" \
    --header "Accept: application/json" \
    --data '{"name": "curl_DL"}' \
    "$DATAPLATFORM_URL/v2/spaces" \
     | grep '"id": ' | awk -F '"' '{ print $4 }'

space_id = space_id.split('\n')[1]
%env SPACE_ID=$space_id

Space creation is asynchronous. This means that you need to check space creation status after creation call. Make sure that your newly created space is active.

<a href="https://cpd-spaces-api.eu-gb.cf.appdomain.cloud/#/Spaces/spaces_get" target="_blank" rel="noopener no referrer">Get space information

%%bash

!curl -sk -X GET \
    --header "Authorization: Bearer $TOKEN" \
    --header "Content-Type: application/json" \
    --header "Accept: application/json" \
    "$DATAPLATFORM_URL/v2/spaces/$SPACE_ID"

2. Experiment / optimizer configuration

Provide input information for AutoAI experiment / optimizer:

  • name - experiment name

  • learning_type - type of the problem

  • label - target column name

  • scorer_for_ranking - optimization metric

  • holdout_param - percentage of training data to use as a holdout [0 - 1]

  • daub_include_only_estimators - list of estimators to use

You can modify parameters section of the following cURL call to change AutoAI experiment settings.

<a href="https://watson-ml-v4-api.mybluemix.net/wml-restapi-cloud.html#/Pipelines/pipelines_create" target="_blank" rel="noopener no referrer">Define AutoAI experiment.

%%bash --out pipeline_payload

PIPELINE_PAYLOAD='{"space_id": "'"$SPACE_ID"'", "name": "Credit Risk Prediction - AutoAI", "description": "", "document": {"doc_type": "pipeline", "version": "2.0", "pipelines": [{"id": "autoai", "runtime_ref": "hybrid", "nodes": [{"id": "automl", "type": "execution_node", "parameters": {"stage_flag": true, "output_logs": true, "input_file_separator": ",", "optimization": {"learning_type": "binary", "label": "Risk", "max_num_daub_ensembles": 1, "daub_include_only_estimators": ["ExtraTreesClassifierEstimator", "GradientBoostingClassifierEstimator", "LGBMClassifierEstimator", "LogisticRegressionEstimator", "RandomForestClassifierEstimator", "XGBClassifierEstimator", "DecisionTreeClassifierEstimator"], "scorer_for_ranking": "roc_auc", "compute_pipeline_notebooks_flag": true, "run_cognito_flag": true, "holdout_param": 0.1}}, "runtime_ref": "autoai", "op": "kube"}]}], "runtimes": [{"id": "autoai", "name": "auto_ai.kb", "app_data": {"wml_data": {"hardware_spec": { "name": "M"}}}, "version": "3.0.2"}],"primary_pipeline": "autoai"}}'
echo $PIPELINE_PAYLOAD | python -m json.tool

%env PIPELINE_PAYLOAD=$pipeline_payload
env: PIPELINE_PAYLOAD={ "space_id": "2219f598-6194-4e9a-9287-0730c69b5f4a", "name": "Credit Risk Prediction - AutoAI", "description": "", "document": { "doc_type": "pipeline", "version": "2.0", "pipelines": [ { "id": "autoai", "runtime_ref": "hybrid", "nodes": [ { "id": "automl", "type": "execution_node", "parameters": { "stage_flag": true, "output_logs": true, "input_file_separator": ",", "optimization": { "learning_type": "binary", "label": "Risk", "max_num_daub_ensembles": 1, "daub_include_only_estimators": [ "ExtraTreesClassifierEstimator", "GradientBoostingClassifierEstimator", "LGBMClassifierEstimator", "LogisticRegressionEstimator", "RandomForestClassifierEstimator", "XGBClassifierEstimator", "DecisionTreeClassifierEstimator" ], "scorer_for_ranking": "roc_auc", "compute_pipeline_notebooks_flag": true, "run_cognito_flag": true, "holdout_param": 0.1 } }, "runtime_ref": "autoai", "op": "kube" } ] } ], "runtimes": [ { "id": "autoai", "name": "auto_ai.kb", "app_data": { "wml_data": { "hardware_spec": { "name": "M" } } }, "version": "3.0.2" } ], "primary_pipeline": "autoai" } } 
%%bash --out pipeline_id

curl -sk -X POST \
    --header "Authorization: Bearer $TOKEN" \
    --header "Content-Type: application/json" \
    --header "Accept: application/json" \
    --data "$PIPELINE_PAYLOAD" \
    "$DATAPLATFORM_URL/ml/v4/pipelines?version=2020-08-01" \
    | grep '"id": ' | awk -F '"' '{ print $4 }' | sed -n 5p

%env PIPELINE_ID=$pipeline_id
env: PIPELINE_ID=a8cbf66e-2e7d-4ec4-b81c-0c24b38ef35f

Get experiment details

To retrieve AutoAI experiment / optimizer configuration you can follow below cURL GET call.

<a href="https://watson-ml-v4-api.mybluemix.net/wml-restapi-cloud.html#/Pipelines/pipelines_get" target="_blank" rel="noopener no referrer">Get experiment / optimizer information

%%bash

curl -sk -X GET \
    --header "Authorization: Bearer $TOKEN" \
    --header "Content-Type: application/json" \
    --header "Accept: application/json" \
    "$DATAPLATFORM_URL/ml/v4/pipelines/$PIPELINE_ID?space_id=$SPACE_ID&version=2020-08-01" \
    | python -m json.tool
{ "entity": { "document": { "doc_type": "pipeline", "pipelines": [ { "id": "autoai", "nodes": [ { "id": "automl", "op": "kube", "parameters": { "input_file_separator": ",", "optimization": { "compute_pipeline_notebooks_flag": true, "daub_include_only_estimators": [ "ExtraTreesClassifierEstimator", "GradientBoostingClassifierEstimator", "LGBMClassifierEstimator", "LogisticRegressionEstimator", "RandomForestClassifierEstimator", "XGBClassifierEstimator", "DecisionTreeClassifierEstimator" ], "holdout_param": 0.1, "label": "Risk", "learning_type": "binary", "max_num_daub_ensembles": 1.0, "run_cognito_flag": true, "scorer_for_ranking": "roc_auc" }, "output_logs": true, "stage_flag": true }, "runtime_ref": "autoai", "type": "execution_node" } ], "runtime_ref": "hybrid" } ], "primary_pipeline": "autoai", "runtimes": [ { "app_data": { "wml_data": { "hardware_spec": { "id": "c076e82c-b2a7-4d20-9c0f-1f0c2fdf5a24", "name": "M" } } }, "id": "autoai", "name": "auto_ai.kb", "version": "3.0.2" } ], "version": "2.0" } }, "metadata": { "created_at": "2020-10-01T06:40:35.876Z", "id": "a8cbf66e-2e7d-4ec4-b81c-0c24b38ef35f", "modified_at": "2020-10-01T06:40:35.876Z", "name": "Credit Risk Prediction - AutoAI", "owner": "1000330999", "space_id": "2219f598-6194-4e9a-9287-0730c69b5f4a" } }

Training data connection

Define connection information to COS bucket and training data CSV file. This example uses the German Credit Risk dataset.

The dataset can be downloaded from here. You can also download it to local filesystem by running the cell below.

Action: Upload training data to COS bucket and enter location information in the next cURL examples.

%%bash

wget https://github.com/IBM/watson-machine-learning-samples/raw/master/cpd3.5/data/credit_risk/credit_risk_training_light.csv \
     -O credit_risk_training_light.csv
--2020-10-01 08:40:42-- https://github.com/IBM/watson-machine-learning-samples/raw/master/data/credit_risk/credit_risk_training_light.csv Resolving github.com (github.com)... 140.82.121.4 Connecting to github.com (github.com)|140.82.121.4|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://raw.githubusercontent.com/IBM/watson-machine-learning-samples/master/data/credit_risk/credit_risk_training_light.csv [following] --2020-10-01 08:40:43-- https://raw.githubusercontent.com/IBM/watson-machine-learning-samples/master/data/credit_risk/credit_risk_training_light.csv Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.36.133 Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.36.133|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 34787 (34K) [text/plain] Saving to: ‘credit_risk_training_light.csv’ 0K .......... .......... .......... ... 100% 925K=0,04s 2020-10-01 08:40:43 (925 KB/s) - ‘credit_risk_training_light.csv’ saved [34787/34787]

Create Data Asset

Upload your local dataset into Data Asset Storage

<a href="https://cloud.ibm.com/apidocs/watson-data-api#createdataassetv2" target="_blank" rel="noopener no referrer">Upload file as Data Asset

%%bash --out data_asset_metadata

DATA_ASSET_METADATA='{"metadata": {"name": "autoai_training_data","description": "desc","asset_type": "data_asset","origin_country": "us","asset_category": "USER"},"entity": {"data_asset": {"mime_type": "text/csv"}}}'
echo $DATA_ASSET_METADATA | python -m json.tool

%env DATA_ASSET_METADATA=$data_asset_metadata
env: DATA_ASSET_METADATA={ "metadata": { "name": "autoai_training_data", "description": "desc", "asset_type": "data_asset", "origin_country": "us", "asset_category": "USER" }, "entity": { "data_asset": { "mime_type": "text/csv" } } } 
%%bash --out asset_id

curl -sk -X POST \
    --header "Authorization: Bearer $TOKEN" \
    --header "Content-Type: application/json" \
    --data "$DATA_ASSET_METADATA" \
    "$DATAPLATFORM_URL/v2/assets?space_id=$SPACE_ID&version=2020-08-01" \
    | cut -d '"' -f 108

%env ASSET_ID=$asset_id
env: ASSET_ID=f5531b4b-d1e8-44e7-a464-67164a2b8a6e 
%%bash --out attachment

ATTACHMENT='{"asset_type": "data_asset", "name": "credit_risk_training_light.csv", "mime": "text/csv"}'
echo $ATTACHMENT | python -m json.tool

%env ATTACHMENT=$attachment
env: ATTACHMENT={ "asset_type": "data_asset", "name": "credit_risk_training_light.csv", "mime": "text/csv" } 
%%bash --out attachment_response

curl -sk -X POST \
    --header "Authorization: Bearer $TOKEN" \
    --header "Content-Type: application/json" \
    --data "$ATTACHMENT" \
    "$DATAPLATFORM_URL/v2/assets/$ASSET_ID/attachments?space_id=$SPACE_ID&version=2020-08-01"


%env ATTACHMENT_RESPONSE=$attachment_response
env: ATTACHMENT_RESPONSE={"attachment_id":"7e81043b-134a-44c3-bb1c-f63628da2447","upload_id":"7be9c4bc-2091-4c75-8f93-d2fe00b115e1","data_partitions":1,"start_part_num":1,"url1":"/v2/asset_files/12667e3f-da77-4e8a-972a-eaf8ad3996e9/f5531b4b-d1e8-44e7-a464-67164a2b8a6e/7e81043b-134a-44c3-bb1c-f63628da2447?space_id=2219f598-6194-4e9a-9287-0730c69b5f4a&signature=lxGFHZYVh77a1OhGYEV6tQ%3D%3D%3ARPIz1KeMVaGUXIbjaj3CkDoDP%2BFnBh0iN9rYPNng7%2FIADu6QtGG4DS49CyXTHbLhCcNojg%2BbNt0cBp6L4foWvGlygXqBElvLpPytVOc0BNn05fXWEdSmIWKm9JbwcS1DjHlK51KJQVa2FClKNeXngoa5y3RQE0dUd6CJgerAUyhl2iBAEVBcLbVmTFg%2BRtwAPvbQZqJbyb24DW4U%2FT1n4syyFUnvk0dwdmwXG%2FA2OgsEJNnxd6aJKGjLrwJ%2FYbvge27eXIClzyPEo33dW5HSf9iEMumt7rP0Mn5iAitcheo44w1wzGGn%2BqI%3D","datasource_type":"81bafdbd-b7c6-45c5-a4fd-6ec135f66f4e","asset_type":"data_asset","attachment_type":"local","is_partitioned":false,"name":"credit_risk_training_light.csv","mime":"text/csv","user_data":{},"href":"/v2/assets/f5531b4b-d1e8-44e7-a464-67164a2b8a6e/attachments/7e81043b-134a-44c3-bb1c-f63628da2447?space_id=2219f598-6194-4e9a-9287-0730c69b5f4a","asset_category":"USER"} 
%%bash --out attachment_id

echo $ATTACHMENT_RESPONSE | cut -d '"' -f 4 | tr -d '\n'

%env ATTACHMENT_ID=$attachment_id
env: ATTACHMENT_ID=7e81043b-134a-44c3-bb1c-f63628da2447 
%%bash --out attachment_url

echo $ATTACHMENT_RESPONSE | cut -d '"' -f 16 | tr -d '\n'

%env ATTACHMENT_URL=$attachment_url
env: ATTACHMENT_URL=/v2/asset_files/12667e3f-da77-4e8a-972a-eaf8ad3996e9/f5531b4b-d1e8-44e7-a464-67164a2b8a6e/7e81043b-134a-44c3-bb1c-f63628da2447?space_id=2219f598-6194-4e9a-9287-0730c69b5f4a&signature=lxGFHZYVh77a1OhGYEV6tQ%3D%3D%3ARPIz1KeMVaGUXIbjaj3CkDoDP%2BFnBh0iN9rYPNng7%2FIADu6QtGG4DS49CyXTHbLhCcNojg%2BbNt0cBp6L4foWvGlygXqBElvLpPytVOc0BNn05fXWEdSmIWKm9JbwcS1DjHlK51KJQVa2FClKNeXngoa5y3RQE0dUd6CJgerAUyhl2iBAEVBcLbVmTFg%2BRtwAPvbQZqJbyb24DW4U%2FT1n4syyFUnvk0dwdmwXG%2FA2OgsEJNnxd6aJKGjLrwJ%2FYbvge27eXIClzyPEo33dW5HSf9iEMumt7rP0Mn5iAitcheo44w1wzGGn%2BqI%3D 
%%bash

curl -sk -X PUT \
    --header "Authorization: Bearer $TOKEN" \
    --header "Content-Type: multipart/form-data" \
    -F 'file=@credit_risk_training_light.csv' \
    "$DATAPLATFORM_URL$ATTACHMENT_URL"
{"status":"Asset created: The asset was successfully uploaded."}

The response should looks like this: {"status":"Asset created: The asset was successfully uploaded."}

%%bash

curl -sk -X POST \
    --header "Authorization: Bearer $TOKEN" \
    --header "Content-Type: application/json" \
    "$DATAPLATFORM_URL/v2/assets/$ASSET_ID/attachments/$ATTACHMENT_ID/complete?space_id=$SPACE_ID&version=2020-08-01"
{"attachment_id":"7e81043b-134a-44c3-bb1c-f63628da2447","asset_type":"data_asset","name":"credit_risk_training_light.csv","mime":"text/csv","size":34787}

3. Experiment run

This section provides samples about how to trigger AutoAI experiment via cURL calls.

<a href="https://watson-ml-v4-api.mybluemix.net/wml-restapi-cloud.html#/Trainings/trainings_create" target="_blank" rel="noopener no referrer">Schedule a training job for AutoAI experiment

%%bash --out random_id

openssl rand -hex 20

%env RANDOM_ID=$random_id
env: RANDOM_ID=eab90a7e0841c00db04266373c27ef71a0f312b5 
%%bash --out training_payload

TRAINING_PAYLOAD='{"space_id": "'"$SPACE_ID"'", "training_data_references": [{"type": "data_asset", "id": "credit_risk_training_light.csv", "connection": {}, "location": {"href": "/v2/assets/'"$ASSET_ID"'?space_id='"$SPACE_ID"'"}}], "results_reference": {"type": "fs", "id": "autoai_results", "connection": {}, "location": {"path": "/spaces/'$SPACE_ID'/assets/auto_ml/auto_ml_curl.'$RANDOM_ID'/wml_data"}}, "tags": [{"value": "autoai"}], "pipeline": {"href": "/v4/pipelines/'"$PIPELINE_ID"'"}}'
echo $TRAINING_PAYLOAD | python -m json.tool

%env TRAINING_PAYLOAD=$training_payload
env: TRAINING_PAYLOAD={ "space_id": "2219f598-6194-4e9a-9287-0730c69b5f4a", "training_data_references": [ { "type": "data_asset", "id": "credit_risk_training_light.csv", "connection": {}, "location": { "href": "/v2/assets/f5531b4b-d1e8-44e7-a464-67164a2b8a6e?space_id=2219f598-6194-4e9a-9287-0730c69b5f4a" } } ], "results_reference": { "type": "fs", "id": "autoai_results", "connection": {}, "location": { "path": "/spaces/2219f598-6194-4e9a-9287-0730c69b5f4a/assets/auto_ml/auto_ml_curl.eab90a7e0841c00db04266373c27ef71a0f312b5/wml_data" } }, "tags": [ { "value": "autoai" } ], "pipeline": { "href": "/v4/pipelines/a8cbf66e-2e7d-4ec4-b81c-0c24b38ef35f" } } 
%%bash --out training_id

curl -sk -X POST \
    --header "Authorization: Bearer $TOKEN" \
    --header "Content-Type: application/json" \
    --header "Accept: application/json" \
    --data "$TRAINING_PAYLOAD" \
    "$DATAPLATFORM_URL/ml/v4/trainings?version=2020-08-01" \
    | awk -F'"id":' '{print $2}' | cut -c2-37

%env TRAINING_ID=$training_id
env: TRAINING_ID=b4a501d9-8da3-47fa-9a2b-197d73fe84fb

Get training details

Training is an asynchronous endpoint. In case you want to monitor training status and details, you need to use a GET method and specify which training you want to monitor by usage of training ID.

<a href="https://watson-ml-v4-api.mybluemix.net/wml-restapi-cloud.html#/Trainings/trainings_get" target="_blank" rel="noopener no referrer">Get information about training job

%%bash

curl -sk -X GET \
    --header "Authorization: Bearer $TOKEN" \
    --header "Content-Type: application/json" \
    --header "Accept: application/json" \
    "$DATAPLATFORM_URL/ml/v4/trainings/$TRAINING_ID?space_id=$SPACE_ID&version=2020-08-01" \
    | python -m json.tool

Get training status

%%bash

STATUS=$(curl -sk -X GET\
    --header "Authorization: Bearer $TOKEN" \
    --header "Content-Type: application/json" \
    --header "Accept: application/json" \
    "$DATAPLATFORM_URL/ml/v4/trainings/$TRAINING_ID?space_id=$SPACE_ID&version=2020-08-01")
    
STATUS=${STATUS#*state\":\"}
STATUS=${STATUS%%\"*}
echo $STATUS
completed

Please make sure that training is completed before you go to the next sections. Monitor state of your training by running above cell couple of times.

4. Historical runs

In this section you will see cURL examples describing how to get historical training runs information.

Output should be similar to the output from training creation but you should see more trainings entries. Listing trainings:

<a href="https://watson-ml-v4-api.mybluemix.net/wml-restapi-cloud.html#/Trainings/trainings_list" target="_blank" rel="noopener no referrer">Get list of historical training jobs information

%%bash

HISTORICAL_TRAINING_LIMIT_TO_GET=2

curl -sk -X GET \
    --header "Authorization: Bearer $TOKEN" \
    --header "Content-Type: application/json" \
    --header "Accept: application/json" \
    "$DATAPLATFORM_URL/ml/v4/trainings?space_id=$SPACE_ID&version=2020-08-01&limit=$HISTORICAL_TRAINING_LIMIT_TO_GET" \
    | python -m json.tool
{ "limit": 2, "next": { "href": "/ml/v4/trainings?start=g1AAAAB4eJzLYWBgYMpgSmHgKy5JLCrJTq2MT8lPzkzJBYqrWCYnGxulmiXrpqampeqapBqk6lqaJZnopiWZJhqnmScZGKcagfRywPQSrSsLAN8IIc0&limit=2&space_id=2219f598-6194-4e9a-9287-0730c69b5f4a" }, "resources": [ { "metadata": { "created_at": "2020-10-01T06:50:05.311Z", "guid": "6e1bee43-062a-4d2b-9eeb-d4afd841a8ba", "id": "6e1bee43-062a-4d2b-9eeb-d4afd841a8ba", "modified_at": "2020-10-01T06:56:07.646Z", "space_id": "2219f598-6194-4e9a-9287-0730c69b5f4a", "tags": [ "autoai" ] }, "entity": { "pipeline": { "href": "/v4/pipelines/a8cbf66e-2e7d-4ec4-b81c-0c24b38ef35f", "id": "a8cbf66e-2e7d-4ec4-b81c-0c24b38ef35f" }, "results_reference": { "location": { "training": "6e1bee43-062a-4d2b-9eeb-d4afd841a8ba", "path": ".", "assets_path": "6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/assets", "training_status": "6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/training-status.json" }, "type": "fs", "connection": {}, "id": "autoai_results" }, "space_id": "2219f598-6194-4e9a-9287-0730c69b5f4a", "status": { "completed_at": "2020-10-01T06:56:07.634Z", "message": { "text": "Training job 6e1bee43-062a-4d2b-9eeb-d4afd841a8ba completed", "level": "info" }, "metrics": [ { "context": { "binary_classification": { "confusion_matrix": [ { "fn": 2, "fp": 4, "tn": 4, "tp": 15, "true_class": "No Risk" }, { "fn": 4, "fp": 2, "tn": 15, "tp": 4, "true_class": "Risk" } ], "roc_curve": [ { "fpr": [ 0.0, 0.0, 0.0, 0.125, 0.125, 0.25, 0.25, 0.375, 0.375, 0.5, 0.5, 1.0 ], "thresholds": [ 1.9959793090820312, 0.9959793090820312, 0.9949495196342468, 0.9937530755996704, 0.9900901913642883, 0.9870622158050537, 0.9241466522216797, 0.8619304299354553, 0.7010563015937805, 0.5894654989242554, 0.23473195731639862, 0.010169494897127151 ], "tpr": [ 0.0, 0.058823529411764705, 0.11764705882352941, 0.11764705882352941, 0.23529411764705882, 0.23529411764705882, 0.47058823529411764, 0.47058823529411764, 0.7058823529411765, 0.7058823529411765, 1.0, 1.0 ], "true_class": "No Risk" }, { "fpr": [ 0.0, 0.0, 0.0, 0.29411764705882354, 0.29411764705882354, 0.4117647058823529, 0.4117647058823529, 0.7058823529411765, 0.7058823529411765, 0.8235294117647058, 0.8235294117647058, 1.0 ], "thresholds": [ 1.9907909631729126, 0.9907909631729126, 0.9192749261856079, 0.37951037287712097, 0.3295697271823883, 0.15989628434181213, 0.11566814035177231, 0.03620442748069763, 0.02297501638531685, 0.010295126587152481, 0.007908087223768234, 0.003513400675728917 ], "tpr": [ 0.0, 0.125, 0.5, 0.5, 0.625, 0.625, 0.75, 0.75, 0.875, 0.875, 1.0, 1.0 ], "true_class": "Risk" } ] }, "classes": [ "No Risk", "Risk" ], "features_importance": [ { "computation_type": "", "features": { "LoanAmount": 0.47, "InstallmentPercent": 0.06, "ExistingSavings": 0.16, "CheckingStatus": 0.05, "LoanPurpose": 0.23, "Job": 0.04, "InstallmentPlans": 0.0, "OthersOnLoan": 0.14, "LoanDuration": 0.24, "Age": 1.0, "CreditHistory": 0.13, "CurrentResidenceDuration": 0.16, "ForeignWorker": 0.01, "Housing": 0.0, "ExistingCreditsCount": 0.07, "Telephone": 0.0, "Sex": 0.01, "Dependents": 0.08, "OwnsProperty": 0.04, "EmploymentDuration": 0.11 }, "stage": "Mean" } ], "intermediate_model": { "composition_steps": [ "TrainingDataset_full_249_20", "Split_TrainingHoldout", "TrainingDataset_full_224_20", "Preprocessor_default", "DAUB" ], "duration": 785, "location": { "model": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/pre_hpo_d_output/Pipeline1/model.pickle", "pipeline": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/pre_hpo_d_output/Pipeline1/pipeline.json", "pipeline_model": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/pre_hpo_d_output/Pipeline1/pipeline-model.json" }, "name": "P1", "notebook_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/pre_hpo_d_output/Pipeline1/notebook.ipynb", "pipeline_nodes": [ "PreprocessingTransformer", "GradientBoostingClassifierEstimator" ], "process": "pre_hpo_d_output", "schema_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/pre_hpo_d_output/Pipeline1/schema.json", "sdk_notebook_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/pre_hpo_d_output/Pipeline1/notebook_sdk.ipynb" }, "phase": "pre_hpo_d_output" }, "ml_metrics": { "training_roc_auc": 0.8283546163980947, "holdout_precision": 0.6666666666666666, "training_average_precision": 0.6996753806435804, "holdout_average_precision": 0.7116994847258005, "training_neg_log_loss": -0.5747366162076947, "holdout_recall": 0.5, "training_precision": 0.6439393939393939, "holdout_accuracy": 0.76, "holdout_balanced_accuracy": 0.6911764705882353, "training_recall": 0.5718050065876153, "holdout_f1": 0.5714285714285715, "holdout_neg_log_loss": -0.7066552317239467, "training_accuracy": 0.7723123123123123, "holdout_roc_auc": 0.7205882352941178, "training_balanced_accuracy": 0.715389682780987, "training_f1": 0.6009839626860902 }, "timestamp": "2020-10-01T06:51:18.000Z" }, { "context": { "binary_classification": { "confusion_matrix": [ { "fn": 2, "fp": 3, "tn": 5, "tp": 15, "true_class": "No Risk" }, { "fn": 3, "fp": 2, "tn": 15, "tp": 5, "true_class": "Risk" } ], "roc_curve": [ { "fpr": [ 0.0, 0.0, 0.0, 0.125, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 1.0 ], "thresholds": [ 1.9118249416351318, 0.9118249416351318, 0.8851636648178101, 0.8839583992958069, 0.8572619557380676, 0.8568741083145142, 0.7981584072113037, 0.7974665760993958, 0.529254138469696, 0.4382297098636627, 0.37201759219169617, 0.24074923992156982 ], "tpr": [ 0.0, 0.058823529411764705, 0.29411764705882354, 0.29411764705882354, 0.35294117647058826, 0.35294117647058826, 0.5882352941176471, 0.5882352941176471, 0.8823529411764706, 0.8823529411764706, 1.0, 1.0 ], "true_class": "No Risk" }, { "fpr": [ 0.0, 0.0, 0.0, 0.11764705882352941, 0.11764705882352941, 0.4117647058823529, 0.4117647058823529, 0.6470588235294118, 0.6470588235294118, 0.7058823529411765, 0.7058823529411765, 1.0 ], "thresholds": [ 1.7592507600784302, 0.7592507600784302, 0.6614384055137634, 0.5711529850959778, 0.529679536819458, 0.29737383127212524, 0.20253342390060425, 0.14502882957458496, 0.14312586188316345, 0.14273804426193237, 0.11604157835245132, 0.08817508071660995 ], "tpr": [ 0.0, 0.125, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 0.875, 0.875, 1.0, 1.0 ], "true_class": "Risk" } ] }, "classes": [ "No Risk", "Risk" ], "features_importance": [ { "computation_type": "", "features": { "LoanAmount": 1.0, "InstallmentPercent": 0.32, "ExistingSavings": 0.09, "CheckingStatus": 0.17, "LoanPurpose": 0.16, "Job": 0.01, "InstallmentPlans": 0.07, "OthersOnLoan": 0.37, "LoanDuration": 0.43, "Age": 0.51, "CreditHistory": 0.26, "CurrentResidenceDuration": 0.18, "ForeignWorker": 0.0, "Housing": 0.08, "ExistingCreditsCount": 0.22, "Telephone": 0.06, "Sex": 0.2, "Dependents": 0.23, "OwnsProperty": 0.15, "EmploymentDuration": 0.22 }, "stage": "Mean" } ], "intermediate_model": { "composition_steps": [ "TrainingDataset_full_249_20", "Split_TrainingHoldout", "TrainingDataset_full_224_20", "Preprocessor_default", "DAUB", "hpo_d" ], "duration": 5318, "location": { "model": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/hpo_d_output/Pipeline1/model.pickle", "pipeline": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/hpo_d_output/Pipeline1/pipeline.json", "pipeline_model": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/hpo_d_output/Pipeline1/pipeline-model.json" }, "name": "P2", "notebook_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/hpo_d_output/Pipeline1/notebook.ipynb", "pipeline_nodes": [ "PreprocessingTransformer", "GradientBoostingClassifierEstimator" ], "process": "hpo_d_output", "schema_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/hpo_d_output/Pipeline1/schema.json", "sdk_notebook_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/hpo_d_output/Pipeline1/notebook_sdk.ipynb" }, "phase": "hpo_d_output" }, "ml_metrics": { "training_roc_auc": 0.8566053511705686, "holdout_precision": 0.7142857142857143, "training_average_precision": 0.7320733111904305, "holdout_average_precision": 0.7039224664224665, "training_neg_log_loss": -0.4465285582613328, "holdout_recall": 0.625, "training_precision": 0.7780214424951266, "holdout_accuracy": 0.8, "holdout_balanced_accuracy": 0.7536764705882353, "training_recall": 0.49736495388669294, "holdout_f1": 0.6666666666666666, "holdout_neg_log_loss": -0.5421721597291818, "training_accuracy": 0.7990390390390391, "holdout_roc_auc": 0.75, "training_balanced_accuracy": 0.7134260666869362, "training_f1": 0.589159891598916 }, "timestamp": "2020-10-01T06:51:40.000Z" }, { "context": { "binary_classification": { "confusion_matrix": [ { "fn": 4, "fp": 3, "tn": 5, "tp": 13, "true_class": "No Risk" }, { "fn": 3, "fp": 4, "tn": 13, "tp": 5, "true_class": "Risk" } ], "roc_curve": [ { "fpr": [ 0.0, 0.0, 0.0, 0.125, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 1.0 ], "thresholds": [ 1.9320768117904663, 0.9320768117904663, 0.9299767017364502, 0.9259238243103027, 0.8939604759216309, 0.8521214127540588, 0.7138300538063049, 0.7113333344459534, 0.45716628432273865, 0.43239983916282654, 0.42159730195999146, 0.3683125376701355, 0.3567972481250763, 0.19027136266231537 ], "tpr": [ 0.0, 0.058823529411764705, 0.11764705882352941, 0.11764705882352941, 0.29411764705882354, 0.29411764705882354, 0.6470588235294118, 0.6470588235294118, 0.8823529411764706, 0.8823529411764706, 0.9411764705882353, 0.9411764705882353, 1.0, 1.0 ], "true_class": "No Risk" }, { "fpr": [ 0.0, 0.0, 0.0, 0.058823529411764705, 0.058823529411764705, 0.11764705882352941, 0.11764705882352941, 0.35294117647058826, 0.35294117647058826, 0.7058823529411765, 0.7058823529411765, 0.8823529411764706, 0.8823529411764706, 1.0 ], "thresholds": [ 1.8097286224365234, 0.8097286224365234, 0.7199095487594604, 0.6432027816772461, 0.6316874623298645, 0.5784026980400085, 0.5589774250984192, 0.3221668303012848, 0.28866666555404663, 0.15159079432487488, 0.14787857234477997, 0.08006153255701065, 0.07407616823911667, 0.06792321056127548 ], "tpr": [ 0.0, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 0.875, 0.875, 1.0, 1.0 ], "true_class": "Risk" } ] }, "classes": [ "No Risk", "Risk" ], "features_importance": [ { "computation_type": "", "features": { "LoanAmount": 0.86, "InstallmentPercent": 0.14, "ExistingSavings": 0.42, "CheckingStatus": 0.1, "LoanPurpose": 0.36, "NewFeature_0_featureagglomeration_0": 0.39, "Job": 0.04, "InstallmentPlans": 0.1, "OthersOnLoan": 0.43, "LoanDuration": 1.0, "Age": 0.99, "CreditHistory": 0.57, "CurrentResidenceDuration": 0.2, "ForeignWorker": 0.0, "Housing": 0.07, "ExistingCreditsCount": 0.59, "Telephone": 0.06, "NewFeature_1_featureagglomeration_1": 0.49, "Sex": 0.09, "Dependents": 0.08, "OwnsProperty": 0.28, "EmploymentDuration": 0.06 }, "stage": "Mean" } ], "intermediate_model": { "composition_steps": [ "TrainingDataset_full_249_20", "Split_TrainingHoldout", "TrainingDataset_full_224_20", "Preprocessor_default", "DAUB", "hpo_d", "cognito" ], "duration": 27061, "location": { "model": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/cognito_output/Pipeline1/model.pickle", "pipeline": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/cognito_output/Pipeline1/pipeline.json", "pipeline_model": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/cognito_output/Pipeline1/pipeline-model.json" }, "name": "P3", "notebook_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/cognito_output/Pipeline1/notebook.ipynb", "pipeline_nodes": [ "PreprocessingTransformer", "autoai_libs.cognito.transforms.transform_utils.featureagglomeration", "autoai_libs.cognito.transforms.transform_utils.FS1", "GradientBoostingClassifierEstimator" ], "process": "cognito_output", "schema_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/cognito_output/Pipeline1/schema.json", "sdk_notebook_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/cognito_output/Pipeline1/notebook_sdk.ipynb" }, "phase": "cognito_output" }, "ml_metrics": { "training_roc_auc": 0.8637123745819397, "holdout_precision": 0.5555555555555556, "training_average_precision": 0.7631005098061324, "holdout_average_precision": 0.6683999400675602, "training_neg_log_loss": -0.436711969891688, "holdout_recall": 0.625, "training_precision": 0.7331140350877193, "holdout_accuracy": 0.72, "holdout_balanced_accuracy": 0.6948529411764706, "training_recall": 0.5882740447957838, "holdout_f1": 0.5882352941176471, "holdout_neg_log_loss": -0.555290777346542, "training_accuracy": 0.8081681681681682, "holdout_roc_auc": 0.7205882352941176, "training_balanced_accuracy": 0.7460600993209688, "training_f1": 0.651842785257718 }, "timestamp": "2020-10-01T06:52:24.000Z" }, { "context": { "binary_classification": { "confusion_matrix": [ { "fn": 3, "fp": 3, "tn": 5, "tp": 14, "true_class": "No Risk" }, { "fn": 3, "fp": 3, "tn": 14, "tp": 5, "true_class": "Risk" } ], "roc_curve": [ { "fpr": [ 0.0, 0.0, 0.0, 0.125, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 1.0 ], "thresholds": [ 1.912172555923462, 0.9121726155281067, 0.9023290276527405, 0.9015228152275085, 0.8295201063156128, 0.8294777870178223, 0.6834437251091003, 0.6798041462898254, 0.4823266565799713, 0.4560561776161194, 0.40774452686309814, 0.40617018938064575, 0.39104729890823364, 0.28977835178375244 ], "tpr": [ 0.0, 0.058823529411764705, 0.17647058823529413, 0.17647058823529413, 0.4117647058823529, 0.4117647058823529, 0.6470588235294118, 0.6470588235294118, 0.8823529411764706, 0.8823529411764706, 0.9411764705882353, 0.9411764705882353, 1.0, 1.0 ], "true_class": "No Risk" }, { "fpr": [ 0.0, 0.0, 0.0, 0.058823529411764705, 0.058823529411764705, 0.11764705882352941, 0.11764705882352941, 0.35294117647058826, 0.35294117647058826, 0.5882352941176471, 0.5882352941176471, 0.8235294117647058, 0.8235294117647058, 1.0 ], "thresholds": [ 1.7102216482162476, 0.7102216482162476, 0.669156014919281, 0.6089527010917664, 0.5938298106193542, 0.5922554731369019, 0.5338093638420105, 0.3384079039096832, 0.32019585371017456, 0.18048915266990662, 0.17052221298217773, 0.11166291683912277, 0.09847716987133026, 0.0878273993730545 ], "tpr": [ 0.0, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 0.875, 0.875, 1.0, 1.0 ], "true_class": "Risk" } ] }, "classes": [ "No Risk", "Risk" ], "features_importance": [ { "computation_type": "", "features": { "LoanAmount": 0.68, "InstallmentPercent": 0.14, "ExistingSavings": 0.42, "CheckingStatus": 0.09, "LoanPurpose": 0.32, "NewFeature_0_featureagglomeration_0": 0.39, "Job": 0.04, "InstallmentPlans": 0.09, "OthersOnLoan": 0.43, "LoanDuration": 1.0, "Age": 0.87, "CreditHistory": 0.54, "CurrentResidenceDuration": 0.2, "ForeignWorker": 0.0, "Housing": 0.07, "ExistingCreditsCount": 0.51, "Telephone": 0.01, "NewFeature_1_featureagglomeration_1": 0.41, "Sex": 0.05, "Dependents": 0.03, "OwnsProperty": 0.26, "EmploymentDuration": 0.06 }, "stage": "Mean" } ], "intermediate_model": { "composition_steps": [ "TrainingDataset_full_249_20", "Split_TrainingHoldout", "TrainingDataset_full_224_20", "Preprocessor_default", "DAUB", "hpo_d", "cognito", "hpo_c" ], "duration": 9795, "location": { "model": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/hpo_c_output/Pipeline1/model.pickle", "pipeline": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/hpo_c_output/Pipeline1/pipeline.json", "pipeline_model": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/hpo_c_output/Pipeline1/pipeline-model.json" }, "name": "P4", "notebook_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/hpo_c_output/Pipeline1/notebook.ipynb", "pipeline_nodes": [ "PreprocessingTransformer", "autoai_libs.cognito.transforms.transform_utils.featureagglomeration", "autoai_libs.cognito.transforms.transform_utils.FS1", "GradientBoostingClassifierEstimator" ], "process": "hpo_c_output", "schema_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/hpo_c_output/Pipeline1/schema.json", "sdk_notebook_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/hpo_c_output/Pipeline1/notebook_sdk.ipynb" }, "phase": "hpo_c_output" }, "ml_metrics": { "training_roc_auc": 0.8656126482213438, "holdout_precision": 0.625, "training_average_precision": 0.7583491228304152, "holdout_average_precision": 0.6757941813088872, "training_neg_log_loss": -0.4447612030777576, "holdout_recall": 0.625, "training_precision": 0.7137566137566139, "holdout_accuracy": 0.76, "holdout_balanced_accuracy": 0.7242647058823529, "training_recall": 0.5586297760210803, "holdout_f1": 0.625, "holdout_neg_log_loss": -0.5520562594922229, "training_accuracy": 0.7947747747747748, "holdout_roc_auc": 0.7426470588235294, "training_balanced_accuracy": 0.7280328367284888, "training_f1": 0.6245680270070514 }, "timestamp": "2020-10-01T06:52:50.000Z" }, { "context": { "binary_classification": { "confusion_matrix": [ { "fn": 2, "fp": 4, "tn": 4, "tp": 15, "true_class": "No Risk" }, { "fn": 4, "fp": 2, "tn": 15, "tp": 4, "true_class": "Risk" } ], "roc_curve": [ { "fpr": [ 0.0, 0.0, 0.0, 0.125, 0.125, 0.25, 0.25, 0.375, 0.375, 0.5, 0.5, 1.0 ], "thresholds": [ 1.9959793090820312, 0.9959793090820312, 0.9949495196342468, 0.9937530755996704, 0.9900901913642883, 0.9870622158050537, 0.9241466522216797, 0.8619304299354553, 0.7010563015937805, 0.5894654989242554, 0.23473195731639862, 0.010169494897127151 ], "tpr": [ 0.0, 0.058823529411764705, 0.11764705882352941, 0.11764705882352941, 0.23529411764705882, 0.23529411764705882, 0.47058823529411764, 0.47058823529411764, 0.7058823529411765, 0.7058823529411765, 1.0, 1.0 ], "true_class": "No Risk" }, { "fpr": [ 0.0, 0.0, 0.0, 0.29411764705882354, 0.29411764705882354, 0.4117647058823529, 0.4117647058823529, 0.7058823529411765, 0.7058823529411765, 0.8235294117647058, 0.8235294117647058, 1.0 ], "thresholds": [ 1.9907909631729126, 0.9907909631729126, 0.9192749261856079, 0.37951037287712097, 0.3295697271823883, 0.15989628434181213, 0.11566814035177231, 0.03620442748069763, 0.02297501638531685, 0.010295126587152481, 0.007908087223768234, 0.003513400675728917 ], "tpr": [ 0.0, 0.125, 0.5, 0.5, 0.625, 0.625, 0.75, 0.75, 0.875, 0.875, 1.0, 1.0 ], "true_class": "Risk" } ] }, "classes": [ "No Risk", "Risk" ], "features_importance": [ { "computation_type": "", "features": { "LoanAmount": 0.68, "InstallmentPercent": 0.14, "ExistingSavings": 0.42, "CheckingStatus": 0.09, "LoanPurpose": 0.32, "NewFeature_0_featureagglomeration_0": 0.39, "Job": 0.04, "InstallmentPlans": 0.09, "OthersOnLoan": 0.43, "LoanDuration": 1.0, "Age": 0.87, "CreditHistory": 0.54, "CurrentResidenceDuration": 0.2, "ForeignWorker": 0.0, "Housing": 0.07, "ExistingCreditsCount": 0.51, "Telephone": 0.01, "NewFeature_1_featureagglomeration_1": 0.41, "Sex": 0.05, "Dependents": 0.03, "OwnsProperty": 0.26, "EmploymentDuration": 0.06 }, "stage": "Mean" } ], "intermediate_model": { "composition_steps": [ "TrainingDataset_full_249_20", "Split_TrainingHoldout", "TrainingDataset_full_224_20", "Preprocessor_default", "DAUB" ], "duration": 109249, "location": { "model": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/compose_model_type_output/Pipeline0/model.pickle", "pipeline": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/compose_model_type_output/Pipeline0/pipeline.json", "pipeline_model": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/compose_model_type_output/Pipeline0/pipeline-model.json" }, "name": "P1", "notebook_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/compose_model_type_output/Pipeline0/notebook.ipynb", "pipeline_nodes": [ "PreprocessingTransformer", "GradientBoostingClassifierEstimator" ], "process": "compose_model_type_output", "schema_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/compose_model_type_output/Pipeline0/schema.json", "sdk_notebook_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/compose_model_type_output/Pipeline0/notebook_sdk.ipynb" }, "phase": "compose_model_type_output" }, "ml_metrics": { "training_roc_auc": 0.8283546163980947, "holdout_precision": 0.6666666666666666, "training_average_precision": 0.6996753806435804, "holdout_average_precision": 0.7116994847258005, "training_neg_log_loss": -0.5747366162076947, "holdout_recall": 0.5, "training_precision": 0.6439393939393939, "holdout_accuracy": 0.76, "holdout_balanced_accuracy": 0.6911764705882353, "training_recall": 0.5718050065876153, "holdout_f1": 0.5714285714285715, "holdout_neg_log_loss": -0.7066552317239467, "training_accuracy": 0.7723123123123123, "holdout_roc_auc": 0.7205882352941178, "training_balanced_accuracy": 0.715389682780987, "training_f1": 0.6009839626860902 }, "timestamp": "2020-10-01T06:53:56.000Z" }, { "context": { "binary_classification": { "confusion_matrix": [ { "fn": 2, "fp": 3, "tn": 5, "tp": 15, "true_class": "No Risk" }, { "fn": 3, "fp": 2, "tn": 15, "tp": 5, "true_class": "Risk" } ], "roc_curve": [ { "fpr": [ 0.0, 0.0, 0.0, 0.125, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 1.0 ], "thresholds": [ 1.9118249416351318, 0.9118249416351318, 0.8851636648178101, 0.8839583992958069, 0.8572619557380676, 0.8568741083145142, 0.7981584072113037, 0.7974665760993958, 0.529254138469696, 0.4382297098636627, 0.37201759219169617, 0.24074923992156982 ], "tpr": [ 0.0, 0.058823529411764705, 0.29411764705882354, 0.29411764705882354, 0.35294117647058826, 0.35294117647058826, 0.5882352941176471, 0.5882352941176471, 0.8823529411764706, 0.8823529411764706, 1.0, 1.0 ], "true_class": "No Risk" }, { "fpr": [ 0.0, 0.0, 0.0, 0.11764705882352941, 0.11764705882352941, 0.4117647058823529, 0.4117647058823529, 0.6470588235294118, 0.6470588235294118, 0.7058823529411765, 0.7058823529411765, 1.0 ], "thresholds": [ 1.7592507600784302, 0.7592507600784302, 0.6614384055137634, 0.5711529850959778, 0.529679536819458, 0.29737383127212524, 0.20253342390060425, 0.14502882957458496, 0.14312586188316345, 0.14273804426193237, 0.11604157835245132, 0.08817508071660995 ], "tpr": [ 0.0, 0.125, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 0.875, 0.875, 1.0, 1.0 ], "true_class": "Risk" } ] }, "classes": [ "No Risk", "Risk" ], "features_importance": [ { "computation_type": "", "features": { "LoanAmount": 0.68, "InstallmentPercent": 0.14, "ExistingSavings": 0.42, "CheckingStatus": 0.09, "LoanPurpose": 0.32, "NewFeature_0_featureagglomeration_0": 0.39, "Job": 0.04, "InstallmentPlans": 0.09, "OthersOnLoan": 0.43, "LoanDuration": 1.0, "Age": 0.87, "CreditHistory": 0.54, "CurrentResidenceDuration": 0.2, "ForeignWorker": 0.0, "Housing": 0.07, "ExistingCreditsCount": 0.51, "Telephone": 0.01, "NewFeature_1_featureagglomeration_1": 0.41, "Sex": 0.05, "Dependents": 0.03, "OwnsProperty": 0.26, "EmploymentDuration": 0.06 }, "stage": "Mean" } ], "intermediate_model": { "composition_steps": [ "TrainingDataset_full_249_20", "Split_TrainingHoldout", "TrainingDataset_full_224_20", "Preprocessor_default", "DAUB", "hpo_d" ], "duration": 109249, "location": { "model": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/compose_model_type_output/Pipeline1/model.pickle", "pipeline": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/compose_model_type_output/Pipeline1/pipeline.json", "pipeline_model": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/compose_model_type_output/Pipeline1/pipeline-model.json" }, "name": "P2", "notebook_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/compose_model_type_output/Pipeline1/notebook.ipynb", "pipeline_nodes": [ "PreprocessingTransformer", "GradientBoostingClassifierEstimator" ], "process": "compose_model_type_output", "schema_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/compose_model_type_output/Pipeline1/schema.json", "sdk_notebook_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/compose_model_type_output/Pipeline1/notebook_sdk.ipynb" }, "phase": "compose_model_type_output" }, "ml_metrics": { "training_roc_auc": 0.8566053511705686, "holdout_precision": 0.7142857142857143, "training_average_precision": 0.7320733111904305, "holdout_average_precision": 0.7039224664224665, "training_neg_log_loss": -0.4465285582613328, "holdout_recall": 0.625, "training_precision": 0.7780214424951266, "holdout_accuracy": 0.8, "holdout_balanced_accuracy": 0.7536764705882353, "training_recall": 0.49736495388669294, "holdout_f1": 0.6666666666666666, "holdout_neg_log_loss": -0.5421721597291818, "training_accuracy": 0.7990390390390391, "holdout_roc_auc": 0.75, "training_balanced_accuracy": 0.7134260666869362, "training_f1": 0.589159891598916 }, "timestamp": "2020-10-01T06:53:56.000Z" }, { "context": { "binary_classification": { "confusion_matrix": [ { "fn": 4, "fp": 3, "tn": 5, "tp": 13, "true_class": "No Risk" }, { "fn": 3, "fp": 4, "tn": 13, "tp": 5, "true_class": "Risk" } ], "roc_curve": [ { "fpr": [ 0.0, 0.0, 0.0, 0.125, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 1.0 ], "thresholds": [ 1.9320768117904663, 0.9320768117904663, 0.9299767017364502, 0.9259238243103027, 0.8939604759216309, 0.8521214127540588, 0.7138300538063049, 0.7113333344459534, 0.45716628432273865, 0.43239983916282654, 0.42159730195999146, 0.3683125376701355, 0.3567972481250763, 0.19027136266231537 ], "tpr": [ 0.0, 0.058823529411764705, 0.11764705882352941, 0.11764705882352941, 0.29411764705882354, 0.29411764705882354, 0.6470588235294118, 0.6470588235294118, 0.8823529411764706, 0.8823529411764706, 0.9411764705882353, 0.9411764705882353, 1.0, 1.0 ], "true_class": "No Risk" }, { "fpr": [ 0.0, 0.0, 0.0, 0.058823529411764705, 0.058823529411764705, 0.11764705882352941, 0.11764705882352941, 0.35294117647058826, 0.35294117647058826, 0.7058823529411765, 0.7058823529411765, 0.8823529411764706, 0.8823529411764706, 1.0 ], "thresholds": [ 1.8097286224365234, 0.8097286224365234, 0.7199095487594604, 0.6432027816772461, 0.6316874623298645, 0.5784026980400085, 0.5589774250984192, 0.3221668303012848, 0.28866666555404663, 0.15159079432487488, 0.14787857234477997, 0.08006153255701065, 0.07407616823911667, 0.06792321056127548 ], "tpr": [ 0.0, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 0.875, 0.875, 1.0, 1.0 ], "true_class": "Risk" } ] }, "classes": [ "No Risk", "Risk" ], "features_importance": [ { "computation_type": "", "features": { "LoanAmount": 0.68, "InstallmentPercent": 0.14, "ExistingSavings": 0.42, "CheckingStatus": 0.09, "LoanPurpose": 0.32, "NewFeature_0_featureagglomeration_0": 0.39, "Job": 0.04, "InstallmentPlans": 0.09, "OthersOnLoan": 0.43, "LoanDuration": 1.0, "Age": 0.87, "CreditHistory": 0.54, "CurrentResidenceDuration": 0.2, "ForeignWorker": 0.0, "Housing": 0.07, "ExistingCreditsCount": 0.51, "Telephone": 0.01, "NewFeature_1_featureagglomeration_1": 0.41, "Sex": 0.05, "Dependents": 0.03, "OwnsProperty": 0.26, "EmploymentDuration": 0.06 }, "stage": "Mean" } ], "intermediate_model": { "composition_steps": [ "TrainingDataset_full_249_20", "Split_TrainingHoldout", "TrainingDataset_full_224_20", "Preprocessor_default", "DAUB", "hpo_d", "cognito" ], "duration": 109249, "location": { "model": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/compose_model_type_output/Pipeline2/model.pickle", "pipeline": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/compose_model_type_output/Pipeline2/pipeline.json", "pipeline_model": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/compose_model_type_output/Pipeline2/pipeline-model.json" }, "name": "P3", "notebook_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/compose_model_type_output/Pipeline2/notebook.ipynb", "pipeline_nodes": [ "PreprocessingTransformer", "autoai_libs.cognito.transforms.transform_utils.featureagglomeration", "autoai_libs.cognito.transforms.transform_utils.FS1", "GradientBoostingClassifierEstimator" ], "process": "compose_model_type_output", "schema_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/compose_model_type_output/Pipeline2/schema.json", "sdk_notebook_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/compose_model_type_output/Pipeline2/notebook_sdk.ipynb" }, "phase": "compose_model_type_output" }, "ml_metrics": { "training_roc_auc": 0.8637123745819397, "holdout_precision": 0.5555555555555556, "training_average_precision": 0.7631005098061324, "holdout_average_precision": 0.6683999400675602, "training_neg_log_loss": -0.436711969891688, "holdout_recall": 0.625, "training_precision": 0.7331140350877193, "holdout_accuracy": 0.72, "holdout_balanced_accuracy": 0.6948529411764706, "training_recall": 0.5882740447957838, "holdout_f1": 0.5882352941176471, "holdout_neg_log_loss": -0.555290777346542, "training_accuracy": 0.8081681681681682, "holdout_roc_auc": 0.7205882352941176, "training_balanced_accuracy": 0.7460600993209688, "training_f1": 0.651842785257718 }, "timestamp": "2020-10-01T06:53:57.000Z" }, { "context": { "binary_classification": { "confusion_matrix": [ { "fn": 3, "fp": 3, "tn": 5, "tp": 14, "true_class": "No Risk" }, { "fn": 3, "fp": 3, "tn": 14, "tp": 5, "true_class": "Risk" } ], "roc_curve": [ { "fpr": [ 0.0, 0.0, 0.0, 0.125, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 1.0 ], "thresholds": [ 1.912172555923462, 0.9121726155281067, 0.9023290276527405, 0.9015228152275085, 0.8295201063156128, 0.8294777870178223, 0.6834437251091003, 0.6798041462898254, 0.4823266565799713, 0.4560561776161194, 0.40774452686309814, 0.40617018938064575, 0.39104729890823364, 0.28977835178375244 ], "tpr": [ 0.0, 0.058823529411764705, 0.17647058823529413, 0.17647058823529413, 0.4117647058823529, 0.4117647058823529, 0.6470588235294118, 0.6470588235294118, 0.8823529411764706, 0.8823529411764706, 0.9411764705882353, 0.9411764705882353, 1.0, 1.0 ], "true_class": "No Risk" }, { "fpr": [ 0.0, 0.0, 0.0, 0.058823529411764705, 0.058823529411764705, 0.11764705882352941, 0.11764705882352941, 0.35294117647058826, 0.35294117647058826, 0.5882352941176471, 0.5882352941176471, 0.8235294117647058, 0.8235294117647058, 1.0 ], "thresholds": [ 1.7102216482162476, 0.7102216482162476, 0.669156014919281, 0.6089527010917664, 0.5938298106193542, 0.5922554731369019, 0.5338093638420105, 0.3384079039096832, 0.32019585371017456, 0.18048915266990662, 0.17052221298217773, 0.11166291683912277, 0.09847716987133026, 0.0878273993730545 ], "tpr": [ 0.0, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 0.875, 0.875, 1.0, 1.0 ], "true_class": "Risk" } ] }, "classes": [ "No Risk", "Risk" ], "features_importance": [ { "computation_type": "", "features": { "LoanAmount": 0.68, "InstallmentPercent": 0.14, "ExistingSavings": 0.42, "CheckingStatus": 0.09, "LoanPurpose": 0.32, "NewFeature_0_featureagglomeration_0": 0.39, "Job": 0.04, "InstallmentPlans": 0.09, "OthersOnLoan": 0.43, "LoanDuration": 1.0, "Age": 0.87, "CreditHistory": 0.54, "CurrentResidenceDuration": 0.2, "ForeignWorker": 0.0, "Housing": 0.07, "ExistingCreditsCount": 0.51, "Telephone": 0.01, "NewFeature_1_featureagglomeration_1": 0.41, "Sex": 0.05, "Dependents": 0.03, "OwnsProperty": 0.26, "EmploymentDuration": 0.06 }, "stage": "Mean" } ], "intermediate_model": { "composition_steps": [ "TrainingDataset_full_249_20", "Split_TrainingHoldout", "TrainingDataset_full_224_20", "Preprocessor_default", "DAUB", "hpo_d", "cognito", "hpo_c" ], "duration": 109249, "location": { "model": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/compose_model_type_output/Pipeline3/model.pickle", "pipeline": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/compose_model_type_output/Pipeline3/pipeline.json", "pipeline_model": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/compose_model_type_output/Pipeline3/pipeline-model.json" }, "name": "P4", "notebook_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/compose_model_type_output/Pipeline3/notebook.ipynb", "pipeline_nodes": [ "PreprocessingTransformer", "autoai_libs.cognito.transforms.transform_utils.featureagglomeration", "autoai_libs.cognito.transforms.transform_utils.FS1", "GradientBoostingClassifierEstimator" ], "process": "compose_model_type_output", "schema_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/compose_model_type_output/Pipeline3/schema.json", "sdk_notebook_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/compose_model_type_output/Pipeline3/notebook_sdk.ipynb" }, "phase": "compose_model_type_output" }, "ml_metrics": { "training_roc_auc": 0.8656126482213438, "holdout_precision": 0.625, "training_average_precision": 0.7583491228304152, "holdout_average_precision": 0.6757941813088872, "training_neg_log_loss": -0.4447612030777576, "holdout_recall": 0.625, "training_precision": 0.7137566137566139, "holdout_accuracy": 0.76, "holdout_balanced_accuracy": 0.7242647058823529, "training_recall": 0.5586297760210803, "holdout_f1": 0.625, "holdout_neg_log_loss": -0.5520562594922229, "training_accuracy": 0.7947747747747748, "holdout_roc_auc": 0.7426470588235294, "training_balanced_accuracy": 0.7280328367284888, "training_f1": 0.6245680270070514 }, "timestamp": "2020-10-01T06:53:57.000Z" }, { "context": { "binary_classification": { "confusion_matrix": [ { "fn": 2, "fp": 4, "tn": 4, "tp": 15, "true_class": "No Risk" }, { "fn": 4, "fp": 2, "tn": 15, "tp": 4, "true_class": "Risk" } ], "roc_curve": [ { "fpr": [ 0.0, 0.0, 0.0, 0.125, 0.125, 0.25, 0.25, 0.375, 0.375, 0.5, 0.5, 1.0 ], "thresholds": [ 1.9959793090820312, 0.9959793090820312, 0.9949495196342468, 0.9937530755996704, 0.9900901913642883, 0.9870622158050537, 0.9241466522216797, 0.8619304299354553, 0.7010563015937805, 0.5894654989242554, 0.23473195731639862, 0.010169494897127151 ], "tpr": [ 0.0, 0.058823529411764705, 0.11764705882352941, 0.11764705882352941, 0.23529411764705882, 0.23529411764705882, 0.47058823529411764, 0.47058823529411764, 0.7058823529411765, 0.7058823529411765, 1.0, 1.0 ], "true_class": "No Risk" }, { "fpr": [ 0.0, 0.0, 0.0, 0.29411764705882354, 0.29411764705882354, 0.4117647058823529, 0.4117647058823529, 0.7058823529411765, 0.7058823529411765, 0.8235294117647058, 0.8235294117647058, 1.0 ], "thresholds": [ 1.9907909631729126, 0.9907909631729126, 0.9192749261856079, 0.37951037287712097, 0.3295697271823883, 0.15989628434181213, 0.11566814035177231, 0.03620442748069763, 0.02297501638531685, 0.010295126587152481, 0.007908087223768234, 0.003513400675728917 ], "tpr": [ 0.0, 0.125, 0.5, 0.5, 0.625, 0.625, 0.75, 0.75, 0.875, 0.875, 1.0, 1.0 ], "true_class": "Risk" } ] }, "classes": [ "No Risk", "Risk" ], "features_importance": [ { "computation_type": "", "features": { "LoanAmount": 0.68, "InstallmentPercent": 0.14, "ExistingSavings": 0.42, "CheckingStatus": 0.09, "LoanPurpose": 0.32, "NewFeature_0_featureagglomeration_0": 0.39, "Job": 0.04, "InstallmentPlans": 0.09, "OthersOnLoan": 0.43, "LoanDuration": 1.0, "Age": 0.87, "CreditHistory": 0.54, "CurrentResidenceDuration": 0.2, "ForeignWorker": 0.0, "Housing": 0.07, "ExistingCreditsCount": 0.51, "Telephone": 0.01, "NewFeature_1_featureagglomeration_1": 0.41, "Sex": 0.05, "Dependents": 0.03, "OwnsProperty": 0.26, "EmploymentDuration": 0.06 }, "stage": "Mean" } ], "intermediate_model": { "composition_steps": [ "TrainingDataset_full_249_20", "Split_TrainingHoldout", "TrainingDataset_full_224_20", "Preprocessor_default", "DAUB" ], "duration": 177440, "location": { "model": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/fold_output/Pipeline0/model.pickle", "pipeline": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/fold_output/Pipeline0/pipeline.json", "pipeline_model": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/fold_output/Pipeline0/pipeline-model.json" }, "name": "P1", "notebook_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/fold_output/Pipeline0/notebook.ipynb", "pipeline_nodes": [ "PreprocessingTransformer", "GradientBoostingClassifierEstimator" ], "process": "fold_output", "schema_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/fold_output/Pipeline0/schema.json", "sdk_notebook_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/fold_output/Pipeline0/notebook_sdk.ipynb" }, "phase": "fold_output" }, "ml_metrics": { "training_roc_auc": 0.8283546163980947, "holdout_precision": 0.6666666666666666, "training_average_precision": 0.6996753806435804, "holdout_average_precision": 0.7116994847258005, "training_neg_log_loss": -0.5747366162076947, "holdout_recall": 0.5, "training_precision": 0.6439393939393939, "holdout_accuracy": 0.76, "holdout_balanced_accuracy": 0.6911764705882353, "training_recall": 0.5718050065876153, "holdout_f1": 0.5714285714285715, "holdout_neg_log_loss": -0.7066552317239467, "training_accuracy": 0.7723123123123123, "holdout_roc_auc": 0.7205882352941178, "training_balanced_accuracy": 0.715389682780987, "training_f1": 0.6009839626860902 }, "timestamp": "2020-10-01T06:55:01.000Z" }, { "context": { "binary_classification": { "confusion_matrix": [ { "fn": 2, "fp": 3, "tn": 5, "tp": 15, "true_class": "No Risk" }, { "fn": 3, "fp": 2, "tn": 15, "tp": 5, "true_class": "Risk" } ], "roc_curve": [ { "fpr": [ 0.0, 0.0, 0.0, 0.125, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 1.0 ], "thresholds": [ 1.9118249416351318, 0.9118249416351318, 0.8851636648178101, 0.8839583992958069, 0.8572619557380676, 0.8568741083145142, 0.7981584072113037, 0.7974665760993958, 0.529254138469696, 0.4382297098636627, 0.37201759219169617, 0.24074923992156982 ], "tpr": [ 0.0, 0.058823529411764705, 0.29411764705882354, 0.29411764705882354, 0.35294117647058826, 0.35294117647058826, 0.5882352941176471, 0.5882352941176471, 0.8823529411764706, 0.8823529411764706, 1.0, 1.0 ], "true_class": "No Risk" }, { "fpr": [ 0.0, 0.0, 0.0, 0.11764705882352941, 0.11764705882352941, 0.4117647058823529, 0.4117647058823529, 0.6470588235294118, 0.6470588235294118, 0.7058823529411765, 0.7058823529411765, 1.0 ], "thresholds": [ 1.7592507600784302, 0.7592507600784302, 0.6614384055137634, 0.5711529850959778, 0.529679536819458, 0.29737383127212524, 0.20253342390060425, 0.14502882957458496, 0.14312586188316345, 0.14273804426193237, 0.11604157835245132, 0.08817508071660995 ], "tpr": [ 0.0, 0.125, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 0.875, 0.875, 1.0, 1.0 ], "true_class": "Risk" } ] }, "classes": [ "No Risk", "Risk" ], "features_importance": [ { "computation_type": "", "features": { "LoanAmount": 0.68, "InstallmentPercent": 0.14, "ExistingSavings": 0.42, "CheckingStatus": 0.09, "LoanPurpose": 0.32, "NewFeature_0_featureagglomeration_0": 0.39, "Job": 0.04, "InstallmentPlans": 0.09, "OthersOnLoan": 0.43, "LoanDuration": 1.0, "Age": 0.87, "CreditHistory": 0.54, "CurrentResidenceDuration": 0.2, "ForeignWorker": 0.0, "Housing": 0.07, "ExistingCreditsCount": 0.51, "Telephone": 0.01, "NewFeature_1_featureagglomeration_1": 0.41, "Sex": 0.05, "Dependents": 0.03, "OwnsProperty": 0.26, "EmploymentDuration": 0.06 }, "stage": "Mean" } ], "intermediate_model": { "composition_steps": [ "TrainingDataset_full_249_20", "Split_TrainingHoldout", "TrainingDataset_full_224_20", "Preprocessor_default", "DAUB", "hpo_d" ], "duration": 177440, "location": { "model": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/fold_output/Pipeline1/model.pickle", "pipeline": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/fold_output/Pipeline1/pipeline.json", "pipeline_model": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/fold_output/Pipeline1/pipeline-model.json" }, "name": "P2", "notebook_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/fold_output/Pipeline1/notebook.ipynb", "pipeline_nodes": [ "PreprocessingTransformer", "GradientBoostingClassifierEstimator" ], "process": "fold_output", "schema_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/fold_output/Pipeline1/schema.json", "sdk_notebook_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/fold_output/Pipeline1/notebook_sdk.ipynb" }, "phase": "fold_output" }, "ml_metrics": { "training_roc_auc": 0.8566053511705686, "holdout_precision": 0.7142857142857143, "training_average_precision": 0.7320733111904305, "holdout_average_precision": 0.7039224664224665, "training_neg_log_loss": -0.4465285582613328, "holdout_recall": 0.625, "training_precision": 0.7780214424951266, "holdout_accuracy": 0.8, "holdout_balanced_accuracy": 0.7536764705882353, "training_recall": 0.49736495388669294, "holdout_f1": 0.6666666666666666, "holdout_neg_log_loss": -0.5421721597291818, "training_accuracy": 0.7990390390390391, "holdout_roc_auc": 0.75, "training_balanced_accuracy": 0.7134260666869362, "training_f1": 0.589159891598916 }, "timestamp": "2020-10-01T06:55:02.000Z" }, { "context": { "binary_classification": { "confusion_matrix": [ { "fn": 4, "fp": 3, "tn": 5, "tp": 13, "true_class": "No Risk" }, { "fn": 3, "fp": 4, "tn": 13, "tp": 5, "true_class": "Risk" } ], "roc_curve": [ { "fpr": [ 0.0, 0.0, 0.0, 0.125, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 1.0 ], "thresholds": [ 1.9320768117904663, 0.9320768117904663, 0.9299767017364502, 0.9259238243103027, 0.8939604759216309, 0.8521214127540588, 0.7138300538063049, 0.7113333344459534, 0.45716628432273865, 0.43239983916282654, 0.42159730195999146, 0.3683125376701355, 0.3567972481250763, 0.19027136266231537 ], "tpr": [ 0.0, 0.058823529411764705, 0.11764705882352941, 0.11764705882352941, 0.29411764705882354, 0.29411764705882354, 0.6470588235294118, 0.6470588235294118, 0.8823529411764706, 0.8823529411764706, 0.9411764705882353, 0.9411764705882353, 1.0, 1.0 ], "true_class": "No Risk" }, { "fpr": [ 0.0, 0.0, 0.0, 0.058823529411764705, 0.058823529411764705, 0.11764705882352941, 0.11764705882352941, 0.35294117647058826, 0.35294117647058826, 0.7058823529411765, 0.7058823529411765, 0.8823529411764706, 0.8823529411764706, 1.0 ], "thresholds": [ 1.8097286224365234, 0.8097286224365234, 0.7199095487594604, 0.6432027816772461, 0.6316874623298645, 0.5784026980400085, 0.5589774250984192, 0.3221668303012848, 0.28866666555404663, 0.15159079432487488, 0.14787857234477997, 0.08006153255701065, 0.07407616823911667, 0.06792321056127548 ], "tpr": [ 0.0, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 0.875, 0.875, 1.0, 1.0 ], "true_class": "Risk" } ] }, "classes": [ "No Risk", "Risk" ], "features_importance": [ { "computation_type": "", "features": { "LoanAmount": 0.68, "InstallmentPercent": 0.14, "ExistingSavings": 0.42, "CheckingStatus": 0.09, "LoanPurpose": 0.32, "NewFeature_0_featureagglomeration_0": 0.39, "Job": 0.04, "InstallmentPlans": 0.09, "OthersOnLoan": 0.43, "LoanDuration": 1.0, "Age": 0.87, "CreditHistory": 0.54, "CurrentResidenceDuration": 0.2, "ForeignWorker": 0.0, "Housing": 0.07, "ExistingCreditsCount": 0.51, "Telephone": 0.01, "NewFeature_1_featureagglomeration_1": 0.41, "Sex": 0.05, "Dependents": 0.03, "OwnsProperty": 0.26, "EmploymentDuration": 0.06 }, "stage": "Mean" } ], "intermediate_model": { "composition_steps": [ "TrainingDataset_full_249_20", "Split_TrainingHoldout", "TrainingDataset_full_224_20", "Preprocessor_default", "DAUB", "hpo_d", "cognito" ], "duration": 177440, "location": { "model": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/fold_output/Pipeline2/model.pickle", "pipeline": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/fold_output/Pipeline2/pipeline.json", "pipeline_model": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/fold_output/Pipeline2/pipeline-model.json" }, "name": "P3", "notebook_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/fold_output/Pipeline2/notebook.ipynb", "pipeline_nodes": [ "PreprocessingTransformer", "autoai_libs.cognito.transforms.transform_utils.featureagglomeration", "autoai_libs.cognito.transforms.transform_utils.FS1", "GradientBoostingClassifierEstimator" ], "process": "fold_output", "schema_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/fold_output/Pipeline2/schema.json", "sdk_notebook_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/fold_output/Pipeline2/notebook_sdk.ipynb" }, "phase": "fold_output" }, "ml_metrics": { "training_roc_auc": 0.8637123745819397, "holdout_precision": 0.5555555555555556, "training_average_precision": 0.7631005098061324, "holdout_average_precision": 0.6683999400675602, "training_neg_log_loss": -0.436711969891688, "holdout_recall": 0.625, "training_precision": 0.7331140350877193, "holdout_accuracy": 0.72, "holdout_balanced_accuracy": 0.6948529411764706, "training_recall": 0.5882740447957838, "holdout_f1": 0.5882352941176471, "holdout_neg_log_loss": -0.555290777346542, "training_accuracy": 0.8081681681681682, "holdout_roc_auc": 0.7205882352941176, "training_balanced_accuracy": 0.7460600993209688, "training_f1": 0.651842785257718 }, "timestamp": "2020-10-01T06:55:02.000Z" }, { "context": { "binary_classification": { "confusion_matrix": [ { "fn": 3, "fp": 3, "tn": 5, "tp": 14, "true_class": "No Risk" }, { "fn": 3, "fp": 3, "tn": 14, "tp": 5, "true_class": "Risk" } ], "roc_curve": [ { "fpr": [ 0.0, 0.0, 0.0, 0.125, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 1.0 ], "thresholds": [ 1.912172555923462, 0.9121726155281067, 0.9023290276527405, 0.9015228152275085, 0.8295201063156128, 0.8294777870178223, 0.6834437251091003, 0.6798041462898254, 0.4823266565799713, 0.4560561776161194, 0.40774452686309814, 0.40617018938064575, 0.39104729890823364, 0.28977835178375244 ], "tpr": [ 0.0, 0.058823529411764705, 0.17647058823529413, 0.17647058823529413, 0.4117647058823529, 0.4117647058823529, 0.6470588235294118, 0.6470588235294118, 0.8823529411764706, 0.8823529411764706, 0.9411764705882353, 0.9411764705882353, 1.0, 1.0 ], "true_class": "No Risk" }, { "fpr": [ 0.0, 0.0, 0.0, 0.058823529411764705, 0.058823529411764705, 0.11764705882352941, 0.11764705882352941, 0.35294117647058826, 0.35294117647058826, 0.5882352941176471, 0.5882352941176471, 0.8235294117647058, 0.8235294117647058, 1.0 ], "thresholds": [ 1.7102216482162476, 0.7102216482162476, 0.669156014919281, 0.6089527010917664, 0.5938298106193542, 0.5922554731369019, 0.5338093638420105, 0.3384079039096832, 0.32019585371017456, 0.18048915266990662, 0.17052221298217773, 0.11166291683912277, 0.09847716987133026, 0.0878273993730545 ], "tpr": [ 0.0, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 0.875, 0.875, 1.0, 1.0 ], "true_class": "Risk" } ] }, "classes": [ "No Risk", "Risk" ], "features_importance": [ { "computation_type": "", "features": { "LoanAmount": 0.68, "InstallmentPercent": 0.14, "ExistingSavings": 0.42, "CheckingStatus": 0.09, "LoanPurpose": 0.32, "NewFeature_0_featureagglomeration_0": 0.39, "Job": 0.04, "InstallmentPlans": 0.09, "OthersOnLoan": 0.43, "LoanDuration": 1.0, "Age": 0.87, "CreditHistory": 0.54, "CurrentResidenceDuration": 0.2, "ForeignWorker": 0.0, "Housing": 0.07, "ExistingCreditsCount": 0.51, "Telephone": 0.01, "NewFeature_1_featureagglomeration_1": 0.41, "Sex": 0.05, "Dependents": 0.03, "OwnsProperty": 0.26, "EmploymentDuration": 0.06 }, "stage": "Mean" } ], "intermediate_model": { "composition_steps": [ "TrainingDataset_full_249_20", "Split_TrainingHoldout", "TrainingDataset_full_224_20", "Preprocessor_default", "DAUB", "hpo_d", "cognito", "hpo_c" ], "duration": 177440, "location": { "model": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/fold_output/Pipeline3/model.pickle", "pipeline": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/fold_output/Pipeline3/pipeline.json", "pipeline_model": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/fold_output/Pipeline3/pipeline-model.json" }, "name": "P4", "notebook_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/fold_output/Pipeline3/notebook.ipynb", "pipeline_nodes": [ "PreprocessingTransformer", "autoai_libs.cognito.transforms.transform_utils.featureagglomeration", "autoai_libs.cognito.transforms.transform_utils.FS1", "GradientBoostingClassifierEstimator" ], "process": "fold_output", "schema_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/fold_output/Pipeline3/schema.json", "sdk_notebook_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/fold_output/Pipeline3/notebook_sdk.ipynb" }, "phase": "fold_output" }, "ml_metrics": { "training_roc_auc": 0.8656126482213438, "holdout_precision": 0.625, "training_average_precision": 0.7583491228304152, "holdout_average_precision": 0.6757941813088872, "training_neg_log_loss": -0.4447612030777576, "holdout_recall": 0.625, "training_precision": 0.7137566137566139, "holdout_accuracy": 0.76, "holdout_balanced_accuracy": 0.7242647058823529, "training_recall": 0.5586297760210803, "holdout_f1": 0.625, "holdout_neg_log_loss": -0.5520562594922229, "training_accuracy": 0.7947747747747748, "holdout_roc_auc": 0.7426470588235294, "training_balanced_accuracy": 0.7280328367284888, "training_f1": 0.6245680270070514 }, "timestamp": "2020-10-01T06:55:02.000Z" }, { "context": { "binary_classification": { "confusion_matrix": [ { "fn": 2, "fp": 4, "tn": 4, "tp": 15, "true_class": "No Risk" }, { "fn": 4, "fp": 2, "tn": 15, "tp": 4, "true_class": "Risk" } ], "roc_curve": [ { "fpr": [ 0.0, 0.0, 0.0, 0.125, 0.125, 0.25, 0.25, 0.375, 0.375, 0.5, 0.5, 1.0 ], "thresholds": [ 1.9959793090820312, 0.9959793090820312, 0.9949495196342468, 0.9937530755996704, 0.9900901913642883, 0.9870622158050537, 0.9241466522216797, 0.8619304299354553, 0.7010563015937805, 0.5894654989242554, 0.23473195731639862, 0.010169494897127151 ], "tpr": [ 0.0, 0.058823529411764705, 0.11764705882352941, 0.11764705882352941, 0.23529411764705882, 0.23529411764705882, 0.47058823529411764, 0.47058823529411764, 0.7058823529411765, 0.7058823529411765, 1.0, 1.0 ], "true_class": "No Risk" }, { "fpr": [ 0.0, 0.0, 0.0, 0.29411764705882354, 0.29411764705882354, 0.4117647058823529, 0.4117647058823529, 0.7058823529411765, 0.7058823529411765, 0.8235294117647058, 0.8235294117647058, 1.0 ], "thresholds": [ 1.9907909631729126, 0.9907909631729126, 0.9192749261856079, 0.37951037287712097, 0.3295697271823883, 0.15989628434181213, 0.11566814035177231, 0.03620442748069763, 0.02297501638531685, 0.010295126587152481, 0.007908087223768234, 0.003513400675728917 ], "tpr": [ 0.0, 0.125, 0.5, 0.5, 0.625, 0.625, 0.75, 0.75, 0.875, 0.875, 1.0, 1.0 ], "true_class": "Risk" } ] }, "classes": [ "No Risk", "Risk" ], "features_importance": [ { "computation_type": "", "features": { "LoanAmount": 0.68, "InstallmentPercent": 0.14, "ExistingSavings": 0.42, "CheckingStatus": 0.09, "LoanPurpose": 0.32, "NewFeature_0_featureagglomeration_0": 0.39, "Job": 0.04, "InstallmentPlans": 0.09, "OthersOnLoan": 0.43, "LoanDuration": 1.0, "Age": 0.87, "CreditHistory": 0.54, "CurrentResidenceDuration": 0.2, "ForeignWorker": 0.0, "Housing": 0.07, "ExistingCreditsCount": 0.51, "Telephone": 0.01, "NewFeature_1_featureagglomeration_1": 0.41, "Sex": 0.05, "Dependents": 0.03, "OwnsProperty": 0.26, "EmploymentDuration": 0.06 }, "stage": "Mean" } ], "intermediate_model": { "composition_steps": [ "TrainingDataset_full_249_20", "Split_TrainingHoldout", "TrainingDataset_full_224_20", "Preprocessor_default", "DAUB" ], "duration": 242615, "location": { "model": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/global_output/Pipeline0/model.pickle", "pipeline": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/global_output/Pipeline0/pipeline.json", "pipeline_model": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/global_output/Pipeline0/pipeline-model.json" }, "name": "P1", "notebook_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/global_output/Pipeline0/notebook.ipynb", "pipeline_nodes": [ "PreprocessingTransformer", "GradientBoostingClassifierEstimator" ], "process": "global_output", "schema_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/global_output/Pipeline0/schema.json", "sdk_notebook_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/global_output/Pipeline0/notebook_sdk.ipynb" }, "phase": "global_output" }, "ml_metrics": { "training_roc_auc": 0.8283546163980947, "holdout_precision": 0.6666666666666666, "training_average_precision": 0.6996753806435804, "holdout_average_precision": 0.7116994847258005, "training_neg_log_loss": -0.5747366162076947, "holdout_recall": 0.5, "training_precision": 0.6439393939393939, "holdout_accuracy": 0.76, "holdout_balanced_accuracy": 0.6911764705882353, "training_recall": 0.5718050065876153, "holdout_f1": 0.5714285714285715, "holdout_neg_log_loss": -0.7066552317239467, "training_accuracy": 0.7723123123123123, "holdout_roc_auc": 0.7205882352941178, "training_balanced_accuracy": 0.715389682780987, "training_f1": 0.6009839626860902 }, "timestamp": "2020-10-01T06:56:06.000Z" }, { "context": { "binary_classification": { "confusion_matrix": [ { "fn": 2, "fp": 3, "tn": 5, "tp": 15, "true_class": "No Risk" }, { "fn": 3, "fp": 2, "tn": 15, "tp": 5, "true_class": "Risk" } ], "roc_curve": [ { "fpr": [ 0.0, 0.0, 0.0, 0.125, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 1.0 ], "thresholds": [ 1.9118249416351318, 0.9118249416351318, 0.8851636648178101, 0.8839583992958069, 0.8572619557380676, 0.8568741083145142, 0.7981584072113037, 0.7974665760993958, 0.529254138469696, 0.4382297098636627, 0.37201759219169617, 0.24074923992156982 ], "tpr": [ 0.0, 0.058823529411764705, 0.29411764705882354, 0.29411764705882354, 0.35294117647058826, 0.35294117647058826, 0.5882352941176471, 0.5882352941176471, 0.8823529411764706, 0.8823529411764706, 1.0, 1.0 ], "true_class": "No Risk" }, { "fpr": [ 0.0, 0.0, 0.0, 0.11764705882352941, 0.11764705882352941, 0.4117647058823529, 0.4117647058823529, 0.6470588235294118, 0.6470588235294118, 0.7058823529411765, 0.7058823529411765, 1.0 ], "thresholds": [ 1.7592507600784302, 0.7592507600784302, 0.6614384055137634, 0.5711529850959778, 0.529679536819458, 0.29737383127212524, 0.20253342390060425, 0.14502882957458496, 0.14312586188316345, 0.14273804426193237, 0.11604157835245132, 0.08817508071660995 ], "tpr": [ 0.0, 0.125, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 0.875, 0.875, 1.0, 1.0 ], "true_class": "Risk" } ] }, "classes": [ "No Risk", "Risk" ], "features_importance": [ { "computation_type": "", "features": { "LoanAmount": 0.68, "InstallmentPercent": 0.14, "ExistingSavings": 0.42, "CheckingStatus": 0.09, "LoanPurpose": 0.32, "NewFeature_0_featureagglomeration_0": 0.39, "Job": 0.04, "InstallmentPlans": 0.09, "OthersOnLoan": 0.43, "LoanDuration": 1.0, "Age": 0.87, "CreditHistory": 0.54, "CurrentResidenceDuration": 0.2, "ForeignWorker": 0.0, "Housing": 0.07, "ExistingCreditsCount": 0.51, "Telephone": 0.01, "NewFeature_1_featureagglomeration_1": 0.41, "Sex": 0.05, "Dependents": 0.03, "OwnsProperty": 0.26, "EmploymentDuration": 0.06 }, "stage": "Mean" } ], "intermediate_model": { "composition_steps": [ "TrainingDataset_full_249_20", "Split_TrainingHoldout", "TrainingDataset_full_224_20", "Preprocessor_default", "DAUB", "hpo_d" ], "duration": 242615, "location": { "model": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/global_output/Pipeline1/model.pickle", "pipeline": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/global_output/Pipeline1/pipeline.json", "pipeline_model": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/global_output/Pipeline1/pipeline-model.json" }, "name": "P2", "notebook_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/global_output/Pipeline1/notebook.ipynb", "pipeline_nodes": [ "PreprocessingTransformer", "GradientBoostingClassifierEstimator" ], "process": "global_output", "schema_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/global_output/Pipeline1/schema.json", "sdk_notebook_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/global_output/Pipeline1/notebook_sdk.ipynb" }, "phase": "global_output" }, "ml_metrics": { "training_roc_auc": 0.8566053511705686, "holdout_precision": 0.7142857142857143, "training_average_precision": 0.7320733111904305, "holdout_average_precision": 0.7039224664224665, "training_neg_log_loss": -0.4465285582613328, "holdout_recall": 0.625, "training_precision": 0.7780214424951266, "holdout_accuracy": 0.8, "holdout_balanced_accuracy": 0.7536764705882353, "training_recall": 0.49736495388669294, "holdout_f1": 0.6666666666666666, "holdout_neg_log_loss": -0.5421721597291818, "training_accuracy": 0.7990390390390391, "holdout_roc_auc": 0.75, "training_balanced_accuracy": 0.7134260666869362, "training_f1": 0.589159891598916 }, "timestamp": "2020-10-01T06:56:06.000Z" }, { "context": { "binary_classification": { "confusion_matrix": [ { "fn": 4, "fp": 3, "tn": 5, "tp": 13, "true_class": "No Risk" }, { "fn": 3, "fp": 4, "tn": 13, "tp": 5, "true_class": "Risk" } ], "roc_curve": [ { "fpr": [ 0.0, 0.0, 0.0, 0.125, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 1.0 ], "thresholds": [ 1.9320768117904663, 0.9320768117904663, 0.9299767017364502, 0.9259238243103027, 0.8939604759216309, 0.8521214127540588, 0.7138300538063049, 0.7113333344459534, 0.45716628432273865, 0.43239983916282654, 0.42159730195999146, 0.3683125376701355, 0.3567972481250763, 0.19027136266231537 ], "tpr": [ 0.0, 0.058823529411764705, 0.11764705882352941, 0.11764705882352941, 0.29411764705882354, 0.29411764705882354, 0.6470588235294118, 0.6470588235294118, 0.8823529411764706, 0.8823529411764706, 0.9411764705882353, 0.9411764705882353, 1.0, 1.0 ], "true_class": "No Risk" }, { "fpr": [ 0.0, 0.0, 0.0, 0.058823529411764705, 0.058823529411764705, 0.11764705882352941, 0.11764705882352941, 0.35294117647058826, 0.35294117647058826, 0.7058823529411765, 0.7058823529411765, 0.8823529411764706, 0.8823529411764706, 1.0 ], "thresholds": [ 1.8097286224365234, 0.8097286224365234, 0.7199095487594604, 0.6432027816772461, 0.6316874623298645, 0.5784026980400085, 0.5589774250984192, 0.3221668303012848, 0.28866666555404663, 0.15159079432487488, 0.14787857234477997, 0.08006153255701065, 0.07407616823911667, 0.06792321056127548 ], "tpr": [ 0.0, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 0.875, 0.875, 1.0, 1.0 ], "true_class": "Risk" } ] }, "classes": [ "No Risk", "Risk" ], "features_importance": [ { "computation_type": "", "features": { "LoanAmount": 0.68, "InstallmentPercent": 0.14, "ExistingSavings": 0.42, "CheckingStatus": 0.09, "LoanPurpose": 0.32, "NewFeature_0_featureagglomeration_0": 0.39, "Job": 0.04, "InstallmentPlans": 0.09, "OthersOnLoan": 0.43, "LoanDuration": 1.0, "Age": 0.87, "CreditHistory": 0.54, "CurrentResidenceDuration": 0.2, "ForeignWorker": 0.0, "Housing": 0.07, "ExistingCreditsCount": 0.51, "Telephone": 0.01, "NewFeature_1_featureagglomeration_1": 0.41, "Sex": 0.05, "Dependents": 0.03, "OwnsProperty": 0.26, "EmploymentDuration": 0.06 }, "stage": "Mean" } ], "intermediate_model": { "composition_steps": [ "TrainingDataset_full_249_20", "Split_TrainingHoldout", "TrainingDataset_full_224_20", "Preprocessor_default", "DAUB", "hpo_d", "cognito" ], "duration": 242615, "location": { "model": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/global_output/Pipeline2/model.pickle", "pipeline": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/global_output/Pipeline2/pipeline.json", "pipeline_model": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/global_output/Pipeline2/pipeline-model.json" }, "name": "P3", "notebook_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/global_output/Pipeline2/notebook.ipynb", "pipeline_nodes": [ "PreprocessingTransformer", "autoai_libs.cognito.transforms.transform_utils.featureagglomeration", "autoai_libs.cognito.transforms.transform_utils.FS1", "GradientBoostingClassifierEstimator" ], "process": "global_output", "schema_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/global_output/Pipeline2/schema.json", "sdk_notebook_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/global_output/Pipeline2/notebook_sdk.ipynb" }, "phase": "global_output" }, "ml_metrics": { "training_roc_auc": 0.8637123745819397, "holdout_precision": 0.5555555555555556, "training_average_precision": 0.7631005098061324, "holdout_average_precision": 0.6683999400675602, "training_neg_log_loss": -0.436711969891688, "holdout_recall": 0.625, "training_precision": 0.7331140350877193, "holdout_accuracy": 0.72, "holdout_balanced_accuracy": 0.6948529411764706, "training_recall": 0.5882740447957838, "holdout_f1": 0.5882352941176471, "holdout_neg_log_loss": -0.555290777346542, "training_accuracy": 0.8081681681681682, "holdout_roc_auc": 0.7205882352941176, "training_balanced_accuracy": 0.7460600993209688, "training_f1": 0.651842785257718 }, "timestamp": "2020-10-01T06:56:07.000Z" }, { "context": { "binary_classification": { "confusion_matrix": [ { "fn": 3, "fp": 3, "tn": 5, "tp": 14, "true_class": "No Risk" }, { "fn": 3, "fp": 3, "tn": 14, "tp": 5, "true_class": "Risk" } ], "roc_curve": [ { "fpr": [ 0.0, 0.0, 0.0, 0.125, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 1.0 ], "thresholds": [ 1.912172555923462, 0.9121726155281067, 0.9023290276527405, 0.9015228152275085, 0.8295201063156128, 0.8294777870178223, 0.6834437251091003, 0.6798041462898254, 0.4823266565799713, 0.4560561776161194, 0.40774452686309814, 0.40617018938064575, 0.39104729890823364, 0.28977835178375244 ], "tpr": [ 0.0, 0.058823529411764705, 0.17647058823529413, 0.17647058823529413, 0.4117647058823529, 0.4117647058823529, 0.6470588235294118, 0.6470588235294118, 0.8823529411764706, 0.8823529411764706, 0.9411764705882353, 0.9411764705882353, 1.0, 1.0 ], "true_class": "No Risk" }, { "fpr": [ 0.0, 0.0, 0.0, 0.058823529411764705, 0.058823529411764705, 0.11764705882352941, 0.11764705882352941, 0.35294117647058826, 0.35294117647058826, 0.5882352941176471, 0.5882352941176471, 0.8235294117647058, 0.8235294117647058, 1.0 ], "thresholds": [ 1.7102216482162476, 0.7102216482162476, 0.669156014919281, 0.6089527010917664, 0.5938298106193542, 0.5922554731369019, 0.5338093638420105, 0.3384079039096832, 0.32019585371017456, 0.18048915266990662, 0.17052221298217773, 0.11166291683912277, 0.09847716987133026, 0.0878273993730545 ], "tpr": [ 0.0, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 0.875, 0.875, 1.0, 1.0 ], "true_class": "Risk" } ] }, "classes": [ "No Risk", "Risk" ], "features_importance": [ { "computation_type": "", "features": { "LoanAmount": 0.68, "InstallmentPercent": 0.14, "ExistingSavings": 0.42, "CheckingStatus": 0.09, "LoanPurpose": 0.32, "NewFeature_0_featureagglomeration_0": 0.39, "Job": 0.04, "InstallmentPlans": 0.09, "OthersOnLoan": 0.43, "LoanDuration": 1.0, "Age": 0.87, "CreditHistory": 0.54, "CurrentResidenceDuration": 0.2, "ForeignWorker": 0.0, "Housing": 0.07, "ExistingCreditsCount": 0.51, "Telephone": 0.01, "NewFeature_1_featureagglomeration_1": 0.41, "Sex": 0.05, "Dependents": 0.03, "OwnsProperty": 0.26, "EmploymentDuration": 0.06 }, "stage": "Mean" } ], "intermediate_model": { "composition_steps": [ "TrainingDataset_full_249_20", "Split_TrainingHoldout", "TrainingDataset_full_224_20", "Preprocessor_default", "DAUB", "hpo_d", "cognito", "hpo_c" ], "duration": 242615, "location": { "model": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/global_output/Pipeline3/model.pickle", "pipeline": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/global_output/Pipeline3/pipeline.json", "pipeline_model": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/global_output/Pipeline3/pipeline-model.json" }, "name": "P4", "notebook_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/global_output/Pipeline3/notebook.ipynb", "pipeline_nodes": [ "PreprocessingTransformer", "autoai_libs.cognito.transforms.transform_utils.featureagglomeration", "autoai_libs.cognito.transforms.transform_utils.FS1", "GradientBoostingClassifierEstimator" ], "process": "global_output", "schema_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/global_output/Pipeline3/schema.json", "sdk_notebook_location": "./6e1bee43-062a-4d2b-9eeb-d4afd841a8ba/data/automl/global_output/Pipeline3/notebook_sdk.ipynb" }, "phase": "global_output" }, "ml_metrics": { "training_roc_auc": 0.8656126482213438, "holdout_precision": 0.625, "training_average_precision": 0.7583491228304152, "holdout_average_precision": 0.6757941813088872, "training_neg_log_loss": -0.4447612030777576, "holdout_recall": 0.625, "training_precision": 0.7137566137566139, "holdout_accuracy": 0.76, "holdout_balanced_accuracy": 0.7242647058823529, "training_recall": 0.5586297760210803, "holdout_f1": 0.625, "holdout_neg_log_loss": -0.5520562594922229, "training_accuracy": 0.7947747747747748, "holdout_roc_auc": 0.7426470588235294, "training_balanced_accuracy": 0.7280328367284888, "training_f1": 0.6245680270070514 }, "timestamp": "2020-10-01T06:56:07.000Z" } ], "running_at": "2020-10-01T06:56:07.000Z", "state": "completed" }, "tags": [ { "value": "autoai" } ], "training_data_references": [ { "location": { "href": "/v2/assets/f5531b4b-d1e8-44e7-a464-67164a2b8a6e?space_id=2219f598-6194-4e9a-9287-0730c69b5f4a" }, "type": "data_asset", "connection": {}, "id": "credit_risk_training_light.csv" } ] } }, { "metadata": { "created_at": "2020-09-30T14:33:40.223Z", "guid": "9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2", "id": "9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2", "modified_at": "2020-09-30T14:39:58.836Z", "space_id": "2219f598-6194-4e9a-9287-0730c69b5f4a", "tags": [ "autoai" ] }, "entity": { "pipeline": { "href": "/v4/pipelines/cf72a5a2-c51e-4c97-bd34-9e80cf818b5b", "id": "cf72a5a2-c51e-4c97-bd34-9e80cf818b5b" }, "results_reference": { "location": { "training": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2", "path": "/", "assets_path": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/assets", "training_status": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/training-status.json" }, "type": "fs", "connection": {}, "id": "autoai_results" }, "space_id": "2219f598-6194-4e9a-9287-0730c69b5f4a", "status": { "completed_at": "2020-09-30T14:39:58.827Z", "message": { "text": "Training job 9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2 completed", "level": "info" }, "metrics": [ { "context": { "binary_classification": { "confusion_matrix": [ { "fn": 2, "fp": 4, "tn": 4, "tp": 15, "true_class": "No Risk" }, { "fn": 4, "fp": 2, "tn": 15, "tp": 4, "true_class": "Risk" } ], "roc_curve": [ { "fpr": [ 0.0, 0.0, 0.0, 0.125, 0.125, 0.25, 0.25, 0.375, 0.375, 0.5, 0.5, 1.0 ], "thresholds": [ 1.9959793090820312, 0.9959793090820312, 0.9949495196342468, 0.9937530755996704, 0.9900901913642883, 0.9870622158050537, 0.9241466522216797, 0.8619304299354553, 0.7010563015937805, 0.5894654989242554, 0.23473195731639862, 0.010169494897127151 ], "tpr": [ 0.0, 0.058823529411764705, 0.11764705882352941, 0.11764705882352941, 0.23529411764705882, 0.23529411764705882, 0.47058823529411764, 0.47058823529411764, 0.7058823529411765, 0.7058823529411765, 1.0, 1.0 ], "true_class": "No Risk" }, { "fpr": [ 0.0, 0.0, 0.0, 0.29411764705882354, 0.29411764705882354, 0.4117647058823529, 0.4117647058823529, 0.7058823529411765, 0.7058823529411765, 0.8235294117647058, 0.8235294117647058, 1.0 ], "thresholds": [ 1.9907909631729126, 0.9907909631729126, 0.9192749261856079, 0.37951037287712097, 0.3295697271823883, 0.15989628434181213, 0.11566814035177231, 0.03620442748069763, 0.02297501638531685, 0.010295126587152481, 0.007908087223768234, 0.003513400675728917 ], "tpr": [ 0.0, 0.125, 0.5, 0.5, 0.625, 0.625, 0.75, 0.75, 0.875, 0.875, 1.0, 1.0 ], "true_class": "Risk" } ] }, "classes": [ "No Risk", "Risk" ], "features_importance": [ { "computation_type": "", "features": { "LoanAmount": 0.47, "InstallmentPercent": 0.06, "ExistingSavings": 0.16, "CheckingStatus": 0.05, "LoanPurpose": 0.23, "Job": 0.04, "InstallmentPlans": 0.0, "OthersOnLoan": 0.14, "LoanDuration": 0.24, "Age": 1.0, "CreditHistory": 0.13, "CurrentResidenceDuration": 0.16, "ForeignWorker": 0.01, "Housing": 0.0, "ExistingCreditsCount": 0.07, "Telephone": 0.0, "Sex": 0.01, "Dependents": 0.08, "OwnsProperty": 0.04, "EmploymentDuration": 0.11 }, "stage": "Mean" } ], "intermediate_model": { "composition_steps": [ "TrainingDataset_full_249_20", "Split_TrainingHoldout", "TrainingDataset_full_224_20", "Preprocessor_default", "DAUB" ], "duration": 760, "location": { "model": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/pre_hpo_d_output/Pipeline1/model.pickle", "pipeline": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/pre_hpo_d_output/Pipeline1/pipeline.json", "pipeline_model": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/pre_hpo_d_output/Pipeline1/pipeline-model.json" }, "name": "P1", "notebook_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/pre_hpo_d_output/Pipeline1/notebook.ipynb", "pipeline_nodes": [ "PreprocessingTransformer", "GradientBoostingClassifierEstimator" ], "process": "pre_hpo_d_output", "schema_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/pre_hpo_d_output/Pipeline1/schema.json", "sdk_notebook_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/pre_hpo_d_output/Pipeline1/notebook_sdk.ipynb" }, "phase": "pre_hpo_d_output" }, "ml_metrics": { "training_roc_auc": 0.8283546163980947, "holdout_precision": 0.6666666666666666, "training_average_precision": 0.6996753806435804, "holdout_average_precision": 0.7116994847258005, "training_neg_log_loss": -0.5747366162076947, "holdout_recall": 0.5, "training_precision": 0.6439393939393939, "holdout_accuracy": 0.76, "holdout_balanced_accuracy": 0.6911764705882353, "training_recall": 0.5718050065876153, "holdout_f1": 0.5714285714285715, "holdout_neg_log_loss": -0.7066552317239467, "training_accuracy": 0.7723123123123123, "holdout_roc_auc": 0.7205882352941178, "training_balanced_accuracy": 0.715389682780987, "training_f1": 0.6009839626860902 }, "timestamp": "2020-09-30T14:35:19.000Z" }, { "context": { "binary_classification": { "confusion_matrix": [ { "fn": 2, "fp": 3, "tn": 5, "tp": 15, "true_class": "No Risk" }, { "fn": 3, "fp": 2, "tn": 15, "tp": 5, "true_class": "Risk" } ], "roc_curve": [ { "fpr": [ 0.0, 0.0, 0.0, 0.125, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 1.0 ], "thresholds": [ 1.9118249416351318, 0.9118249416351318, 0.8851636648178101, 0.8839583992958069, 0.8572619557380676, 0.8568741083145142, 0.7981584072113037, 0.7974665760993958, 0.529254138469696, 0.4382297098636627, 0.37201759219169617, 0.24074923992156982 ], "tpr": [ 0.0, 0.058823529411764705, 0.29411764705882354, 0.29411764705882354, 0.35294117647058826, 0.35294117647058826, 0.5882352941176471, 0.5882352941176471, 0.8823529411764706, 0.8823529411764706, 1.0, 1.0 ], "true_class": "No Risk" }, { "fpr": [ 0.0, 0.0, 0.0, 0.11764705882352941, 0.11764705882352941, 0.4117647058823529, 0.4117647058823529, 0.6470588235294118, 0.6470588235294118, 0.7058823529411765, 0.7058823529411765, 1.0 ], "thresholds": [ 1.7592507600784302, 0.7592507600784302, 0.6614384055137634, 0.5711529850959778, 0.529679536819458, 0.29737383127212524, 0.20253342390060425, 0.14502882957458496, 0.14312586188316345, 0.14273804426193237, 0.11604157835245132, 0.08817508071660995 ], "tpr": [ 0.0, 0.125, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 0.875, 0.875, 1.0, 1.0 ], "true_class": "Risk" } ] }, "classes": [ "No Risk", "Risk" ], "features_importance": [ { "computation_type": "", "features": { "LoanAmount": 1.0, "InstallmentPercent": 0.32, "ExistingSavings": 0.09, "CheckingStatus": 0.17, "LoanPurpose": 0.16, "Job": 0.01, "InstallmentPlans": 0.07, "OthersOnLoan": 0.37, "LoanDuration": 0.43, "Age": 0.51, "CreditHistory": 0.26, "CurrentResidenceDuration": 0.18, "ForeignWorker": 0.0, "Housing": 0.08, "ExistingCreditsCount": 0.22, "Telephone": 0.06, "Sex": 0.2, "Dependents": 0.23, "OwnsProperty": 0.15, "EmploymentDuration": 0.22 }, "stage": "Mean" } ], "intermediate_model": { "composition_steps": [ "TrainingDataset_full_249_20", "Split_TrainingHoldout", "TrainingDataset_full_224_20", "Preprocessor_default", "DAUB", "hpo_d" ], "duration": 5151, "location": { "model": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/hpo_d_output/Pipeline1/model.pickle", "pipeline": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/hpo_d_output/Pipeline1/pipeline.json", "pipeline_model": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/hpo_d_output/Pipeline1/pipeline-model.json" }, "name": "P2", "notebook_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/hpo_d_output/Pipeline1/notebook.ipynb", "pipeline_nodes": [ "PreprocessingTransformer", "GradientBoostingClassifierEstimator" ], "process": "hpo_d_output", "schema_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/hpo_d_output/Pipeline1/schema.json", "sdk_notebook_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/hpo_d_output/Pipeline1/notebook_sdk.ipynb" }, "phase": "hpo_d_output" }, "ml_metrics": { "training_roc_auc": 0.8566053511705686, "holdout_precision": 0.7142857142857143, "training_average_precision": 0.7320733111904305, "holdout_average_precision": 0.7039224664224665, "training_neg_log_loss": -0.4465285582613328, "holdout_recall": 0.625, "training_precision": 0.7780214424951266, "holdout_accuracy": 0.8, "holdout_balanced_accuracy": 0.7536764705882353, "training_recall": 0.49736495388669294, "holdout_f1": 0.6666666666666666, "holdout_neg_log_loss": -0.5421721597291818, "training_accuracy": 0.7990390390390391, "holdout_roc_auc": 0.75, "training_balanced_accuracy": 0.7134260666869362, "training_f1": 0.589159891598916 }, "timestamp": "2020-09-30T14:35:40.000Z" }, { "context": { "binary_classification": { "confusion_matrix": [ { "fn": 4, "fp": 3, "tn": 5, "tp": 13, "true_class": "No Risk" }, { "fn": 3, "fp": 4, "tn": 13, "tp": 5, "true_class": "Risk" } ], "roc_curve": [ { "fpr": [ 0.0, 0.0, 0.0, 0.125, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 1.0 ], "thresholds": [ 1.9320768117904663, 0.9320768117904663, 0.9299767017364502, 0.9259238243103027, 0.8939604759216309, 0.8521214127540588, 0.7138300538063049, 0.7113333344459534, 0.45716628432273865, 0.43239983916282654, 0.42159730195999146, 0.3683125376701355, 0.3567972481250763, 0.19027136266231537 ], "tpr": [ 0.0, 0.058823529411764705, 0.11764705882352941, 0.11764705882352941, 0.29411764705882354, 0.29411764705882354, 0.6470588235294118, 0.6470588235294118, 0.8823529411764706, 0.8823529411764706, 0.9411764705882353, 0.9411764705882353, 1.0, 1.0 ], "true_class": "No Risk" }, { "fpr": [ 0.0, 0.0, 0.0, 0.058823529411764705, 0.058823529411764705, 0.11764705882352941, 0.11764705882352941, 0.35294117647058826, 0.35294117647058826, 0.7058823529411765, 0.7058823529411765, 0.8823529411764706, 0.8823529411764706, 1.0 ], "thresholds": [ 1.8097286224365234, 0.8097286224365234, 0.7199095487594604, 0.6432027816772461, 0.6316874623298645, 0.5784026980400085, 0.5589774250984192, 0.3221668303012848, 0.28866666555404663, 0.15159079432487488, 0.14787857234477997, 0.08006153255701065, 0.07407616823911667, 0.06792321056127548 ], "tpr": [ 0.0, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 0.875, 0.875, 1.0, 1.0 ], "true_class": "Risk" } ] }, "classes": [ "No Risk", "Risk" ], "features_importance": [ { "computation_type": "", "features": { "LoanAmount": 0.86, "InstallmentPercent": 0.14, "ExistingSavings": 0.42, "CheckingStatus": 0.1, "LoanPurpose": 0.36, "NewFeature_0_featureagglomeration_0": 0.39, "Job": 0.04, "InstallmentPlans": 0.1, "OthersOnLoan": 0.43, "LoanDuration": 1.0, "Age": 0.99, "CreditHistory": 0.57, "CurrentResidenceDuration": 0.2, "ForeignWorker": 0.0, "Housing": 0.07, "ExistingCreditsCount": 0.59, "Telephone": 0.06, "NewFeature_1_featureagglomeration_1": 0.49, "Sex": 0.09, "Dependents": 0.08, "OwnsProperty": 0.28, "EmploymentDuration": 0.06 }, "stage": "Mean" } ], "intermediate_model": { "composition_steps": [ "TrainingDataset_full_249_20", "Split_TrainingHoldout", "TrainingDataset_full_224_20", "Preprocessor_default", "DAUB", "hpo_d", "cognito" ], "duration": 26778, "location": { "model": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/cognito_output/Pipeline1/model.pickle", "pipeline": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/cognito_output/Pipeline1/pipeline.json", "pipeline_model": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/cognito_output/Pipeline1/pipeline-model.json" }, "name": "P3", "notebook_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/cognito_output/Pipeline1/notebook.ipynb", "pipeline_nodes": [ "PreprocessingTransformer", "autoai_libs.cognito.transforms.transform_utils.featureagglomeration", "autoai_libs.cognito.transforms.transform_utils.FS1", "GradientBoostingClassifierEstimator" ], "process": "cognito_output", "schema_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/cognito_output/Pipeline1/schema.json", "sdk_notebook_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/cognito_output/Pipeline1/notebook_sdk.ipynb" }, "phase": "cognito_output" }, "ml_metrics": { "training_roc_auc": 0.8637123745819397, "holdout_precision": 0.5555555555555556, "training_average_precision": 0.7631005098061324, "holdout_average_precision": 0.6683999400675602, "training_neg_log_loss": -0.436711969891688, "holdout_recall": 0.625, "training_precision": 0.7331140350877193, "holdout_accuracy": 0.72, "holdout_balanced_accuracy": 0.6948529411764706, "training_recall": 0.5882740447957838, "holdout_f1": 0.5882352941176471, "holdout_neg_log_loss": -0.555290777346542, "training_accuracy": 0.8081681681681682, "holdout_roc_auc": 0.7205882352941176, "training_balanced_accuracy": 0.7460600993209688, "training_f1": 0.651842785257718 }, "timestamp": "2020-09-30T14:36:23.000Z" }, { "context": { "binary_classification": { "confusion_matrix": [ { "fn": 3, "fp": 3, "tn": 5, "tp": 14, "true_class": "No Risk" }, { "fn": 3, "fp": 3, "tn": 14, "tp": 5, "true_class": "Risk" } ], "roc_curve": [ { "fpr": [ 0.0, 0.0, 0.0, 0.125, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 1.0 ], "thresholds": [ 1.912172555923462, 0.9121726155281067, 0.9023290276527405, 0.9015228152275085, 0.8295201063156128, 0.8294777870178223, 0.6834437251091003, 0.6798041462898254, 0.4823266565799713, 0.4560561776161194, 0.40774452686309814, 0.40617018938064575, 0.39104729890823364, 0.28977835178375244 ], "tpr": [ 0.0, 0.058823529411764705, 0.17647058823529413, 0.17647058823529413, 0.4117647058823529, 0.4117647058823529, 0.6470588235294118, 0.6470588235294118, 0.8823529411764706, 0.8823529411764706, 0.9411764705882353, 0.9411764705882353, 1.0, 1.0 ], "true_class": "No Risk" }, { "fpr": [ 0.0, 0.0, 0.0, 0.058823529411764705, 0.058823529411764705, 0.11764705882352941, 0.11764705882352941, 0.35294117647058826, 0.35294117647058826, 0.5882352941176471, 0.5882352941176471, 0.8235294117647058, 0.8235294117647058, 1.0 ], "thresholds": [ 1.7102216482162476, 0.7102216482162476, 0.669156014919281, 0.6089527010917664, 0.5938298106193542, 0.5922554731369019, 0.5338093638420105, 0.3384079039096832, 0.32019585371017456, 0.18048915266990662, 0.17052221298217773, 0.11166291683912277, 0.09847716987133026, 0.0878273993730545 ], "tpr": [ 0.0, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 0.875, 0.875, 1.0, 1.0 ], "true_class": "Risk" } ] }, "classes": [ "No Risk", "Risk" ], "features_importance": [ { "computation_type": "", "features": { "LoanAmount": 0.68, "InstallmentPercent": 0.14, "ExistingSavings": 0.42, "CheckingStatus": 0.09, "LoanPurpose": 0.32, "NewFeature_0_featureagglomeration_0": 0.39, "Job": 0.04, "InstallmentPlans": 0.09, "OthersOnLoan": 0.43, "LoanDuration": 1.0, "Age": 0.87, "CreditHistory": 0.54, "CurrentResidenceDuration": 0.2, "ForeignWorker": 0.0, "Housing": 0.07, "ExistingCreditsCount": 0.51, "Telephone": 0.01, "NewFeature_1_featureagglomeration_1": 0.41, "Sex": 0.05, "Dependents": 0.03, "OwnsProperty": 0.26, "EmploymentDuration": 0.06 }, "stage": "Mean" } ], "intermediate_model": { "composition_steps": [ "TrainingDataset_full_249_20", "Split_TrainingHoldout", "TrainingDataset_full_224_20", "Preprocessor_default", "DAUB", "hpo_d", "cognito", "hpo_c" ], "duration": 9322, "location": { "model": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/hpo_c_output/Pipeline1/model.pickle", "pipeline": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/hpo_c_output/Pipeline1/pipeline.json", "pipeline_model": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/hpo_c_output/Pipeline1/pipeline-model.json" }, "name": "P4", "notebook_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/hpo_c_output/Pipeline1/notebook.ipynb", "pipeline_nodes": [ "PreprocessingTransformer", "autoai_libs.cognito.transforms.transform_utils.featureagglomeration", "autoai_libs.cognito.transforms.transform_utils.FS1", "GradientBoostingClassifierEstimator" ], "process": "hpo_c_output", "schema_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/hpo_c_output/Pipeline1/schema.json", "sdk_notebook_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/hpo_c_output/Pipeline1/notebook_sdk.ipynb" }, "phase": "hpo_c_output" }, "ml_metrics": { "training_roc_auc": 0.8656126482213438, "holdout_precision": 0.625, "training_average_precision": 0.7583491228304152, "holdout_average_precision": 0.6757941813088872, "training_neg_log_loss": -0.4447612030777576, "holdout_recall": 0.625, "training_precision": 0.7137566137566139, "holdout_accuracy": 0.76, "holdout_balanced_accuracy": 0.7242647058823529, "training_recall": 0.5586297760210803, "holdout_f1": 0.625, "holdout_neg_log_loss": -0.5520562594922229, "training_accuracy": 0.7947747747747748, "holdout_roc_auc": 0.7426470588235294, "training_balanced_accuracy": 0.7280328367284888, "training_f1": 0.6245680270070514 }, "timestamp": "2020-09-30T14:36:48.000Z" }, { "context": { "binary_classification": { "confusion_matrix": [ { "fn": 2, "fp": 4, "tn": 4, "tp": 15, "true_class": "No Risk" }, { "fn": 4, "fp": 2, "tn": 15, "tp": 4, "true_class": "Risk" } ], "roc_curve": [ { "fpr": [ 0.0, 0.0, 0.0, 0.125, 0.125, 0.25, 0.25, 0.375, 0.375, 0.5, 0.5, 1.0 ], "thresholds": [ 1.9959793090820312, 0.9959793090820312, 0.9949495196342468, 0.9937530755996704, 0.9900901913642883, 0.9870622158050537, 0.9241466522216797, 0.8619304299354553, 0.7010563015937805, 0.5894654989242554, 0.23473195731639862, 0.010169494897127151 ], "tpr": [ 0.0, 0.058823529411764705, 0.11764705882352941, 0.11764705882352941, 0.23529411764705882, 0.23529411764705882, 0.47058823529411764, 0.47058823529411764, 0.7058823529411765, 0.7058823529411765, 1.0, 1.0 ], "true_class": "No Risk" }, { "fpr": [ 0.0, 0.0, 0.0, 0.29411764705882354, 0.29411764705882354, 0.4117647058823529, 0.4117647058823529, 0.7058823529411765, 0.7058823529411765, 0.8235294117647058, 0.8235294117647058, 1.0 ], "thresholds": [ 1.9907909631729126, 0.9907909631729126, 0.9192749261856079, 0.37951037287712097, 0.3295697271823883, 0.15989628434181213, 0.11566814035177231, 0.03620442748069763, 0.02297501638531685, 0.010295126587152481, 0.007908087223768234, 0.003513400675728917 ], "tpr": [ 0.0, 0.125, 0.5, 0.5, 0.625, 0.625, 0.75, 0.75, 0.875, 0.875, 1.0, 1.0 ], "true_class": "Risk" } ] }, "classes": [ "No Risk", "Risk" ], "features_importance": [ { "computation_type": "", "features": { "LoanAmount": 0.68, "InstallmentPercent": 0.14, "ExistingSavings": 0.42, "CheckingStatus": 0.09, "LoanPurpose": 0.32, "NewFeature_0_featureagglomeration_0": 0.39, "Job": 0.04, "InstallmentPlans": 0.09, "OthersOnLoan": 0.43, "LoanDuration": 1.0, "Age": 0.87, "CreditHistory": 0.54, "CurrentResidenceDuration": 0.2, "ForeignWorker": 0.0, "Housing": 0.07, "ExistingCreditsCount": 0.51, "Telephone": 0.01, "NewFeature_1_featureagglomeration_1": 0.41, "Sex": 0.05, "Dependents": 0.03, "OwnsProperty": 0.26, "EmploymentDuration": 0.06 }, "stage": "Mean" } ], "intermediate_model": { "composition_steps": [ "TrainingDataset_full_249_20", "Split_TrainingHoldout", "TrainingDataset_full_224_20", "Preprocessor_default", "DAUB" ], "duration": 105598, "location": { "model": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/compose_model_type_output/Pipeline0/model.pickle", "pipeline": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/compose_model_type_output/Pipeline0/pipeline.json", "pipeline_model": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/compose_model_type_output/Pipeline0/pipeline-model.json" }, "name": "P1", "notebook_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/compose_model_type_output/Pipeline0/notebook.ipynb", "pipeline_nodes": [ "PreprocessingTransformer", "GradientBoostingClassifierEstimator" ], "process": "compose_model_type_output", "schema_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/compose_model_type_output/Pipeline0/schema.json", "sdk_notebook_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/compose_model_type_output/Pipeline0/notebook_sdk.ipynb" }, "phase": "compose_model_type_output" }, "ml_metrics": { "training_roc_auc": 0.8283546163980947, "holdout_precision": 0.6666666666666666, "training_average_precision": 0.6996753806435804, "holdout_average_precision": 0.7116994847258005, "training_neg_log_loss": -0.5747366162076947, "holdout_recall": 0.5, "training_precision": 0.6439393939393939, "holdout_accuracy": 0.76, "holdout_balanced_accuracy": 0.6911764705882353, "training_recall": 0.5718050065876153, "holdout_f1": 0.5714285714285715, "holdout_neg_log_loss": -0.7066552317239467, "training_accuracy": 0.7723123123123123, "holdout_roc_auc": 0.7205882352941178, "training_balanced_accuracy": 0.715389682780987, "training_f1": 0.6009839626860902 }, "timestamp": "2020-09-30T14:37:51.000Z" }, { "context": { "binary_classification": { "confusion_matrix": [ { "fn": 2, "fp": 3, "tn": 5, "tp": 15, "true_class": "No Risk" }, { "fn": 3, "fp": 2, "tn": 15, "tp": 5, "true_class": "Risk" } ], "roc_curve": [ { "fpr": [ 0.0, 0.0, 0.0, 0.125, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 1.0 ], "thresholds": [ 1.9118249416351318, 0.9118249416351318, 0.8851636648178101, 0.8839583992958069, 0.8572619557380676, 0.8568741083145142, 0.7981584072113037, 0.7974665760993958, 0.529254138469696, 0.4382297098636627, 0.37201759219169617, 0.24074923992156982 ], "tpr": [ 0.0, 0.058823529411764705, 0.29411764705882354, 0.29411764705882354, 0.35294117647058826, 0.35294117647058826, 0.5882352941176471, 0.5882352941176471, 0.8823529411764706, 0.8823529411764706, 1.0, 1.0 ], "true_class": "No Risk" }, { "fpr": [ 0.0, 0.0, 0.0, 0.11764705882352941, 0.11764705882352941, 0.4117647058823529, 0.4117647058823529, 0.6470588235294118, 0.6470588235294118, 0.7058823529411765, 0.7058823529411765, 1.0 ], "thresholds": [ 1.7592507600784302, 0.7592507600784302, 0.6614384055137634, 0.5711529850959778, 0.529679536819458, 0.29737383127212524, 0.20253342390060425, 0.14502882957458496, 0.14312586188316345, 0.14273804426193237, 0.11604157835245132, 0.08817508071660995 ], "tpr": [ 0.0, 0.125, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 0.875, 0.875, 1.0, 1.0 ], "true_class": "Risk" } ] }, "classes": [ "No Risk", "Risk" ], "features_importance": [ { "computation_type": "", "features": { "LoanAmount": 0.68, "InstallmentPercent": 0.14, "ExistingSavings": 0.42, "CheckingStatus": 0.09, "LoanPurpose": 0.32, "NewFeature_0_featureagglomeration_0": 0.39, "Job": 0.04, "InstallmentPlans": 0.09, "OthersOnLoan": 0.43, "LoanDuration": 1.0, "Age": 0.87, "CreditHistory": 0.54, "CurrentResidenceDuration": 0.2, "ForeignWorker": 0.0, "Housing": 0.07, "ExistingCreditsCount": 0.51, "Telephone": 0.01, "NewFeature_1_featureagglomeration_1": 0.41, "Sex": 0.05, "Dependents": 0.03, "OwnsProperty": 0.26, "EmploymentDuration": 0.06 }, "stage": "Mean" } ], "intermediate_model": { "composition_steps": [ "TrainingDataset_full_249_20", "Split_TrainingHoldout", "TrainingDataset_full_224_20", "Preprocessor_default", "DAUB", "hpo_d" ], "duration": 105598, "location": { "model": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/compose_model_type_output/Pipeline1/model.pickle", "pipeline": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/compose_model_type_output/Pipeline1/pipeline.json", "pipeline_model": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/compose_model_type_output/Pipeline1/pipeline-model.json" }, "name": "P2", "notebook_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/compose_model_type_output/Pipeline1/notebook.ipynb", "pipeline_nodes": [ "PreprocessingTransformer", "GradientBoostingClassifierEstimator" ], "process": "compose_model_type_output", "schema_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/compose_model_type_output/Pipeline1/schema.json", "sdk_notebook_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/compose_model_type_output/Pipeline1/notebook_sdk.ipynb" }, "phase": "compose_model_type_output" }, "ml_metrics": { "training_roc_auc": 0.8566053511705686, "holdout_precision": 0.7142857142857143, "training_average_precision": 0.7320733111904305, "holdout_average_precision": 0.7039224664224665, "training_neg_log_loss": -0.4465285582613328, "holdout_recall": 0.625, "training_precision": 0.7780214424951266, "holdout_accuracy": 0.8, "holdout_balanced_accuracy": 0.7536764705882353, "training_recall": 0.49736495388669294, "holdout_f1": 0.6666666666666666, "holdout_neg_log_loss": -0.5421721597291818, "training_accuracy": 0.7990390390390391, "holdout_roc_auc": 0.75, "training_balanced_accuracy": 0.7134260666869362, "training_f1": 0.589159891598916 }, "timestamp": "2020-09-30T14:37:51.000Z" }, { "context": { "binary_classification": { "confusion_matrix": [ { "fn": 4, "fp": 3, "tn": 5, "tp": 13, "true_class": "No Risk" }, { "fn": 3, "fp": 4, "tn": 13, "tp": 5, "true_class": "Risk" } ], "roc_curve": [ { "fpr": [ 0.0, 0.0, 0.0, 0.125, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 1.0 ], "thresholds": [ 1.9320768117904663, 0.9320768117904663, 0.9299767017364502, 0.9259238243103027, 0.8939604759216309, 0.8521214127540588, 0.7138300538063049, 0.7113333344459534, 0.45716628432273865, 0.43239983916282654, 0.42159730195999146, 0.3683125376701355, 0.3567972481250763, 0.19027136266231537 ], "tpr": [ 0.0, 0.058823529411764705, 0.11764705882352941, 0.11764705882352941, 0.29411764705882354, 0.29411764705882354, 0.6470588235294118, 0.6470588235294118, 0.8823529411764706, 0.8823529411764706, 0.9411764705882353, 0.9411764705882353, 1.0, 1.0 ], "true_class": "No Risk" }, { "fpr": [ 0.0, 0.0, 0.0, 0.058823529411764705, 0.058823529411764705, 0.11764705882352941, 0.11764705882352941, 0.35294117647058826, 0.35294117647058826, 0.7058823529411765, 0.7058823529411765, 0.8823529411764706, 0.8823529411764706, 1.0 ], "thresholds": [ 1.8097286224365234, 0.8097286224365234, 0.7199095487594604, 0.6432027816772461, 0.6316874623298645, 0.5784026980400085, 0.5589774250984192, 0.3221668303012848, 0.28866666555404663, 0.15159079432487488, 0.14787857234477997, 0.08006153255701065, 0.07407616823911667, 0.06792321056127548 ], "tpr": [ 0.0, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 0.875, 0.875, 1.0, 1.0 ], "true_class": "Risk" } ] }, "classes": [ "No Risk", "Risk" ], "features_importance": [ { "computation_type": "", "features": { "LoanAmount": 0.68, "InstallmentPercent": 0.14, "ExistingSavings": 0.42, "CheckingStatus": 0.09, "LoanPurpose": 0.32, "NewFeature_0_featureagglomeration_0": 0.39, "Job": 0.04, "InstallmentPlans": 0.09, "OthersOnLoan": 0.43, "LoanDuration": 1.0, "Age": 0.87, "CreditHistory": 0.54, "CurrentResidenceDuration": 0.2, "ForeignWorker": 0.0, "Housing": 0.07, "ExistingCreditsCount": 0.51, "Telephone": 0.01, "NewFeature_1_featureagglomeration_1": 0.41, "Sex": 0.05, "Dependents": 0.03, "OwnsProperty": 0.26, "EmploymentDuration": 0.06 }, "stage": "Mean" } ], "intermediate_model": { "composition_steps": [ "TrainingDataset_full_249_20", "Split_TrainingHoldout", "TrainingDataset_full_224_20", "Preprocessor_default", "DAUB", "hpo_d", "cognito" ], "duration": 105598, "location": { "model": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/compose_model_type_output/Pipeline2/model.pickle", "pipeline": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/compose_model_type_output/Pipeline2/pipeline.json", "pipeline_model": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/compose_model_type_output/Pipeline2/pipeline-model.json" }, "name": "P3", "notebook_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/compose_model_type_output/Pipeline2/notebook.ipynb", "pipeline_nodes": [ "PreprocessingTransformer", "autoai_libs.cognito.transforms.transform_utils.featureagglomeration", "autoai_libs.cognito.transforms.transform_utils.FS1", "GradientBoostingClassifierEstimator" ], "process": "compose_model_type_output", "schema_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/compose_model_type_output/Pipeline2/schema.json", "sdk_notebook_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/compose_model_type_output/Pipeline2/notebook_sdk.ipynb" }, "phase": "compose_model_type_output" }, "ml_metrics": { "training_roc_auc": 0.8637123745819397, "holdout_precision": 0.5555555555555556, "training_average_precision": 0.7631005098061324, "holdout_average_precision": 0.6683999400675602, "training_neg_log_loss": -0.436711969891688, "holdout_recall": 0.625, "training_precision": 0.7331140350877193, "holdout_accuracy": 0.72, "holdout_balanced_accuracy": 0.6948529411764706, "training_recall": 0.5882740447957838, "holdout_f1": 0.5882352941176471, "holdout_neg_log_loss": -0.555290777346542, "training_accuracy": 0.8081681681681682, "holdout_roc_auc": 0.7205882352941176, "training_balanced_accuracy": 0.7460600993209688, "training_f1": 0.651842785257718 }, "timestamp": "2020-09-30T14:37:51.000Z" }, { "context": { "binary_classification": { "confusion_matrix": [ { "fn": 3, "fp": 3, "tn": 5, "tp": 14, "true_class": "No Risk" }, { "fn": 3, "fp": 3, "tn": 14, "tp": 5, "true_class": "Risk" } ], "roc_curve": [ { "fpr": [ 0.0, 0.0, 0.0, 0.125, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 1.0 ], "thresholds": [ 1.912172555923462, 0.9121726155281067, 0.9023290276527405, 0.9015228152275085, 0.8295201063156128, 0.8294777870178223, 0.6834437251091003, 0.6798041462898254, 0.4823266565799713, 0.4560561776161194, 0.40774452686309814, 0.40617018938064575, 0.39104729890823364, 0.28977835178375244 ], "tpr": [ 0.0, 0.058823529411764705, 0.17647058823529413, 0.17647058823529413, 0.4117647058823529, 0.4117647058823529, 0.6470588235294118, 0.6470588235294118, 0.8823529411764706, 0.8823529411764706, 0.9411764705882353, 0.9411764705882353, 1.0, 1.0 ], "true_class": "No Risk" }, { "fpr": [ 0.0, 0.0, 0.0, 0.058823529411764705, 0.058823529411764705, 0.11764705882352941, 0.11764705882352941, 0.35294117647058826, 0.35294117647058826, 0.5882352941176471, 0.5882352941176471, 0.8235294117647058, 0.8235294117647058, 1.0 ], "thresholds": [ 1.7102216482162476, 0.7102216482162476, 0.669156014919281, 0.6089527010917664, 0.5938298106193542, 0.5922554731369019, 0.5338093638420105, 0.3384079039096832, 0.32019585371017456, 0.18048915266990662, 0.17052221298217773, 0.11166291683912277, 0.09847716987133026, 0.0878273993730545 ], "tpr": [ 0.0, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 0.875, 0.875, 1.0, 1.0 ], "true_class": "Risk" } ] }, "classes": [ "No Risk", "Risk" ], "features_importance": [ { "computation_type": "", "features": { "LoanAmount": 0.68, "InstallmentPercent": 0.14, "ExistingSavings": 0.42, "CheckingStatus": 0.09, "LoanPurpose": 0.32, "NewFeature_0_featureagglomeration_0": 0.39, "Job": 0.04, "InstallmentPlans": 0.09, "OthersOnLoan": 0.43, "LoanDuration": 1.0, "Age": 0.87, "CreditHistory": 0.54, "CurrentResidenceDuration": 0.2, "ForeignWorker": 0.0, "Housing": 0.07, "ExistingCreditsCount": 0.51, "Telephone": 0.01, "NewFeature_1_featureagglomeration_1": 0.41, "Sex": 0.05, "Dependents": 0.03, "OwnsProperty": 0.26, "EmploymentDuration": 0.06 }, "stage": "Mean" } ], "intermediate_model": { "composition_steps": [ "TrainingDataset_full_249_20", "Split_TrainingHoldout", "TrainingDataset_full_224_20", "Preprocessor_default", "DAUB", "hpo_d", "cognito", "hpo_c" ], "duration": 105598, "location": { "model": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/compose_model_type_output/Pipeline3/model.pickle", "pipeline": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/compose_model_type_output/Pipeline3/pipeline.json", "pipeline_model": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/compose_model_type_output/Pipeline3/pipeline-model.json" }, "name": "P4", "notebook_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/compose_model_type_output/Pipeline3/notebook.ipynb", "pipeline_nodes": [ "PreprocessingTransformer", "autoai_libs.cognito.transforms.transform_utils.featureagglomeration", "autoai_libs.cognito.transforms.transform_utils.FS1", "GradientBoostingClassifierEstimator" ], "process": "compose_model_type_output", "schema_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/compose_model_type_output/Pipeline3/schema.json", "sdk_notebook_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/compose_model_type_output/Pipeline3/notebook_sdk.ipynb" }, "phase": "compose_model_type_output" }, "ml_metrics": { "training_roc_auc": 0.8656126482213438, "holdout_precision": 0.625, "training_average_precision": 0.7583491228304152, "holdout_average_precision": 0.6757941813088872, "training_neg_log_loss": -0.4447612030777576, "holdout_recall": 0.625, "training_precision": 0.7137566137566139, "holdout_accuracy": 0.76, "holdout_balanced_accuracy": 0.7242647058823529, "training_recall": 0.5586297760210803, "holdout_f1": 0.625, "holdout_neg_log_loss": -0.5520562594922229, "training_accuracy": 0.7947747747747748, "holdout_roc_auc": 0.7426470588235294, "training_balanced_accuracy": 0.7280328367284888, "training_f1": 0.6245680270070514 }, "timestamp": "2020-09-30T14:37:51.000Z" }, { "context": { "binary_classification": { "confusion_matrix": [ { "fn": 2, "fp": 4, "tn": 4, "tp": 15, "true_class": "No Risk" }, { "fn": 4, "fp": 2, "tn": 15, "tp": 4, "true_class": "Risk" } ], "roc_curve": [ { "fpr": [ 0.0, 0.0, 0.0, 0.125, 0.125, 0.25, 0.25, 0.375, 0.375, 0.5, 0.5, 1.0 ], "thresholds": [ 1.9959793090820312, 0.9959793090820312, 0.9949495196342468, 0.9937530755996704, 0.9900901913642883, 0.9870622158050537, 0.9241466522216797, 0.8619304299354553, 0.7010563015937805, 0.5894654989242554, 0.23473195731639862, 0.010169494897127151 ], "tpr": [ 0.0, 0.058823529411764705, 0.11764705882352941, 0.11764705882352941, 0.23529411764705882, 0.23529411764705882, 0.47058823529411764, 0.47058823529411764, 0.7058823529411765, 0.7058823529411765, 1.0, 1.0 ], "true_class": "No Risk" }, { "fpr": [ 0.0, 0.0, 0.0, 0.29411764705882354, 0.29411764705882354, 0.4117647058823529, 0.4117647058823529, 0.7058823529411765, 0.7058823529411765, 0.8235294117647058, 0.8235294117647058, 1.0 ], "thresholds": [ 1.9907909631729126, 0.9907909631729126, 0.9192749261856079, 0.37951037287712097, 0.3295697271823883, 0.15989628434181213, 0.11566814035177231, 0.03620442748069763, 0.02297501638531685, 0.010295126587152481, 0.007908087223768234, 0.003513400675728917 ], "tpr": [ 0.0, 0.125, 0.5, 0.5, 0.625, 0.625, 0.75, 0.75, 0.875, 0.875, 1.0, 1.0 ], "true_class": "Risk" } ] }, "classes": [ "No Risk", "Risk" ], "features_importance": [ { "computation_type": "", "features": { "LoanAmount": 0.68, "InstallmentPercent": 0.14, "ExistingSavings": 0.42, "CheckingStatus": 0.09, "LoanPurpose": 0.32, "NewFeature_0_featureagglomeration_0": 0.39, "Job": 0.04, "InstallmentPlans": 0.09, "OthersOnLoan": 0.43, "LoanDuration": 1.0, "Age": 0.87, "CreditHistory": 0.54, "CurrentResidenceDuration": 0.2, "ForeignWorker": 0.0, "Housing": 0.07, "ExistingCreditsCount": 0.51, "Telephone": 0.01, "NewFeature_1_featureagglomeration_1": 0.41, "Sex": 0.05, "Dependents": 0.03, "OwnsProperty": 0.26, "EmploymentDuration": 0.06 }, "stage": "Mean" } ], "intermediate_model": { "composition_steps": [ "TrainingDataset_full_249_20", "Split_TrainingHoldout", "TrainingDataset_full_224_20", "Preprocessor_default", "DAUB" ], "duration": 170647, "location": { "model": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/fold_output/Pipeline0/model.pickle", "pipeline": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/fold_output/Pipeline0/pipeline.json", "pipeline_model": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/fold_output/Pipeline0/pipeline-model.json" }, "name": "P1", "notebook_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/fold_output/Pipeline0/notebook.ipynb", "pipeline_nodes": [ "PreprocessingTransformer", "GradientBoostingClassifierEstimator" ], "process": "fold_output", "schema_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/fold_output/Pipeline0/schema.json", "sdk_notebook_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/fold_output/Pipeline0/notebook_sdk.ipynb" }, "phase": "fold_output" }, "ml_metrics": { "training_roc_auc": 0.8283546163980947, "holdout_precision": 0.6666666666666666, "training_average_precision": 0.6996753806435804, "holdout_average_precision": 0.7116994847258005, "training_neg_log_loss": -0.5747366162076947, "holdout_recall": 0.5, "training_precision": 0.6439393939393939, "holdout_accuracy": 0.76, "holdout_balanced_accuracy": 0.6911764705882353, "training_recall": 0.5718050065876153, "holdout_f1": 0.5714285714285715, "holdout_neg_log_loss": -0.7066552317239467, "training_accuracy": 0.7723123123123123, "holdout_roc_auc": 0.7205882352941178, "training_balanced_accuracy": 0.715389682780987, "training_f1": 0.6009839626860902 }, "timestamp": "2020-09-30T14:38:54.000Z" }, { "context": { "binary_classification": { "confusion_matrix": [ { "fn": 2, "fp": 3, "tn": 5, "tp": 15, "true_class": "No Risk" }, { "fn": 3, "fp": 2, "tn": 15, "tp": 5, "true_class": "Risk" } ], "roc_curve": [ { "fpr": [ 0.0, 0.0, 0.0, 0.125, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 1.0 ], "thresholds": [ 1.9118249416351318, 0.9118249416351318, 0.8851636648178101, 0.8839583992958069, 0.8572619557380676, 0.8568741083145142, 0.7981584072113037, 0.7974665760993958, 0.529254138469696, 0.4382297098636627, 0.37201759219169617, 0.24074923992156982 ], "tpr": [ 0.0, 0.058823529411764705, 0.29411764705882354, 0.29411764705882354, 0.35294117647058826, 0.35294117647058826, 0.5882352941176471, 0.5882352941176471, 0.8823529411764706, 0.8823529411764706, 1.0, 1.0 ], "true_class": "No Risk" }, { "fpr": [ 0.0, 0.0, 0.0, 0.11764705882352941, 0.11764705882352941, 0.4117647058823529, 0.4117647058823529, 0.6470588235294118, 0.6470588235294118, 0.7058823529411765, 0.7058823529411765, 1.0 ], "thresholds": [ 1.7592507600784302, 0.7592507600784302, 0.6614384055137634, 0.5711529850959778, 0.529679536819458, 0.29737383127212524, 0.20253342390060425, 0.14502882957458496, 0.14312586188316345, 0.14273804426193237, 0.11604157835245132, 0.08817508071660995 ], "tpr": [ 0.0, 0.125, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 0.875, 0.875, 1.0, 1.0 ], "true_class": "Risk" } ] }, "classes": [ "No Risk", "Risk" ], "features_importance": [ { "computation_type": "", "features": { "LoanAmount": 0.68, "InstallmentPercent": 0.14, "ExistingSavings": 0.42, "CheckingStatus": 0.09, "LoanPurpose": 0.32, "NewFeature_0_featureagglomeration_0": 0.39, "Job": 0.04, "InstallmentPlans": 0.09, "OthersOnLoan": 0.43, "LoanDuration": 1.0, "Age": 0.87, "CreditHistory": 0.54, "CurrentResidenceDuration": 0.2, "ForeignWorker": 0.0, "Housing": 0.07, "ExistingCreditsCount": 0.51, "Telephone": 0.01, "NewFeature_1_featureagglomeration_1": 0.41, "Sex": 0.05, "Dependents": 0.03, "OwnsProperty": 0.26, "EmploymentDuration": 0.06 }, "stage": "Mean" } ], "intermediate_model": { "composition_steps": [ "TrainingDataset_full_249_20", "Split_TrainingHoldout", "TrainingDataset_full_224_20", "Preprocessor_default", "DAUB", "hpo_d" ], "duration": 170647, "location": { "model": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/fold_output/Pipeline1/model.pickle", "pipeline": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/fold_output/Pipeline1/pipeline.json", "pipeline_model": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/fold_output/Pipeline1/pipeline-model.json" }, "name": "P2", "notebook_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/fold_output/Pipeline1/notebook.ipynb", "pipeline_nodes": [ "PreprocessingTransformer", "GradientBoostingClassifierEstimator" ], "process": "fold_output", "schema_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/fold_output/Pipeline1/schema.json", "sdk_notebook_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/fold_output/Pipeline1/notebook_sdk.ipynb" }, "phase": "fold_output" }, "ml_metrics": { "training_roc_auc": 0.8566053511705686, "holdout_precision": 0.7142857142857143, "training_average_precision": 0.7320733111904305, "holdout_average_precision": 0.7039224664224665, "training_neg_log_loss": -0.4465285582613328, "holdout_recall": 0.625, "training_precision": 0.7780214424951266, "holdout_accuracy": 0.8, "holdout_balanced_accuracy": 0.7536764705882353, "training_recall": 0.49736495388669294, "holdout_f1": 0.6666666666666666, "holdout_neg_log_loss": -0.5421721597291818, "training_accuracy": 0.7990390390390391, "holdout_roc_auc": 0.75, "training_balanced_accuracy": 0.7134260666869362, "training_f1": 0.589159891598916 }, "timestamp": "2020-09-30T14:38:54.000Z" }, { "context": { "binary_classification": { "confusion_matrix": [ { "fn": 4, "fp": 3, "tn": 5, "tp": 13, "true_class": "No Risk" }, { "fn": 3, "fp": 4, "tn": 13, "tp": 5, "true_class": "Risk" } ], "roc_curve": [ { "fpr": [ 0.0, 0.0, 0.0, 0.125, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 1.0 ], "thresholds": [ 1.9320768117904663, 0.9320768117904663, 0.9299767017364502, 0.9259238243103027, 0.8939604759216309, 0.8521214127540588, 0.7138300538063049, 0.7113333344459534, 0.45716628432273865, 0.43239983916282654, 0.42159730195999146, 0.3683125376701355, 0.3567972481250763, 0.19027136266231537 ], "tpr": [ 0.0, 0.058823529411764705, 0.11764705882352941, 0.11764705882352941, 0.29411764705882354, 0.29411764705882354, 0.6470588235294118, 0.6470588235294118, 0.8823529411764706, 0.8823529411764706, 0.9411764705882353, 0.9411764705882353, 1.0, 1.0 ], "true_class": "No Risk" }, { "fpr": [ 0.0, 0.0, 0.0, 0.058823529411764705, 0.058823529411764705, 0.11764705882352941, 0.11764705882352941, 0.35294117647058826, 0.35294117647058826, 0.7058823529411765, 0.7058823529411765, 0.8823529411764706, 0.8823529411764706, 1.0 ], "thresholds": [ 1.8097286224365234, 0.8097286224365234, 0.7199095487594604, 0.6432027816772461, 0.6316874623298645, 0.5784026980400085, 0.5589774250984192, 0.3221668303012848, 0.28866666555404663, 0.15159079432487488, 0.14787857234477997, 0.08006153255701065, 0.07407616823911667, 0.06792321056127548 ], "tpr": [ 0.0, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 0.875, 0.875, 1.0, 1.0 ], "true_class": "Risk" } ] }, "classes": [ "No Risk", "Risk" ], "features_importance": [ { "computation_type": "", "features": { "LoanAmount": 0.68, "InstallmentPercent": 0.14, "ExistingSavings": 0.42, "CheckingStatus": 0.09, "LoanPurpose": 0.32, "NewFeature_0_featureagglomeration_0": 0.39, "Job": 0.04, "InstallmentPlans": 0.09, "OthersOnLoan": 0.43, "LoanDuration": 1.0, "Age": 0.87, "CreditHistory": 0.54, "CurrentResidenceDuration": 0.2, "ForeignWorker": 0.0, "Housing": 0.07, "ExistingCreditsCount": 0.51, "Telephone": 0.01, "NewFeature_1_featureagglomeration_1": 0.41, "Sex": 0.05, "Dependents": 0.03, "OwnsProperty": 0.26, "EmploymentDuration": 0.06 }, "stage": "Mean" } ], "intermediate_model": { "composition_steps": [ "TrainingDataset_full_249_20", "Split_TrainingHoldout", "TrainingDataset_full_224_20", "Preprocessor_default", "DAUB", "hpo_d", "cognito" ], "duration": 170647, "location": { "model": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/fold_output/Pipeline2/model.pickle", "pipeline": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/fold_output/Pipeline2/pipeline.json", "pipeline_model": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/fold_output/Pipeline2/pipeline-model.json" }, "name": "P3", "notebook_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/fold_output/Pipeline2/notebook.ipynb", "pipeline_nodes": [ "PreprocessingTransformer", "autoai_libs.cognito.transforms.transform_utils.featureagglomeration", "autoai_libs.cognito.transforms.transform_utils.FS1", "GradientBoostingClassifierEstimator" ], "process": "fold_output", "schema_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/fold_output/Pipeline2/schema.json", "sdk_notebook_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/fold_output/Pipeline2/notebook_sdk.ipynb" }, "phase": "fold_output" }, "ml_metrics": { "training_roc_auc": 0.8637123745819397, "holdout_precision": 0.5555555555555556, "training_average_precision": 0.7631005098061324, "holdout_average_precision": 0.6683999400675602, "training_neg_log_loss": -0.436711969891688, "holdout_recall": 0.625, "training_precision": 0.7331140350877193, "holdout_accuracy": 0.72, "holdout_balanced_accuracy": 0.6948529411764706, "training_recall": 0.5882740447957838, "holdout_f1": 0.5882352941176471, "holdout_neg_log_loss": -0.555290777346542, "training_accuracy": 0.8081681681681682, "holdout_roc_auc": 0.7205882352941176, "training_balanced_accuracy": 0.7460600993209688, "training_f1": 0.651842785257718 }, "timestamp": "2020-09-30T14:38:55.000Z" }, { "context": { "binary_classification": { "confusion_matrix": [ { "fn": 3, "fp": 3, "tn": 5, "tp": 14, "true_class": "No Risk" }, { "fn": 3, "fp": 3, "tn": 14, "tp": 5, "true_class": "Risk" } ], "roc_curve": [ { "fpr": [ 0.0, 0.0, 0.0, 0.125, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 1.0 ], "thresholds": [ 1.912172555923462, 0.9121726155281067, 0.9023290276527405, 0.9015228152275085, 0.8295201063156128, 0.8294777870178223, 0.6834437251091003, 0.6798041462898254, 0.4823266565799713, 0.4560561776161194, 0.40774452686309814, 0.40617018938064575, 0.39104729890823364, 0.28977835178375244 ], "tpr": [ 0.0, 0.058823529411764705, 0.17647058823529413, 0.17647058823529413, 0.4117647058823529, 0.4117647058823529, 0.6470588235294118, 0.6470588235294118, 0.8823529411764706, 0.8823529411764706, 0.9411764705882353, 0.9411764705882353, 1.0, 1.0 ], "true_class": "No Risk" }, { "fpr": [ 0.0, 0.0, 0.0, 0.058823529411764705, 0.058823529411764705, 0.11764705882352941, 0.11764705882352941, 0.35294117647058826, 0.35294117647058826, 0.5882352941176471, 0.5882352941176471, 0.8235294117647058, 0.8235294117647058, 1.0 ], "thresholds": [ 1.7102216482162476, 0.7102216482162476, 0.669156014919281, 0.6089527010917664, 0.5938298106193542, 0.5922554731369019, 0.5338093638420105, 0.3384079039096832, 0.32019585371017456, 0.18048915266990662, 0.17052221298217773, 0.11166291683912277, 0.09847716987133026, 0.0878273993730545 ], "tpr": [ 0.0, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 0.875, 0.875, 1.0, 1.0 ], "true_class": "Risk" } ] }, "classes": [ "No Risk", "Risk" ], "features_importance": [ { "computation_type": "", "features": { "LoanAmount": 0.68, "InstallmentPercent": 0.14, "ExistingSavings": 0.42, "CheckingStatus": 0.09, "LoanPurpose": 0.32, "NewFeature_0_featureagglomeration_0": 0.39, "Job": 0.04, "InstallmentPlans": 0.09, "OthersOnLoan": 0.43, "LoanDuration": 1.0, "Age": 0.87, "CreditHistory": 0.54, "CurrentResidenceDuration": 0.2, "ForeignWorker": 0.0, "Housing": 0.07, "ExistingCreditsCount": 0.51, "Telephone": 0.01, "NewFeature_1_featureagglomeration_1": 0.41, "Sex": 0.05, "Dependents": 0.03, "OwnsProperty": 0.26, "EmploymentDuration": 0.06 }, "stage": "Mean" } ], "intermediate_model": { "composition_steps": [ "TrainingDataset_full_249_20", "Split_TrainingHoldout", "TrainingDataset_full_224_20", "Preprocessor_default", "DAUB", "hpo_d", "cognito", "hpo_c" ], "duration": 170647, "location": { "model": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/fold_output/Pipeline3/model.pickle", "pipeline": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/fold_output/Pipeline3/pipeline.json", "pipeline_model": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/fold_output/Pipeline3/pipeline-model.json" }, "name": "P4", "notebook_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/fold_output/Pipeline3/notebook.ipynb", "pipeline_nodes": [ "PreprocessingTransformer", "autoai_libs.cognito.transforms.transform_utils.featureagglomeration", "autoai_libs.cognito.transforms.transform_utils.FS1", "GradientBoostingClassifierEstimator" ], "process": "fold_output", "schema_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/fold_output/Pipeline3/schema.json", "sdk_notebook_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/fold_output/Pipeline3/notebook_sdk.ipynb" }, "phase": "fold_output" }, "ml_metrics": { "training_roc_auc": 0.8656126482213438, "holdout_precision": 0.625, "training_average_precision": 0.7583491228304152, "holdout_average_precision": 0.6757941813088872, "training_neg_log_loss": -0.4447612030777576, "holdout_recall": 0.625, "training_precision": 0.7137566137566139, "holdout_accuracy": 0.76, "holdout_balanced_accuracy": 0.7242647058823529, "training_recall": 0.5586297760210803, "holdout_f1": 0.625, "holdout_neg_log_loss": -0.5520562594922229, "training_accuracy": 0.7947747747747748, "holdout_roc_auc": 0.7426470588235294, "training_balanced_accuracy": 0.7280328367284888, "training_f1": 0.6245680270070514 }, "timestamp": "2020-09-30T14:38:55.000Z" }, { "context": { "binary_classification": { "confusion_matrix": [ { "fn": 2, "fp": 4, "tn": 4, "tp": 15, "true_class": "No Risk" }, { "fn": 4, "fp": 2, "tn": 15, "tp": 4, "true_class": "Risk" } ], "roc_curve": [ { "fpr": [ 0.0, 0.0, 0.0, 0.125, 0.125, 0.25, 0.25, 0.375, 0.375, 0.5, 0.5, 1.0 ], "thresholds": [ 1.9959793090820312, 0.9959793090820312, 0.9949495196342468, 0.9937530755996704, 0.9900901913642883, 0.9870622158050537, 0.9241466522216797, 0.8619304299354553, 0.7010563015937805, 0.5894654989242554, 0.23473195731639862, 0.010169494897127151 ], "tpr": [ 0.0, 0.058823529411764705, 0.11764705882352941, 0.11764705882352941, 0.23529411764705882, 0.23529411764705882, 0.47058823529411764, 0.47058823529411764, 0.7058823529411765, 0.7058823529411765, 1.0, 1.0 ], "true_class": "No Risk" }, { "fpr": [ 0.0, 0.0, 0.0, 0.29411764705882354, 0.29411764705882354, 0.4117647058823529, 0.4117647058823529, 0.7058823529411765, 0.7058823529411765, 0.8235294117647058, 0.8235294117647058, 1.0 ], "thresholds": [ 1.9907909631729126, 0.9907909631729126, 0.9192749261856079, 0.37951037287712097, 0.3295697271823883, 0.15989628434181213, 0.11566814035177231, 0.03620442748069763, 0.02297501638531685, 0.010295126587152481, 0.007908087223768234, 0.003513400675728917 ], "tpr": [ 0.0, 0.125, 0.5, 0.5, 0.625, 0.625, 0.75, 0.75, 0.875, 0.875, 1.0, 1.0 ], "true_class": "Risk" } ] }, "classes": [ "No Risk", "Risk" ], "features_importance": [ { "computation_type": "", "features": { "LoanAmount": 0.68, "InstallmentPercent": 0.14, "ExistingSavings": 0.42, "CheckingStatus": 0.09, "LoanPurpose": 0.32, "NewFeature_0_featureagglomeration_0": 0.39, "Job": 0.04, "InstallmentPlans": 0.09, "OthersOnLoan": 0.43, "LoanDuration": 1.0, "Age": 0.87, "CreditHistory": 0.54, "CurrentResidenceDuration": 0.2, "ForeignWorker": 0.0, "Housing": 0.07, "ExistingCreditsCount": 0.51, "Telephone": 0.01, "NewFeature_1_featureagglomeration_1": 0.41, "Sex": 0.05, "Dependents": 0.03, "OwnsProperty": 0.26, "EmploymentDuration": 0.06 }, "stage": "Mean" } ], "intermediate_model": { "composition_steps": [ "TrainingDataset_full_249_20", "Split_TrainingHoldout", "TrainingDataset_full_224_20", "Preprocessor_default", "DAUB" ], "duration": 234120, "location": { "model": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/global_output/Pipeline0/model.pickle", "pipeline": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/global_output/Pipeline0/pipeline.json", "pipeline_model": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/global_output/Pipeline0/pipeline-model.json" }, "name": "P1", "notebook_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/global_output/Pipeline0/notebook.ipynb", "pipeline_nodes": [ "PreprocessingTransformer", "GradientBoostingClassifierEstimator" ], "process": "global_output", "schema_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/global_output/Pipeline0/schema.json", "sdk_notebook_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/global_output/Pipeline0/notebook_sdk.ipynb" }, "phase": "global_output" }, "ml_metrics": { "training_roc_auc": 0.8283546163980947, "holdout_precision": 0.6666666666666666, "training_average_precision": 0.6996753806435804, "holdout_average_precision": 0.7116994847258005, "training_neg_log_loss": -0.5747366162076947, "holdout_recall": 0.5, "training_precision": 0.6439393939393939, "holdout_accuracy": 0.76, "holdout_balanced_accuracy": 0.6911764705882353, "training_recall": 0.5718050065876153, "holdout_f1": 0.5714285714285715, "holdout_neg_log_loss": -0.7066552317239467, "training_accuracy": 0.7723123123123123, "holdout_roc_auc": 0.7205882352941178, "training_balanced_accuracy": 0.715389682780987, "training_f1": 0.6009839626860902 }, "timestamp": "2020-09-30T14:39:58.000Z" }, { "context": { "binary_classification": { "confusion_matrix": [ { "fn": 2, "fp": 3, "tn": 5, "tp": 15, "true_class": "No Risk" }, { "fn": 3, "fp": 2, "tn": 15, "tp": 5, "true_class": "Risk" } ], "roc_curve": [ { "fpr": [ 0.0, 0.0, 0.0, 0.125, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 1.0 ], "thresholds": [ 1.9118249416351318, 0.9118249416351318, 0.8851636648178101, 0.8839583992958069, 0.8572619557380676, 0.8568741083145142, 0.7981584072113037, 0.7974665760993958, 0.529254138469696, 0.4382297098636627, 0.37201759219169617, 0.24074923992156982 ], "tpr": [ 0.0, 0.058823529411764705, 0.29411764705882354, 0.29411764705882354, 0.35294117647058826, 0.35294117647058826, 0.5882352941176471, 0.5882352941176471, 0.8823529411764706, 0.8823529411764706, 1.0, 1.0 ], "true_class": "No Risk" }, { "fpr": [ 0.0, 0.0, 0.0, 0.11764705882352941, 0.11764705882352941, 0.4117647058823529, 0.4117647058823529, 0.6470588235294118, 0.6470588235294118, 0.7058823529411765, 0.7058823529411765, 1.0 ], "thresholds": [ 1.7592507600784302, 0.7592507600784302, 0.6614384055137634, 0.5711529850959778, 0.529679536819458, 0.29737383127212524, 0.20253342390060425, 0.14502882957458496, 0.14312586188316345, 0.14273804426193237, 0.11604157835245132, 0.08817508071660995 ], "tpr": [ 0.0, 0.125, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 0.875, 0.875, 1.0, 1.0 ], "true_class": "Risk" } ] }, "classes": [ "No Risk", "Risk" ], "features_importance": [ { "computation_type": "", "features": { "LoanAmount": 0.68, "InstallmentPercent": 0.14, "ExistingSavings": 0.42, "CheckingStatus": 0.09, "LoanPurpose": 0.32, "NewFeature_0_featureagglomeration_0": 0.39, "Job": 0.04, "InstallmentPlans": 0.09, "OthersOnLoan": 0.43, "LoanDuration": 1.0, "Age": 0.87, "CreditHistory": 0.54, "CurrentResidenceDuration": 0.2, "ForeignWorker": 0.0, "Housing": 0.07, "ExistingCreditsCount": 0.51, "Telephone": 0.01, "NewFeature_1_featureagglomeration_1": 0.41, "Sex": 0.05, "Dependents": 0.03, "OwnsProperty": 0.26, "EmploymentDuration": 0.06 }, "stage": "Mean" } ], "intermediate_model": { "composition_steps": [ "TrainingDataset_full_249_20", "Split_TrainingHoldout", "TrainingDataset_full_224_20", "Preprocessor_default", "DAUB", "hpo_d" ], "duration": 234120, "location": { "model": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/global_output/Pipeline1/model.pickle", "pipeline": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/global_output/Pipeline1/pipeline.json", "pipeline_model": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/global_output/Pipeline1/pipeline-model.json" }, "name": "P2", "notebook_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/global_output/Pipeline1/notebook.ipynb", "pipeline_nodes": [ "PreprocessingTransformer", "GradientBoostingClassifierEstimator" ], "process": "global_output", "schema_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/global_output/Pipeline1/schema.json", "sdk_notebook_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/global_output/Pipeline1/notebook_sdk.ipynb" }, "phase": "global_output" }, "ml_metrics": { "training_roc_auc": 0.8566053511705686, "holdout_precision": 0.7142857142857143, "training_average_precision": 0.7320733111904305, "holdout_average_precision": 0.7039224664224665, "training_neg_log_loss": -0.4465285582613328, "holdout_recall": 0.625, "training_precision": 0.7780214424951266, "holdout_accuracy": 0.8, "holdout_balanced_accuracy": 0.7536764705882353, "training_recall": 0.49736495388669294, "holdout_f1": 0.6666666666666666, "holdout_neg_log_loss": -0.5421721597291818, "training_accuracy": 0.7990390390390391, "holdout_roc_auc": 0.75, "training_balanced_accuracy": 0.7134260666869362, "training_f1": 0.589159891598916 }, "timestamp": "2020-09-30T14:39:58.000Z" }, { "context": { "binary_classification": { "confusion_matrix": [ { "fn": 4, "fp": 3, "tn": 5, "tp": 13, "true_class": "No Risk" }, { "fn": 3, "fp": 4, "tn": 13, "tp": 5, "true_class": "Risk" } ], "roc_curve": [ { "fpr": [ 0.0, 0.0, 0.0, 0.125, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 1.0 ], "thresholds": [ 1.9320768117904663, 0.9320768117904663, 0.9299767017364502, 0.9259238243103027, 0.8939604759216309, 0.8521214127540588, 0.7138300538063049, 0.7113333344459534, 0.45716628432273865, 0.43239983916282654, 0.42159730195999146, 0.3683125376701355, 0.3567972481250763, 0.19027136266231537 ], "tpr": [ 0.0, 0.058823529411764705, 0.11764705882352941, 0.11764705882352941, 0.29411764705882354, 0.29411764705882354, 0.6470588235294118, 0.6470588235294118, 0.8823529411764706, 0.8823529411764706, 0.9411764705882353, 0.9411764705882353, 1.0, 1.0 ], "true_class": "No Risk" }, { "fpr": [ 0.0, 0.0, 0.0, 0.058823529411764705, 0.058823529411764705, 0.11764705882352941, 0.11764705882352941, 0.35294117647058826, 0.35294117647058826, 0.7058823529411765, 0.7058823529411765, 0.8823529411764706, 0.8823529411764706, 1.0 ], "thresholds": [ 1.8097286224365234, 0.8097286224365234, 0.7199095487594604, 0.6432027816772461, 0.6316874623298645, 0.5784026980400085, 0.5589774250984192, 0.3221668303012848, 0.28866666555404663, 0.15159079432487488, 0.14787857234477997, 0.08006153255701065, 0.07407616823911667, 0.06792321056127548 ], "tpr": [ 0.0, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 0.875, 0.875, 1.0, 1.0 ], "true_class": "Risk" } ] }, "classes": [ "No Risk", "Risk" ], "features_importance": [ { "computation_type": "", "features": { "LoanAmount": 0.68, "InstallmentPercent": 0.14, "ExistingSavings": 0.42, "CheckingStatus": 0.09, "LoanPurpose": 0.32, "NewFeature_0_featureagglomeration_0": 0.39, "Job": 0.04, "InstallmentPlans": 0.09, "OthersOnLoan": 0.43, "LoanDuration": 1.0, "Age": 0.87, "CreditHistory": 0.54, "CurrentResidenceDuration": 0.2, "ForeignWorker": 0.0, "Housing": 0.07, "ExistingCreditsCount": 0.51, "Telephone": 0.01, "NewFeature_1_featureagglomeration_1": 0.41, "Sex": 0.05, "Dependents": 0.03, "OwnsProperty": 0.26, "EmploymentDuration": 0.06 }, "stage": "Mean" } ], "intermediate_model": { "composition_steps": [ "TrainingDataset_full_249_20", "Split_TrainingHoldout", "TrainingDataset_full_224_20", "Preprocessor_default", "DAUB", "hpo_d", "cognito" ], "duration": 234120, "location": { "model": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/global_output/Pipeline2/model.pickle", "pipeline": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/global_output/Pipeline2/pipeline.json", "pipeline_model": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/global_output/Pipeline2/pipeline-model.json" }, "name": "P3", "notebook_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/global_output/Pipeline2/notebook.ipynb", "pipeline_nodes": [ "PreprocessingTransformer", "autoai_libs.cognito.transforms.transform_utils.featureagglomeration", "autoai_libs.cognito.transforms.transform_utils.FS1", "GradientBoostingClassifierEstimator" ], "process": "global_output", "schema_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/global_output/Pipeline2/schema.json", "sdk_notebook_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/global_output/Pipeline2/notebook_sdk.ipynb" }, "phase": "global_output" }, "ml_metrics": { "training_roc_auc": 0.8637123745819397, "holdout_precision": 0.5555555555555556, "training_average_precision": 0.7631005098061324, "holdout_average_precision": 0.6683999400675602, "training_neg_log_loss": -0.436711969891688, "holdout_recall": 0.625, "training_precision": 0.7331140350877193, "holdout_accuracy": 0.72, "holdout_balanced_accuracy": 0.6948529411764706, "training_recall": 0.5882740447957838, "holdout_f1": 0.5882352941176471, "holdout_neg_log_loss": -0.555290777346542, "training_accuracy": 0.8081681681681682, "holdout_roc_auc": 0.7205882352941176, "training_balanced_accuracy": 0.7460600993209688, "training_f1": 0.651842785257718 }, "timestamp": "2020-09-30T14:39:58.000Z" }, { "context": { "binary_classification": { "confusion_matrix": [ { "fn": 3, "fp": 3, "tn": 5, "tp": 14, "true_class": "No Risk" }, { "fn": 3, "fp": 3, "tn": 14, "tp": 5, "true_class": "Risk" } ], "roc_curve": [ { "fpr": [ 0.0, 0.0, 0.0, 0.125, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 1.0 ], "thresholds": [ 1.912172555923462, 0.9121726155281067, 0.9023290276527405, 0.9015228152275085, 0.8295201063156128, 0.8294777870178223, 0.6834437251091003, 0.6798041462898254, 0.4823266565799713, 0.4560561776161194, 0.40774452686309814, 0.40617018938064575, 0.39104729890823364, 0.28977835178375244 ], "tpr": [ 0.0, 0.058823529411764705, 0.17647058823529413, 0.17647058823529413, 0.4117647058823529, 0.4117647058823529, 0.6470588235294118, 0.6470588235294118, 0.8823529411764706, 0.8823529411764706, 0.9411764705882353, 0.9411764705882353, 1.0, 1.0 ], "true_class": "No Risk" }, { "fpr": [ 0.0, 0.0, 0.0, 0.058823529411764705, 0.058823529411764705, 0.11764705882352941, 0.11764705882352941, 0.35294117647058826, 0.35294117647058826, 0.5882352941176471, 0.5882352941176471, 0.8235294117647058, 0.8235294117647058, 1.0 ], "thresholds": [ 1.7102216482162476, 0.7102216482162476, 0.669156014919281, 0.6089527010917664, 0.5938298106193542, 0.5922554731369019, 0.5338093638420105, 0.3384079039096832, 0.32019585371017456, 0.18048915266990662, 0.17052221298217773, 0.11166291683912277, 0.09847716987133026, 0.0878273993730545 ], "tpr": [ 0.0, 0.125, 0.25, 0.25, 0.375, 0.375, 0.625, 0.625, 0.75, 0.75, 0.875, 0.875, 1.0, 1.0 ], "true_class": "Risk" } ] }, "classes": [ "No Risk", "Risk" ], "features_importance": [ { "computation_type": "", "features": { "LoanAmount": 0.68, "InstallmentPercent": 0.14, "ExistingSavings": 0.42, "CheckingStatus": 0.09, "LoanPurpose": 0.32, "NewFeature_0_featureagglomeration_0": 0.39, "Job": 0.04, "InstallmentPlans": 0.09, "OthersOnLoan": 0.43, "LoanDuration": 1.0, "Age": 0.87, "CreditHistory": 0.54, "CurrentResidenceDuration": 0.2, "ForeignWorker": 0.0, "Housing": 0.07, "ExistingCreditsCount": 0.51, "Telephone": 0.01, "NewFeature_1_featureagglomeration_1": 0.41, "Sex": 0.05, "Dependents": 0.03, "OwnsProperty": 0.26, "EmploymentDuration": 0.06 }, "stage": "Mean" } ], "intermediate_model": { "composition_steps": [ "TrainingDataset_full_249_20", "Split_TrainingHoldout", "TrainingDataset_full_224_20", "Preprocessor_default", "DAUB", "hpo_d", "cognito", "hpo_c" ], "duration": 234120, "location": { "model": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/global_output/Pipeline3/model.pickle", "pipeline": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/global_output/Pipeline3/pipeline.json", "pipeline_model": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/global_output/Pipeline3/pipeline-model.json" }, "name": "P4", "notebook_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/global_output/Pipeline3/notebook.ipynb", "pipeline_nodes": [ "PreprocessingTransformer", "autoai_libs.cognito.transforms.transform_utils.featureagglomeration", "autoai_libs.cognito.transforms.transform_utils.FS1", "GradientBoostingClassifierEstimator" ], "process": "global_output", "schema_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/global_output/Pipeline3/schema.json", "sdk_notebook_location": "//9cc32e6c-eefe-4e0e-96b4-fb5a3f7b03e2/data/automl/global_output/Pipeline3/notebook_sdk.ipynb" }, "phase": "global_output" }, "ml_metrics": { "training_roc_auc": 0.8656126482213438, "holdout_precision": 0.625, "training_average_precision": 0.7583491228304152, "holdout_average_precision": 0.6757941813088872, "training_neg_log_loss": -0.4447612030777576, "holdout_recall": 0.625, "training_precision": 0.7137566137566139, "holdout_accuracy": 0.76, "holdout_balanced_accuracy": 0.7242647058823529, "training_recall": 0.5586297760210803, "holdout_f1": 0.625, "holdout_neg_log_loss": -0.5520562594922229, "training_accuracy": 0.7947747747747748, "holdout_roc_auc": 0.7426470588235294, "training_balanced_accuracy": 0.7280328367284888, "training_f1": 0.6245680270070514 }, "timestamp": "2020-09-30T14:39:58.000Z" } ], "running_at": "2020-09-30T14:39:58.000Z", "state": "completed" }, "tags": [ { "value": "autoai" } ], "training_data_references": [ { "location": { "href": "/v2/assets/127d4a1d-0d5f-41bc-bd7f-ed78ffea6165?space_id=2219f598-6194-4e9a-9287-0730c69b5f4a" }, "type": "data_asset", "connection": {}, "id": "credit_risk_training_light.csv" } ] } } ] }

Cancel training run

Tip: If you want to cancel your training, please convert below cell to code, specify training ID and run.

<a href="https://watson-ml-v4-api.mybluemix.net/wml-restapi-cloud.html#/Trainings/trainings_delete" target="_blank" rel="noopener no referrer">Canceling training

%%bash

TRAINING_ID_TO_CANCEL=...

curl -sk -X DELETE \
    --header "Authorization: Bearer $TOKEN" \
    --header "Content-Type: application/json" \
    --header "Accept: application/json" \
    "$DATAPLATFORM_URL/ml/v4/trainings/$TRAINING_ID_TO_CANCEL?space_id=$SPACE_ID&version=2020-08-01"

5. Deploy and Score

In this section you will learn how to deploy and score pipeline model as webservice using WML instance.

Before deployment creation, you need store your model in WML repository. Please see below cURL call example how to do it. Remember that you need to specify where your chosen model is stored in COS.

Store AutoAI model

Store information about your model to WML repository.

<a href="https://watson-ml-v4-api.mybluemix.net/wml-restapi-cloud.html#/Models/models_create" target="_blank" rel="noopener no referrer">Model storing

%%bash --out model_payload

curl -sk -X GET \
    --header "Authorization: Bearer $TOKEN" \
    --header "Content-Type: application/json" \
    --header "Accept: application/json" \
    "$DATAPLATFORM_URL/v2/asset_files/auto_ml/auto_ml_curl.$RANDOM_ID/wml_data/$TRAINING_ID/assets/$TRAINING_ID""_P1_global_output/resources/wml_model/request.json?space_id=$SPACE_ID" \
    | python -m json.tool

%env MODEL_PAYLOAD=$model_payload
env: MODEL_PAYLOAD={ "content_location": { "connection": {}, "contents": [ { "content_format": "native", "file_name": "pipeline_model.json", "location": "/spaces/2219f598-6194-4e9a-9287-0730c69b5f4a/assets/auto_ml/auto_ml_curl.eab90a7e0841c00db04266373c27ef71a0f312b5/wml_data/b4a501d9-8da3-47fa-9a2b-197d73fe84fb/assets/b4a501d9-8da3-47fa-9a2b-197d73fe84fb_P1_global_output/resources/wml_model/pipeline_model.json" }, { "content_format": "pipeline-node", "file_name": "P1_automl.zip", "location": "/spaces/2219f598-6194-4e9a-9287-0730c69b5f4a/assets/auto_ml/auto_ml_curl.eab90a7e0841c00db04266373c27ef71a0f312b5/wml_data/b4a501d9-8da3-47fa-9a2b-197d73fe84fb/assets/b4a501d9-8da3-47fa-9a2b-197d73fe84fb_P1_global_output/resources/wml_model/P1_automl.zip", "pipeline_node_id": "automl" } ], "location": { "training": "/spaces/2219f598-6194-4e9a-9287-0730c69b5f4a/assets/auto_ml/auto_ml_curl.eab90a7e0841c00db04266373c27ef71a0f312b5/wml_data/b4a501d9-8da3-47fa-9a2b-197d73fe84fb", "path": "/spaces/2219f598-6194-4e9a-9287-0730c69b5f4a/assets/auto_ml/auto_ml_curl.eab90a7e0841c00db04266373c27ef71a0f312b5/wml_data", "training_status": "/spaces/2219f598-6194-4e9a-9287-0730c69b5f4a/assets/auto_ml/auto_ml_curl.eab90a7e0841c00db04266373c27ef71a0f312b5/wml_data/b4a501d9-8da3-47fa-9a2b-197d73fe84fb/training-status.json", "assets_path": "/spaces/2219f598-6194-4e9a-9287-0730c69b5f4a/assets/auto_ml/auto_ml_curl.eab90a7e0841c00db04266373c27ef71a0f312b5/wml_data/b4a501d9-8da3-47fa-9a2b-197d73fe84fb/assets" }, "type": "fs" }, "hybrid_pipeline_software_specs": [ { "name": "autoai-kb_3.1-py3.7" } ], "name": "P1", "pipeline": { "id": "a8cbf66e-2e7d-4ec4-b81c-0c24b38ef35f" }, "schemas": { "input": [ { "fields": [ { "name": "CheckingStatus", "type": "other" }, { "name": "LoanDuration", "type": "integer" }, { "name": "CreditHistory", "type": "other" }, { "name": "LoanPurpose", "type": "other" }, { "name": "LoanAmount", "type": "integer" }, { "name": "ExistingSavings", "type": "other" }, { "name": "EmploymentDuration", "type": "other" }, { "name": "InstallmentPercent", "type": "integer" }, { "name": "Sex", "type": "other" }, { "name": "OthersOnLoan", "type": "other" }, { "name": "CurrentResidenceDuration", "type": "integer" }, { "name": "OwnsProperty", "type": "other" }, { "name": "Age", "type": "integer" }, { "name": "InstallmentPlans", "type": "other" }, { "name": "Housing", "type": "other" }, { "name": "ExistingCreditsCount", "type": "integer" }, { "name": "Job", "type": "other" }, { "name": "Dependents", "type": "integer" }, { "name": "Telephone", "type": "other" }, { "name": "ForeignWorker", "type": "other" } ], "id": "auto_ai_kb_input_schema" } ], "output": [] }, "software_spec": { "name": "hybrid_0.1" }, "space_id": "2219f598-6194-4e9a-9287-0730c69b5f4a", "training_data_references": [ { "connection": {}, "id": "credit_risk_training_light.csv", "location": { "href": "/v2/assets/f5531b4b-d1e8-44e7-a464-67164a2b8a6e?space_id=2219f598-6194-4e9a-9287-0730c69b5f4a" }, "type": "data_asset" } ], "type": "wml-hybrid_0.1" } 
%%bash --out model_details

curl -sk -X POST \
    --header "Authorization: Bearer $TOKEN" \
    --header "Content-Type: application/json" \
    --header "Accept: application/json" \
    --data "$MODEL_PAYLOAD" \
    "$DATAPLATFORM_URL/ml/v4/models?version=2020-08-01&space_id=$SPACE_ID"

%env MODEL_DETAILS=$model_details
env: MODEL_DETAILS={ "entity": { "content_import_state": "running", "hybrid_pipeline_software_specs": [{ "id": "632d4b22-10aa-5180-88f0-f52dfb6444d7", "name": "autoai-kb_3.1-py3.7" }], "pipeline": { "id": "a8cbf66e-2e7d-4ec4-b81c-0c24b38ef35f" }, "schemas": { "input": [{ "fields": [{ "name": "CheckingStatus", "type": "other" }, { "name": "LoanDuration", "type": "integer" }, { "name": "CreditHistory", "type": "other" }, { "name": "LoanPurpose", "type": "other" }, { "name": "LoanAmount", "type": "integer" }, { "name": "ExistingSavings", "type": "other" }, { "name": "EmploymentDuration", "type": "other" }, { "name": "InstallmentPercent", "type": "integer" }, { "name": "Sex", "type": "other" }, { "name": "OthersOnLoan", "type": "other" }, { "name": "CurrentResidenceDuration", "type": "integer" }, { "name": "OwnsProperty", "type": "other" }, { "name": "Age", "type": "integer" }, { "name": "InstallmentPlans", "type": "other" }, { "name": "Housing", "type": "other" }, { "name": "ExistingCreditsCount", "type": "integer" }, { "name": "Job", "type": "other" }, { "name": "Dependents", "type": "integer" }, { "name": "Telephone", "type": "other" }, { "name": "ForeignWorker", "type": "other" }], "id": "auto_ai_kb_input_schema" }], "output": [] }, "software_spec": { "id": "8c1a58c6-62b5-4dc4-987a-df751c2756b6", "name": "hybrid_0.1" }, "training_data_references": [{ "connection": { }, "id": "credit_risk_training_light.csv", "location": { "href": "/v2/assets/f5531b4b-d1e8-44e7-a464-67164a2b8a6e?space_id=2219f598-6194-4e9a-9287-0730c69b5f4a" }, "type": "data_asset" }], "type": "wml-hybrid_0.1" }, "metadata": { "created_at": "2020-10-01T09:34:45.069Z", "id": "3f6336a3-b20d-4fd1-9ec6-07797344c799", "modified_at": "2020-10-01T09:34:45.069Z", "name": "P1", "owner": "1000330999", "space_id": "2219f598-6194-4e9a-9287-0730c69b5f4a" } } 
%%bash --out model_id

echo $MODEL_DETAILS | awk -F '"id": ' '{ print $7 }' | cut -d '"' -f 2

%env MODEL_ID=$model_id
env: MODEL_ID=3f6336a3-b20d-4fd1-9ec6-07797344c799

Download model content

If you want to download your saved model, please make the following call.

<a href="https://watson-ml-v4-api.mybluemix.net/wml-restapi-cloud.html#/Models/models_filtered_download" target="_blank" rel="noopener no referrer">Download model content

%%bash

curl -sk -X GET \
    --header "Authorization: Bearer $TOKEN" \
    --output "model.tar.gz" \
    "$DATAPLATFORM_URL/ml/v4/models/$MODEL_ID/download?space_id=$SPACE_ID&version=2020-08-01"

!ls -l model.tar.gz
-rw-r--r-- 1 [email protected] staff 272 Oct 1 11:35 model.tar.gz

Deployment creation

An AutoAI Batch deployment creation is presented below.

<a href="https://watson-ml-v4-api.mybluemix.net/wml-restapi-cloud.html#/Deployments/deployments_create" target="_blank" rel="noopener no referrer">Create deployment

%%bash --out deployment_payload

DEPLOYMENT_PAYLOAD='{"space_id": "'"$SPACE_ID"'","name": "AutoAI deployment","description": "This is description","batch": {}, "hybrid_pipeline_hardware_specs": [{"node_runtime_id": "auto_ai.kb", "hardware_spec": {"name": "M"}}],"asset": {"id": "'"$MODEL_ID"'"}}'
echo $DEPLOYMENT_PAYLOAD | python -m json.tool

%env DEPLOYMENT_PAYLOAD=$deployment_payload
env: DEPLOYMENT_PAYLOAD={ "space_id": "2219f598-6194-4e9a-9287-0730c69b5f4a", "name": "AutoAI deployment", "description": "This is description", "batch": {}, "hybrid_pipeline_hardware_specs": [ { "node_runtime_id": "auto_ai.kb", "hardware_spec": { "name": "M" } } ], "asset": { "id": "3f6336a3-b20d-4fd1-9ec6-07797344c799" } } 
%%bash --out deployment_details

curl -sk -X POST \
    --header "Authorization: Bearer $TOKEN" \
    --header "Content-Type: application/json" \
    --header "Accept: application/json" \
    --data "$DEPLOYMENT_PAYLOAD" \
    "$DATAPLATFORM_URL/ml/v4/deployments?version=2020-08-01"

%env DEPLOYMENT_DETAILS=$deployment_details
env: DEPLOYMENT_DETAILS={ "entity": { "asset": { "id": "3f6336a3-b20d-4fd1-9ec6-07797344c799" }, "batch": { }, "custom": { }, "deployed_asset_type": "model", "description": "This is description", "hybrid_pipeline_hardware_specs": [{ "hardware_spec": { "name": "M" }, "node_runtime_id": "auto_ai.kb" }], "name": "AutoAI deployment", "space_id": "2219f598-6194-4e9a-9287-0730c69b5f4a", "status": { "state": "ready" } }, "metadata": { "created_at": "2020-10-01T09:37:37.176Z", "description": "This is description", "id": "aa2cf720-a21f-4fa9-80d4-a4d1f845a7ee", "modified_at": "2020-10-01T09:37:37.176Z", "name": "AutoAI deployment", "owner": "1000330999", "space_id": "2219f598-6194-4e9a-9287-0730c69b5f4a" } } 
%%bash --out deployment_id

echo $DEPLOYMENT_DETAILS | awk -F '"id": ' '{ print $3 }' | cut -d '"' -f 2

%env DEPLOYMENT_ID=$deployment_id
env: DEPLOYMENT_ID=aa2cf720-a21f-4fa9-80d4-a4d1f845a7ee

Get deployment details

As deployment API is asynchronous, please make sure your deployment is in ready state before going to the next points.

<a href="https://watson-ml-v4-api.mybluemix.net/wml-restapi-cloud.html#/Deployments/deployments_get" target="_blank" rel="noopener no referrer">Get deployment details

%%bash

curl -sk -X GET \
    --header "Authorization: Bearer $TOKEN" \
    --header "Content-Type: application/json" \
    "$DATAPLATFORM_URL/ml/v4/deployments/$DEPLOYMENT_ID?space_id=$SPACE_ID&version=2020-08-01" \
    | python -m json.tool
{ "entity": { "asset": { "id": "3f6336a3-b20d-4fd1-9ec6-07797344c799" }, "batch": {}, "custom": {}, "deployed_asset_type": "model", "description": "This is description", "hybrid_pipeline_hardware_specs": [ { "hardware_spec": { "name": "M" }, "node_runtime_id": "auto_ai.kb" } ], "name": "AutoAI deployment", "space_id": "2219f598-6194-4e9a-9287-0730c69b5f4a", "status": { "state": "ready" } }, "metadata": { "created_at": "2020-10-01T09:37:37.176Z", "description": "This is description", "id": "aa2cf720-a21f-4fa9-80d4-a4d1f845a7ee", "modified_at": "2020-10-01T09:37:37.176Z", "name": "AutoAI deployment", "owner": "1000330999", "space_id": "2219f598-6194-4e9a-9287-0730c69b5f4a" } }

Score your Batch deployment

Scoring for Batch deployment is done by creating jobs. User can specify job payload as a json or as data connection to eg. COS.

<a href="https://watson-ml-v4-api.mybluemix.net/wml-restapi-cloud.html#/Deployment Jobs/deployment_jobs_create" target="_blank" rel="noopener no referrer">Create deployment job

%%bash --out job_payload

JOB_PAYLOAD='{"name": "AutoAI job", "space_id": "'"$SPACE_ID"'","deployment": {"id": "'"$DEPLOYMENT_ID"'"}, "hybrid_pipeline_hardware_specs": [{"node_runtime_id": "auto_ai.kb", "hardware_spec": {"name": "M"}}], "scoring": {"input_data": [{"fields": ["CheckingStatus", "LoanDuration", "CreditHistory", "LoanPurpose", "LoanAmount", "ExistingSavings", "EmploymentDuration", "InstallmentPercent", "Sex", "OthersOnLoan", "CurrentResidenceDuration", "OwnsProperty", "Age", "InstallmentPlans", "Housing", "ExistingCreditsCount", "Job", "Dependents", "Telephone", "ForeignWorker"], "values": [["less_0", 6, "all_credits_paid_back", "car_used", 250, "less_100", "1_to_4", 2, "male", "none", 2, "savings_insurance", 28, "stores", "rent", 1, "skilled", 1, "none", "yes"]]}]}}'
echo $JOB_PAYLOAD | python -m json.tool

%env JOB_PAYLOAD=$job_payload
env: JOB_PAYLOAD={ "name": "AutoAI job", "space_id": "2219f598-6194-4e9a-9287-0730c69b5f4a", "deployment": { "id": "aa2cf720-a21f-4fa9-80d4-a4d1f845a7ee" }, "hybrid_pipeline_hardware_specs": [ { "node_runtime_id": "auto_ai.kb", "hardware_spec": { "name": "M" } } ], "scoring": { "input_data": [ { "fields": [ "CheckingStatus", "LoanDuration", "CreditHistory", "LoanPurpose", "LoanAmount", "ExistingSavings", "EmploymentDuration", "InstallmentPercent", "Sex", "OthersOnLoan", "CurrentResidenceDuration", "OwnsProperty", "Age", "InstallmentPlans", "Housing", "ExistingCreditsCount", "Job", "Dependents", "Telephone", "ForeignWorker" ], "values": [ [ "less_0", 6, "all_credits_paid_back", "car_used", 250, "less_100", "1_to_4", 2, "male", "none", 2, "savings_insurance", 28, "stores", "rent", 1, "skilled", 1, "none", "yes" ] ] } ] } } 
%%bash --out job_id

curl -sk -X POST \
    --header "Authorization: Bearer $TOKEN" \
    --header "Content-Type: application/json" \
    --header "Accept: application/json" \
    --data "$JOB_PAYLOAD" \
    "$DATAPLATFORM_URL/ml/v4/deployment_jobs?version=2020-08-01" \
    | grep '"id": ' | awk -F '"' '{ print $4 }' | sed -n 2p

%env JOB_ID=$job_id
env: JOB_ID=1ec9d794-0a12-4632-a9cd-4e5a2b842dcf

Listing all Batch jobs

%%bash

curl -sk -X GET \
    --header "Authorization: Bearer $TOKEN" \
    --header "Content-Type: application/json" \
    --header "Accept: application/json" \
    "$DATAPLATFORM_URL/ml/v4/deployment_jobs?space_id=$SPACE_ID&version=2020-08-01" \
    | python -m json.tool
{ "resources": [ { "entity": { "deployment": { "id": "aa2cf720-a21f-4fa9-80d4-a4d1f845a7ee" }, "hybrid_pipeline_hardware_specs": [ { "hardware_spec": { "name": "M" }, "node_runtime_id": "auto_ai.kb" } ], "platform_job": { "job_id": "1a9a3796-3a77-43d8-9fe7-8a92f7199166", "run_id": "b10a72d5-30d2-43b9-9694-7a2a928b8190" }, "scoring": { "input_data": [ { "fields": [ "CheckingStatus", "LoanDuration", "CreditHistory", "LoanPurpose", "LoanAmount", "ExistingSavings", "EmploymentDuration", "InstallmentPercent", "Sex", "OthersOnLoan", "CurrentResidenceDuration", "OwnsProperty", "Age", "InstallmentPlans", "Housing", "ExistingCreditsCount", "Job", "Dependents", "Telephone", "ForeignWorker" ], "values": [ [ "less_0", 6, "all_credits_paid_back", "car_used", 250, "less_100", "1_to_4", 2, "male", "none", 2, "savings_insurance", 28, "stores", "rent", 1, "skilled", 1, "none", "yes" ] ] } ], "status": { "completed_at": "", "running_at": "", "state": "queued" } } }, "metadata": { "created_at": "2020-10-01T09:43:16.576Z", "id": "1ec9d794-0a12-4632-a9cd-4e5a2b842dcf", "name": "AutoAI job", "space_id": "2219f598-6194-4e9a-9287-0730c69b5f4a" } } ] }

Get particular job details

<a href="https://watson-ml-v4-api.mybluemix.net/wml-restapi-cloud.html#/Deployment Jobs/deployment_jobs_get" target="_blank" rel="noopener no referrer">Get job details

%%bash

curl -sk -X GET \
    --header "Authorization: Bearer $TOKEN" \
    --header "Content-Type: application/json" \
    --header "Accept: application/json" \
    "$DATAPLATFORM_URL/ml/v4/deployment_jobs/$JOB_ID?space_id=$SPACE_ID&version=2020-08-01" \
    | python -m json.tool
{ "entity": { "deployment": { "id": "aa2cf720-a21f-4fa9-80d4-a4d1f845a7ee" }, "hybrid_pipeline_hardware_specs": [ { "hardware_spec": { "name": "M" }, "node_runtime_id": "auto_ai.kb" } ], "platform_job": { "job_id": "1a9a3796-3a77-43d8-9fe7-8a92f7199166", "run_id": "b10a72d5-30d2-43b9-9694-7a2a928b8190" }, "scoring": { "input_data": [ { "fields": [ "CheckingStatus", "LoanDuration", "CreditHistory", "LoanPurpose", "LoanAmount", "ExistingSavings", "EmploymentDuration", "InstallmentPercent", "Sex", "OthersOnLoan", "CurrentResidenceDuration", "OwnsProperty", "Age", "InstallmentPlans", "Housing", "ExistingCreditsCount", "Job", "Dependents", "Telephone", "ForeignWorker" ], "values": [ [ "less_0", 6, "all_credits_paid_back", "car_used", 250, "less_100", "1_to_4", 2, "male", "none", 2, "savings_insurance", 28, "stores", "rent", 1, "skilled", 1, "none", "yes" ] ] } ], "status": { "completed_at": "", "running_at": "", "state": "queued" } } }, "metadata": { "created_at": "2020-10-01T09:43:16.576Z", "id": "1ec9d794-0a12-4632-a9cd-4e5a2b842dcf", "name": "AutoAI job", "space_id": "2219f598-6194-4e9a-9287-0730c69b5f4a" } }

Cancel job

Tip: You can cancel running job by calling delete method. Just convert below cell to code and run it.

%%bash

curl -sk -X DELETE \
    --header "Authorization: Bearer $TOKEN" \
    --header "Content-Type: application/json" \
    --header "Accept: application/json" \
    "$DATAPLATFORM_URL/ml/v4/deployment_jobs/$JOB_ID?space_id=$SPACE_ID&version=2020-08-01"

Listing all deployments

<a href="https://watson-ml-v4-api.mybluemix.net/wml-restapi-cloud.html#/Deployments/deployments_list" target="_blank" rel="noopener no referrer">List deployments details

%%bash

curl -sk -X GET \
    --header "Authorization: Bearer $TOKEN" \
    --header "Content-Type: application/json" \
    "$DATAPLATFORM_URL/ml/v4/deployments?space_id=$SPACE_ID&version=2020-08-01" \
    | python -m json.tool
{ "resources": [ { "entity": { "asset": { "id": "3f6336a3-b20d-4fd1-9ec6-07797344c799" }, "batch": {}, "custom": {}, "deployed_asset_type": "model", "description": "This is description", "hybrid_pipeline_hardware_specs": [ { "hardware_spec": { "name": "M" }, "node_runtime_id": "auto_ai.kb" } ], "name": "AutoAI deployment", "space_id": "2219f598-6194-4e9a-9287-0730c69b5f4a", "status": { "state": "ready" } }, "metadata": { "created_at": "2020-10-01T09:35:43.215Z", "description": "This is description", "id": "4f3ba600-bbe4-4b13-a25d-7fe6a0cbe74b", "modified_at": "2020-10-01T09:35:43.215Z", "name": "AutoAI deployment", "owner": "1000330999", "space_id": "2219f598-6194-4e9a-9287-0730c69b5f4a" } }, { "entity": { "asset": { "id": "04f3cc1c-51b9-4f3d-a196-de58961e2002" }, "custom": {}, "deployed_asset_type": "model", "hardware_spec": { "id": "c076e82c-b2a7-4d20-9c0f-1f0c2fdf5a24", "name": "M", "num_nodes": 1 }, "hybrid_pipeline_hardware_specs": [ { "hardware_spec": { "name": "S", "num_nodes": 1 }, "node_runtime_id": "auto_ai.kb" } ], "name": "Iris AutoAI Deployment tests", "online": {}, "space_id": "2219f598-6194-4e9a-9287-0730c69b5f4a", "status": { "online_url": { "url": "https://sep29b93-cpd-sep29b93.apps.ocwmlfvt454.cp.fyre.ibm.com/ml/v4/deployments/513da634-bf73-4c74-91a9-2c02dafa70a2/predictions" }, "state": "ready" } }, "metadata": { "created_at": "2020-10-01T07:05:44.639Z", "id": "513da634-bf73-4c74-91a9-2c02dafa70a2", "modified_at": "2020-10-01T07:05:44.639Z", "name": "Iris AutoAI Deployment tests", "owner": "1000330999", "space_id": "2219f598-6194-4e9a-9287-0730c69b5f4a" } }, { "entity": { "asset": { "id": "3f6336a3-b20d-4fd1-9ec6-07797344c799" }, "batch": {}, "custom": {}, "deployed_asset_type": "model", "description": "This is description", "hybrid_pipeline_hardware_specs": [ { "hardware_spec": { "name": "M" }, "node_runtime_id": "auto_ai.kb" } ], "name": "AutoAI deployment", "space_id": "2219f598-6194-4e9a-9287-0730c69b5f4a", "status": { "state": "ready" } }, "metadata": { "created_at": "2020-10-01T09:36:56.873Z", "description": "This is description", "id": "546b95fb-1be0-4f3a-9238-93b2bc19b858", "modified_at": "2020-10-01T09:36:56.873Z", "name": "AutoAI deployment", "owner": "1000330999", "space_id": "2219f598-6194-4e9a-9287-0730c69b5f4a" } }, { "entity": { "asset": { "id": "3f6336a3-b20d-4fd1-9ec6-07797344c799" }, "batch": {}, "custom": {}, "deployed_asset_type": "model", "description": "This is description", "hybrid_pipeline_hardware_specs": [ { "hardware_spec": { "name": "M" }, "node_runtime_id": "auto_ai.kb" } ], "name": "AutoAI deployment", "space_id": "2219f598-6194-4e9a-9287-0730c69b5f4a", "status": { "state": "ready" } }, "metadata": { "created_at": "2020-10-01T09:37:37.176Z", "description": "This is description", "id": "aa2cf720-a21f-4fa9-80d4-a4d1f845a7ee", "modified_at": "2020-10-01T09:37:37.176Z", "name": "AutoAI deployment", "owner": "1000330999", "space_id": "2219f598-6194-4e9a-9287-0730c69b5f4a" } } ] }

6. Cleaning section

Below section is useful when you want to clean all of your previous work within this notebook. Just convert below cells into the code and run them.

Delete training run

Tip: You can completely delete a training run with its metadata.

<a href="https://watson-ml-v4-api.mybluemix.net/wml-restapi-cloud.html#/Trainings/trainings_delete" target="_blank" rel="noopener no referrer">Deleting training

%%bash

TRAINING_ID_TO_DELETE=...

curl -sk -X DELETE \
    --header "Authorization: Bearer $TOKEN" \
    --header "Content-Type: application/json" \
    --header "Accept: application/json" \
    "$DATAPLATFORM_URL/ml/v4/trainings/$TRAINING_ID_TO_DELETE?space_id=$SPACE_ID&version=2020-08-01&hard_delete=true"

Delete job

Tip: If you want remove job completely (with metadata), just specify hard_delete to True.

%%bash

curl -sk -X DELETE \
    --header "Authorization: Bearer $TOKEN" \
    --header "Content-Type: application/json" \
    --header "Accept: application/json" \
    "$DATAPLATFORM_URL/ml/v4/deployment_jobs/$JOB_ID?space_id=$SPACE_ID&version=2020-08-01&hard_delete=true"

Deleting deployment

Tip: You can delete existing deployment by calling DELETE method.

%%bash

curl -sk -X DELETE \
    --header "Authorization: Bearer $TOKEN" \
    --header "Content-Type: application/json" \
    --header "Accept: application/json" \
    "$DATAPLATFORM_URL/ml/v4/deployments/$DEPLOYMENT_ID?space_id=$SPACE_ID&version=2020-08-01"

Delete model from repository

Tip: If you want to completely remove your stored model and model metadata, just use a DELETE method.

<a href="https://watson-ml-v4-api.mybluemix.net/wml-restapi-cloud.html#/Models/models_delete" target="_blank" rel="noopener no referrer">Delete model from repository

%%bash

curl -sk -X DELETE \
    --header "Authorization: Bearer $TOKEN" \
    --header "Content-Type: application/json" \
    "$DATAPLATFORM_URL/ml/v4/models/$MODEL_ID?space_id=$SPACE_ID&version=2020-08-01"

7. Summary and next steps

You successfully completed this notebook!.

You learned how to use cURL calls to store, deploy and score a AutoAI model in WML.

Authors

Amadeusz Masny, Python Software Developer in Watson Machine Learning at IBM Jan Sołtysik, Intern in Watson Machine Learning at IBM

Copyright © 2020-2025 IBM. This notebook and its source code are released under the terms of the MIT License.