Book a Demo!
CoCalc Logo Icon
StoreFeaturesDocsShareSupportNewsAboutPoliciesSign UpSign In
YStrano
GitHub Repository: YStrano/DataScience_GA
 Path: blob/master/lessons/lesson_10-sub-Jacob_Koehler/03-ROC-AUC - done.ipynb
1904 views
Kernel: Python 3

More Evaluation Metrics

  • Precision/Recall Curves

  • ROC and AUC

  • Multiclass Evaluation

Recall Oriented TaskPrecision Oriented Task
Tumor DetectionSearch Engine Results
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from sklearn.datasets import load_breast_cancer, load_digits
from sklearn.model_selection import train_test_split

Adapting the Classifier

Depending on the application, we may want to change the classifier and its threshold for class membership. Here, we look at the ability of the classifier to detect 3's and discuss how we might be interested in changing the threshold.

digits = load_digits()

X = digits.data
y = digits.target
X_test, X_train, y_test, y_train = train_test_split(X, y)

from sklearn.model_selection import cross_val_score

y_train = (y_train == 3)
y_test = (y_test == 3)

from sklearn.linear_model import LogisticRegression

lgr = LogisticRegression()

cross_val_score(lgr, X_train, y_train, cv = 3, scoring = 'accuracy')
array([0.98666667, 0.99333333, 0.98 ])
lgr.fit(X_train, y_train)
LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True, intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1, penalty='l2', random_state=None, solver='liblinear', tol=0.0001, verbose=0, warm_start=False)
lgr.predict(X_train[312].reshape(1,-1))
array([False])
y_score = lgr.decision_function(X_train[312].reshape(1,-1))

y_score
array([-32.83098589])
X_train[312]
array([ 0., 0., 0., 7., 14., 0., 0., 0., 0., 0., 4., 16., 5., 0., 0., 0., 0., 0., 14., 9., 0., 0., 10., 3., 0., 7., 15., 0., 0., 9., 15., 0., 0., 12., 15., 8., 10., 15., 10., 0., 0., 8., 15., 12., 16., 12., 1., 0., 0., 0., 0., 5., 15., 3., 0., 0., 0., 0., 0., 9., 13., 0., 0., 0.])
plt.imshow(X_train[312].reshape(8,8)) 

#using single value of 312 to see if it is a 3 or not and it tells you false, so no
<matplotlib.image.AxesImage at 0x1a22f83b38>
Image in a Jupyter notebook
threshold = 0
y_digit_predict = (y_score > threshold)
y_digit_predict
array([False])
#you can adjust what you are willing to accept as a member of a class by adjusting the threshold

threshold = 2000 
y_digit_predict = (y_score > threshold)
y_digit_predict
array([False])

Adjusting the Threshold

y_pred_prob = lgr.predict_proba(X_test)

y_pred_prob
array([[1.00000000e+00, 1.27821843e-11], [1.00000000e+00, 7.08886452e-12], [9.99999999e-01, 1.44998030e-09], ..., [9.99999920e-01, 7.97564955e-08], [1.76808997e-04, 9.99823191e-01], [9.99999910e-01, 8.95395114e-08]])
y_pred_adjusted = []
for probability in y_pred_prob[1]:
    if probability > .2:
        y_pred_adjusted.append(1)
    else:
        y_pred_adjusted.append(0)

len(y_pred_prob)
1347
len(y_pred_adjusted)
2
from sklearn.metrics import precision_recall_curve
from sklearn.model_selection import cross_val_predict

y_scores = cross_val_predict(lgr, X_train, y_train, cv = 3, method = 'decision_function')

precisions, recalls, thresholds = precision_recall_curve(y_train, y_scores)

plt.plot(thresholds, precisions[:-1], label = 'Precision')
plt.plot(thresholds, recalls[:-1], label = 'Recall')
plt.xlabel('Threshold Values');
plt.legend(frameon = False);
Image in a Jupyter notebook
from sklearn.metrics import confusion_matrix

confusion_matrix(y_test, lgr.predict(X_test))
array([[1211, 4], [ 16, 116]])

Cancer Example

We want to explore a classification problem using breast cancer data. Here, our goal is to classify a tumor as malignant or not based on measurements of the tumor. In this example, we want to consider the nature of the classifier examined, and determine how to alter the boundary to better the classifier to our liking.

  • Load and examine data

  • Compare LogisticRegression, SGDClassifier, and a DummyClassifier

  • Examine Precision vs. Recall curve

  • Examine ROC Curve

  • Shift Decision Boundary and evaluate

cancer = load_breast_cancer()

print(cancer.DESCR)
Breast Cancer Wisconsin (Diagnostic) Database ============================================= Notes ----- Data Set Characteristics: :Number of Instances: 569 :Number of Attributes: 30 numeric, predictive attributes and the class :Attribute Information: - radius (mean of distances from center to points on the perimeter) - texture (standard deviation of gray-scale values) - perimeter - area - smoothness (local variation in radius lengths) - compactness (perimeter^2 / area - 1.0) - concavity (severity of concave portions of the contour) - concave points (number of concave portions of the contour) - symmetry - fractal dimension ("coastline approximation" - 1) The mean, standard error, and "worst" or largest (mean of the three largest values) of these features were computed for each image, resulting in 30 features. For instance, field 3 is Mean Radius, field 13 is Radius SE, field 23 is Worst Radius. - class: - WDBC-Malignant - WDBC-Benign :Summary Statistics: ===================================== ====== ====== Min Max ===================================== ====== ====== radius (mean): 6.981 28.11 texture (mean): 9.71 39.28 perimeter (mean): 43.79 188.5 area (mean): 143.5 2501.0 smoothness (mean): 0.053 0.163 compactness (mean): 0.019 0.345 concavity (mean): 0.0 0.427 concave points (mean): 0.0 0.201 symmetry (mean): 0.106 0.304 fractal dimension (mean): 0.05 0.097 radius (standard error): 0.112 2.873 texture (standard error): 0.36 4.885 perimeter (standard error): 0.757 21.98 area (standard error): 6.802 542.2 smoothness (standard error): 0.002 0.031 compactness (standard error): 0.002 0.135 concavity (standard error): 0.0 0.396 concave points (standard error): 0.0 0.053 symmetry (standard error): 0.008 0.079 fractal dimension (standard error): 0.001 0.03 radius (worst): 7.93 36.04 texture (worst): 12.02 49.54 perimeter (worst): 50.41 251.2 area (worst): 185.2 4254.0 smoothness (worst): 0.071 0.223 compactness (worst): 0.027 1.058 concavity (worst): 0.0 1.252 concave points (worst): 0.0 0.291 symmetry (worst): 0.156 0.664 fractal dimension (worst): 0.055 0.208 ===================================== ====== ====== :Missing Attribute Values: None :Class Distribution: 212 - Malignant, 357 - Benign :Creator: Dr. William H. Wolberg, W. Nick Street, Olvi L. Mangasarian :Donor: Nick Street :Date: November, 1995 This is a copy of UCI ML Breast Cancer Wisconsin (Diagnostic) datasets. https://goo.gl/U2Uwz2 Features are computed from a digitized image of a fine needle aspirate (FNA) of a breast mass. They describe characteristics of the cell nuclei present in the image. Separating plane described above was obtained using Multisurface Method-Tree (MSM-T) [K. P. Bennett, "Decision Tree Construction Via Linear Programming." Proceedings of the 4th Midwest Artificial Intelligence and Cognitive Science Society, pp. 97-101, 1992], a classification method which uses linear programming to construct a decision tree. Relevant features were selected using an exhaustive search in the space of 1-4 features and 1-3 separating planes. The actual linear program used to obtain the separating plane in the 3-dimensional space is that described in: [K. P. Bennett and O. L. Mangasarian: "Robust Linear Programming Discrimination of Two Linearly Inseparable Sets", Optimization Methods and Software 1, 1992, 23-34]. This database is also available through the UW CS ftp server: ftp ftp.cs.wisc.edu cd math-prog/cpo-dataset/machine-learn/WDBC/ References ---------- - W.N. Street, W.H. Wolberg and O.L. Mangasarian. Nuclear feature extraction for breast tumor diagnosis. IS&T/SPIE 1993 International Symposium on Electronic Imaging: Science and Technology, volume 1905, pages 861-870, San Jose, CA, 1993. - O.L. Mangasarian, W.N. Street and W.H. Wolberg. Breast cancer diagnosis and prognosis via linear programming. Operations Research, 43(4), pages 570-577, July-August 1995. - W.H. Wolberg, W.N. Street, and O.L. Mangasarian. Machine learning techniques to diagnose breast cancer from fine-needle aspirates. Cancer Letters 77 (1994) 163-171. 
from sklearn.linear_model import LogisticRegression, SGDClassifier
from sklearn.dummy import DummyClassifier

cancer.target
array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0, 1, 0, 0, 1, 1, 1, 1, 0, 1, 0, 0, 1, 0, 1, 0, 0, 1, 1, 1, 0, 0, 1, 0, 0, 0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 1, 1, 1, 0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 0, 0, 1, 0, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 0, 1, 1, 0, 0, 0, 1, 0, 1, 0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 0, 1, 1, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 1, 1, 0, 1, 1, 0, 0, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 0, 1, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 0, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 0, 1, 0, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 0, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 0, 1, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 1, 0, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1])
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix, classification_report

X_train, X_test, y_train, y_test = train_test_split(cancer.data, cancer.target)

lgr = LogisticRegression()
lgr.fit(X_train, y_train)
lg_pred = lgr.predict(X_train)
print(confusion_matrix(y_train, lg_pred))
[[145 10] [ 5 266]] 
print(classification_report(y_train, lg_pred))
precision recall f1-score support 0 0.97 0.94 0.95 155 1 0.96 0.98 0.97 271 avg / total 0.96 0.96 0.96 426 
from sklearn.model_selection import cross_val_predict

lgr_scores = cross_val_predict(lgr, X_train, y_train, cv = 5, method = 'decision_function')

lgr_scores[:10]
array([ 4.74538118, 8.28651392, 5.95784711, -116.57742553, 2.56762554, 10.11739245, 3.011605 , -3.86397083, 5.80765056, 5.08453476])

Comparing Precision and Recall

We can visualize the changes that occur accross these metrics together. To begin, we plot the 

from sklearn.metrics import precision_recall_curve, roc_curve, roc_auc_score, auc

from sklearn.metrics import precision_recall_curve
lgr.fit(X_train, y_train)
LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True, intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1, penalty='l2', random_state=None, solver='liblinear', tol=0.0001, verbose=0, warm_start=False)
y_scr = cross_val_predict(lgr, X_train, y_train, cv = 3, method = 'decision_function')
y_scr.shape
(426,)
y_train.shape
(426,)
precisions, recall, threshold = precision_recall_curve(y_test, lgr.predict(X_test))

plt.figure(figsize = (7, 7))
plt.xlim([0.0, 1.01])
plt.ylim([0.0, 1.01])
plt.plot(precisions, recall, label='Precision-Recall Curve')
plt.xlabel('Precision', fontsize=16)
plt.ylabel('Recall', fontsize=16)

#optimal place for recall is as close as you can get to top right corner of graph
#aim towards perfect precision and perfect recall
#in this case (1,1)
Text(0,0.5,'Recall')
Image in a Jupyter notebook
y_score_lr = lgr.fit(X_train, y_train).decision_function(X_test)
fpr_lr, tpr_lr, _ = roc_curve(y_test, y_score_lr)
roc_auc_lr = auc(fpr_lr, tpr_lr)

plt.figure(figsize = (10, 7))
plt.xlim([-0.01, 1.00])
plt.ylim([-0.01, 1.01])
plt.plot(fpr_lr, tpr_lr, lw=3, label='LogRegr ROC curve (area = {:0.2f})'.format(roc_auc_lr))
plt.xlabel('False Positive Rate', fontsize=16)
plt.ylabel('True Positive Rate', fontsize=16)
plt.title('ROC curve (Cancer Data)', fontsize=16)
plt.legend(loc='lower right', fontsize=13)
plt.plot([0, 1], [0, 1], color='navy', lw=3, linestyle='--') #this is the dummy line which represents pure randomness like a flop of a cin, 0 or 1

#in this case, want to be as close as possible to upper left corner
#the larger the area above the dummy dotted line, the better the score or AUC (AUC is area under the curve)
[<matplotlib.lines.Line2D at 0x1a23826198>]
Image in a Jupyter notebook

Other Classifiers

sgd = SGDClassifier(max_iter=1000)
sgd.fit(X_train, y_train)
sgd_pred = sgd.predict(X_train)
print(confusion_matrix(y_train, sgd_pred))
print(classification_report(y_train, sgd_pred))
[[136 19] [ 6 265]] precision recall f1-score support 0 0.96 0.88 0.92 155 1 0.93 0.98 0.95 271 avg / total 0.94 0.94 0.94 426 
dum = DummyClassifier(strategy='most_frequent')
dum.fit(X_train, y_train)
dum_pred = dum.predict(X_train)
print(confusion_matrix(y_train, dum_pred))
print(classification_report(y_train, dum_pred))
[[ 0 155] [ 0 271]] precision recall f1-score support 0 0.00 0.00 0.00 155 1 0.64 1.00 0.78 271 avg / total 0.40 0.64 0.49 426
/anaconda3/lib/python3.6/site-packages/sklearn/metrics/classification.py:1135: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples. 'precision', 'predicted', average, warn_for)

Digits and Multi-Class Classification

What are the difficult digits to see?

from sklearn.datasets import load_digits
import seaborn as sns
dataset = load_digits()
X, y = dataset.data, dataset.target

lgr.fit(X, y)
LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True, intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1, penalty='l2', random_state=None, solver='liblinear', tol=0.0001, verbose=0, warm_start=False)
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y)

lgr.fit(X_train, y_train)
lg_pred = lgr.predict(X_test)
mat = confusion_matrix(y_test, lg_pred)
plt.figure(figsize = (12, 9))
sns.heatmap(mat, annot = True,fmt="d", cmap="viridis", alpha = 0.6) #annot pumps the values in as annotation
<matplotlib.axes._subplots.AxesSubplot at 0x11034feb8>
Image in a Jupyter notebook
print(classification_report(y_test, lg_pred))
precision recall f1-score support 0 1.00 1.00 1.00 45 1 0.93 1.00 0.96 39 2 0.98 1.00 0.99 42 3 0.98 0.91 0.94 44 4 1.00 0.96 0.98 48 5 0.95 0.95 0.95 44 6 1.00 0.98 0.99 52 7 0.96 0.98 0.97 50 8 0.97 0.92 0.95 38 9 0.94 1.00 0.97 48 avg / total 0.97 0.97 0.97 450

Review

dataset = load_digits()
X, y = dataset.data, dataset.target

y_binary_imbalanced = y.copy()
y_binary_imbalanced[y_binary_imbalanced != 1] = 0

X_train, X_test, y_train, y_test = train_test_split(X, y_binary_imbalanced)

from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression

dum = DummyClassifier().fit(X_train, y_train)
lgr = LogisticRegression().fit(X_train, y_train)

dum.score(X_test, y_test)
0.8266666666666667
lgr.score(X_test, y_test)
0.9577777777777777
from sklearn.metrics import confusion_matrix, classification_report

confusion_matrix(y_test, lgr.predict(X_test))
array([[397, 10], [ 9, 34]])
print(classification_report(y_test, lgr.predict(X_test)))
precision recall f1-score support 0 0.98 0.98 0.98 407 1 0.77 0.79 0.78 43 avg / total 0.96 0.96 0.96 450 
#decision function
X_train, X_test, y_train, y_test = train_test_split(X, y_binary_imbalanced, random_state=0)
y_scores_lr = lgr.fit(X_train, y_train).decision_function(X_test)
y_score_list = list(zip(y_test[0:20], y_scores_lr[0:20]))

# show the decision_function scores for first 20 instances
y_score_list
[(0, -23.17672219032446), (0, -13.541097998188556), (0, -21.72259438367228), (0, -18.90745367451705), (0, -19.735861448886578), (0, -9.749531202700574), (1, 5.234582714211504), (0, -19.307421141904705), (0, -25.101013223364184), (0, -21.826901181142077), (0, -24.151051279273634), (0, -19.5767098694618), (0, -22.574755095082132), (0, -10.823600209909634), (0, -11.912481921586272), (0, -10.97950057771539), (1, 11.205943104028172), (0, -27.64580991535633), (0, -12.859075688221054), (0, -25.84859078059094)]
X_train, X_test, y_train, y_test = train_test_split(X, y_binary_imbalanced, random_state=0)
y_proba_lr = lgr.fit(X_train, y_train).predict_proba(X_test)
y_proba_list = list(zip(y_test[0:20], y_proba_lr[0:20,1]))

# show the probability of positive class for first 20 instances
y_proba_list
[(0, 8.637821465159233e-11), (0, 1.3137136149911269e-06), (0, 3.7000322509313465e-10), (0, 6.173681547140952e-09), (0, 2.6916731258575595e-09), (0, 5.850568274992845e-05), (1, 0.9946908056379261), (0, 4.117464349701057e-09), (0, 1.2575140460785078e-11), (0, 3.32462716719908e-10), (0, 3.2701730624135435e-11), (0, 3.1405855846856138e-09), (0, 1.580164156431959e-10), (0, 1.9946973566405738e-05), (0, 6.7386087141757545e-06), (0, 1.709320352634685e-05), (1, 0.9999864166717638), (0, 9.869399441180764e-13), (0, 2.605173131907056e-06), (0, 5.946829447190687e-12)]
#precision recall curve
from sklearn.metrics import precision_recall_curve

precision, recall, thresholds = precision_recall_curve(y_test, y_scores_lr)
closest_zero = np.argmin(np.abs(thresholds))
closest_zero_p = precision[closest_zero]
closest_zero_r = recall[closest_zero]

plt.figure()
plt.xlim([0.0, 1.01])
plt.ylim([0.0, 1.01])
plt.plot(precision, recall, label='Precision-Recall Curve')
plt.plot(closest_zero_p, closest_zero_r, 'o', markersize = 12, fillstyle = 'none', c='r', mew=3)
plt.xlabel('Precision', fontsize=16)
plt.ylabel('Recall', fontsize=16)
Text(0,0.5,'Recall')
Image in a Jupyter notebook
from sklearn.metrics import roc_curve, auc

X_train, X_test, y_train, y_test = train_test_split(X, y_binary_imbalanced, random_state=0)

y_score_lr = lgr.fit(X_train, y_train).decision_function(X_test)
fpr_lr, tpr_lr, _ = roc_curve(y_test, y_score_lr)
roc_auc_lr = auc(fpr_lr, tpr_lr)

plt.figure()
plt.xlim([-0.01, 1.00])
plt.ylim([-0.01, 1.01])
plt.plot(fpr_lr, tpr_lr, lw=3, label='LogRegr ROC curve (area = {:0.2f})'.format(roc_auc_lr))
plt.xlabel('False Positive Rate', fontsize=16)
plt.ylabel('True Positive Rate', fontsize=16)
plt.title('ROC curve (1-of-10 digits classifier)', fontsize=16)
plt.legend(loc='lower right', fontsize=13)
plt.plot([0, 1], [0, 1], color='navy', lw=3, linestyle='--')
[<matplotlib.lines.Line2D at 0x1a22720438>]
Image in a Jupyter notebook
from sklearn.model_selection import cross_val_score
from sklearn.svm import SVC

dataset = load_digits()
# again, making this a binary problem with 'digit 1' as positive class 
# and 'not 1' as negative class
X, y = dataset.data, dataset.target == 1
clf = LogisticRegression()

# accuracy is the default scoring metric
print('Cross-validation (accuracy)', cross_val_score(clf, X, y, cv=5))
# use AUC as scoring metric
print('Cross-validation (AUC)', cross_val_score(clf, X, y, cv=5, scoring = 'roc_auc'))
# use recall as scoring metric
print('Cross-validation (recall)', cross_val_score(clf, X, y, cv=5, scoring = 'recall'))
Cross-validation (accuracy) [0.91666667 0.97777778 0.96657382 0.98885794 0.94986072] Cross-validation (AUC) [0.97447912 0.99372437 0.99303406 0.99914001 0.97205022] Cross-validation (recall) [0.91891892 0.91891892 0.75 0.91666667 0.80555556] 
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import roc_auc_score

dataset = load_digits()
X, y = dataset.data, dataset.target == 1
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

clf = LogisticRegression()
grid_values = {'C': [0.001, 0.01, 0.05, 0.1, 1, 10, 100]}

# default metric to optimize over grid parameters: accuracy
grid_clf_acc = GridSearchCV(clf, param_grid = grid_values)
grid_clf_acc.fit(X_train, y_train)
y_decision_fn_scores_acc = grid_clf_acc.decision_function(X_test) 

print('Grid best parameter (max. accuracy): ', grid_clf_acc.best_params_)
print('Grid best score (accuracy): ', grid_clf_acc.best_score_)

# alternative metric to optimize over grid parameters: AUC
grid_clf_auc = GridSearchCV(clf, param_grid = grid_values, scoring = 'roc_auc')
grid_clf_auc.fit(X_train, y_train)
y_decision_fn_scores_auc = grid_clf_auc.decision_function(X_test) 

print('Test set AUC: ', roc_auc_score(y_test, y_decision_fn_scores_auc))
print('Grid best parameter (max. AUC): ', grid_clf_auc.best_params_)
print('Grid best score (AUC): ', grid_clf_auc.best_score_)
Grid best parameter (max. accuracy): {'C': 0.1} Grid best score (accuracy): 0.9732739420935412 Test set AUC: 0.995085995085995 Grid best parameter (max. AUC): {'C': 0.01} Grid best score (AUC): 0.9930270809018346 
from sklearn.metrics.scorer import SCORERS

print(sorted(list(SCORERS.keys())))
['accuracy', 'adjusted_mutual_info_score', 'adjusted_rand_score', 'average_precision', 'completeness_score', 'explained_variance', 'f1', 'f1_macro', 'f1_micro', 'f1_samples', 'f1_weighted', 'fowlkes_mallows_score', 'homogeneity_score', 'log_loss', 'mean_absolute_error', 'mean_squared_error', 'median_absolute_error', 'mutual_info_score', 'neg_log_loss', 'neg_mean_absolute_error', 'neg_mean_squared_error', 'neg_mean_squared_log_error', 'neg_median_absolute_error', 'normalized_mutual_info_score', 'precision', 'precision_macro', 'precision_micro', 'precision_samples', 'precision_weighted', 'r2', 'recall', 'recall_macro', 'recall_micro', 'recall_samples', 'recall_weighted', 'roc_auc', 'v_measure_score'] 
lgr = LogisticRegression()

lgr.fit(X_train, y_train) #C=1.0 is the default
LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True, intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1, penalty='l2', random_state=None, solver='liblinear', tol=0.0001, verbose=0, warm_start=False)
param_grid = {'C': [0.01, 0.1, 1.0, 10, 100]}

#this causes the fit to fail

#param_grid = [{'C': [0.01, 0.1, 1.0, 10, 100]}, {'class_weight': ['balanced', 'imbalanced']}, {'fit_intercept': [True, False]}]

grid = GridSearchCV(lgr, param_grid = param_grid, cv = 5)

grid.fit(X_train, y_train)
GridSearchCV(cv=5, error_score='raise', estimator=LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True, intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1, penalty='l2', random_state=None, solver='liblinear', tol=0.0001, verbose=0, warm_start=False), fit_params=None, iid=True, n_jobs=1, param_grid={'C': [0.01, 0.1, 1.0, 10, 100]}, pre_dispatch='2*n_jobs', refit=True, return_train_score='warn', scoring=None, verbose=0)
grid.best_params_
{'C': 0.1}
grid.best_estimator_
LogisticRegression(C=0.1, class_weight=None, dual=False, fit_intercept=True, intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1, penalty='l2', random_state=None, solver='liblinear', tol=0.0001, verbose=0, warm_start=False)
super_model = grid.best_estimator_

super_model.fit(X_train, y_train)
LogisticRegression(C=0.1, class_weight=None, dual=False, fit_intercept=True, intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1, penalty='l2', random_state=None, solver='liblinear', tol=0.0001, verbose=0, warm_start=False)