Unit 02 · Chapter 11

Model Evaluation

Model Evaluation01

Deciding on which Model

We’ve seen a large number of different models now, as well as a number of different use cases. How should we decide which model to use and when?

Models for different Tasks:

Classification:

  • kNN
  • Decision Trees
  • Perceptrons
  • Logistic Regression
  • SVMs

Regression:

  • kNN
  • Decision Trees
  • Linear Regression
  • SVRs

Clustering:

  • kMeans
  • Hierarchical
Model Evaluation02

Model Evaluation

Lets suppose we were training a classifier to predict whether or not Notre Dame won a football game

classifierfeaturesunseen games
kNN (k = 5)yards81.6%
decision tree (depth 3)yards82.3%
logistic regressionyards83.7%
random forest (200 trees)15 stats83.6%
logistic regression15 stats85.8%

How good are these classifiers? Thus far we've been using  
(how close a given set of predictions are to their  )

Model Evaluation · Confusion Matrices04

Model Evaluation

However, using only   can be misleading 

classifierfeaturesunseen games
always guess "win"none75.2%
kNN (k = 5)yards81.6%
decision tree (depth 3)yards82.3%
logistic regressionyards83.7%
random forest (200 trees)15 stats83.6%
logistic regression15 stats85.8%

Notre Dame wins 3 out of 4 of its football games, resulting in an imbalanced dataset. Out of our 141 samples, only 35 of them are losses. This means that we could get an accuracy of 75% by just predicting Notre Dame to win every game.

Model Evaluation · Confusion Matrices05

Confusion Matrices

A confusion matrix helps assess a classification model's performance by tabulating each of the four possible outcomes from a binary prediction task

99 true positives 7 false negatives 16 false positives 19 true negatives predicted win predicted loss won lost 106 wins 35 losses

  refer to data points belonging to the target class (win), while   refer to data points belonging to the non-target class (loss)

Model Evaluation · Confusion Matrices06

Confusion Matrices and Imbalanced Datasets

Notre Dame wins 3 games in 4, so the games worth predicting are the losses

19 true positives 16 false negatives 7 false positives 99 true negatives predicted loss predicted win lost won 35 losses 106 wins
Model Evaluation · Confusion Matrices07

Type 1 vs. Type 2 Errors

Type 2 errors are called "false negatives"

predicted loss predicted win lost won 35 losses 106 wins

Type 1 errors are called "false positives"

Depending on the task at hand, we may care more about these types of errors, but only considering the overall accuracy may hide them from us.

Consider a task in which we're performing a cancer screening, with the presence of cancer being the positive class.

If we have a dataset with 998 healthy examples, and 2 cancerous examples, we could get an overall accuracy of 99.8% by just predicting that everyone is healthy!

0 true positives 2 false negatives 0 false positives 998 true negatives predicted cancer predicted healthy cancer healthy 2 sick 998 healthy

However, missing the 2 sick patients would be false negatives, and if we're trying to catch deadly diseases, making these type 2 errors are extremely costly!

Model Evaluation · Confusion Matrices08

Type 1 vs. Type 2 Errors

Type 1 errors are called "false positives"

Type 2 errors are called "false negatives"

Model Evaluation · Confusion Matrices09

Type 1 and Type 2 Errors in the Body

Our immune system is a classifier: every cell and molecule it meets is either "self" or "not self".

The body can't push both to zero, so it trades one against the other, the same as moving a decision threshold. ROC curves started out with radar operators, and Green and Swets (1966) brought them into psychology to measure how well people hear faint tones.

Model Evaluation · Confusion Matrices10

Accuracy

\text{Accuracy} = \frac{\green{TP} + \green{TN}}{\green{TP} + \green{TN} + \red{FP} + \red{FN}} \text{Accuracy} = \frac{\green{TP} + \green{TN}}{\green{TP} + \green{TN} + \red{FP} + \red{FN}} 0 35 0 106 16 19 7 99 16 19 6 100 19 16 7 99 loss win lost won always "win" loss win lost won kNN (k = 5) loss win lost won decision tree (depth 3) loss win lost won logistic regression

Accuracy:

  • Always "win":  
  • kNN:  
  • Decision Tree:  
  • Logistic Regression:  

Depending on what we care about, there are several other metrics we can use to evaluate our models other than accuracy.

  • Accuracy
  •  
  •  
  •  
Model Evaluation · Confusion Matrices11

K-Fold Cross-Validation

The best way to get an accurate accuracy is to use a method called K-Fold Cross-Validation

Test Train Train Train Train Train Test Train Train Train Train Train Test Train Train Train Train Train Test Train Train Train Train Train Test Iteration 1 Iteration 2 Iteration 3 Iteration 4 Iteration 5

In K-Fold Cross-Validation, we split our dataset into K folds, with one of the folds acting as the testing data, and the remaining folds serving as the training data.

We can then report the average accuracy across all K results.

Model Evaluation · Grid Search12

Accuracy and K-Fold in sklearn

Like usual, it's very easy to run K-fold cross-validation and produce a confusion matrix in sklearn

import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_predict
from sklearn.metrics import confusion_matrix
games = pd.read_csv("nd_football.csv")
X = games[["nd_total_yards", "opp_total_yards"]] / 100
y = 1 - games["win"] # a loss is our positive class
model = LogisticRegression(C=1e9)
pred = cross_val_predict(model, X, y, cv=5)
print(confusion_matrix(y, pred, labels=[1, 0]))
Output
[[19 16]
 [ 7 99]]
Model Evaluation · Grid Search13

Precision

Precision is how close the measurements are to  

\blue{\text{Precision}} = \frac{\green{TP}}{\green{TP} + \red{FP}} \blue{\text{Precision}} = \frac{\green{TP}}{\green{TP} + \red{FP}}

We typically use precision when false positives (type 1 errors) are costly 

0 35 0 106 16 19 7 99 16 19 6 100 19 16 7 99 loss win lost won always "win" loss win lost won kNN (k = 5) loss win lost won decision tree (depth 3) loss win lost won logistic regression

Accuracy:

  • Always "win":  
  • kNN:  
  • Decision Tree:  
  • Logistic Regression:  

Precision:

  • Always "win":  
  • kNN:  
  • Decision Tree:  
  • Logistic Regression:  
Model Evaluation · Grid Search14

Accuracy and Precision

Accuracy:

how close your predictions are to the true values.

Precision:

how close your predictions are to each other.

Accuracy Precision low high low high
Model Evaluation · Grid Search15

Recall

The proportion of actual positives that were classified  .

\purple{\text{Recall}} = \frac{\green{TP}}{\green{TP} + \red{FN}} \purple{\text{Recall}} = \frac{\green{TP}}{\green{TP} + \red{FN}}

We typically use recall when false negatives (type 2 errors) are costly 

0 35 0 106 16 19 7 99 16 19 6 100 19 16 7 99 loss win lost won always "win" loss win lost won kNN (k = 5) loss win lost won decision tree (depth 3) loss win lost won logistic regression

Accuracy:

  • Always "win":  
  • kNN:  
  • Tree:  
  • Log. reg.:  

Precision:

  • Always "win":  
  • kNN:  
  • Tree:  
  • Log. reg.:  

Recall:

  • Always "win":  
  • kNN:  
  • Tree:  
  • Log. reg.:  
Model Evaluation · Grid Search16

Precision vs. Recall

losses (35) wins (106) Precision = = 19 / 26 = 0.731 Recall = = 19 / 35 = 0.543 called losses (26)

Precision only looks inside the circle, where 19 of the 26 games are losses. Recall only looks at the left half, where 19 of the 35 losses made it into the circle.

Model Evaluation · Grid Search17

Precision vs. Recall

Notice what happens if the circle gets bigger. It picks up more of the missed losses, so recall goes up, but it also picks up more wins, so precision goes down. Shrink it and the opposite happens. Usually we can trade one for the other

threshold 0.75 losses (35) wins (106) called losses: 12 precision 10 / 12 = 0.83 recall 10 / 35 = 0.29 threshold 0.5 losses (35) wins (106) called losses: 26 precision 19 / 26 = 0.73 recall 19 / 35 = 0.54 threshold 0.25 losses (35) wins (106) called losses: 54 precision 28 / 54 = 0.52 recall 28 / 35 = 0.80
Model Evaluation · Grid Search18

F1-Score

F1-Score is a better measure to use if we need to   between Precision and Recall AND there is  

F_1 = 2 \cdot \frac{\blue{\text{Precision}} \cdot \purple{\text{Recall}}}{\blue{\text{Precision}} + \purple{\text{Recall}}} F_1 = 2 \cdot \frac{\blue{\text{Precision}} \cdot \purple{\text{Recall}}}{\blue{\text{Precision}} + \purple{\text{Recall}}} 0 35 0 106 16 19 7 99 16 19 6 100 19 16 7 99 loss win lost won always "win" loss win lost won kNN (k = 5) loss win lost won decision tree (depth 3) loss win lost won logistic regression

F1-Score:

  • Always "win":  
  • kNN:  
  • Decision Tree:  
  • Logistic Regression:  
Model Evaluation · Grid Search19

Classification Reports

In sklearn, classification_report prints precision, recall and F1 with each class taking a turn as the positive one:

import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_predict
from sklearn.metrics import classification_report
games = pd.read_csv("nd_football.csv")
X = games[["nd_total_yards", "opp_total_yards"]] / 100
y = 1 - games["win"] # a loss is our positive class
model = LogisticRegression(C=1e9)
pred = cross_val_predict(model, X, y, cv=5)
print(classification_report(y, pred, digits=3,
target_names=["win", "loss"]))
Output
              precision    recall  f1-score   support

         win      0.861     0.934     0.896       106
        loss      0.731     0.543     0.623        35

    accuracy                          0.837       141
   macro avg      0.796     0.738     0.759       141
weighted avg      0.829     0.837     0.828       141

support is the number of true samples of each class in the data being scored.

Model Evaluation · Grid Search20

Accuracy vs. Precision vs. Recall

Model Evaluation · Grid Search21

Evaluating Decision Thresholds

Another parameter that we could change is where the decision threshold lies, thus far for logistic regression we've just set it to 0.5

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 threshold: call it a loss when p(loss) is at least this 0.0 0.2 0.4 0.6 0.8 1.0 threshold 0.5 precision (losses) recall (losses)

Every threshold gives a different confusion matrix, and raising recall costs precision.

Rather than pick one threshold, we can plot all of them. This is a receiver operating characteristic (ROC) curve. From this we can get a single number called the AUC (Area Under Curve)

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ 0.0 0.2 0.4 0.6 0.8 1.0 false positive rate (wins called losses) 0.0 0.2 0.4 0.6 0.8 1.0 true positive rate (losses called losses) threshold 0.5 kNN, k = 5 (AUC 0.72) tree, depth 3 (AUC 0.74) logistic, yards (AUC 0.86) logistic, 15 stats (AUC 0.91)
Model Evaluation · Grid Search22

Evaluating Regression Tasks

Remember that for regression tasks we are trying to predict  

Because regression tasks are supervised, we have ground-truth so we can tell how far away our predicted answers are from the ground truth.

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ 2 3 4 5 6 Notre Dame's yards (hundreds) 10 15 20 25 30 35 40 45 50 points

These gaps are called the   (R) and the goal of regression is to minimize the residuals.

Model Evaluation · Assessing Regression23

Residuals Plot

A residual plot is a graph that shows the residuals on the vertical axis and the independent variable on the horizontal axis.

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ 0 20 40 60 predicted points 0 10 20 30 40 50 60 70 actual points 200 400 600 Notre Dame's yards −30 −20 −10 0 10 20 30 residual (points)

If the points in a residual plot are randomly dispersed around the horizontal axis, a linear regression model is appropriate for the data, otherwise, a non-linear model is more appropriate.

Model Evaluation · Assessing Regression24

Regression Tasks - R²

R² indicates the proportion of variance in the dependent variable that can be explained by the independent variable(s) in a model.

R^2 = 1 - \frac{\sum_i (y_i - \yhat_i)^2}{\sum_i (y_i - \bar{y})^2} R^2 = 1 - \frac{\sum_i (y_i - \yhat_i)^2}{\sum_i (y_i - \bar{y})^2}

R² shows how well the regression line fits the data, with a value ranging from 0 (poor fit) to 1 (perfect fit).

Model Evaluation · Assessing Regression25

Mean Squared Error

Another option is, using the residuals, we can write a number of different error functions for regression tasks:

  • Residual Sum of Squares (RSS)
  • Mean Absolute Error (MAE)
  •   (MSE)
  • Mean Percentage Error (MPE)
  • Mean Absolute Percentage Error (MAPE)
  • Root Mean Squared Error (RMSE)

MSE measures the average squared difference between the estimated values and the actual value

\text{MSE} = \frac{1}{n}\sum_{i=1}^{n} (y_i - \yhat_i)^2 \text{MSE} = \frac{1}{n}\sum_{i=1}^{n} (y_i - \yhat_i)^2
Model Evaluation · Assessing Regression26

Learning Curves - Machine Learning

Learning curves are a plot of model learning performance over  

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ 20 40 60 80 100 number of training games 75 80 85 90 95 100 accuracy (%) games it was fit to unseen games

For parametric models, we can also plot learning curves over  

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ 1 0 0 1 0 1 1 0 2 1 0 3 epoch 0.0 0.1 0.2 0.3 0.4 0.5 0.6 binary cross-entropy training loss (2012-2019) validation loss (2020-2022)
Model Evaluation · Assessing General Learning Methods27

Assessing Clustering

Assessing unsupervised learning is difficult as there are no   against which we can compare results.

We often don't know what the right answer is!

Model Evaluation · Assessing Unsupervised Methods28

Silhouette Score

Silhouette score is a numerical value that measures how well a data point belongs to its cluster and how separated it is from other clusters.

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ 2 3 4 5 6 7 8 9 10 number of clusters (k) 0.0 0.1 0.2 0.3 0.4 0.5 0.6 silhouette score

The score ranges from -1 to 1, with higher values indicating better clustering.

Model Evaluation · Assessing Unsupervised Methods29

Dendrograms

A dendrogram is a branching diagram that shows the relationships between classes

To assess a clustering, cut the tree where the vertical lines are longest. Here the last merge (1.15) is far above the one before it (0.66), so two clusters is the natural cut.

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ 2022 2012 2019 2018 2020 2015 2017 2014 2021 2013 2016 0.0 0.2 0.4 0.6 0.8 1.0 1.2 height (Ward distance)
Model Evaluation · Assessing Unsupervised Methods30

Elbow Plots

Previously, we have used elbow plots to estimate the number of   to use on a data set.

image/svg+xml Matplotlib v3.11.2, https://matplotlib.org/ 1 2 3 4 5 6 7 8 9 10 number of clusters (k) 0 50 100 150 200 250 300 sum of squared distances

In cluster analysis, we can plot the sum of squared distances from each point to its centroid as a function of the number of clusters and pick the elbow of the curve as the number of clusters to use.

Model Evaluation · Assessing Unsupervised Methods31