Model Evaluation
Deciding on which Model
We’ve seen a large number of different models now, as well as a number of different use cases. How should we decide which model to use and when?
Models for different Tasks:
Classification:
- kNN
- Decision Trees
- Perceptrons
- Logistic Regression
- SVMs
Regression:
- kNN
- Decision Trees
- Linear Regression
- SVRs
Clustering:
- kMeans
- Hierarchical
Grid Search
The simplest way to decide which model to use on a given task is to test different models and configurations on the task and compare their results
Grid Search is simply the term for systematically testing different combinations of hyperparameters
Model Evaluation
Lets suppose we were training a classifier to predict whether or not Notre Dame won a football game
| classifier | features | unseen games |
|---|---|---|
| kNN (k = 5) | yards | 81.6% |
| decision tree (depth 3) | yards | 82.3% |
| logistic regression | yards | 83.7% |
| random forest (200 trees) | 15 stats | 83.6% |
| logistic regression | 15 stats | 85.8% |
How good are these classifiers? Thus far we've been using accuracy
(how close a given set of predictions are to their ground truth values)
Model Evaluation
However, using only accuracy can be misleading
| classifier | features | unseen games |
|---|---|---|
| always guess "win" | none | 75.2% |
| kNN (k = 5) | yards | 81.6% |
| decision tree (depth 3) | yards | 82.3% |
| logistic regression | yards | 83.7% |
| random forest (200 trees) | 15 stats | 83.6% |
| logistic regression | 15 stats | 85.8% |
Notre Dame wins 3 out of 4 of its football games, resulting in an imbalanced dataset. Out of our 141 samples, only 35 of them are losses. This means that we could get an accuracy of 75% by just predicting Notre Dame to win every game.
Confusion Matrices
A confusion matrix helps assess a classification model's performance by tabulating each of the four possible outcomes from a binary prediction task
"Positives" refer to data points belonging to the target class (win), while "negatives" refer to data points belonging to the non-target class (loss)
Confusion Matrices and Imbalanced Datasets
Notre Dame wins 3 games in 4, so the games worth predicting are the losses
Type 1 vs. Type 2 Errors
Type 2 errors are called "false negatives"
Type 1 errors are called "false positives"
Depending on the task at hand, we may care more about these types of errors, but only considering the overall accuracy may hide them from us.
Type 1 vs. Type 2 Errors
Consider a task in which we're performing a cancer screening, with the presence of cancer being the positive class.
If we have a dataset with 998 healthy examples, and 2 cancerous examples, we could get an overall accuracy of 99.8% by just predicting that everyone is healthy!
However, missing the 2 sick patients would be false negatives, and if we're trying to catch deadly diseases, making these type 2 errors are extremely costly!
Type 1 vs. Type 2 Errors
Type 1 errors are called "false positives"
- A test says a healthy dog has heart disease
- A spam filter sends a real email to spam
- We predict a loss and Notre Dame wins (2015 USC)
Type 2 errors are called "false negatives"
- A test misses a dog that has heart disease
- A spam email ends up in your inbox
- We predict a win and Notre Dame loses (2015 Clemson)
Type 1 and Type 2 Errors in the Body
Our immune system is a classifier: every cell and molecule it meets is either "self" or "not self".
- False positive: attacking our own cells, which is what happens in autoimmune diseases like type 1 diabetes
- False negative: missing a virus or bacterium, and getting sick
The body can't push both to zero, so it trades one against the other, the same as moving a decision threshold. ROC curves started out with radar operators, and Green and Swets (1966) brought them into psychology to measure how well people hear faint tones.
Accuracy
Accuracy:
- Always "win": 106/141 = 0.752
- kNN: 115/141 = 0.816
- Decision Tree: 116/141 = 0.823
- Logistic Regression: 118/141 = 0.837
Accuracy
Depending on what we care about, there are several other metrics we can use to evaluate our models other than accuracy.
- Accuracy
- Recall
- Precision
- F1-Score
K-Fold Cross-Validation
The best way to get an accurate accuracy is to use a method called K-Fold Cross-Validation
In K-Fold Cross-Validation, we split our dataset into K folds, with one of the folds acting as the testing data, and the remaining folds serving as the training data.
We can then report the average accuracy across all K results.
Accuracy and K-Fold in sklearn
Like usual, it's very easy to run K-fold cross-validation and produce a confusion matrix in sklearn
[[19 16] [ 7 99]]
Precision
Precision is how close the measurements are to each other
We typically use precision when false positives (type 1 errors) are costly
Precision
Accuracy:
- Always "win": 106/141 = 0.752
- kNN: 115/141 = 0.816
- Decision Tree: 116/141 = 0.823
- Logistic Regression: 118/141 = 0.837
Precision:
- Always "win": 0/0 (never calls a loss)
- kNN: 16/23 = 0.696
- Decision Tree: 16/22 = 0.727
- Logistic Regression: 19/26 = 0.731
Accuracy and Precision
Accuracy:
how close your predictions are to the true values.
Precision:
how close your predictions are to each other.
Recall
The proportion of actual positives that were classified correctly (as positive).
We typically use recall when false negatives (type 2 errors) are costly
Recall
Accuracy:
- Always "win": 0.752
- kNN: 0.816
- Tree: 0.823
- Log. reg.: 0.837
Precision:
- Always "win": none
- kNN: 0.696
- Tree: 0.727
- Log. reg.: 0.731
Recall:
- Always "win": 0/35 = 0
- kNN: 16/35 = 0.457
- Tree: 16/35 = 0.457
- Log. reg.: 19/35 = 0.543
Precision vs. Recall
Precision only looks inside the circle, where 19 of the 26 games are losses. Recall only looks at the left half, where 19 of the 35 losses made it into the circle.
Precision vs. Recall
Notice what happens if the circle gets bigger. It picks up more of the missed losses, so recall goes up, but it also picks up more wins, so precision goes down. Shrink it and the opposite happens. Usually we can trade one for the other
F1-Score
F1-Score is a better measure to use if we need to seek a balance between Precision and Recall AND there is an uneven class distribution
F1-Score
F1-Score:
- Always "win": 0
- kNN: 0.552
- Decision Tree: 0.561
- Logistic Regression: 0.623
Classification Reports
In sklearn, classification_report prints precision, recall and F1 with each class taking a turn as the positive one:
precision recall f1-score support
win 0.861 0.934 0.896 106
loss 0.731 0.543 0.623 35
accuracy 0.837 141
macro avg 0.796 0.738 0.759 141
weighted avg 0.829 0.837 0.828 141support is the number of true samples of each class in the data being scored.
Accuracy vs. Precision vs. Recall
- Accuracy - when class distribution is balanced and false positives and false negatives have similar consequences.
- Precision - when false positives are costly or undesirable
- Recall - when false negatives are more critical than false positives
- F1 - when you need a balance between precision and recall and the class distribution is imbalanced.
Evaluating Decision Thresholds
Another parameter that we could change is where the decision threshold lies, thus far for logistic regression we've just set it to 0.5
Every threshold gives a different confusion matrix, and raising recall costs precision.
Evaluating Decision Thresholds
Rather than pick one threshold, we can plot all of them. This is a receiver operating characteristic (ROC) curve. From this we can get a single number called the AUC (Area Under Curve)
Evaluating Regression Tasks
Remember that for regression tasks we are trying to predict a continuous value
Because regression tasks are supervised, we have ground-truth so we can tell how far away our predicted answers are from the ground truth.
These gaps are called the residuals (R) and the goal of regression is to minimize the residuals.
Residuals Plot
A residual plot is a graph that shows the residuals on the vertical axis and the independent variable on the horizontal axis.
If the points in a residual plot are randomly dispersed around the horizontal axis, a linear regression model is appropriate for the data, otherwise, a non-linear model is more appropriate.
Regression Tasks - R²
R² indicates the proportion of variance in the dependent variable that can be explained by the independent variable(s) in a model.
R² shows how well the regression line fits the data, with a value ranging from 0 (poor fit) to 1 (perfect fit).
Mean Squared Error
Another option is, using the residuals, we can write a number of different error functions for regression tasks:
- Residual Sum of Squares (RSS)
- Mean Absolute Error (MAE)
- Mean Squared Error (MSE)
- Mean Percentage Error (MPE)
- Mean Absolute Percentage Error (MAPE)
- Root Mean Squared Error (RMSE)
Mean Squared Error
MSE measures the average squared difference between the estimated values and the actual value
Learning Curves - Machine Learning
Learning curves are a plot of model learning performance over the number of training samples
Learning Curves - Machine Learning
For parametric models, we can also plot learning curves over time (epochs)
Assessing Clustering
Assessing unsupervised learning is difficult as there are no ground-truth labels against which we can compare results.
We often don't know what the right answer is!
Silhouette Score
Silhouette score is a numerical value that measures how well a data point belongs to its cluster and how separated it is from other clusters.
The score ranges from -1 to 1, with higher values indicating better clustering.
Dendrograms
A dendrogram is a branching diagram that shows the relationships between classes
To assess a clustering, cut the tree where the vertical lines are longest. Here the last merge (1.15) is far above the one before it (0.66), so two clusters is the natural cut.
Elbow Plots
Previously, we have used elbow plots to estimate the number of principal components to use on a data set.
In cluster analysis, we can plot the sum of squared distances from each point to its centroid as a function of the number of clusters and pick the elbow of the curve as the number of clusters to use.