Skip to main content

Evaluating model performance

Instruction and application
Complete

Success in Optimization​

If your goal is to optimize a ML model, how do you know if you have been successful?

Do you trust your instincts and select the model which “feels” best, or rely on performance metrics that give you an objective view? In this section, we will explore the essential metrics for building and assessing effective ML models.

Hand pointing illustration

Classification metrics​

Classification is a supervised ML technique used to predict labels, such as detecting spam or customer churn. To decide how effective a model is, we compare predictions against actual known outcomes.

Scenario: Spam filtering​

Imagine you work for a mobile provider. Your team filters spam messages. You train a model on 5,572 messages, where 13.4% are spam. After training, you test it to see how many it correctly identifies.

Classification Concept

Confusion matrix​

A confusion matrix tracks four possible outcomes:

  • True Positive: Spam correctly labeled "spam".
  • True Negative: Non-spam correctly labeled "not spam".
  • False Negative: Spam incorrectly labeled "not spam".
  • False Positive: Non-spam incorrectly labeled "spam".

Core performance metrics​

1. Accuracy​

The proportion of data points correctly labeled.

Formula: Accuracy = (TP + TN) / N

In our example, 4824 non-spam and 689 spam were correctly identified, resulting in an accuracy of 0.989. However, this must always be compared to the baseline (86.6% in this case).

2. Precision​

Answers: "Of all instances predicted as positive, how many were correct?"

Formula: Precision = TP / (TP + FP)

In our example, 689/690 = 99.9%. Precision determines how much we can trust a specific positive prediction.

3. Recall​

Answers: "Of all actual positive instances, how many did the model correctly identify?"

Formula: Recall = TP / (TP + FN)

In our example, 689/747 = 92.2%. Recall determines how effective the model is at detecting a specific label.

The Precision-Recall Trade-off​

In high-stakes scenarios like cancer detection, a False Negative (missing a case) is much worse than aFalse Positive. In such cases, we optimize forRecall, even if it causesPrecision to decrease.

4. F1-Score​

The harmonic mean of precision and recall, providing a balanced view, especially for imbalanced datasets.

Formula: F1 Score = 2 * (Precision * Recall) / (Precision + Recall)

Action item: Reflection​

Consider the following questions regarding model evaluation.

Questions & Reflections
1. Which of the four metrics would be most important in a scenario where false negatives have serious consequences, and why?

Type your reflection here...

2. How might your choice of metric change when working with a highly imbalanced dataset?

Type your reflection here...