Introduction to model evaluation fundamentals
Instruction and application
In ProgressIntroduction to model evaluation fundamentals
Before you can decide how well your model is doing, you need to know when and where to measure it.Model evaluation isn’t just a technical task—it’s a key part of building ML systems that are useful and trustworthy. In this section, we’ll walk through the key concepts that form the foundation of model evaluation. You’ll learn where evaluation fits in the ML process, and how to distinguish between the metrics used during training, validation, and testing.
These ideas will set you up for success as you begin to match performance with business goals.
Where evaluation fits in the ML lifecycle
Evaluation isn’t just a final checkpoint—it’s embedded in the entire lifecycle of a machine learning solution. From initial experimentation to long-term monitoring, evaluation plays a critical role in helping you build models that are not only technically sound but also aligned with real-world needs.
The diagram below highlights the four key stages where evaluation plays a critical role in the ML lifecycle. Use the tabs to explore how evaluation supports each stage in more detail.
** Model selection** During development, evaluation helps you compare models and select the best-performing candidate based on validation metrics. This might involve choosing between different algorithms or tuning hyperparameters to optimize performance.** Risk assessment** Evaluation metrics can reveal trade-offs that impact business outcomes. For example, understanding the cost of false negatives vs. false positives is critical in contexts like fraud detection, healthcare, or loan approvals.** Business alignment** Metrics are the bridge between technical performance and business priorities. A model with high accuracy may still underperform in practice if the metric doesn't reflect what matters to stakeholders—such as reducing customer churn or increasing safety.** Model monitoring** After deployment, continuous evaluation is necessary to track model performance in production. Real-world data can shift over time (a phenomenon known as** data drift**), and evaluation helps you detect and respond to this. Without monitoring, even a great model can quietly degrade and introduce business risk.** Training, validation, and test metrics: What’s the difference?**
Now that you’ve seen how evaluation fits across the full ML lifecycle, let’s zoom in on model development. At each stage—training, validation, and testing—you’ll apply different metrics that serve distinct purposes.
Let’s review how the metrics used at each ML pipeline stage help you make better decisions throughout the ML development process.
Training metrics: Learning the patterns
Training metrics show how well your model is learning from the data it was trained on. These metrics (such as accuracy or loss) help you track your model’s progress during development and determine whether the algorithm is improving.
But there’s a catch: training metrics can be misleading if used in isolation. A model might perform perfectly on training data by memorizing it, rather than learning general patterns—a problem known as ** overfitting** . That’s why we don’t stop at training metrics.
Validation metrics: Tuning and comparison
Validation metrics help you evaluate model performance on data the model hasn’t seen before—but that’s still within your development pipeline. These metrics are essential for:
- Tuning hyperparameters (e.g. learning rate, number of layers, regularization).
- Comparing different model architectures or training runs.
- Checking for overfitting or underfitting. Validation is where you start asking: Is this model learning something generalizable? A high training score but low validation score often signals that your model is not generalizing well.
Test metrics: The real-world preview
Once you’ve chosen the best-performing model using validation results, it’s time to test. Test metrics are calculated on a completely held-out dataset that ** was not used** during training or validation. This is your unbiased view into how the model is likely to perform in a real-world setting.
You typically report test metrics at the end of development to stakeholders or as part of the production readiness assessment. These metrics inform business and technical decisions about whether to deploy, retrain, or revisit your approach.
Example
Let’s say you're building a machine learning model to detect fraudulent credit card transactions.
- ** During training,** your model achieves 99% accuracy—a promising start.
- ** On validation data,** however, you notice that recall drops to 60%, meaning it’s missing a significant number of actual fraud cases.
- ** On test data,** performance remains consistent with validation, confirming that your model struggles to detect fraud in new data.** What does this tell you?**
Your model may have learned to predict the most common (non-fraudulent) cases very well, but it isn’t picking up on the rare patterns that indicate fraud. High training accuracy isn’t enough. Without validation and test metrics—especially recall, which is crucial for catching fraud—you might deploy a model that leaves your users vulnerable and your business exposed to risk.
Action item: Reflection
You’re working on a machine learning project to recommend health and wellness products to users. Your team is testing multiple models. One has the highest training accuracy, another performs better on validation recall. You’re preparing a presentation for stakeholders.
- Which metric would you prioritize in your presentation, and why?
- How would you explain your choice in a way that connects with non-technical business leaders? Use the form below to capture your thoughts.