Cross-validation

Instruction and application

In Progress

Cross-validation

A high validation score might look good—but is it reliable?

Without the right evaluation approach, it’s easy to overfit, misjudge performance, or choose a model that won’t hold up in production.

**

Data splitting and cross-validation

At its simplest, model evaluation starts with a ** train/test split**—you train your model on one part of the data and test it on the rest. While simple and fast, this approach can be** highly sensitive to how the split is made** , especially with small or imbalanced datasets.** Cross-validation** addresses this by running your model multiple times on different splits, then averaging the results. This reduces variance in performance estimates and makes your conclusions more reliable.** K-Fold cross-validation** K-fold cross-validation is one of the most commonly used model evaluation techniques. It provides a more reliable estimate of model performance by reducing the chance that your results are skewed by a single random train-test split.** Here’s how it works (see diagram below):**- Split your dataset into k equal-sized folds (for example k = 5).

• For each of the k iterations:Use one fold as the test set.
• Use the remaining k – 1 folds as the training set.
• Train and evaluate your model k times, and average the performance scores to get a final estimate.

This approach gives every data point a chance to be in the test set, helping reduce variance in performance metrics and making your evaluation more stable and trustworthy.

When to use it:

• For most regression or balanced classification tasks with randomly distributed data.
• When you want to compare models or tune hyperparameters without biasing results toward a particular data split.

Trade-off:

• More robust than using a single train-test split.
• But it requires k full training cycles, which can be computationally expensive, especially for large datasets or complex models like deep learning.

Key point

K = 5 or 10 are common defaults. Lower values reduce compute time, while higher values offer more stable performance estimates—but at greater cost.** Stratified K-Fold cross-validation** In classification tasks, especially with imbalanced classes, standard K-fold can lead to test sets that don’t represent the true class distribution.** Stratified K-fold** solves this by ensuring that** each fold maintains the same class distribution as the full dataset** .

In other words, if 10% of your data is labelled as “fraud,” then each fold will also contain 10% fraud cases. This helps ensure that every training and evaluation run reflects the real-world challenge of dealing with rare classes.

When to use it:

• Classification problems with class imbalance, especially where minority classes carry high business risk.
• Examples include fraud detection, churn prediction, medical diagnosis, or defect classification in manufacturing.

Trade-off:

• Slightly more complex to implement, but many modern ML libraries (like scikit-learn) offer built-in support.
• Essential for ** trustworthy evaluation**—especially if your model’s success depends on correctly identifying rare events.

Why this matters

Without stratification, your model might perform well simply because most test samples are from the majority class. That leads to misleading metrics like inflated accuracy—masking poor performance where it matters most.

Nested cross-validation: When tuning and testing need to stay separate

Stratified K-Fold helps preserve class balance—but what if you’re also tuning hyperparameters? Using the same validation set for both tuning and evaluation can inflate your performance metrics.** Nested cross-validation** solves this by clearly separating** model selection** from** final evaluation** , giving you an unbiased estimate of how well your tuned model will perform on unseen data.

It works in two loops (see diagram below):

• The ** inner loop**(green folds) tunes hyperparameters or selects models.
• The ** outer loop**(blue folds) evaluates performance on truly unseen data, with one fold held out each time (shown in red).

When to use it:

• When comparing models or tuning complex algorithms (e.g., XGBoost, neural networks).
• In research, regulated industries, or any setting where ** evaluation rigour matters** .

Trade-off:

• ** Computationally expensive** , since each outer fold runs a full inner tuning cycle.
• But provides one of the ** most reliable ways** to report performance without bias.** Example:** You’re testing a tuned random forest vs. a tuned logistic regression. For each outer fold, the model is tuned using an inner loop and then tested on an untouched outer fold. This approach gives you an honest view of which model generalises best.

Specialised cross-validation techniques

Not every dataset fits neatly into standard K-fold cross-validation. Some are time-dependent, others are too small for traditional splits, and some require uncertainty estimates. Select the tabs to learn about techniques to help you handle these edge cases with greater care—so your model evaluation is both reliable and realistic.** Leave-One-Out Cross-Validation (LOOCV)** LOOCV is a specific case of K-fold where k = n, meaning the model is trained n times—each time leaving out just one data point for testing and using all the others for training.

As shown in the diagram below, each iteration holds out one data point (red) as the test set while all remaining points (blue) are used for training. The process repeats until every point has been tested once.

** When to use it:** LOOCV is often used when you have very small datasets, such as in biomedical research or pilot studies, where maximizing training data is critical.** Why it matters:** It gives the most thorough use of data, but it also has high variance in results (since each test is based on one point) and high computational cost (since the model is trained once per data point).** Example:** Evaluating a model on a dataset of 50 rare disease cases, where every example provides meaningful signal.** Time Series Cross-Validation** Time series data has a built-in dependency: the past precedes the future. Randomly shuffling or splitting this type of data can introduce** data leakage**—where future values influence training.** Time Series CV** uses** rolling** or** expanding windows** to maintain temporal order. For each fold:

• Train on data up to a certain time point.
• Test on the immediate next block of time. As shown in the diagram below, training always uses past data (blue), while the test set (red) is the next block in sequence. This process repeats, moving forward in time, until all data points have been covered.

** When to use it:** Forecasting electricity demand, predicting sales trends, or detecting equipment failure—all require respect for time order.** Why it matters:** Preserves the real-world deployment scenario, where future data is unknown during training.** Example:** A retail company uses historical sales data from January to May to predict June, then trains on January to June to predict July, and so on.** Bootstrapping**** Bootstrapping** takes a different approach: instead of fixed folds, it repeatedly samples the dataset with replacement to create multiple training sets. Each resampled set is used to train a model, and performance is evaluated on the data points** not included** in that sample (known as the "out-of-bag" set).

As shown in the diagram below, training sets (blue) are drawn with replacement, so some data points appear multiple times while others are left out. The points not included in each sample (red) form the out-of-bag test set. This process is repeated many times to build a distribution of performance scores.

** When to use it:** Especially helpful with** small or noisy datasets** , or when you want to calculate** confidence intervals** for model metrics.** Why it matters:** Bootstrapping gives you a distribution of performance scores, not just a single number—useful when uncertainty needs to be measured or communicated.** Example:** In a risk modeling scenario, you might bootstrap your training set 1,000 times and build confidence intervals for precision, recall, or AUC to better understand how much variation is expected in production.** Cross-validation under resource limits** Cross-validation isn't just about testing accuracy—it’s about choosing evaluation methods that align with your resource limits and model goals. Whether you're working with deep models, limited compute, or business-critical deadlines, the right technique helps you balance** model complexity** ,** hardware requirements** , and** performance insights** .

Use the table below to compare how each method performs under different constraints:** Technique**** Strengths**** Trade-offs** ** When to use it**** K-Fold** Reliable average performance estimateSlower than a single split; cost scales with model sizeGeneral-purpose evaluation of regression or balanced classification models.** Stratified K-Fold** Keeps class ratios stable in each foldSlightly more setup; needed for imbalanced classesClassification problems with imbalanced datasets (e.g., fraud, churn).** Nested CV** Unbiased evaluation when tuning modelsExtremely compute-heavy—may not scale for deep learningComparing/tuning complex algorithms where unbiased performance is critical.** Time Series CV** Respects temporal structure for forecastingLess flexible; increases training time per foldForecasting and sequence problems (e.g., sales, demand, equipment failure).** LOOCV** Maximizes data use on small datasetsVery high compute cost and high result varianceVery small datasets (e.g., medical research, pilot studies).** Bootstrapping** Provides performance variability insightsCan inflate compute use and distort rare class signalsSmall or noisy datasets; when confidence intervals for metrics are needed.

Tip

The more complex your model and the tighter your deployment timeline, the more you need to weigh the cost of deeper validation against the value of its insights.

When refining models, your evaluation strategy plays a critical role in determining how much performance insight you gain—without overspending compute or time.

The right strategy ensures that ** you’re not wasting resources** ,** over-committing hardware** , or** missing performance risks** that only show up at scale.

Action item: Cross-validation in context — Quick check

** Scenario:** You’re training a model to forecast ** hourly energy usage** using time-stamped sensor data from multiple buildings. You’ve applied** random K-Fold cross-validation** and achieved strong performance, but once deployed, the model’s accuracy drops significantly. Answer the following questions to assess what went wrong—and how to choose a better cross-validation strategy for time-dependent data.