Capacity planning frameworks for ML systems

Instruction and application

In Progress

Capacity planning isn’t just a technical exercise — it’s the difference between a smooth user experience and a system that grinds to a halt under pressure.

Whether it’s fraud detection in milliseconds or a language model serving millions of queries, your system’s success depends on anticipating and provisioning resources effectively.

In this section, we’ll define capacity planning in ML, examine core system components and review methods to keep deployments stable, efficient and cost-effective.

Defining operational requirements

Before you can plan capacity, you need to clearly define what 'success' looks like for your system. Common requirements include:

• Throughput: How many predictions your system must deliver per second, minute or hour. For example, an ad recommendation engine might require 5,000 predictions per second during peak shopping hours.
• Latency: The time it takes to serve a single request from input to output. A credit card fraud detection system may have a hard limit of 200 milliseconds per transaction.
• Availability: The percentage of time your system must remain operational (e.g. 99.9% uptime means less than nine hours of downtime per year).
• Scalability: The ability to handle growing workloads without noticeable performance degradation.
• Cost-effectiveness: Balancing your performance targets with budget constraints, ensuring you’re not overprovisioning resources that sit idle. Clearly defining these metrics up front sets the target for all future planning decisions.

Components of ML system capacity

After defining your operational requirements, the next step is to map those requirements across the ML pipeline. The diagram below illustrates how capacity planning touches every stage, with each stage carrying its own demands and potential bottlenecks.

• Data ingestion and pre-processing: ETL pipelines and feature engineering need steady CPU/memory, and real-time tasks may require GPUs. Poor design here slows the entire pipeline.
• Model training: Training is resource-heavy, requiring GPUs/TPUs or CPU clusters plus storage for datasets and checkpoints. Insufficient resources cause failures or delays.
• Model inference: Training is resource-heavy, requiring GPUs/TPUs or CPU clusters plus storage for datasets and checkpoints. Insufficient resources cause failures or delays.
• Monitoring and logging: Logs, metrics and alerts need storage and compute, adding background load that competes with training and inference if unplanned.

Capacity planning methodologies

Once you’ve mapped your system’s components, the next step is anticipating and provisioning for future demand. Capacity planning is an ongoing process of measurement, modelling and testing to ensure your system handles current workloads and future surges without wasteful over-provisioning.

• Workload characterisation: Study current and projected usage patterns, capturing both peak and average loads. Incorporate seasonality or predictable spikes, such as end-of-quarter reporting in financial forecasting models or holiday season demand for recommendation engines.
• Load testing and stress testing: Simulate high-traffic scenarios to uncover bottlenecks before they happen in production. Stress testing goes further by pushing systems beyond their expected limits to reveal breaking points.
• Performance benchmarking: Compare hardware, frameworks or deployment strategies, such as different GPU models, batch sizes or container orchestration approaches, to determine the most efficient configuration for your workloads.
• Resource profiling: Continuously measure CPU, memory, GPU and I/O usage across training and inference stages. This helps pinpoint underutilised resources that you can scale down or overutilised components that need reinforcement.
• Queuing theory basics: Understand how request queues build up under load and how increasing throughput, optimising batch processing or reducing service time can improve latency and system stability.

Example: Scaling an ML-based fraud detection system

A bank’s fraud detection model typically processes 200 transactions per second during business hours, with an average model inference latency of50 milliseconds. However, during a high-profile promotional campaign, incoming transactions surge to1,200 per second.Without proper workload characterisation and stress testing: Queues build up, causing inference latency to rise from 50 ms to over 500 ms.

• Delayed decisions lead to missed fraud alerts or false positives, eroding customer trust.
• Overloaded nodes begin dropping requests, forcing costly manual review of unprocessed transactions. With robust capacity planning: Load tests conducted before the campaign reveal the need for autoscaling rules tied to transaction volume.
• Performance benchmarks guide the selection of GPU instances that deliver low latency under burst conditions.
• Resource profiling ensures the monitoring system scales in tandem, avoiding bottlenecks in fraud event logging. By simulating demand surges before they happen, the bank keeps fraud detection accurate, responsive and cost-efficient, even under extreme load.

Scaling strategies

After defining operational requirements, pinpointing bottlenecks and validating capacity needs, the next step is selecting the right scaling approach to match workload demands efficiently.

The scaling approaches below outline the most common ways ML systems adapt to changing workloads.

Horizontal scaling

Add servers, VMs or containers to distribute workloads, improving fault tolerance and handling sudden traffic spikes. This is particularly effective for stateless inference services, where requests can be processed in parallel without shared state overhead.

Vertical scaling

Upgrade a single machine’s CPU, RAM or GPU when workloads aren’t easily parallelised, such as certain training jobs or batch inference pipelines. While simpler to implement, it has physical and cost limitations that make it less sustainable for long-term growth.

Autoscaling

Dynamically adjust resources with tools like Kubernetes HPA or AWS Auto Scaling Groups, scaling up for peak events and down during low demand. This minimises wasted resources while maintaining performance during unpredictable traffic patterns.

Load balancing

Evenly route requests to prevent bottlenecks and maintain consistent performance across instances. Effective load balancing also improves resilience by automatically rerouting traffic if a node fails.

Practical tip

Most ML systems use a combination of these strategies: Horizontal for elasticity, vertical for specialised workloads, autoscaling for cost control and load balancing for reliability.

Action item: Quiz — capacity planning for ML systems

Test your understanding of operational requirements, pipeline components, planning methodologies and scaling strategies for ML in production.