Relating model capacity, resources, and operational needs

Learning

Complete

A powerful ML model is only useful if it runs fast enough, often enough, and cheaply enough to matter.

Now that you’ve explored the hardware and compute strategies that power ML platforms, it’s time to zoom out. This section focuses on how model complexity, infrastructure resources, and operational constraints work together — and sometimes against each other.

Understanding this interplay is key to making deployment decisions that are both effective and efficient.

Model capacity and resource consumption

The more powerful and complex the model, the more it typically consumes. This increased complexity impacts how you design your infrastructure, scale your system, and manage cost.

Key factors that influence resource consumption in ML systems include:

• **Model size:**The number of parameters determines memory and storage requirements. For instance, a model with 500 million parameters might require over 2 GB of memory just to load for inference.
• **Computational intensity:**Measured in FLOPs (Floating Point Operations), this reflects how much raw computation is needed per training or inference step. Deep models with attention layers or recurrent structures often demand high FLOPs.
• **Data volume and velocity:**High-throughput systems — such as fraud detection on financial transactions — require fast data ingestion and low-latency response times, which impacts infrastructure bandwidth and I/O design.

Impact of ML method choice

• Beyond just model size and FLOPs, the fundamental ML method chosen dictates these factors. For example:Simple Models (e.g., Linear Regression, Decision Trees): Often have smaller model sizes, lower computational intensity, and faster inference. They can be suitable for high-throughput, low-latency applications on CPUs but may sacrifice some predictive power.
• Complex Models (e.g., Deep Neural Networks, Large Language Models): Tend to have massive parameter counts, high computational intensity, and require significant GPU/TPU resources for both training and inference. While they offer higher accuracy or richer capabilities, they come with a substantial infrastructure cost and complexity burden.Trade-off consideration: Critically evaluating which ML method to use involves not only predictive performance but also assessing if its resource footprint aligns with the budget, latency requirements, and complexity your organisation is prepared to manage. Sometimes, a slightly less accurate but far more efficient model provides greater overall organisational impact due to lower operational costs and faster deployment.

<g></g><defs><clipPath><rect width="24" height="24" fill="white"></rect></clipPath></defs>## Case study: Scaling complex models for real-time ML A global fintech company develops a machine learning model to detect fraudulent credit card transactions in real time. The original system was based on a gradient-boosted decision tree ensemble.

However, after thorough evaluation, the team transitioned to a more complex neural network architecture with attention mechanisms and temporal features. This change was driven by evidence that the new architecture would significantly improve fraud detection accuracy — delivering a projected higher reduction in false negatives and building greater customer trust — despite the additional infrastructure demands.

Challenges faced:

• Model size: The new model has over 250 million parameters, requiring 1.2 GB of memory per instance just to serve predictions. Initial CPU-based deployments struggled to keep up with the demand.
• Computational intensity: Each prediction required significant processing (measured in billions of FLOPs), slowing inference time when traffic spiked, especially during peak hours in global markets.
• Data volume and velocity: The system ingested tens of thousands of transaction records per second, each requiring immediate classification. The infrastructure needed to support both batch training jobs and sub-300 ms inference latency.Solution: The team migrated to a GPU-based inference cluster designed for horizontal scalability. Real-time scoring was reserved for high-risk transactions, while batch-serving was introduced for lower-priority tasks. They streamlined preprocessing pipelines to reduce I/O bottlenecks and applied model compression techniques, cutting the memory footprint by 30% without impacting accuracy.

This solution struck a balance between performance, scalability, and cost — supporting the more advanced architecture while delivering business-critical fraud prevention outcomes.

Operational requirements and performance benchmarking

Understanding model complexity and resource consumption is essential, but it’s only part of the equation. To deploy ML systems successfully, you must also account for operational requirements that directly impact user experience, system design, and infrastructure decisions.

These include:

• **Latency:**How fast each prediction is returned. A self-driving car system may require responses in under 100 ms, whereas a document summarisation model might tolerate a few seconds.
• **Throughput:**How many predictions can be made per second. For example, a streaming platform might need to serve 10,000+ recommendations per second during peak usage.
• **Cost efficiency:**Striking a balance between performance and infrastructure cost. Deploying a small model on a GPU cluster might be overkill if a CPU-based setup meets your needs.**Scalability:**Can the system handle an increase in requests, data volume, or both? Horizontal scaling (adding more instances) and vertical scaling (using more powerful machines) support different needs.
• **Reliability and availability:**Mission-critical systems often require high uptime (e.g., 99.99%). This impacts infrastructure redundancy, monitoring, and failover strategies.

Key point

To monitor how well your system meets operational expectations,** use performance metrics such as:**

• CPU/GPU utilisation.
• Memory and disk I/O usage.
• Latency percentiles (e.g., 95th percentile response time).
• Request success rate and uptime.

Scaling strategies

Once you've benchmarked performance and identified where your system is falling short — whether it's the latency, throughput, or reliability — it’s time to scale. The right scaling strategy ensures your ML system can keep up with demand without overspending on infrastructure.

Select the headings below to learn about the two primary scaling approaches.

Vertical scalingScale up by using more powerful machines with greater CPU, GPU, memory, or storage capacity.Best for:

• Single-instance workloads with tight latency requirements.

• Simpler deployment pipelines that don't need load balancing. **Example:**Upgrading a CPU-based inference server to a GPU-backed machine to reduce response time for a medium-sized deep learning model.Horizontal scalingScale out by distributing workloads across multiple machines, containers, or nodes.Best for:

• High-throughput systems (e.g., recommendation engines, fraud detection).

• Use cases that require redundancy and failover for high availability. **Example:**Running multiple instances of a model-serving API behind a load balancer to support thousands of concurrent user requests.

Action item: ** Estimate deployment resources check point**

Let’s apply what you’ve learned. Below, you’ll evaluate an ML model’s demands and determine what compute setup is needed to meet operational requirements.

Scenario: You’ve been given a trained model with the following characteristics:

• Model size: 100 million parameters

• Inference time on a single CPU: 800 ms

• Average prediction requests per day: 1.2 millionOperational requirements:

• Target latency per prediction: ≤500 ms

• Max infrastructure budget: $250/day

• Availability target: 99.9% uptime