Skip to main content

Async review

Instruction and application
In Progress

Recap core topics:​

  • Unit 3: ML/AI Platform Architecture and Resource Allocation

Unit 3: ML/AI Platform Architecture and Resource Allocation​

In Unit 3, you explored…

  • ML/AI platform architecture basics: The essential components that support model deployment at scale — compute, storage, model serving layers, and data pipelines.
  • Operational performance metrics: How latency (time per prediction) and throughput (predictions per second) guide deployment decisions, especially for real-time applications.
  • Scaling strategies: The difference between vertical scaling (using more powerful machines) and horizontal scaling (distributing load across many machines), and how each impacts performance and cost.
  • Resource selection: The trade-offs between CPUs, GPUs, and accelerators, and how different workloads (e.g., batch vs. real-time inference) require different hardware strategies.
  • Cost-performance balancing: How to align architectural choices with business goals by optimising for speed, scalability, and resource efficiency — ensuring models are not just accurate but also affordable to run at scale.

The performance–cost balancing act​

  • ML platforms must deliver fast predictions (low latency) andmany predictions (high throughput).
  • But higher performance often means higher cost, through more powerful machines or more infrastructure.
  • Smart deployment means balancing:Latency: How fast is each prediction?
  • Throughput: How many predictions can run at once?
  • Cost: What resources are required to support both?

Scaling strategies: Vertical vs. horizontal​

To balance these performance and cost trade-offs, teams typically scale their systems in one of two ways:

**Vertical scaling (scale up)**Upgrade to more powerful hardware (e.g., faster CPUs, more RAM, better GPUs).

  • Can improve latency for single predictions.

  • Has limits and becomes expensive quickly.

  • Best for: Low request volumes, latency-critical tasks.**Horizontal scaling (scale out)**Add more machines or instances to handle traffic in parallel.

  • Increases throughput by distributing load.

  • Often more cost-efficient at scale.

  • Requires load balancing and orchestration.

  • Best for: High request volumes, scalable cloud systems.

Action item: Scaling strategies poll​

Let’s do a quick poll to explore trade-offs in ML inference scaling. Choose the answer that you think makes the most practical sense. There’s no single right answer, just different priorities depending on the use case.