Skip to main content

Powering ML platforms: Hardware and compute strategies

Learning
Complete

Even the best-designed ML platform architecture is only as effective as the compute power it utilises.

Microscope illustration

Within any ML platform, selecting the right computational resources is just as critical as the models or data pipelines you build. The wrong setup can bottleneck performance, inflate costs, or introduce latency that breaks real-time use cases.

In this section, you'll explore the specialised hardware and distributed computing strategies that power today’s most demanding ML/AI workloads — from small-scale classifiers to massive deep learning models trained across clusters.

Specialised hardware for ML​

Choosing the right hardware depends on the task at hand. The computational demands of training a deep learning model are very different from serving lightweight predictions in real time, so aligning hardware capabilities with workload requirements is essential.

Here are common processor types and when to use them:

  • GPUs (Graphics Processing Units): Designed for parallel operations, making them excellent for deep learning workloads. Suitable for both training and inference of convolutional and transformer models. **Example:**Training a convolutional neural network on thousands of labeled images.
  • TPUs (Tensor Processing Units): Custom hardware developed by Google, highly optimised for TensorFlow-based neural network operations. Offers exceptional speed for specific deep learning tasks. **Example:**Scaling BERT training for natural language processing in Google Cloud.
  • FPGAs (Field-Programmable Gate Arrays): Hardware that can be reprogrammed for specific functions. Useful for ultra-low-latency inference, real-time signal processing, or hardware-constrained environments. **Example:**Running a compressed model on an edge device for drone navigation.

Distributed computing concepts for ML​

When datasets grow or models become too large for a single machine, distributed computing comes into play. These strategies allow ML workloads to scale efficiently by splitting tasks across multiple compute nodes, enabling faster training and broader model deployment.

Select the headings below to learn about three foundational concepts that underpin distributed ML workflows. Each plays a key role in ensuring scalability, reliability, and resource efficiency across compute environments:

Data parallelismThe same model is trained across different slices of the dataset on multiple nodes. Each node updates a copy of the model, and updates are aggregated.Used when: The model fits in memory, but the data does not.Example: Training a ResNet model on millions of images across eight GPUs.Model parallelismThe model itself is split across compute nodes — each handles part of the architecture (e.g., layers or attention blocks).Used when: The model is too large to fit on one device.Example: Training a custom deep learning model for multilingual document summarisation with hundreds of millions of parameters, where the model’s size exceeds the memory of a single GPU.Cluster management and schedulingTools like Kubernetes, Apache Spark, or Ray coordinate workloads across distributed infrastructure, ensuring compute resources are efficiently allocated and tasks are completed reliably.Used when: You need to run and manage ML jobs across multiple machines, automate workflow execution, or scale compute dynamically in response to workload demands.Example: A data science team schedules a nightly training job using Kubeflow running on Kubernetes clusters.

Let’s bring the core concepts of distributed computing to life with a practical scenario. This example shows how data parallelism, model parallelism, and cluster management work together in a real ML platform architecture.

<g></g><defs><clipPath><rect width="24" height="24" fill="white"></rect></clipPath></defs>## Case study A global e-commerce company is training a recommendation engine to personalise product suggestions for millions of users based on browsing history, purchase patterns, and clickstream data.

  • The training dataset includes petabytes of user interactions — too large for a single machine. The ML engineers usedata parallelism to split the dataset across multiple GPUs, allowing the same model to be trained simultaneously on smaller chunks of data.
  • The deep learning model itself is complex, incorporating multiple embeddings, attention layers, and custom ranking logic. Because it exceeds the memory of a single GPU, the team also usesmodel parallelism to distribute parts of the model across nodes, e.g., one GPU handles the embedding layers, while another handles attention and output layers.
  • To manage this at scale, the team uses Kubernetes andKubeflow Pipelines to orchestrate the training workflow, monitor job health, and automatically reschedule failed jobs. Cluster autoscaling ensures resources are provisioned only when needed, saving costs during low-usage periods. Without distributed computing, this kind of large-scale training would take days — or be entirely infeasible.

Action item: ** Matching ML workloads to compute resources checkpoint**​

Let's test your ability to align ML tasks with the most appropriate compute setups.