Skip to main content

Operational considerations

Instruction and application
Complete

Training bigger, faster models sounds like a no-brainer—until the energy bill lands and your GPUs start sweating. Scaling up comes with real-world costs: money, complexity, and carbon. So how do you get the performance you need without burning through your budget—or the planet?

Illustration

In this section, you'll learn how to do just that.

Cost/benefit considerations​

Adopting distributed training allows us to train larger models more quickly that can scale with our growing datasets. They also allow us to utilise our resources more effectively.

However, they also require increased hardware investment and communication overheads which drives up the costs. Additionally, running multiple high-power devices in parallel naturally leads to higher energy costs and the ethical implications that come with it.

If you want to use a distributed system, think through these questions:

  • Are you dealing with datasets and models that truly necessitate distributed training?
  • How critical is it to reduce training time?
  • Can your organisation afford the hardware and engineering effort required?
  • Do you have the skills to set up and manage a distributed training environment effectively?

Resource optimisation strategies​

Instead of investing in a distributed system by adding more hardware, you could focus on making your existing hardware work smarter, allowing you to train larger or more complex models without additional hardware.

Here are some strategies you can use:

Mixed precision trainingMixed precision training involves using different numerical precisions for different parts of the neural network during training.

Traditionally, most deep learning models use 32-bit floating-point numbers (FP32) for weights, activations, and gradients. Mixed precision strategically uses lower precision formats like 16-bit floating-point (FP16) or even 8-bit integers (INT8) for the majority of computations, while keeping FP32 for numerically sensitive operations (like gradient accumulation and certain layers).

It helps us circumvent hardware constraints by:

  • Reducing memory footprint: Lower precision formats require less memory to store weights, activations, and gradients. This allows you to fit larger models or larger batch sizes into the limited memory of your GPUs.
  • Speeding up computations: Modern GPUs have specialised hardware units (like Tensor Cores on NVIDIA GPUs) that are significantly faster when performing computations with lower precision formats like FP16. This can lead to substantial speedups in training time.
  • Increasing throughput: The combination of reduced memory usage and faster computations can lead to higher overall throughput, allowing you to process more data in the same amount of time. Mixed precision focuses on optimising the efficiency of training on a single machine, meaning we don’t need to run multiple processors in parallel. However, mixed precision training can be combined with distributed learning to achieve even greater performance gains.

Gradient accumulationGradient accumulation is a technique that allows you to simulate training with a large batch size even when your hardware (specifically GPU memory) cannot accommodate it directly.

Instead of processing a large batch all at once, you process the data in smaller "micro-batches" sequentially. After each micro-batch, you calculate the gradients, but instead of immediately applying them to update the model's weights, you accumulate these gradients.

Once you have processed a certain number of micro-batches (effectively simulating the desired larger batch size), you then perform a single weight update using the accumulated gradients.

It helps us circumvent hardware constraints by:

  • Overcoming memory limitations: By processing data in smaller micro-batches, the memory required to store intermediate activations and gradients during the backward pass is significantly reduced, allowing you to train with an effective batch size that would otherwise lead to out-of-memory errors.
  • Simulating large batch training: Training with larger batch sizes can sometimes lead to more stable training and better generalisation. Gradient accumulation allows you to reap these potential benefits without requiring the memory for a physically large batch. As this is a sequential process within a single training process, it can be used in conjunction with parallelism where each worker uses gradient accumulation to further increase its effective batch size.

Dynamic batch sizingDynamic batch sizing involves adjusting the batch size during the training process based on factors like the current training stage, the loss landscape, or available memory.

Instead of using a fixed batch size throughout training, the system dynamically increases or decreases it.

It helps us circumvent hardware constraints by:

  • Managing memory: During the initial stages of training or when the loss landscape is relatively flat, you might be able to use larger batch sizes to speed up training. As training progresses or when the loss landscape becomes more complex, reducing the batch size can sometimes improve stability and prevent oscillations, potentially fitting within stricter memory constraints at those later stages.
  • Improving training efficiency: By using larger batch sizes when possible, you can potentially reduce the number of weight updates needed to converge. Conversely, smaller batch sizes can introduce more noise, which can help the model escape local minima. Dynamically adjusting this can lead to more efficient training within the given hardware limitations. Again, this method can be used with distributed learning where each device optimises its use of available resources (memory) over time.

**Efficient architecture search (EAS)**Efficient Architecture Search (EAS) encompasses a range of techniques aimed at automatically discovering neural network architectures that achieve high performance while having a smaller number of parameters and lower computational cost.

Traditional manual architecture design can be time-consuming and may not always yield the most resource-efficient models.

EAS methods use algorithms (like evolutionary algorithms, reinforcement learning, or gradient-based methods) to explore the vast space of possible network architectures and identify those that offer the best trade-off between accuracy and resource usage (e.g., FLOPs, number of parameters).

It helps us circumvent hardware constraints by:

  • Using smaller models: EAS can discover architectures with significantly fewer parameters, which directly reduces the memory footprint of the model during training and inference. This allows you to train and deploy more complex tasks on devices with limited memory.
  • Reducing computational cost: The discovered architectures often have lower FLOPs (floating-point operations per second), leading to faster training times on the same hardware. This can make training feasible within your hardware limitations even for complex tasks. EAS focuses on finding a more efficient model to begin with, resulting in a more efficient architecture being trained on a single machine that might not require as much computational power as distributed learning.

Each of these strategies can improve the performance of your model training but may require significant investment from your organisation. If you would like to propose one for your organisation, make sure you have considered the cost vs benefit of the approach and can explain clearly to a non-technical stakeholder what benefit it will provide.

Environmental and sustainability considerations​

Each of the training methods and alternate strategies we have looked at have focused on computational requirements, but as we increase the strength of our ML processes we also need to think about the environmental impact.

For example, offloading our computation to a data center will result in an increase in power consumption.

So how do we balance our model training needs against sustainable practices?

Carbon footprint awareness

Training large-scale ML models, especially deep learning models with billions of parameters, can be extremely computationally intensive.

This high computational demand translates directly to significant energy consumption, which in turn contributes to carbon emissions, especially if the electricity used is not from renewable sources.

The awareness of this "carbon footprint" of AI is growing, prompting researchers and practitioners to consider the environmental impact of their work.

So how does this impact our model training?

  • Increased complexity consideration: The awareness of the carbon footprint encourages us to be more critical in the evaluation of model complexity.

We might be more inclined to explore simpler architectures or apply model compression techniques if the marginal gains in accuracy from increasingly complex models come at a disproportionately high energy cost.

  • Trade-off with powerful processors: While more powerful processors (like high-end GPUs and TPUs) can accelerate training and potentially reduce the overall energy consumed for a specific training run by finishing it faster, they themselves consume significant power.

The environmental benefit depends on the efficiency gains outweighing the increased power draw of these processors and the source of the electricity powering them. Training efficiency metrics

When we think about evaluating our models, we tend to do so in terms of performance - how accurate/correct are predictions or responses?

What if we also considered the efficiency of our models?

Models that can process complex tasks quicker or use less parameters will be more environmentally efficient.

As we build models, we should be encouraged to also explore these metrics in a bid to create more sustainable practices:

  • Time to convergence: How long does it take to reach a desired accuracy level?
  • Energy consumption (kWh): The total energy used during the training process.
  • FLOPs (Floating Point Operations): A measure of the computational work performed.
  • Carbon emissions (kg CO2e): An estimate of the greenhouse gases released due to the energy consumption.
  • Parameter efficiency: The number of parameters in the model relative to its performance. So how can we make our models more efficient? Here are some ideas:

TechniqueHow it works Architectural innovationsDesign inherently more efficient network architectures (e.g., using depthwise separable convolutions, attention mechanisms more judiciously, exploring sparse networks).Hyperparameter optimisationFind optimal learning rates, batch sizes, and other hyperparameters to achieve faster convergence and reduced overall computation.Model compression techniquesTechniques like pruning, quantisation and distillation can reduce the model size and computational requirements.Early stoppingMonitor validation performance and stop training when it plateaus to prevent unnecessary computation.Transfer learning and fine-tuningReusing knowledge from pre-trained models can significantly reduce the amount of data and computation needed for a new task.Data efficiencyUse techniques like data augmentation or active learning to get the most out of smaller datasets.Algorithmic efficiencyExplore more efficient optimisation algorithms or training schedules.Green computing options

Green computing refers to the practice of using computing resources in an environmentally sustainable way.

This involves strategies to minimise the energy consumption and environmental impact of training and deploying models.

Options include:

Utilising renewable energy sources​

Powering training infrastructure with electricity generated from solar, wind, or hydroelectric sources significantly reduces the carbon footprint. Cloud providers are increasingly offering regions powered by renewable energy.

Optimised hardware usage​

Efficiently utilising hardware resources, avoiding idle compute instances, and maximising the utilisation of accelerators like GPUs.

Energy-efficient hardware​

Choosing hardware with better energy efficiency (performance per watt).

Cloud computing with green initiatives​

Selecting cloud providers that have strong commitments to sustainability and invest in renewable energy and energy-efficient infrastructure.

Federated learning​

Training models on decentralised devices (like smartphones) can reduce the need for large, centralised data centers and their associated energy consumption, although communication efficiency becomes a key consideration.

Edge computing​

Deploying models on edge devices (closer to the data source) can reduce the need for energy-intensive data transfer to and from the cloud.

Software and algorithm optimisation​

The training efficiency metrics discussed above directly contribute to green computing by reducing energy consumption.

Key points​

As you start to consider more complex models and the hardware requirements needed to achieve your goals, also think about the environmental impact and what you can do to promote more sustainable practices.

For example, if you need to use a cloud-service, can you recommend one which has committed to using renewable energy or carbon-zero practices?

Knowledge check​

Let's see what you've learned, try answering the questions below.