Welcome to the workshop!
Workshop
In ProgressOptimising ML Inference Scaling for Performance and Cost
Today's icebreaker:
One giant or many small?
If you had to serve one million predictions per second, would you rather use one super-powerful computer or many smaller computers?
Share your response in the chat!
Today's agenda:
- **Review:**Recap key concepts.
- **Practice:**Simulating inference scaling trade-offs.
- **Closing:**Key takeaways and next steps.
Today's learning objectives:
- Understand the fundamental trade-offs between latency, throughput, and resource allocation for ML inference workloads.
- Simulate the impact of different scaling strategies (vertical vs. horizontal) on the performance and theoretical cost of an ML inference service.
- Critically assess how various computational resources relate to operational requirements in ML/AI systems.
- Reinforce concepts of performance optimisation and cost-efficiency in designing scalable ML platform architectures.