Data's impact on ML system scalability

Instruction and application

In Progress

Data sits at the heart of every ML system’s performance. How it flows, scales and varies determines whether a system runs smoothly or stalls under pressure. In this section, we’ll explore how volume and velocity affect processing speed, how variety and formats shape storage and architecture and how data quality drives both performance and resource demands.

Through practical examples and strategies, you’ll see how to keep your system growing without losing accuracy, efficiency or cost control.

Data volume and velocity

Two of the biggest scalability challenges in ML systems are the volume andvelocity of data.

• Volume is the total amount of data collected and stored. As datasets grow, so do demands on compute, memory and storage, leading to longer training times and slower inference if pipelines aren’t optimised.
• Velocity is the rate at which a system generates and ingests new data. High-velocity streams require systems that can process information in real time without lag.

Example: When volume outpaces capacity

A recommendation system processes 10 million daily interactions, retraining in just a few hours.

• As activity grows to 50 million, the retraining window exceeds the available time.
• Without distributed training or other scalability measures, updates may be skipped, leading to less relevant recommendations and reduced user engagement. As data volumes grow and velocity increases, the way you process that data becomes just as important as how much of it you store. The choice of processing approach can directly affect scalability, responsiveness and infrastructure requirements. Select the tabs to learn more. Streaming… Processes data in near real time, enabling instant decisions such as fraud detection. It supports rapid response but demands always-on infrastructure and careful scaling to handle traffic spikes.Batch processing… Handles data in scheduled intervals, such as daily sales aggregations. It’s easier to scale for throughput, but it doesn’t provide instant insights.

Techniques for handling large volumes

Even with streaming or batch processing, growing datasets can overwhelm storage, training and inference pipelines. The following strategies help maintain scalability without sacrificing performance or cost efficiency:

• Sampling reduces the dataset to a representative subset, allowing faster experimentation and iteration without losing key patterns.
• Distributed storage solutions such as the Hadoop distributed file system (HDFS), Amazon S3 or Azure Blob Storage provide scalable, fault-tolerant storage that can support batch and streaming workflows.
• Data partitioning or sharding splits datasets into smaller, more manageable chunks that can be processed in parallel, significantly reducing training and query times.

Practical tip

The right approach often involves a hybrid strategy, using streaming for critical, time-sensitive data while processing less urgent information in batches, all supported by scalable storage and efficient partitioning.

Data variety and formats

Volume and velocity define how much data you process and how fast, but variety dictates how you handle it. Different data types require different handling and their characteristics directly influence infrastructure design:

• Structured data (e.g. SQL tables) is predictable, schema-bound and easy to query, making it the most straightforward to scale.Semi-structured data (e.g. JSON, XML) offers flexibility but often requires parsing, schema alignment and validation before use.
• Unstructured data (e.g. text, images, videos) can be the most storage- and compute-intensive, requiring specialised pipelines for pre-processing and feature extraction.

Common data formats in ML

The format in which a system stores and transfers data can have a substantial impact on performance, especially at scale:

• CSV: Simple and widely supported, but inefficient for large-scale storage or high-performance querying.
• Parquet, ORC: Columnar formats optimised for analytical queries and compression, reducing both storage footprint and I/O time.
• JSON, Avro, Protobuf: Ideal for hierarchical or nested data, common in event logs, APIs and microservices communication.

Serialisation/deserialisation overheads

Some formats, particularly text-based ones like JSON, introduce significant parsing overheads, especially when dealing with deeply nested structures. This can slow both training and inference, as the system must repeatedly convert between in-memory objects and storage formats.

At scale, these overheads can accumulate into noticeable delays and higher compute costs.

Security implications of data variety and formats

Beyond scalability and cost, the variety and format of your data also introduce specific security challenges that you must address:

• Unstructured data (text, images, audio): This data often contains hidden sensitive information (e.g. PII, PHI) that is difficult to automatically detect, redact or encrypt. Unlike structured data, where columns can be easily anonymised, identifying sensitive elements within free text or images requires advanced techniques (e.g. natural language processing (NLP) for PII and computer vision for facial recognition data) and can be prone to errors, increasing data leakage risks.
• Semi-structured (JSON/XML) and complex formats: Nested structures can make it challenging to apply fine-grained access controls or consistent masking rules. Security scanners might struggle to parse and identify sensitive data deep within complex JSON objects, potentially creating blind spots.
• Data format-specific vulnerabilities: Malformed input data in certain formats could be exploited to cause denial-of-service, buffer overflows or other vulnerabilities if not rigorously validated during ingestion and processing. Different serialisation formats might also have known security weaknesses.

Strategies for handling diverse data

Managing variety effectively often requires combining architectural and operational practices:

• Apply schema enforcement to maintain consistency across datasets, preventing unexpected parsing errors during model training.
• Use data lakes to store diverse raw data in its native format for flexible, on-demand processing.
• Implement feature stores to centralise and reuse processed features across multiple models, reducing redundancy and improving efficiency. The goal is to ensure that, no matter the type or format of incoming data, it can flow through the pipeline efficiently — supporting both the scalability and adaptability of your ML systems.

Beyond general schema enforcement, employ specialised tools for:

• PII detection and redaction: For unstructured and semi-structured data, use NLP or image processing techniques to identify and redact sensitive information before it reaches the ML pipeline or storage.
• Format-specific input sanitisation: Implement robust sanitisation routines during data ingestion to prevent malformed data from causing system vulnerabilities or compromising integrity.
• Granular access control: Design access policies that consider data variety and formats, ensuring that access to sensitive elements within complex data structures is tightly controlled.

Data quality and its resource implications

While variety shapes how data flows through an ML system, quality decides if it’s worth processing at all. Poor quality not only weakens model performance but also drives up resource use and operational risk. The diagram below highlights the six core characteristics of high-quality data. They set the foundation for reliable, scalable ML pipelines.

Impact on ML performance and resources

The consequences of compromised data quality extend well beyond inaccurate outputs. They affect how efficiently models are trained, how often you must update them and how much infrastructure is necessary to keep systems operational. Model performance impacts-Inaccurate or biased predictions: Flawed data skews the model's understanding of patterns, leading to results that can harm decision-making or cause reputational damage. For example, mislabelled training examples in a fraud detection model could lead to false positives and false negatives.

• Wasted compute in training cycles: Time and resources are spent learning from noise or irrelevant patterns, increasing costs without improving outcomes.
• Frequent retraining or reactive fixes: Pipeline updating must happen more often to correct for quality issues, which disrupts planned release schedules and consumes engineering bandwidth.Resource and infrastructure impacts-Increased computational demands: Additional pre-processing steps, such as deduplication, missing value imputation or format corrections, place extra load on processing clusters.
• More complex validation pipelines: Enforcing quality standards at scale often means adding validation stages that require dedicated processing power and monitoring.
• Higher storage costs: Redundant, outdated or erroneous data takes up capacity in data lakes or warehouses, inflating storage expenses.
• Longer debugging and maintenance cycles: Quality problems that aren’t immediately visible can degrade performance over time, requiring extended investigation and remediation.
• Increased security vulnerabilities: Poor data quality (e.g. invalid data, unexpected formats) can lead to processing errors that create security vulnerabilities, such as unexpected behaviour that could be exploited, or bypassing data validation layers designed to protect against malicious inputs. In short, poor-quality data acts as a multiplier for cost, complexity and risk, reducing scalability, slowing performance, increasing operational burden and exposing ML systems to potential security vulnerabilities.

Strategies to ensure quality

Addressing data quality at scale requires embedding controls directly into the pipeline:

• Set data validation rules: Define clear thresholds and constraints (e.g. acceptable ranges, mandatory fields and schema compliance) to catch errors before they propagate through the system.
• Perform data profiling: Regularly scan datasets to identify anomalies, missing values, inconsistent formats or outliers that could bias the model or cause processing errors.
• Use automated anomaly detection for streaming data: Implement algorithms that monitor live feeds for unexpected patterns, enabling near real-time intervention before poor-quality data accumulates.Enforce data governance frameworks: Apply standardised processes and ownership structures to ensure consistent practices across teams, regions and systems. This reduces the likelihood of siloed, inconsistent datasets entering production.
• Integrate automated quality checks into ETL/ELT pipelines: Include validation steps at key points in data ingestion and transformation workflows so that only clean, verified data moves forward in the pipeline.