Activity: Analysing data characteristics for scalability
Instruction and application
In ProgressIn this activity, you will step into the role of an ML engineer preparing data for a new product recommendation engine for an e-commerce platform. The system will analyse past purchases, browsing history and user interactions to personalise product rankings in real time. Your task is to assess two datasets that will feed into this ML pipeline, evaluating their scalability and identifying potential resource bottlenecks.
This exercise will help you practise key skills, including:
- Recognising how dataset format and size influence scalability and resource usage.
- Selecting appropriate distributed storage solutions for different data types.
- Linking data quality issues to increased compute, memory and I/O demands in ML pipelines. Download and access the datasets: To begin, download the following datasets:
Downloads
M11_U2_dataset_A.csv Historical transaction records from the platform’s database (1 GB). This tabular data contains purchase IDs, product categories, timestamps, prices and customer IDs, but also has many missing values and inconsistent date formats.
M11_U2_dataset_B.jsonl User activity logs captured from the platform’s mobile and web apps (100 MB). This unstructured data includes nested fields for device information, page visits and clickstream sequences.
Your task
Using the provided datasets, complete the following steps. Use the form below to record your answers.Analyse data characteristics: For each dataset, describe the primary challenges its format and size present for the ML system (e.g. storage efficiency, parsing complexity, schema evolution).Select storage solutions:
Suggest one appropriate distributed storage solution for each dataset type and justify your choice. Assess data quality impacts (for dataset_A.csv only):
- Identify two specific data quality issues.
- Explain how each issue would increase resource requirements during model training or inference.