{"cells":[{"cell_type":"markdown","id":"9c846041","metadata":{"id":"9c846041"},"source":["# AI ML Fellowship | Module 10 | Workshop 1\n","\n","## Detecting data drift in production\n","\n","## Objectives:\n","- Identify data drift using visual and statistical checks.\n","- Run a Kolmogorov–Smirnov test to compare baseline and production distributions.\n","- Simulate a simple monitoring loop that can trigger alerts.\n","\n","## Scenario:\n","A model worked perfectly in testing but failed in production. The key question: what changed?\n","\n","## Dataset Information\n","- **Name**: Data Drift Simulation Datasets\n","- **Source**: Synthetic datasets\n","- **Description**: These datasets are designed to simulate real-world scenarios where input data distributions shift over time, affecting model performance in production environments. The data represents simplified demographic and financial information for individuals and is used to demonstrate visual and statistical detection of data drift.\n","    - **[baseline_data.csv](https://drive.google.com/file/d/1B49XJmfzDIs_bi490UkM8QH097dlE8HU/view?usp=drive_link)** represents the reference dataset used for model training. It contains customer attributes such as age, income, and region.\n","    - **[production_data.csv](https://drive.google.com/file/d/1YYsYlR3kg8C4g1F2XeepoEn-dlS6b5sA/view?usp=drive_link)** represents the same population after deployment, where income distribution has shifted upward and regional proportions have changed slightly, introducing measurable drift. It also includes timestamps simulating incoming production data batches for monitoring exercises.\n","- **Format**: CSV\n","- **Size**:\n","    - **baseline_data.csv**: 5,000 records\n","    - **production_data.csv**: 5,000 records"]},{"cell_type":"markdown","id":"24cb6c9d","metadata":{"id":"24cb6c9d"},"source":["## Setup\n","Run this cell to import libraries and set file paths."]},{"cell_type":"code","execution_count":null,"id":"1e552eb3","metadata":{"id":"1e552eb3"},"outputs":[],"source":["import pandas as pd\n","import numpy as np\n","import matplotlib.pyplot as plt\n","from scipy.stats import ks_2samp\n","\n","%matplotlib inline\n","\n","BASELINE_PATH = 'baseline_data.csv'\n","PRODUCTION_PATH = 'production_data.csv'\n","\n","# Choose the feature to analyse (e.g., 'income' or 'age').\n","FEATURE = 'income'\n","FEATURE"]},{"cell_type":"markdown","id":"81750349","metadata":{"id":"81750349"},"source":["## 1. Visualize the baseline distribution\n","- Load the [baseline_data.csv file](https://drive.google.com/file/d/1B49XJmfzDIs_bi490UkM8QH097dlE8HU/view?usp=drive_link) into your workspace.\n","- Select a key feature from the dataset.\n","- Plot a histogram to visualize the distribution of that feature.\n","- Observe this distribution as the model’s reference baseline."]},{"cell_type":"code","execution_count":null,"id":"94bc5433","metadata":{"id":"94bc5433"},"outputs":[],"source":["# Load baseline data\n","baseline = pd.read_csv(BASELINE_PATH)\n","baseline.head()"]},{"cell_type":"code","execution_count":null,"id":"5bbb407f","metadata":{"id":"5bbb407f"},"outputs":[],"source":["# Plot baseline histogram for the selected FEATURE\n","plt.figure()\n","plt.hist(baseline[FEATURE], bins=30, alpha=0.7)\n","plt.title(f'Baseline {FEATURE} distribution')\n","plt.xlabel(FEATURE)\n","plt.ylabel('Count')\n","plt.show()"]},{"cell_type":"markdown","id":"71ce3655","metadata":{"id":"71ce3655"},"source":["## 2. Compare baseline and production data\n","- Load the [production_data.csv file](https://drive.google.com/file/d/1YYsYlR3kg8C4g1F2XeepoEn-dlS6b5sA/view?usp=drive_link).\n","- Plot a histogram of the same feature used in the baseline analysis.\n","- Overlay this histogram on the baseline histogram to visually compare the distributions.\n","- Look for visible drift or changes in the shape, centre, or spread of the feature distribution."]},{"cell_type":"code","execution_count":null,"id":"f84683bf","metadata":{"id":"f84683bf"},"outputs":[],"source":["# Load production data (with timestamps for realism)\n","production = pd.read_csv(PRODUCTION_PATH, parse_dates=['event_time'])\n","production.head()"]},{"cell_type":"code","execution_count":null,"id":"2796d3e5","metadata":{"id":"2796d3e5"},"outputs":[],"source":["# Overlay histograms on the same axis (visual drift)\n","plt.figure()\n","plt.hist(baseline[FEATURE], bins=30, alpha=0.5, label='Baseline')\n","plt.hist(production[FEATURE], bins=30, alpha=0.5, label='Production')\n","plt.title(f'{FEATURE} distribution: baseline vs. production')\n","plt.xlabel(FEATURE)\n","plt.ylabel('Count')\n","plt.legend()\n","plt.show()"]},{"cell_type":"markdown","id":"85060853","metadata":{"id":"85060853"},"source":["## 3. Detect statistical drift\n","Use the Kolmogorov–Smirnov (KS) test to compare the baseline and production distributions:\n","- Import `ks_2samp` from `scipy.stats` (already imported in Setup).\n","- Write a function `detect_drift()` that returns the p-value.\n","- Run the function and interpret the result (p < 0.05 indicates drift)."]},{"cell_type":"code","execution_count":null,"id":"6c4048ff","metadata":{"id":"6c4048ff"},"outputs":[],"source":["def detect_drift(baseline_df: pd.DataFrame, production_df: pd.DataFrame, feature: str) -> float:\n","    \"\"\"\n","    Compare distributions of `feature` in baseline vs production using the two-sample KS test.\n","    Returns the p-value (float). Lower p-values indicate stronger evidence of distributional difference.\n","    \"\"\"\n","    x = baseline_df[feature].dropna().values\n","    y = production_df[feature].dropna().values\n","    _, p = ks_2samp(x, y)\n","    return float(p)\n","\n","p_value = detect_drift(baseline, production, FEATURE)\n","print(f'KS test p-value for {FEATURE}: {p_value:.6f}')\n","if p_value < 0.05:\n","    print('Interpretation: p < 0.05 → Statistical drift detected.')\n","else:\n","    print('Interpretation: p ≥ 0.05 → No statistical drift detected.')"]},{"cell_type":"markdown","id":"23d46dad","metadata":{"id":"23d46dad"},"source":["## 4. Create a simple monitoring loop\n","Simulate a batch stream by creating a loop that processes subsets of production data.\n","- Within each loop iteration:\n","  - Call `detect_drift()` on the current batch vs. the baseline.\n","  - Print the p-value.\n","  - Print an \"ALERT!\" message if p-value < 0.05.\n","- Observe how the loop mimics real-time monitoring and drift detection."]},{"cell_type":"code","execution_count":null,"id":"9655532f","metadata":{"id":"9655532f"},"outputs":[],"source":["THRESHOLD = 0.05\n","BATCH_SIZE = 200\n","\n","def iter_batches(df: pd.DataFrame, batch_size: int):\n","    for i in range(0, len(df), batch_size):\n","        yield df.iloc[i:i+batch_size]\n","\n","alerts = []\n","for batch in iter_batches(production.sort_values('event_time'), BATCH_SIZE):\n","    p = detect_drift(baseline, batch, FEATURE)\n","    ts = batch['event_time'].iloc[-1]\n","    print(f\"Batch ending {ts}: p-value={p:.6f}\")\n","    if p < THRESHOLD:\n","        print(f\"ALERT! Drift detected at {ts} (p={p:.6f}).\")\n","        alerts.append({'time': ts, 'p_value': p})\n","\n","pd.DataFrame(alerts) if alerts else pd.DataFrame(columns=['time', 'p_value'])"]},{"cell_type":"markdown","id":"18199c25","metadata":{"id":"18199c25"},"source":["## 5. Share your findings\n","Submit your work showing your plots, drift detection code, and monitoring loop. Include comments or explanations that clarify your approach.\n","- What did you observe about the baseline vs production distributions?\n","- Was the KS test p-value below 0.05? What does that imply?\n","- How did the monitoring loop behave across batches?"]},{"cell_type":"markdown","id":"9bea6309","metadata":{"id":"9bea6309"},"source":["## Reflection\n","- How does the statistical test used in this activity support real-time monitoring and logging in a deployed ML system?\n","- From a risk management perspective, how could automated drift detection like this help prevent major model failures?\n","- If this monitoring system triggered an alert, what kind of rollback strategy would make sense in response?"]}],"metadata":{"language_info":{"name":"python"},"colab":{"provenance":[]}},"nbformat":4,"nbformat_minor":5}