{"cells":[{"cell_type":"markdown","id":"560cc720","metadata":{"id":"560cc720"},"source":["# AI ML Fellowship | Module 11 | Unit 1\n","\n","# Lab: Detecting Data Drift"]},{"cell_type":"markdown","id":"00b8c54c","metadata":{"id":"00b8c54c"},"source":["**Objective**\n","\n","In this lab, you will practice detecting data drift between a training dataset and a production dataset. You will work with a synthetic customer subscription dataset containing both numerical and categorical features.\n","\n","**Scenario:**  \n","You are managing a machine learning model that predicts customer churn. Recently, business stakeholders raised concerns about a potential shift in customer demographics and subscription patterns, which could affect model performance. Your task is to compare the training dataset with recent production data to identify any signs of drift in key features.\n","\n","You will calculate:\n","- The Population Stability Index (PSI) for a numerical feature (`age`).\n","- The Chi-squared statistic for a categorical feature (`subscription_type`).\n","- Interpret the results to determine if data drift is present.\n"]},{"cell_type":"markdown","id":"7a6c78cf","metadata":{"id":"7a6c78cf"},"source":["\n","## Dataset Information\n","\n","You will work with two datasets in this lab:\n","1. [**Training Data**](https://drive.google.com/file/d/1cof1_XAQEBPkRC31Z_BhoITDBoBoySBy/view?usp=drive_link) – Historical data used to train the churn prediction model.\n","2. [**Production Data**](https://drive.google.com/file/d/1OZQClPszK2ZmAjpxxP7MN30c4Qor36tM/view?usp=drive_link) – Recent data collected after model deployment.\n","\n","Each dataset contains the following features:\n","- **age (Numerical):** Customer's age.\n","- **subscription_type (Categorical):** Type of subscription plan (`Basic`, `Premium`, or `Enterprise`).\n","\n","The goal is to compare these datasets and detect if there are any significant shifts in data distributions that might impact model performance.\n"]},{"cell_type":"markdown","id":"6b004bff","metadata":{"id":"6b004bff"},"source":["\n","## Steps to Follow\n","\n","Follow these steps to complete the lab:\n","1. **Load and inspect** both the [training](https://drive.google.com/file/d/1cof1_XAQEBPkRC31Z_BhoITDBoBoySBy/view?usp=drive_link) and [production](https://drive.google.com/file/d/1OZQClPszK2ZmAjpxxP7MN30c4Qor36tM/view?usp=drive_link) datasets to understand their structure.\n","2. **Calculate the Population Stability Index (PSI)** for the `age` feature to detect any distribution drift in numerical data.\n","3. **Calculate the Chi-squared statistic** for the `subscription_type` feature to assess shifts in categorical distributions.\n","4. **Interpret the results** to determine if significant data drift has occurred.\n","5. Reflect on how these drift indicators might inform your next actions for maintaining model performance.\n"]},{"cell_type":"markdown","id":"a173ddd0","metadata":{"id":"a173ddd0"},"source":["## Step 1: Load and Inspect the Datasets\n","In this step, you will load the provided training and production datasets and inspect their structure. This will help you understand the features you'll be working with and identify any immediate visual differences."]},{"cell_type":"code","execution_count":null,"id":"aed68eeb","metadata":{"id":"aed68eeb"},"outputs":[],"source":["\n","import pandas as pd\n","\n","# Load training and production datasets\n","training_data = pd.read_csv('training_data_drift_activity.csv')\n","production_data = pd.read_csv('production_data_drift_activity.csv')\n","\n","# Preview datasets\n","print(\"Training Data Sample:\")\n","print(training_data.head())\n","\n","print(\"\\nProduction Data Sample:\")\n","print(production_data.head())\n"]},{"cell_type":"markdown","id":"56add546","metadata":{"id":"56add546"},"source":["## Step 2: Calculate PSI for the 'age' Feature\n","Let's now calculate the Population Stability Index (PSI) for the numerical feature `age`. PSI helps quantify shifts in feature distributions between training and production data."]},{"cell_type":"code","execution_count":null,"id":"08cb6550","metadata":{"id":"08cb6550"},"outputs":[],"source":["\n","import numpy as np\n","\n","def calculate_psi(expected, actual, buckets=10):\n","    breakpoints = np.linspace(0, 100, buckets + 1)\n","    expected_percents = np.histogram(expected, bins=np.percentile(expected, breakpoints))[0] / len(expected)\n","    actual_percents = np.histogram(actual, bins=np.percentile(expected, breakpoints))[0] / len(actual)\n","    psi_value = np.sum((expected_percents - actual_percents) * np.log((expected_percents + 1e-6) / (actual_percents + 1e-6)))\n","    return psi_value\n","\n","psi_age = calculate_psi(training_data['age'], production_data['age'])\n","print(f\"PSI for 'age': {psi_age:.4f}\")\n"]},{"cell_type":"markdown","id":"1dc17fd1","metadata":{"id":"1dc17fd1"},"source":["## Step 3: Calculate Chi-squared Statistic for 'subscription_type'\n","Next, let's assess drift in the categorical feature `subscription_type` using a Chi-squared test. This will determine if the category distributions have shifted significantly."]},{"cell_type":"code","execution_count":null,"id":"1784bb1f","metadata":{"id":"1784bb1f"},"outputs":[],"source":["\n","from scipy.stats import chi2_contingency\n","\n","# Create frequency tables\n","train_counts = training_data['subscription_type'].value_counts()\n","prod_counts = production_data['subscription_type'].value_counts()\n","\n","# Align categories\n","categories = sorted(list(set(train_counts.index) | set(prod_counts.index)))\n","train_counts = train_counts.reindex(categories, fill_value=0)\n","prod_counts = prod_counts.reindex(categories, fill_value=0)\n","\n","# Build contingency table\n","contingency_table = pd.DataFrame({'Training': train_counts, 'Production': prod_counts})\n","\n","# Run Chi-squared test\n","chi2, p_value, _, _ = chi2_contingency(contingency_table.T)\n","print(f\"Chi-squared statistic for 'subscription_type': {chi2:.4f}, p-value: {p_value:.4f}\")\n"]},{"cell_type":"markdown","id":"c8df3f0b","metadata":{"id":"c8df3f0b"},"source":["## Step 4: Interpretation\n","Finally, interpret the PSI and Chi-squared test results to determine if significant data drift has occurred. This reflection will guide your decision on whether model maintenance actions are needed."]},{"cell_type":"markdown","id":"36a18770","metadata":{"id":"36a18770"},"source":["\n","- **PSI for 'age':** Values above 0.1 typically indicate moderate drift, while values above 0.25 suggest significant drift. Interpret your PSI result to determine if age distribution has shifted meaningfully.\n","- **Chi-squared test for 'subscription_type':** A low p-value (typically < 0.05) suggests a significant difference in category distributions between training and production data.\n","\n","Based on these results:\n","- Has the 'age' feature drifted in production?\n","- Has the distribution of 'subscription_type' changed significantly?\n","- What would be your next steps to ensure your model remains accurate?\n"]}],"metadata":{"language_info":{"name":"python"},"colab":{"provenance":[]}},"nbformat":4,"nbformat_minor":5}