{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# AI ML Fellowship | Module 7 Asynchronous Unit 1\n",
        "\n",
        "> Model Training and Optimization\n",
        "\n",
        "# Skillable Lab: The Optimization Diagnosis"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n",
        "## Objective\n",
        "\n",
        "In this lab, you will diagnose and improve the training behavior of a neural network for binary classification. You will:\n",
        "- Compare optimizers (SGD vs. Adam) for convergence and generalization.\n",
        "- Apply regularization (L2 and dropout) to reduce overfitting.\n",
        "- Interpret training and validation curves to justify optimization decisions.\n",
        "- Recommend one concrete change to improve stability or performance.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n",
        "## Scenario\n",
        "\n",
        "You have joined **MedPredict**, a health technology startup building predictive models for early disease detection.\n",
        "The current neural network achieves high training accuracy but generalizes poorly to unseen patients. Your task is to\n",
        "analyze the training dynamics, select an appropriate optimizer, and apply regularization to achieve stable, well-generalized performance.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n",
        "## Dataset information\n",
        "\n",
        "You will use a synthetic dataset representing health indicators for 5,000 patients with 20 numerical features and a binary target `disease_present`.\n",
        "- **Rows:** 5,000 samples.\n",
        "- **Features:** 20 standardized numerical inputs.\n",
        "- **Target:** `disease_present` (1 = disease, 0 = no disease).\n",
        "- **Balance:** Approximately 50/50 positive and negative cases.\n",
        "- **Note:** All features are synthetic and standardized; they represent plausible (not real) medical measures.\n",
        "\n",
        "If a local file named `medpredict_synthetic_dataset.csv` is present, the notebook will load it.\n",
        "If not, the dataset will be generated programmatically to keep the lab self-contained.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n",
        "## Skillable Lab: The Optimization Diagnosis — Steps\n",
        "\n",
        "1. Load or generate the dataset.\n",
        "\n",
        "2. Prepare train/validation splits and standardize features.\n",
        "\n",
        "3. Build a baseline neural network.\n",
        "\n",
        "4. Compare optimizers (SGD vs. Adam) and plot training/validation curves.\n",
        "\n",
        "5. Apply regularization (L2 and dropout) and evaluate generalization.\n",
        "\n",
        "6. Summarize results and recommend one improvement.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n",
        "## Step 1: Load or generate the dataset\n",
        "\n",
        "In this step, you will load `medpredict_synthetic_dataset.csv` if available or generate the dataset with comparable properties.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "\n",
        "import os\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "# Try to load dataset if provided alongside the notebook\n",
        "csv_path = \"medpredict_synthetic_dataset.csv\"\n",
        "if os.path.exists(csv_path):\n",
        "    df = pd.read_csv(csv_path)\n",
        "else:\n",
        "    # Generate synthetic dataset (mirrors the separate resource)\n",
        "    from sklearn.datasets import make_classification\n",
        "\n",
        "    X, y = make_classification(\n",
        "        n_samples=5000,\n",
        "        n_features=20,\n",
        "        n_informative=10,\n",
        "        n_redundant=5,\n",
        "        n_repeated=0,\n",
        "        n_classes=2,\n",
        "        n_clusters_per_class=2,\n",
        "        class_sep=1.2,\n",
        "        flip_y=0.01,\n",
        "        random_state=42,\n",
        "    )\n",
        "\n",
        "    feature_names = [\n",
        "        \"age\",\"bmi\",\"blood_pressure\",\"cholesterol\",\"glucose\",\"insulin\",\"heart_rate\",\n",
        "        \"oxygen_saturation\",\"body_temperature\",\"wbc_count\",\"rbc_count\",\"platelet_count\",\n",
        "        \"creatinine\",\"urea\",\"sodium\",\"potassium\",\"calcium\",\"hemoglobin\",\"smoking_index\",\"activity_level\",\n",
        "    ]\n",
        "    df = pd.DataFrame(X, columns=feature_names)\n",
        "    df[\"disease_present\"] = y\n",
        "\n",
        "df.head()\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n",
        "## Step 2: Prepare train/validation splits and standardize features\n",
        "\n",
        "Split the data into train and validation sets, then standardize input features to help the neural network train reliably.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "\n",
        "# Features/target\n",
        "X = df.drop(columns=[\"disease_present\"]).values\n",
        "y = df[\"disease_present\"].values\n",
        "\n",
        "# Train/validation split\n",
        "X_train, X_val, y_train, y_val = train_test_split(\n",
        "    X, y, test_size=0.2, random_state=42, stratify=y\n",
        ")\n",
        "\n",
        "# Standardize\n",
        "scaler = StandardScaler()\n",
        "X_train_scaled = scaler.fit_transform(X_train)\n",
        "X_val_scaled = scaler.transform(X_val)\n",
        "\n",
        "X_train_scaled.shape, X_val_scaled.shape\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n",
        "## Step 3: Build a baseline neural network\n",
        "\n",
        "We define a helper function to build a simple feedforward network with optional **L2 weight decay** and **dropout**.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "\n",
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "from tensorflow.keras import layers, regularizers\n",
        "\n",
        "def build_model(input_dim, l2_lambda=0.0, dropout_rate=0.0):\n",
        "    model = keras.Sequential([\n",
        "        layers.Input(shape=(input_dim,)),\n",
        "        layers.Dense(64, activation=\"relu\", kernel_regularizer=regularizers.l2(l2_lambda) if l2_lambda>0 else None),\n",
        "        layers.Dropout(dropout_rate) if dropout_rate>0 else layers.Lambda(lambda x: x),\n",
        "        layers.Dense(32, activation=\"relu\", kernel_regularizer=regularizers.l2(l2_lambda) if l2_lambda>0 else None),\n",
        "        layers.Dropout(dropout_rate) if dropout_rate>0 else layers.Lambda(lambda x: x),\n",
        "        layers.Dense(1, activation=\"sigmoid\")\n",
        "    ])\n",
        "    return model\n",
        "\n",
        "input_dim = X_train_scaled.shape[1]\n",
        "input_dim\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n",
        "## Step 4: Compare optimizers (SGD vs. Adam)\n",
        "\n",
        "Train two baseline models (no regularization) with **SGD** and **Adam**. Compare convergence speed and generalization using training/validation metrics.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "def compile_and_train(optimizer_name, epochs=15, batch_size=64):\n",
        "    model = build_model(input_dim=input_dim, l2_lambda=0.0, dropout_rate=0.0)\n",
        "    if optimizer_name.lower()==\"sgd\":\n",
        "        optimizer = keras.optimizers.SGD(learning_rate=0.01, momentum=0.0, nesterov=False)\n",
        "    elif optimizer_name.lower()==\"adam\":\n",
        "        optimizer = keras.optimizers.Adam(learning_rate=0.001)\n",
        "    else:\n",
        "        raise ValueError(\"Unsupported optimizer\")\n",
        "    model.compile(optimizer=optimizer, loss=\"binary_crossentropy\", metrics=[\"accuracy\"])\n",
        "    history = model.fit(\n",
        "        X_train_scaled, y_train,\n",
        "        validation_data=(X_val_scaled, y_val),\n",
        "        epochs=15, batch_size=64, verbose=0\n",
        "    )\n",
        "    val_loss, val_acc = model.evaluate(X_val_scaled, y_val, verbose=0)\n",
        "    return model, history, val_loss, val_acc\n",
        "\n",
        "# Train with SGD and Adam\n",
        "sgd_model, sgd_hist, sgd_vloss, sgd_vacc = compile_and_train(\"sgd\")\n",
        "adam_model, adam_hist, adam_vloss, adam_vacc = compile_and_train(\"adam\")\n",
        "\n",
        "print(f\"SGD validation accuracy: {sgd_vacc:.4f} | loss: {sgd_vloss:.4f}\")\n",
        "print(f\"Adam validation accuracy: {adam_vacc:.4f} | loss: {adam_vloss:.4f}\")\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "\n",
        "# Plot training vs validation loss for both optimizers\n",
        "def plot_history(hist, label_prefix):\n",
        "    h = hist.history\n",
        "    epochs = range(1, len(h[\"loss\"])+1)\n",
        "    plt.figure()\n",
        "    plt.plot(epochs, h[\"loss\"], label=f\"{label_prefix} train loss\")\n",
        "    plt.plot(epochs, h[\"val_loss\"], label=f\"{label_prefix} val loss\")\n",
        "    plt.xlabel(\"Epoch\")\n",
        "    plt.ylabel(\"Loss\")\n",
        "    plt.title(f\"{label_prefix}: Training vs Validation Loss\")\n",
        "    plt.legend()\n",
        "    plt.show()\n",
        "\n",
        "plot_history(sgd_hist, \"SGD\")\n",
        "plot_history(adam_hist, \"Adam\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n",
        "## Step 5: Apply regularization (L2 and dropout)\n",
        "\n",
        "Add **L2 weight decay** and **dropout** to the model, then retrain with the better-performing optimizer from Step 4.\n",
        "Evaluate the impact on validation accuracy and stability.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "\n",
        "# Choose the better optimizer based on validation accuracy\n",
        "better_opt = \"adam\" if adam_vacc >= sgd_vacc else \"sgd\"\n",
        "print(\"Selected optimizer for regularized model:\", better_opt.upper())\n",
        "\n",
        "def train_with_regularization(optimizer_name, l2_lambda=1e-4, dropout_rate=0.2, epochs=15):\n",
        "    model = build_model(input_dim=input_dim, l2_lambda=l2_lambda, dropout_rate=dropout_rate)\n",
        "    if optimizer_name.lower()==\"sgd\":\n",
        "        optimizer = keras.optimizers.SGD(learning_rate=0.01)\n",
        "    else:\n",
        "        optimizer = keras.optimizers.Adam(learning_rate=0.001)\n",
        "    model.compile(optimizer=optimizer, loss=\"binary_crossentropy\", metrics=[\"accuracy\"])\n",
        "    callbacks = [\n",
        "        keras.callbacks.EarlyStopping(patience=3, restore_best_weights=True, monitor=\"val_loss\")\n",
        "    ]\n",
        "    history = model.fit(\n",
        "        X_train_scaled, y_train,\n",
        "        validation_data=(X_val_scaled, y_val),\n",
        "        epochs=20, batch_size=64, verbose=0, callbacks=callbacks\n",
        "    )\n",
        "    val_loss, val_acc = model.evaluate(X_val_scaled, y_val, verbose=0)\n",
        "    return model, history, val_loss, val_acc\n",
        "\n",
        "reg_model, reg_hist, reg_vloss, reg_vacc = train_with_regularization(better_opt, l2_lambda=1e-4, dropout_rate=0.2)\n",
        "print(f\"Regularized model ({better_opt.upper()}) validation accuracy: {reg_vacc:.4f} | loss: {reg_vloss:.4f}\")\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "\n",
        "# Plot regularized model curves\n",
        "plot_history(reg_hist, f\"Regularized ({better_opt.upper()})\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n",
        "## Step 6: Summarize results and recommend an improvement\n",
        "\n",
        "Use the results to answer the questions below:\n",
        "- Which optimizer converged faster? Which generalized better on validation?\n",
        "- How did L2 and dropout affect validation accuracy and loss curves?\n",
        "- Recommend one specific change (e.g., learning rate, optimizer choice, dropout rate, L2 strength, early stopping) and justify your decision in 2–3 sentences.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "\n",
        "# Quick summary table\n",
        "import pandas as pd\n",
        "\n",
        "summary_df = pd.DataFrame([\n",
        "    {\"model\":\"SGD (no reg)\", \"val_acc\": sgd_vacc, \"val_loss\": sgd_vloss},\n",
        "    {\"model\":\"Adam (no reg)\", \"val_acc\": adam_vacc, \"val_loss\": adam_vloss},\n",
        "    {\"model\":f\"{better_opt.upper()} + L2+Dropout\", \"val_acc\": reg_vacc, \"val_loss\": reg_vloss},\n",
        "]).sort_values(\"val_acc\", ascending=False).reset_index(drop=True)\n",
        "\n",
        "summary_df\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n",
        "### Reflection (enter your notes here)\n",
        "\n",
        "- Optimizer comparison (speed and generalization):  \n",
        "- Regularization impact (L2 and dropout):  \n",
        "- Final recommendation and rationale:  \n"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.x"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}