{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f2edc6d0",
   "metadata": {},
   "source": [
    "# AI ML Fellowship | Asynchronous Unit 2\n",
    "\n",
    "> Advanced Training Strategies\n",
    "\n",
    "# Skillable Lab: The performance lab"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3299c829",
   "metadata": {},
   "source": [
    "## Objective\n",
    "Apply hyperparameter tuning, ensemble learning, and model calibration to improve the accuracy, robustness, and probability reliability of a credit risk model used for loan default prediction."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aba074de",
   "metadata": {},
   "source": [
    "## Scenario\n",
    "You have joined **SafeFinance**, a digital lending company. The current credit risk model performs well on accuracy but **overpredicts default risk**, causing lost business. Your task is to refine the training process to improve predictive performance **and** align predicted probabilities with real-world outcomes."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a85b06f5",
   "metadata": {},
   "source": [
    "## Dataset Information\n",
    "- Source: Synthetic SafeFinance credit risk dataset prepared for this unit.\n",
    "- Size: 10,000 records; target `default` (1 = default, 0 = repaid).\n",
    "- Default rate: Approximately 2–3%.\n",
    "- Features span demographics, financials, credit history/behaviour, application details, and device signals.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3d4330f",
   "metadata": {},
   "source": [
    "## Skillable Lab: Improving SafeFinance Credit Risk Predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f4fde18",
   "metadata": {},
   "source": [
    "## Step 1: Load the dataset\n",
    "Load the dataset and create a train/test split. Display basic class balance and preview the data.\n",
    "**Deliverable:** Clean load with summary stats and class distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9cfdb821",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "path = '/mnt/data/safefinance_credit_risk.csv'\n",
    "df = pd.read_csv(path)\n",
    "print(df.shape)\n",
    "display(df.head())\n",
    "\n",
    "# Class balance\n",
    "class_counts = df['default'].value_counts().sort_index()\n",
    "print(class_counts)\n",
    "\n",
    "# Simple bar plot of class distribution\n",
    "plt.figure()\n",
    "class_counts.plot(kind='bar')\n",
    "plt.title('Class distribution: default')\n",
    "plt.xlabel('default')\n",
    "plt.ylabel('count')\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c57fef07",
   "metadata": {},
   "source": [
    "## Step 2: Establish a baseline\n",
    "Train a simple baseline model and record ROC AUC. This gives you a reference point for improvements.\n",
    "**Deliverable:** Baseline AUC."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "638d30b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import OneHotEncoder\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import roc_auc_score, RocCurveDisplay\n",
    "\n",
    "# Define features/target\n",
    "y = df['default']\n",
    "X = df.drop(columns=['default'])\n",
    "\n",
    "num_cols = X.select_dtypes(include=['int64','float64','int32','float32']).columns.tolist()\n",
    "cat_cols = X.select_dtypes(include=['object']).columns.tolist()\n",
    "\n",
    "preprocess = ColumnTransformer([\n",
    "    ('cat', OneHotEncoder(handle_unknown='ignore'), cat_cols)\n",
    "], remainder='passthrough')\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42, stratify=y)\n",
    "\n",
    "baseline = Pipeline([\n",
    "    ('prep', preprocess),\n",
    "    ('clf', LogisticRegression(max_iter=1000, solver='liblinear'))\n",
    "])\n",
    "baseline.fit(X_train, y_train)\n",
    "probs = baseline.predict_proba(X_test)[:,1]\n",
    "auc = roc_auc_score(y_test, probs)\n",
    "print(f'Baseline ROC AUC: {auc:.3f}')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "692317f4",
   "metadata": {},
   "source": [
    "## Step 3: Hyperparameter tuning\n",
    "Use grid or random search to tune the baseline model for ROC AUC. Record the best parameters and score.\n",
    "**Deliverable:** Best parameters and improved AUC."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "296f4064",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "pipe = Pipeline([\n",
    "    ('prep', preprocess),\n",
    "    ('clf', LogisticRegression(max_iter=2000, solver='liblinear'))\n",
    "])\n",
    "\n",
    "param_grid = {\n",
    "    'clf__C': [0.01, 0.1, 1.0, 10.0],\n",
    "    'clf__penalty': ['l1','l2']\n",
    "}\n",
    "\n",
    "grid = GridSearchCV(pipe, param_grid=param_grid, scoring='roc_auc', cv=5, n_jobs=-1)\n",
    "grid.fit(X_train, y_train)\n",
    "best = grid.best_estimator_\n",
    "best_probs = best.predict_proba(X_test)[:,1]\n",
    "best_auc = roc_auc_score(y_test, best_probs)\n",
    "print('Best params:', grid.best_params_)\n",
    "print(f'Tuned ROC AUC: {best_auc:.3f}')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cae97f82",
   "metadata": {},
   "source": [
    "## Step 4: Compare an ensemble model\n",
    "Train a Random Forest (or XGBoost if available) and compare ROC AUC with the tuned model. Note any interpretability or efficiency trade-offs.\n",
    "**Deliverable:** Ensemble AUC and a brief comparison."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f725a9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "ens_pipe = Pipeline([\n",
    "    ('prep', preprocess),\n",
    "    ('rf', RandomForestClassifier(n_estimators=200, random_state=42))\n",
    "])\n",
    "ens_pipe.fit(X_train, y_train)\n",
    "ens_probs = ens_pipe.predict_proba(X_test)[:,1]\n",
    "ens_auc = roc_auc_score(y_test, ens_probs)\n",
    "print(f'Random Forest ROC AUC: {ens_auc:.3f}')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f83c4921",
   "metadata": {},
   "source": [
    "## Step 5: Calibrate probabilities and visualize reliability\n",
    "Calibrate your best-performing model using **Platt scaling** or **isotonic regression**. Plot calibration curves and compute Brier scores before vs. after.\n",
    "**Deliverable:** Calibration plot and Brier score comparison."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4cea3587",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.calibration import CalibratedClassifierCV, calibration_curve\n",
    "from sklearn.metrics import brier_score_loss\n",
    "\n",
    "# Choose the stronger of tuned logistic vs ensemble for calibration\n",
    "use_ensemble = ens_auc >= best_auc\n",
    "model_to_cal = ens_pipe if use_ensemble else best\n",
    "\n",
    "cal = CalibratedClassifierCV(model_to_cal, method='isotonic', cv=5)\n",
    "cal.fit(X_train, y_train)\n",
    "\n",
    "probs_before = (model_to_cal.predict_proba(X_test)[:,1])\n",
    "probs_after = cal.predict_proba(X_test)[:,1]\n",
    "\n",
    "true_before, pred_before = calibration_curve(y_test, probs_before, n_bins=10)\n",
    "true_after, pred_after = calibration_curve(y_test, probs_after, n_bins=10)\n",
    "\n",
    "plt.figure()\n",
    "plt.plot(pred_before, true_before, label='Before calibration')\n",
    "plt.plot(pred_after, true_after, label='After calibration')\n",
    "plt.plot([0,1],[0,1],'--', label='Perfectly calibrated')\n",
    "plt.xlabel('Predicted probability')\n",
    "plt.ylabel('True probability')\n",
    "plt.title('Calibration curves')\n",
    "plt.legend()\n",
    "plt.show()\n",
    "\n",
    "b_before = brier_score_loss(y_test, probs_before)\n",
    "b_after = brier_score_loss(y_test, probs_after)\n",
    "print(f'Brier score before: {b_before:.4f}')\n",
    "print(f'Brier score after:  {b_after:.4f}')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a60577d",
   "metadata": {},
   "source": [
    "## Step 6: Reflect on your implementation\n",
    "**Prompt:** In 100–150 words, summarize how tuning, ensembling, and calibration each affected performance and trust. Discuss trade-offs between accuracy, interpretability, and computational efficiency, and recommend the model you would ship for SafeFinance."
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}