{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "32d9f2ee",
   "metadata": {},
   "source": [
    "# AI ML Fellowship | Module 10 | Workshop 2\n",
    "\n",
    "# Simulating inference scaling trade-offs\n",
    "\n",
    "**Objectives**:\n",
    "- Measure baseline latency for a single inference.\n",
    "- Simulate concurrent requests and identify bottlenecks.\n",
    "- Compare vertical scaling (bigger machine) vs. horizontal scaling (more machines).\n",
    "- Evaluate simple cost metrics (e.g., cost per 1,000 predictions) to inform architecture choices.\n",
    "\n",
    "## Scenario:\n",
    "A team ships a blazing fast model on a small dev server, then production traffic spikes and latency goes through the roof. The key question: how do we scale intelligently for both performance and cost?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5bdaf97c",
   "metadata": {},
   "source": [
    "## Getting started\n",
    "- No external datasets are required for this activity.\n",
    "- Run the cells in order from top to bottom.\n",
    "- Adjust the configuration values in the **Setup** cell (processing time, cores, number of requests, and simple cost multipliers) to explore different scenarios.\n",
    "- Use the optional histogram cells to visualize latency distribution.\n",
    "- Record key observations (latency, throughput, and cost) to discuss during the debrief.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bba1041d",
   "metadata": {},
   "source": [
    "## Setup\n",
    "Run this cell to import libraries and configure default parameters for the simulations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d485d6dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import math\n",
    "from concurrent.futures import ThreadPoolExecutor, as_completed\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "# Baseline configuration\n",
    "BASE_PROCESSING_TIME_MS = 40  # baseline per-inference compute time on 1x instance\n",
    "BASE_CORES = 4                # threads available to a single baseline instance\n",
    "BASE_COST = 1.0               # relative hourly cost for a baseline instance\n",
    "\n",
    "# Workload configuration\n",
    "NUM_REQUESTS = 20             # total requests to simulate\n",
    "REQUEST_JITTER_MS = 5         # optional jitter added to processing time to mimic variance\n",
    "\n",
    "def simulate_inference(processing_time_ms: int) -> float:\n",
    "    \"\"\"Simulate a single inference call by sleeping for processing_time_ms.\n",
    "    Returns the observed latency in milliseconds.\"\"\"\n",
    "    start = time.perf_counter()\n",
    "    time.sleep(processing_time_ms / 1000.0)\n",
    "    end = time.perf_counter()\n",
    "    return (end - start) * 1000.0\n",
    "\n",
    "def run_concurrent_requests(n_requests: int, processing_time_ms: int, max_workers: int) -> dict:\n",
    "    \"\"\"Run n_requests concurrently with a thread pool and measure timings.\n",
    "    Returns a dict with per-request latencies and summary stats.\"\"\"\n",
    "    latencies = []\n",
    "    start = time.perf_counter()\n",
    "    with ThreadPoolExecutor(max_workers=max_workers) as executor:\n",
    "        futures = [executor.submit(simulate_inference, processing_time_ms + int(np.random.uniform(0, REQUEST_JITTER_MS)))\n",
    "                   for _ in range(n_requests)]\n",
    "        for f in as_completed(futures):\n",
    "            latencies.append(f.result())\n",
    "    end = time.perf_counter()\n",
    "\n",
    "    total_time_ms = (end - start) * 1000.0\n",
    "    throughput_rps = n_requests / ((end - start) if (end - start) > 0 else 1)\n",
    "    return {\n",
    "        'latencies_ms': latencies,\n",
    "        'total_time_ms': total_time_ms,\n",
    "        'avg_latency_ms': float(np.mean(latencies)) if latencies else float('nan'),\n",
    "        'p95_latency_ms': float(np.percentile(latencies, 95)) if latencies else float('nan'),\n",
    "        'throughput_rps': throughput_rps,\n",
    "    }\n",
    "\n",
    "def summarize_run(tag: str, stats: dict):\n",
    "    print(f\"{tag}\")\n",
    "    print(f\"  Total time: {stats['total_time_ms']:.1f} ms\")\n",
    "    print(f\"  Avg latency: {stats['avg_latency_ms']:.1f} ms\")\n",
    "    print(f\"  P95 latency: {stats['p95_latency_ms']:.1f} ms\")\n",
    "    print(f\"  Throughput: {stats['throughput_rps']:.2f} req/s\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46e9acd5",
   "metadata": {},
   "source": [
    "## Section 1: Baseline single inference performance (5 minutes)\n",
    "Define a `simulate_inference(processing_time_ms)` function and measure the time for a single inference call.\n",
    "\n",
    "**Instructions and tips:** Establish a base latency. Note that `processing_time_ms` represents computational complexity on a basic unit of hardware._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "178e60e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Single-inference measurement\n",
    "latency_ms = simulate_inference(BASE_PROCESSING_TIME_MS)\n",
    "print(f\"Single inference latency: {latency_ms:.1f} ms (baseline)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "638ba693",
   "metadata": {},
   "source": [
    "## Section 2: Simulating concurrent requests and bottlenecks (10 minutes)\n",
    "Use `ThreadPoolExecutor` to simulate processing multiple inference requests concurrently on a single machine with limited cores or threads. Simulate 10–20 concurrent requests and observe how individual request latency and total time change under load due to resource contention.\n",
    "\n",
    "**Instructions and tips:** Show how response times and total time grow as concurrency approaches the core limit on a single machine. This highlights bottlenecks._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89d355fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Concurrent requests on a single baseline instance\n",
    "stats_baseline = run_concurrent_requests(\n",
    "    n_requests=NUM_REQUESTS,\n",
    "    processing_time_ms=BASE_PROCESSING_TIME_MS,\n",
    "    max_workers=BASE_CORES,\n",
    ")\n",
    "summarize_run(\"Baseline instance (vertical=1x, cores={}):\".format(BASE_CORES), stats_baseline)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a0364396",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Optional: visualize latency distribution for the baseline run\n",
    "plt.figure()\n",
    "plt.hist(stats_baseline['latencies_ms'], bins=20)\n",
    "plt.title('Latency distribution (baseline single instance)')\n",
    "plt.xlabel('Latency (ms)')\n",
    "plt.ylabel('Count')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f2f421d",
   "metadata": {},
   "source": [
    "## Section 3: Vertical scaling – more powerful hardware (10 minutes)\n",
    "Modify the inference simulation to represent a more powerful single machine (for example, halve `processing_time_ms`). Simulate the same workload and introduce a simple cost multiplier for the larger machine.\n",
    "\n",
    "**Instructions and tips:** Performance improves at higher per-unit cost. Discuss when this is preferable, such as extremely latency-sensitive workloads._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e2c0e13b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Vertical scaling: faster single machine\n",
    "VERTICAL_COST_MULTIPLIER = 3.0\n",
    "vertical_processing_ms = max(1, BASE_PROCESSING_TIME_MS // 2)  # halve processing time\n",
    "\n",
    "stats_vertical = run_concurrent_requests(\n",
    "    n_requests=NUM_REQUESTS,\n",
    "    processing_time_ms=vertical_processing_ms,\n",
    "    max_workers=BASE_CORES,  # same cores, faster per-core\n",
    ")\n",
    "\n",
    "vertical_hourly_cost = BASE_COST * VERTICAL_COST_MULTIPLIER\n",
    "print(f\"Vertical machine cost (relative): {vertical_hourly_cost:.1f}x baseline\")\n",
    "summarize_run(\"Vertical scaling (faster single instance):\", stats_vertical)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a92fb86b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Optional: visualize latency distribution for the vertical run\n",
    "plt.figure()\n",
    "plt.hist(stats_vertical['latencies_ms'], bins=20)\n",
    "plt.title('Latency distribution (vertical single instance)')\n",
    "plt.xlabel('Latency (ms)')\n",
    "plt.ylabel('Count')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac2de084",
   "metadata": {},
   "source": [
    "## Section 4: Horizontal scaling – multiple smaller instances (15 minutes)\n",
    "Revert to baseline per-core processing time and distribute requests across multiple baseline instances. Compare average latency, total time, throughput, and cost versus the vertical scaling scenario.\n",
    "\n",
    "**Instructions and tips:** This is the core demonstration. Show how distributing load across instances can sustain low per-request latency and high throughput, and discuss cost per prediction._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a188dec",
   "metadata": {},
   "outputs": [],
   "source": [
    "def distribute_requests_across_instances(total_requests: int, instances: int) -> list:\n",
    "    \"\"\"Compute a near-even split of total_requests across instances.\"\"\"\n",
    "    base = total_requests // instances\n",
    "    remainder = total_requests % instances\n",
    "    batches = [base + (1 if i < remainder else 0) for i in range(instances)]\n",
    "    return batches\n",
    "\n",
    "def run_horizontal_scaling(total_requests: int, instances: int, processing_time_ms: int, cores_per_instance: int):\n",
    "    \"\"\"Simulate horizontal scaling by running multiple instances in parallel and aggregating results.\"\"\"\n",
    "    requests_per_instance = distribute_requests_across_instances(total_requests, instances)\n",
    "    instance_stats = []\n",
    "    start = time.perf_counter()\n",
    "    futures = []\n",
    "    with ThreadPoolExecutor(max_workers=instances) as orchestrator:\n",
    "        for reqs in requests_per_instance:\n",
    "            futures.append(\n",
    "                orchestrator.submit(\n",
    "                    run_concurrent_requests, n_requests=reqs,\n",
    "                    processing_time_ms=processing_time_ms,\n",
    "                    max_workers=cores_per_instance\n",
    "                )\n",
    "            )\n",
    "        for f in as_completed(futures):\n",
    "            instance_stats.append(f.result())\n",
    "    end = time.perf_counter()\n",
    "\n",
    "    # Aggregate\n",
    "    all_latencies = [l for s in instance_stats for l in s['latencies_ms']]\n",
    "    wall_time_ms = (end - start) * 1000.0\n",
    "    throughput_rps = total_requests / max((end - start), 1e-9)\n",
    "    return {\n",
    "        'latencies_ms': all_latencies,\n",
    "        'total_time_ms': wall_time_ms,\n",
    "        'avg_latency_ms': float(np.mean(all_latencies)) if all_latencies else float('nan'),\n",
    "        'p95_latency_ms': float(np.percentile(all_latencies, 95)) if all_latencies else float('nan'),\n",
    "        'throughput_rps': throughput_rps,\n",
    "    }\n",
    "\n",
    "# Try different horizontal scales\n",
    "H_INSTANCES = 3  # number of baseline instances\n",
    "stats_horizontal = run_horizontal_scaling(\n",
    "    total_requests=NUM_REQUESTS,\n",
    "    instances=H_INSTANCES,\n",
    "    processing_time_ms=BASE_PROCESSING_TIME_MS,\n",
    "    cores_per_instance=BASE_CORES,\n",
    ")\n",
    "\n",
    "horizontal_hourly_cost = H_INSTANCES * BASE_COST\n",
    "print(f\"Horizontal cluster cost (relative): {horizontal_hourly_cost:.1f}x baseline\")\n",
    "summarize_run(f\"Horizontal scaling ({H_INSTANCES} instances):\", stats_horizontal)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5d2fd612",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Optional: visualize latency distribution for the horizontal run\n",
    "plt.figure()\n",
    "plt.hist(stats_horizontal['latencies_ms'], bins=20)\n",
    "plt.title('Latency distribution (horizontal multi-instance)')\n",
    "plt.xlabel('Latency (ms)')\n",
    "plt.ylabel('Count')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7cb31f3d",
   "metadata": {},
   "source": [
    "## Compare performance and cost\n",
    "Use a simple cost-efficiency view to compare vertical versus horizontal scaling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c268144e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def cost_per_1000_predictions(hourly_cost: float, throughput_rps: float) -> float:\n",
    "    if throughput_rps <= 0:\n",
    "        return float('inf')\n",
    "    preds_per_hour = throughput_rps * 3600.0\n",
    "    return (hourly_cost / preds_per_hour) * 1000.0\n",
    "\n",
    "vertical_hourly_cost = BASE_COST * VERTICAL_COST_MULTIPLIER\n",
    "vertical_cph = cost_per_1000_predictions(vertical_hourly_cost, stats_vertical['throughput_rps'])\n",
    "horizontal_cph = cost_per_1000_predictions(horizontal_hourly_cost, stats_horizontal['throughput_rps'])\n",
    "\n",
    "summary_df = pd.DataFrame([\n",
    "    {\n",
    "        'scenario': 'Baseline single instance',\n",
    "        'hourly_cost': BASE_COST,\n",
    "        'avg_latency_ms': stats_baseline['avg_latency_ms'],\n",
    "        'p95_latency_ms': stats_baseline['p95_latency_ms'],\n",
    "        'throughput_rps': stats_baseline['throughput_rps'],\n",
    "        'cost_per_1000_preds': cost_per_1000_predictions(BASE_COST, stats_baseline['throughput_rps']),\n",
    "    },\n",
    "    {\n",
    "        'scenario': 'Vertical (faster single instance)',\n",
    "        'hourly_cost': vertical_hourly_cost,\n",
    "        'avg_latency_ms': stats_vertical['avg_latency_ms'],\n",
    "        'p95_latency_ms': stats_vertical['p95_latency_ms'],\n",
    "        'throughput_rps': stats_vertical['throughput_rps'],\n",
    "        'cost_per_1000_preds': vertical_cph,\n",
    "    },\n",
    "    {\n",
    "        'scenario': f'Horizontal ({H_INSTANCES} instances)',\n",
    "        'hourly_cost': horizontal_hourly_cost,\n",
    "        'avg_latency_ms': stats_horizontal['avg_latency_ms'],\n",
    "        'p95_latency_ms': stats_horizontal['p95_latency_ms'],\n",
    "        'throughput_rps': stats_horizontal['throughput_rps'],\n",
    "        'cost_per_1000_preds': horizontal_cph,\n",
    "    },\n",
    "])\n",
    "summary_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cebe1f2",
   "metadata": {},
   "source": [
    "## Reflection\n",
    "- Based on our simulations, what are the primary advantages and disadvantages of vertical scaling versus horizontal scaling for an ML inference service?\n",
    "- How does the cost aspect influence your architectural decisions for ML platforms, especially when considering performance targets?\n",
    "- Can you think of real-world cloud services or technologies that embody these vertical and horizontal scaling concepts (for example, larger VM instances versus Kubernetes or serverless functions)?"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}