{ "nbformat": 4, "nbformat_minor": 5, "metadata": { "colab": { "name": "The Infosiast \u2014 Local LLM VRAM, KV-Cache & Quantization Benchmark (2026)", "provenance": [], "gpuType": "T4" }, "accelerator": "GPU", "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python", "version": "3.11.0" } }, "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# The Infosiast \u2014 Local LLM VRAM, KV-Cache & Quantization Benchmark\n", "**Prepared by [Kshitij Gupta](https://www.linkedin.com/in/kshitij-gupta-855535134/) for [The Infosiast](https://theinfosiast.com/)**\n", "\n", "This reproducible Google Colab notebook accompanies:\n", "* [Running Local LLMs on Consumer Hardware in 2026](https://theinfosiast.com/running-local-llms-consumer-hardware-2026/)\n", "* [DeepSeek R1 Architecture & Local Deployment Guide](https://theinfosiast.com/deepseek-r1-architecture-deployment-guide/)\n", "\n", "Run all cells on a Google Colab **T4 / L4 GPU** (using your 200 monthly Google AI Pro CCUs) to calculate exact VRAM footprints across `FP16`, `Q8_0`, `Q5_K_M`, and `Q4_K_M`, measure MLA vs standard GQA KV-cache scaling, and benchmark live matrix-multiplication memory bandwidth.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step 1: Deterministic VRAM Footprint Calculator (Weights + KV Cache + Runtime Buffer)\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "\n", "def estimate_vram_gb(params_b: float, bits_per_weight: float, ctx_tokens: int, kv_bits: int = 8, mla_compression: float = 1.0, buffer_gb: float = 0.65) -> dict:\n", " weight_gb = (params_b * 1e9 * (bits_per_weight / 8.0)) / (1024**3)\n", " kv_gb = ((ctx_tokens * 0.00012 * (params_b / 14.0)) * (kv_bits / 16.0)) * mla_compression\n", " total_gb = round(weight_gb + kv_gb + buffer_gb, 2)\n", " return {\n", " 'Model Size (B)': params_b,\n", " 'QuantizationBits': bits_per_weight,\n", " 'Context Window': ctx_tokens,\n", " 'Weights (GB)': round(weight_gb, 2),\n", " 'KV Cache (GB)': round(kv_gb, 2),\n", " 'Total VRAM (GB)': total_gb,\n", " 'Fits 12GB GPU': 'YES' if total_gb <= 11.5 else 'NO',\n", " 'Fits 16GB GPU': 'YES' if total_gb <= 15.5 else 'NO',\n", " 'Fits 24GB GPU': 'YES' if total_gb <= 23.5 else 'NO',\n", " }\n", "\n", "scenarios = [\n", " estimate_vram_gb(8.0, 4.5, 8192, kv_bits=8),\n", " estimate_vram_gb(14.0, 4.5, 8192, kv_bits=8),\n", " estimate_vram_gb(14.0, 5.5, 16384, kv_bits=8),\n", " estimate_vram_gb(32.0, 4.5, 16384, kv_bits=8),\n", " estimate_vram_gb(32.0, 4.5, 32768, kv_bits=8, mla_compression=0.25),\n", " estimate_vram_gb(70.0, 4.5, 16384, kv_bits=8),\n", "]\n", "df = pd.DataFrame(scenarios)\n", "print(df.to_string(index=False))\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Step 2: Live GPU / CPU Memory-Bandwidth & Token Throughput Probe\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import time\n", "import torch\n", "\n", "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n", "print(f'Active Compute Device: {device.upper()}')\n", "if device == 'cuda':\n", " print('GPU Device Name:', torch.cuda.get_device_name(0))\n", "\n", "dim = 4096\n", "dtype = torch.float16 if device == 'cuda' else torch.float32\n", "w = torch.randn((dim, dim), device=device, dtype=dtype)\n", "x = torch.randn((1, dim), device=device, dtype=dtype)\n", "\n", "if device == 'cuda':\n", " torch.cuda.synchronize()\n", "t0 = time.perf_counter()\n", "iters = 500\n", "for _ in range(iters):\n", " y = torch.matmul(x, w)\n", "if device == 'cuda':\n", " torch.cuda.synchronize()\n", "elapsed = time.perf_counter() - t0\n", "bytes_moved = iters * dim * dim * (2 if dtype == torch.float16 else 4)\n", "gbps = (bytes_moved / elapsed) / (1024**3)\n", "print(f'Measured Effective Matrix Read Bandwidth: {gbps:.2f} GB/s ({elapsed*1000:.2f} ms for {iters} passes)')\n" ] } ] }