KV cache size (GiB) vs context length for Llama 3.1 8B, 70B, and 405B on log-log axes, against A10G / H100 VRAM ceilings. Toggle BF16, INT4, or DeltaKV compression and drag the cursor — the panel shows whether the live KV state fits on a single GPU.
Long-context AI's real bottleneck isn't proof overhead — it's the KV cache. At 128K tokens, Llama 3.1 8B's KV state equals its own weights (15 GiB), pinned to one node per request. The formula, the GPU math, and why prefix caching demands stickiness that decentralized routing can't provide.