The Memory Wall: KV Cache and Why Long-Context AI Can't Decentralize

The Llama 3.1 8B model weighs roughly 15 GiB in BF16. Serve it at its maximum context of 128K tokens and its KV cache weighs the same. Every request at maximum context carries two models’ worth of data — and unlike the weights, that memory belongs to exactly one user and must stay on exactly one node.

Most conversation about what makes decentralized AI hard focuses on verification: how do you prove the right model ran? But there is a subtler and more immediate problem. Before any proof is generated, before any consensus is needed, the serving system has to fit the request into memory and keep it alive for the duration of the generation. At long context lengths, that constraint alone rules out the routing patterns decentralized networks depend on.

What the KV cache is and why you can’t avoid it

A transformer generates tokens one at a time, autoregressively. To produce token N+1 it attends over all N previous tokens: every layer computes queries against every prior key, and value vectors are summed by the attention scores. If you re-computed those keys and values from scratch on every step, inference would be O(N²) per token — unusable at long context.

The KV cache stores the computed key and value vectors for every past token at every layer, so each new token only needs one forward pass through the network. The size of that cache is exact:

bytes = 2 × L × H_kv × D_head × N × elem_bytes

where L is the number of layers, H_kv is the number of key-value heads (with GQA, fewer than the query heads), D_head is the head dimension, N is the context length, and elem_bytes is 2 for BF16, 1 for INT8, 0.5 for INT4. The 2 at the front accounts for separate key and value tensors.

For Llama 3.1 — the 8B model has 32 layers, 8 KV heads, and a head dimension of 128 — the per-token cost in BF16 is:

2 × 32 × 8 × 128 × 2 bytes = 131,072 bytes = 128 KiB per token

That number is fixed by the architecture. Every additional token costs exactly 128 KiB of VRAM, regardless of what those tokens say.

The crossover

At 128K tokens, the Llama 3.1 8B KV cache reaches:

128 KiB/token × 131,072 tokens = 16 GiB

The model weights in BF16 are about 15 GiB. At maximum context, the KV cache equals the model itself. This is not a coincidence — Meta designed Llama 3.1’s architecture knowing the KV budget, and 128K is close to the crossover point given the target VRAM envelopes.

The 70B model has 80 layers and the same KV head configuration, so its per-token cost is 80/32 × the 8B cost = 320 KiB per token in BF16. Its crossover — where KV cache equals model weights (~131 GiB) — happens around 448K tokens, well beyond its current 128K limit. The 405B model, with 126 layers, crosses over around 1.5M tokens.

⬢ loading artifact…

The chart makes the structural problem visible. Drag the cursor to 128K tokens and look at the 8B model: its KV column reaches the weight line. Slide to longer contexts — available with sliding-window and other extension methods — and the curves march upward past every GPU’s VRAM ceiling. The colored horizontal lines (A10G at 24 GiB, A100 40G, H100 80G) show how quickly a single request fills whatever card it lands on.

Why batching collapses at long context

The existing article The Batch You Can’t Fill covered the throughput case: a single-tenant node running one request at a time wastes GPU parallelism and drives cost per token up by 100x compared to a saturated batch. The KV cache adds a capacity version of the same problem.

On an H100 with 80 GiB of VRAM:

• Llama 3.1 70B in 4-bit quantization occupies roughly 35 GiB.
• That leaves 45 GiB for KV caches.
• At 128K context, each 70B request needs 40 GiB of KV — so exactly one long-context request fits, and only barely.

Effective batch size at maximum context: 1. You’ve paid for a high-memory GPU, loaded a large model, and can serve exactly one user at a time before VRAM is exhausted. On a centralized provider, the solution is simple: route short-context requests to one pool, long-context to another, and pack accordingly. On a decentralized network where requests arrive at random nodes with random sizes, there is no safe packing policy.

Prefix caching: the fix that requires stickiness

The dominant optimization for long-context serving is prefix caching: if two requests share a common prefix — the same system prompt, the same document, the same agent instructions — the KV cache for that prefix can be computed once and reused. vLLM and SGLang both implement prefix caching via a radix-tree structure that stores and retrieves KV tensors keyed by their token hash.

The savings can be enormous. Enterprise deployments where thousands of users send the same 4K-token system prompt followed by their own messages reuse the system-prompt KV on every request. Instead of 4K × 128 KiB = 512 MiB computed per request, that block is computed once and served from cache. KVFlow (arXiv:2507.07400) showed up to 1.83× speedup from prefix-cache optimization for single agentic workflows, and 2.19× for concurrent multi-agent workloads — just from managing which KV tensors to keep in memory versus evict.

But prefix caching has a hard requirement: the same node must serve all requests that share a prefix. The KV tensors live in that node’s VRAM. If a request is routed to a different node — one that hasn’t computed or cached the prefix — the cache is cold, and the full prefix must be recomputed, defeating the optimization.

Centralized providers route users with sticky sessions: once you start a conversation, you stay on the same server. Decentralized inference networks, by design, distribute load across heterogeneous nodes and cannot guarantee that consecutive requests from the same user land on the same machine.

How large is the KV transfer problem?

Could a decentralized network just pass the KV cache between nodes? The numbers make this brutal.

For a Llama 3.1 70B request at 128K context in BF16, the KV cache is ~40 GiB. A 10 Gbps link (typical enterprise, fast for most DePIN setups) transfers 1.25 GB/s. Moving that KV cache takes:

40 GiB / 1.25 GB/s ≈ 34 seconds

That is the transfer overhead to change the serving node for a single active 70B conversation at 128K. At INT4 quantization (4× reduction), it’s still ~8.5 seconds for a 70B model. No interactive application tolerates an 8-second routing pause.

The only link speed where KV transfer becomes practical is NVLink (400–600 GB/s), which is how multi-GPU inference works within a single chassis. Across machines, even over 100 GbE InfiniBand (~12 GB/s), moving a 40 GiB blob takes over 3 seconds. Decentralized networks connecting commodity GPUs over the open internet see 0.1–1 Gbps, making cross-node KV transfer architecturally infeasible for long-context workloads.

Compression: how much it helps

Recent work has produced impressive KV cache compression ratios while preserving most accuracy:

• DeltaKV (arXiv:2602.08005, Feb 2026): residual-based compression that reduces KV to 29% of original size (~3.4×) while maintaining near-lossless accuracy on LongBench and SCBench. It also pairs with a custom inference engine (Sparse-vLLM) that achieves 2× throughput improvement over vLLM by managing sparse KV layouts in hardware.
• ARKV (arXiv:2603.08727, Feb 2026): adaptive per-layer, per-token precision allocation across Original/Quantized/Evicted states. Tested on LLaMA3 and Qwen3, it achieves 4× memory reduction while preserving ~97% of baseline accuracy on long-context benchmarks.
• INT4 quantization: applying INT4 to the KV cache gives 4× reduction with typically 1–3% degradation on standard benchmarks, the same as quantizing model weights.

Toggle between these in the chart above to see how the curves shift. At INT4, the 8B crossover moves from 128K to ~512K; at DeltaKV (3.4×), to ~435K. The model weight lines stay fixed — compression only moves the KV curves, not the weight floor.

Compression helps, but it doesn’t solve the stickiness problem. A compressed KV cache on node A is still on node A. Moving it to node B still incurs the full transfer overhead. And even a 4× compressed 70B cache at 128K context is ~10 GiB — still 8+ seconds over a 1 Gbps link.

Agents make the problem worse

Agentic workflows compound the KV cache problem in a specific way: agents accumulate context over many turns. An agent that starts a task, calls tools, receives results, and continues reasoning accumulates context fast. KVFlow’s analysis of agentic serving found that agents invoke the same base model repeatedly with prompts that share a long prefix — the task description, the tool definitions, the conversation history up to the current step. The research showed that LRU eviction of prefix caches in standard schedulers discards entries just before they are reused, triggering expensive recomputes. A workflow-aware eviction policy (their “steps-to-execution” heuristic) captures the reuse and delivers up to 1.83× speedup.

The deeper finding is architectural: agents need session stickiness even more than single-turn users do. Their accumulated context is the KV cache, and evicting or migrating it breaks the continuity of the task. On a decentralized network that doesn’t know the agent’s execution plan, there is no workflow-aware eviction — every node manages its own VRAM independently, and agents pay the recompute tax on every routing boundary.

The structural tension

There is a clean formal argument. The requirements of long-context serving pull in the opposite direction from the requirements of decentralized routing:

These are not engineering details to optimize around; they are structural contradictions. Centralized providers resolve them by taking the left column and building architecture around it. Decentralized networks are premised on the right column.

This doesn’t mean long-context AI can never work on a decentralized network. But it means the current generation of decentralized inference marketplaces — which treat serving as stateless token-production tasks assigned to available nodes — cannot match centralized providers at long context without a fundamentally different design. Proposals include confidential prefix caching via TEEs (the cache stays on one attested node, and other users’ requests route to it without the node seeing plaintext), or explicit KV-as-a-service layers where cached prefix tensors are sold as priced assets between nodes. Neither exists at production scale.

Takeaways

• KV cache grows at a fixed bytes-per-token rate determined by the model architecture alone. For Llama 3.1 8B in BF16, that rate is 128 KiB/token. At 128K tokens it equals the model’s weight — a genuine crossover.
• At long context, VRAM capacity forces batch size to 1 or near it, eliminating the throughput advantage of large GPUs. The batch tax and the KV wall compound.
• Prefix caching is the main optimization for long-context workloads (1.8–2.2× speedup at scale), but it requires request stickiness that decentralized routing cannot guarantee.
• Cross-node KV transfer at 40 GiB costs 30+ seconds on typical DePIN links — architecturally infeasible for interactive use.
• State-of-the-art compression (DeltaKV 3.4×, ARKV 4×) moves the crossover point but doesn’t fix the stickiness problem.
• The constraint gets harder as context windows grow. 1M-token models are in active development. Every doubling of context length doubles the per-request KV budget, tightening the vise further.

The verification problem for on-chain AI gets most of the ink. The memory problem is already here, already limiting what decentralized networks can offer, and it scales with exactly the direction the field is moving.