Decode speed of a frontier MoE model versus interconnect bandwidth. Each layer's expert routing is an all-to-all shuffle bounded by the slowest link. Drag from NVLink down to home broadband: the line crosses the interactive floor at the datacenter edge, then falls to seconds per token.
Frontier open models are now Mixture-of-Experts — cheap to serve in a datacenter, nearly impossible across the internet. The all-to-all expert shuffle moves ~9.5 MiB/token and caps decode at ~67 tok/s even on 400G InfiniBand; on internet links it falls to seconds per token.