Inference at the Margin: The Token Budget Problem for On-Chain AI Agents

Every on-chain action has a cost floor set by gas. But AI-driven agents add a second, murkier cost: inference. Unlike gas — priced at submission time and bounded by the EVM — token costs arrive after the fact, scale non-linearly with task complexity, and vary up to 30× across identical runs of the same model on the same problem.

That 30× figure isn’t hyperbole. It comes from Bai et al.’s April 2026 paper on SWE-bench Verified, which logged every token emitted by eight frontier LLMs across ~500 software-engineering tasks. The paper’s headline finding is that agentic workflows consume 1,000× more tokens than a simple code-chat call. But the within-task variance is the engineering problem nobody talks about enough.

What the distribution actually looks like

Token consumption across runs of the same agentic task follows a lognormal distribution. This isn’t obvious from first principles, but it emerges consistently from the data: there is a hard floor (you can’t solve anything in zero tokens), a soft ceiling (context windows, task difficulty), and the multiplicative nature of reasoning chains — each step conditioned on all prior steps — that pushes the tails log-normally rather than normally.

For a lognormal with median m and natural-log standard deviation σ, the P10 and P90 token counts are:

P10 = m × exp(−1.282σ)
P90 = m × exp(+1.282σ)

With σ = 1.5 (the Bai et al. figure for multi-protocol arbitrage tasks), that ratio is exp(2.564σ) ≈ 47×. So the cheapest 10% of runs cost about one-seventh the median, and the priciest 10% cost seven times the median. Budget for the median and you’ll be underwater nearly half the time on complex tasks.

The interactive calculator below anchors four task complexity scenarios to the paper’s structural findings. Drag the profit-per-decision slider to see how the profitable (green) zone shifts.

⬢ loading artifact…

The circular oracle problem

Here’s the trap that makes this genuinely hard for on-chain agents: you need inference to decide whether inference is worth calling.

A simple threshold rule — “call the LLM only when expected profit > expected cost” — requires knowing expected cost before the call. But cost is what varies 30–50× across runs. The paper’s self-prediction correlation is 0.39, meaning the model’s own estimates of how many tokens it will use explain less than 16% of the variance in actual usage. Models also have a systematic bias: they underestimate.

This is structurally identical to the oracle problem in DeFi. You need a price feed to decide whether to execute a trade, but the feed itself is expensive to query. The difference is that an oracle’s cost is deterministic; inference cost is stochastic with a fat right tail.

The naive approach — estimate cost, compare to profit, proceed if positive — fails because it ignores variance. A 60% profitable invocation rate sounds fine until you realize the 40% loss-making calls tend to be the longest ones. The expected loss from an expensive failed run can easily exceed the expected gain from a cheap successful one.

Break-even analysis under uncertainty

Define the break-even token count b as:

b = profit_USD × 1,000,000 / costPerMk

where costPerMk is your model’s price per million tokens (inputs + outputs blended). The fraction of profitable invocations is then the lognormal CDF evaluated at b:

P(profitable) = Φ((ln(b) − μ) / σ)

with μ = ln(median tokens) and σ from empirical task variance.

Some concrete numbers. Tier 2 model (Sonnet-class, $3/Mk), swap-routing task (median 50K tokens, σ = 1.4), profit target $0.50:

b      = 0.50 × 1,000,000 / 3.0  = 166,667 tokens
μ      = ln(50,000)               = 10.82
z      = (ln(166,667) − 10.82) / 1.4
       = (12.02 − 10.82) / 1.4   = 0.86
P      = Φ(0.86)                  ≈ 80.5 %

80% profitable is workable. But raise the model tier to Opus-class ($15/Mk) without changing the profit target:

b  = 0.50 × 1,000,000 / 15.0  = 33,333 tokens
z  = (ln(33,333) − 10.82) / 1.4 = (10.41 − 10.82) / 1.4 = −0.29
P  = Φ(−0.29)  ≈ 38.6 %

38% profitable means you’re losing money on nearly two out of three calls. The model tier choice isn’t cosmetic — it shifts the entire distribution of outcomes.

Practical mitigations

Tier routing based on task complexity. The 1,000× agentic-vs-chat gap means that a task which could be answered with a single retrieval and a cheap model should never reach an Opus-class agent. Build a lightweight classifier (a Haiku-class call or even a heuristic rule over on-chain state) to route tasks before they hit the expensive agent. The classifier’s token cost is tiny; its value is preventing 15× overkill.

Caching at the session boundary. Bai et al. found that models re-read the same context — tool schemas, system prompts, previous observations — in every call of a multi-step agent loop. On a 1,200K token portfolio rebalance task, 30–40% of tokens are often boilerplate replay. Structured caching (KV-cache prefills, prompt caching) cuts median cost without touching the tail.

Adaptive max-token budgets with early termination. Rather than letting an agent run to completion, set a hard token cap per action and design your tool-calling loop to emit a partial result or a “budget exhausted” signal when the cap is hit. The economic intuition: a 90th-percentile run is already evidence the task is hard; paying another 2× beyond that is typically a sunk cost, not a path to success.

Calibrate with real traffic, not vibe estimates. The paper’s 0.39 self-prediction correlation is damning: model self-reports are barely better than guessing. The only way to parameterize μ and σ for your specific task distribution is to log actual invocations, fit a lognormal, and refresh the fit as your agent’s tool set evolves. A week of production traffic is worth more than any synthetic benchmark.

The token as economic unit

The May 2026 companion paper (Ding et al.) frames tokens as a three-layer economic primitive: production factor (the input the model consumes to produce a result), exchange medium (the basis on which you’re billed), and unit of account (the natural measure of cognitive load). This triple role is what makes inference budgeting genuinely different from other cloud compute.

Gas has a similar duality: it’s both the resource consumed and the price signal. But gas consumption is deterministic given the bytecode path. Token consumption adds a third dimension: the model’s internal reasoning trajectory, which varies probabilistically even with identical inputs.

For on-chain agents to be viable at scale, token cost must become a first-class protocol concern — not a line item in a team’s cloud budget. That probably means on-chain inference pricing oracles, budgeted agent calls with refund mechanics for under-spend, and agent specifications that declare expected token ranges so downstream contracts can gate execution on economic viability before the call is made.

Takeaways

• Token consumption in agentic workflows is lognormally distributed with σ typically 1.2–1.6, producing P90/P10 ratios of 15–50×.
• The break-even token count determines what fraction of invocations are profitable; raise it by raising profit per call or lowering model tier.
• Self-prediction correlation is 0.39: models cannot reliably estimate their own costs. Budget from empirical distributions, not model reports.
• Practical handles: tier routing by complexity, prompt caching for boilerplate context, hard token caps with partial-result signals, and continuous distribution fitting from production logs.
• Token economics is an emerging sub-discipline. The right frame is not “how expensive is this call?” but “what is the distribution of costs, and does the profitable fraction of the distribution justify the expected cost?”