Token budgets are an estimation problem
Jaya Gupta (@JayaGup10) wrote a sharp piece on where enterprise AI is heading. The core argument is that inference has stopped being an experiment and has become an operating infrastructure, and the companies that figure out token-to-outcome attribution will control where AI spending flows.
I agree with the thoughts and like where we are headed, and we are trying to understand the real cost and value of the AI in our work. That is how we mature. The token allocation and budgeting are the logical next steps in this direction.
But this entire token budget war is the most difficult battle for software engineering teams for one reason: estimation.
Jaya’s article talks about cost calculation in retrospect. But in practice, when a software engineer gets a budget of tokens, they need to know how much a task would cost in tokens, at least in parts, if not end-to-end, before they start execution.
The planning room.
Imagine a sprint planning session. The team collaboratively comes up with the list of things to work on next. Now the team has to answer the next important question of the planning session: how much time do we need? And the follow-up question: how many tokens do we need to build it?
The room is silent now, and nobody has a reasonable answer.
The engineering team has the overall budget of allocated time and tokens. But the operating cost of a specific workflow, at the task level, before the work starts, is unknown. That gap is where a lot of token budget wars will become extremely difficult.
Estimation is hard, and AI is making it even harder.
Software engineering, or any stream of engineering, is complex and has multiple moving parts. We, as an industry, know how bad we are at estimating. Countless studies prove this point.
And software teams do not (and should not, IMO) operate like assembly lines. No matter how much we try to operate in a modular, mechanical way.
We invented agile methodologies, scrum, story points, velocity tracking, and planning poker because effort is hard to predict in advance.
Now, bring token allocation into the equation. Just when we tried to make peace with it, token estimation now adds a different problem on top of the existing one. Predicting the token usage for the cost is almost impossible. Because of the design. AI systems operate in a probabilistic way.
An agent can thrash. A model can over-reason. A retrieval step can pull fifty documents when five would do. The context window inflates. Retries compound.
Types of tokens and the control we have.
There are five types of tokens: input, deliberation, output, evaluation, and maintenance.
- Input: Everything the model sees before it starts: the prompt, system instructions, retrieved context, tool definitions, conversation history, and examples. This is the only category we have meaningful control over.
- Deliberation: The model’s internal work: reasoning, planning, tool call decisions, self-correction, and backtracking. This is where variance lives, and it is largely outside our control.
- Output: The model’s response, including tool call syntax, format scaffolding, and streaming overhead. We can constrain it, but not fully predict it.
- Evaluation: A separate LLM call to verify or gate another call’s output. Most teams forget to budget for this until they are already over budget.
- Maintenance: System-level token use independent of any specific task: context compaction, session state, cache priming, and periodic re-injection of system prompts in long-running sessions.
Most teams are shielded from this right now. Subscription pricing on coding agents from frontier model companies hides the underlying token cost. You see credits consumed, or even worse, the available usage limit, not the deliberation and evaluation layers running underneath. That abstraction will not survive enterprise-scale budget scrutiny.
Amazon’s Kiro editor signalled this early. Their plan mode offered flat per-task pricing. Outcome-based, not token-based. Essentially, a vendor absorbs the estimation problem, so teams would not have to. But they revised it to usage-based pricing now. Even the companies building the tool could not control the variance.
The question nobody can answer yet.
“How much will this sprint cost in tokens?”
Jaya’s layer tells you what already happened. That is necessary. Helps us understand the cost in retrospect.
But when the budget is allocated to the engineering teams, people have to know, at least roughly, the cost to run and build before committing to the work. And we have no clear way to calculate it yet.
It is a planning problem. And nobody in the sprint room owns it yet.