Token Cost Isn’t a Prompting Problem, It’s an Architecture Problem

The Number Everyone Watches (& The One They Don’t)

Ask ten people building agents right now how to control token cost, and nine of them will give you the same answer: use a cheaper model, or write a shorter prompt.

Both of those move the needle. Neither is the lever that actually matters, and neither one fixes anything if the underlying architecture is charging the model for work it never needed to do.

Here’s what nobody tells you when you’re staring at a rising token bill: not every step of an agent’s turn should touch the model. Some of it shouldn’t cost a single token, not because it’s free, but because it never needed to reach the LLM in the first place.

That’s what AI agent context management actually means in practice: deciding what does and doesn’t reach the model, before a single prompt gets written.

What Actually Costs Tokens (& What Doesn’t)

Break down a single agent turn and you’ll find two categories of work happening.

The model spends tokens deciding which tool to call, constructing the arguments for that call, and reasoning over whatever comes back. That’s the entire list of things that should cost tokens.

Everything else, the actual API request, the script running on a box somewhere, the round trip to a device, should happen outside the model entirely. No LLM turn should occur during execution.

Should is doing a lot of work in that sentence, because plenty of agent stacks don’t actually draw that line. If your framework routes tool execution through the same reasoning loop as everything else, and a lot of DIY stacks do exactly that, you’re paying model tokens for work that has nothing to do with reasoning. It’s not a bug you can prompt your way around. It’s a structural choice somebody made when they built the framework.

This is precisely why FlowAI’s execution model is deterministic by design. Tool execution runs through Itential Gateway and the platform’s governed execution engine, not through the LLM. Tokens fund the thinking. They do not fund the doing. That’s not a policy on top of the architecture. It is the architecture.

Carry this into every agent build, on any platform: reasoning costs tokens, execution shouldn’t. If your framework can’t tell you which is which, that’s the first thing to fix, before you touch a single prompt.

The Four Silent Token Taxes

Once you separate reasoning from execution, the actual sources of token bloat get a lot easier to spot. In practice, it’s almost always one of four things, and all four are decisions a platform makes for you or leaves you to solve yourself.

Bad Tool Schemas That Trigger Retries

An ambiguous or malformed schema doesn’t fail quietly. It makes the model guess, get it wrong, and try again, and every retry re-pays the full context cost of that turn. FlowAI’s built-in explicit tool definition mechanism exists specifically to close this gap. You don’t re-describe tools or payload structure in the prompt to get structure the schema should already provide. Many frameworks leave that entirely to you.

Over-Specified Prompts

Spelling out every tool, every payload shape, and every edge case in the prompt feels safe. It isn’t free. That detail gets paid for on every single turn, not once. On FlowAI, when explicit callouts really are necessary, decorators tighten a tool’s schema at the source instead of forcing you into verbose manual prompting. That’s the difference between fixing the problem once and repeating the fix on every call.

Duplicated Tool Descriptions

If a tool’s schema already carries a description of what it does and how to call it, restating that in the prompt doubles the cost for zero benefit. The schema gets sent once. The prompt gets sent every turn. Let the schema do its job and build on a platform where the schema is actually built to carry that weight.

Unfiltered Results Flowing Back into Context

If a tool’s schema already carries a description of what it does and how to call it, restating that in the prompt doubles the cost for zero benefit. The schema gets sent once. The prompt gets sent every turn. Let the schema do its job and build on a platform where the schema is actually built to carry that weight.

None of these four are exotic. They’re also, in my experience, exactly the four things nobody checks until the bill shows up, and exactly the four things a well-built platform should have already solved for you.

Design for What Comes Back, Not Just What Goes Out

Most teams building agents spend their design energy on the prompt: what to tell the model, how to phrase the instructions, how explicit to be. That’s the smaller half of the problem, and it’s the half every framework lets you obsess over while the bigger lever sits untouched.

The bigger lever is designing what comes back. A tool or a sub-agent doesn’t need to hand the calling agent its entire output. It needs to hand back enough to confirm success and enough to look up more detail later if something goes wrong: a status, an identifier, a short summary. The full transcript, the full payload, a raw and parsed copy of the same result, none of that needs to re-enter context just because it was available. The sub-agent’s own execution still costs tokens on its own side. What’s saved is the parent’s context budget, not the total cost of the work performed.

This matters more as agent chains get longer. Every additional agent in a chain is another hop the orchestrator has to absorb the result of. A well-scoped specialist sub-agent, one that internally handles a complex integration and returns only a lean result, is a net token saver, not just added overhead, because the orchestrator never has to load that tool’s full schema or context. The cost stays contained inside the specialist. A chain where every hop dumps its full output back up the line gets expensive fast, and in most DIY stacks, keeping that contained is something your team has to build and maintain by hand.

Targeted retrieval works the same way. Query for exactly what’s needed instead of pulling an entire document or dataset into context. Fetching ten times more data than necessary costs roughly ten times the tokens for that step.

That’s not a rounding error. That’s the multiplier, and it’s the same discipline whether you’re querying a knowledge base or a network device through Itential Gateway.

The Lever Most Frameworks Don’t Have Yet

There’s a capability starting to show up across agent platforms that’s worth knowing about even before it’s fully mature: filtering a tool’s raw response before it ever reaches the model, instead of dumping the whole payload into context and hoping the prompt tells the model to ignore most of it.

Today, most tool calls return everything, whether the agent needs five percent of that payload or all of it. Without a native way to filter that response, teams end up wrapping tool calls in custom scripts just to pre-filter data before it reaches the agent, which quietly gives up the point-and-click value of a native integration in exchange for glue code somebody now owns.

This is exactly the gap tool execution transforms are built to close on FlowAI: giving builders control over what enters the context window for reasoning, instead of paying for the full payload on every call.

It’s not a prompting technique. It’s an architectural one, and it’s the kind of capability that separates a platform built for this problem from a framework that’s had it bolted on after the fact.

A Short Checklist Before You Estimate Cost

Before scoping token cost for any agent build, on any platform, four questions get you most of the way to an honest number.

• How many agents are in the chain, and could any of them be collapsed or reshaped?
• How verbose are the tool schemas and prompts actually in use, and is that verbosity something you’re maintaining by hand?
• Is retrieval targeted, or is it pulling more than the step needs?
• Is anything filtering or trimming a tool’s raw output before it re-enters context, or is the model seeing everything because nothing in the stack does that for you?

Walk through those four and a rough guess turns into an estimate you can actually defend, and it usually becomes obvious pretty fast how much of your token bill is really a build-and-maintain bill in disguise.

One more thing worth saying plainly, because it gets missed: fewer tokens isn’t automatically a better agent. A cheaper model can burn roughly the same token count per turn as a stronger one and still produce a worse result. If you’re comparing cost across models or architectures, weigh what the agent actually accomplishes alongside what it costs to get there.

An agent that’s cheap and wrong isn’t a cost win, and neither is a platform that’s cheap because it quietly shifted the real cost onto your engineering team.

The Real Optimization Target

Token cost isn’t a single dial. It’s the sum of a few specific architecture decisions: whether tool schemas carry their own weight, whether sub-agents return lean results by default, whether retrieval is targeted, and whether anything filters a payload before it hits the model.

Get those right at the platform level and the token bill takes care of itself. That’s the real choice underneath every token conversation: keep patching a framework that wasn’t built to make this distinction, or build on one that was. FlowAI was built on the second answer, on purpose, from the start.