Pip: Welcome to Azure Advice, where the chat window looks free and the bill disagrees.
Mara: Christoph Corder has been writing about the hidden mechanics of AI assistants — how context windows fill up, what that costs you, and whether anyone should be paying for wrong answers in the first place. Let's start with the token bill you didn't know you were running up.
Context And Memory Limits
Pip: The premise here is that most people think of AI chat the way they think of texting — just a conversation, no meter running. But there is absolutely a meter running, and it resets in the worst possible direction every single turn.
Mara: The architecture is the key thing to understand. As one post puts it: "Every turn you send re-tokenizes the entire conversation above it: system prompt, memories, style preferences, every previous message, every previous response. Turn one is cheap. Turn twenty is paying for turns one through nineteen."
Pip: So the cost doesn't grow linearly — it compounds. Every new question drags the entire history behind it like a sled full of old Kusto queries nobody needs anymore.
Mara: The numbers make that concrete. A thirty-turn kitchen-sink thread — one where you've debugged a query, asked a command, drafted an email, all in the same session — runs roughly four hundred sixty-five thousand input tokens. Break that same work into six focused five-turn threads and you spend closer to ninety thousand. Same output, one-fifth the cost.
Pip: One-fifth. That is not a rounding error across a team and a year.
Mara: And there's a quality cost before the money cost even registers. The post makes this point clearly: irrelevant earlier context still influences the response. A forty-turn thread that has drifted across four topics produces worse answers than a clean, tightly framed chat — because the model is trying to serve all of it at once.
Mara: The companion post, "Why Your AI Assistant Acts Like a Goldfish," explains why this happens structurally. The model is stateless. There is no ongoing session in its memory. Each turn is, as the post puts it, "a cold read by a stranger who happens to have your entire file open." The continuity you feel is the app doing bookkeeping, not the model remembering you.
Pip: Which reframes the whole thing. You're not having a conversation. You're handing someone a growing stack of paper before every question.
Mara: That post also surfaces something most users get wrong about what actually fills the context window. Chat history is often the smallest item. System prompts, tool schemas, MCP server descriptions, custom instructions you set once and forgot — those frequently outweigh the conversation itself. Your question is, in the post's phrase, "footnotes in your own conversation."
Pip: Right — and the fix isn't exotic. Fresh chat per task. Upload files instead of pasting them, because a pasted twenty-thousand-token file dragged through twenty follow-up turns costs four hundred thousand input tokens for a file that hasn't changed since turn one.
Mara: And turn extended thinking and web search off by default. Ten thousand reasoning tokens per response across twenty replies is two hundred thousand reasoning tokens billed at output rates — the expensive kind. One web search that pulls three sources adds roughly fifteen thousand tokens to every subsequent turn until you close the thread.
Pip: The discipline underneath all three habits is the same: don't carry weight the current turn doesn't need.
Mara: Which is also, it turns out, the discipline that affects whether the answers are any good — not just whether the bill is reasonable. And that connects directly to whether you can trust what you're getting back.
Hallucinations And User Trust
Pip: So the context window explains the cost structure. But there's a separate problem: what happens when what comes out of that context window is wrong, and you paid for it anyway.
Mara: "Congratulations, You Paid Twice" opens on exactly that scenario: "You ask a question. The model answers with total confidence. You point out it's wrong. It agrees enthusiastically. Then it generates the corrected answer. Congratulations. You just paid twice. Once for the mistake, once for the apology."
Pip: The post doesn't stop at outrage, though — it actually traces why refunds are structurally hard to offer.
Mara: The core argument is what it calls the adjudication problem. Outcome-based pricing already exists for support tickets and booked meetings — binary, timestamped, cheap to verify. It breaks down anywhere the question is "was this answer correct," because the moment money attaches to correctness, the definition of correct gets contested. The proposed fix is claims that carry their own evidence: grounded facts, labeled inference, labeled guesses — so pricing can follow provability rather than bolting a warranty onto a system that can't say what it's warranting.
Pip: The vendor that can actually offer that guarantee won't need to lean on it. The guarantee is the proof the verification layer exists.
Mara: Exactly where the market is heading next.
Pip: Context costs, trust costs, and the bill compounds quietly in both directions.
Mara: Trim the stack, start fresh chats, and watch what you're actually paying for. More from Azure Advice next time.
Azure Advice 