sky4me presents
The LLM Context Window
How it works, what breaks when you overflow it, and how compaction keeps agents reliable.
01 Part One
What it actually is
Everything the model sees lives in one window
- ◆ The system prompt and your standing instructions.
- ◆ Every message so far in the conversation.
- ◆ Tool definitions the model is allowed to call.
- ◆ Tool calls and the results they return.
- ◆ Retrieved documents, images, and files.
- ◆ The model's own output, reasoning tokens included.
/ Measured in tokens
The window is counted in tokens, not words
~4
characters per token in English, about three-quarters of a word
100 tokens
roughly 75 words of English
2,048 tokens
roughly 1,500 words, a long email thread
Working memory.
everything the model can hold at once/ How big it is now
Windows are large, and still finite
200K tokens
a common baseline window today (Claude Sonnet 4.5)
1M tokens
frontier default (Claude Opus, GPT-4.1, Gemini)
2M tokens
current ceiling (Gemini 1.5 Pro)
02 Part Two
Bigger didn't fix it
Four ways a full window hurts you
- ◆ Truncation: past the limit, the oldest tokens are silently dropped.
- ◆ Lost in the middle: facts buried mid-context get overlooked.
- ◆ Decay with length: quality slides well before the window is full.
- ◆ Cost and latency: every extra token is paid for on every step.
/ The evidence
Longer input, worse answers
18 models
every frontier model tested loses accuracy as input grows (Chroma)
14-85 %
accuracy lost as length grows, even with perfect retrieval
O(n²)
attention cost grows with the square of input length
Long context quietly collapses reasoning
99.3% Short context (base)
69.7% At 32K tokens
85 %
of models (11 of 13)
Below half their base score at 32K
Held their score
The real danger: it fails quietly
- ◆ No error fires. You get a confident, worse answer.
- ◆ Overflow drops your oldest tokens: instructions and safety checks included.
- ◆ Degradation starts long before the limit, not at it.
- ◆ GPT-4o above is one example: 99.3% down to 69.7% at 32K.
It fails silently.
overflow doesn't throw, it degrades 03 Part Three
Managing the window on purpose
Compaction, step by step
context 149K / 200K tokensthreshold reached, compactingsummarizing earlier conversationkept: architecture decisions, open bugs, current taskdropped: redundant tool outputs, stale messagesnew window seeded with summary, work continues
Compaction is deliberate forgetting
Kept
The high-signal state the agent still needs.
- ◆ Architectural and design decisions
- ◆ Unresolved bugs and the current task
- ◆ Key implementation details
Dropped
Tokens that no longer earn their place.
- ◆ Redundant or stale tool outputs
- ◆ Superseded intermediate messages
- ◆ Raw detail the summary now carries
Compaction is one move in context engineering
- ◆ Compaction: summarize old turns, keep the signal.
- ◆ Context editing: automatically clear stale tool results.
- ◆ Memory tool: persist notes in files outside the window.
- ◆ Just-in-time retrieval: load data only when it's needed.
/ Measured impact
What disciplined context management buys
39 %
gain from the memory tool plus context editing vs baseline
29 %
gain from context editing alone
84 %
fewer tokens used over a 100-turn evaluation
Build agents that stay sharp
Context is a resource. Engineer it.
sky4me builds AI systems that manage their context deliberately, so they stay reliable well past the first few thousand tokens.
Slide 1 of 18