EB
erabot.ai

The Hidden Cost of Long Context Windows: Why Bigger Isn't Always Better

"200K context window!" When Claude announced it, developers celebrated. When OpenAI pushed GPT-4o to 128K tokens, the consensus was clear: more context is always better.

Except it is not. Not when every token you send costs money.

The dirty secret of long context windows is that they quietly encourage a pattern we call context stuffing -- sending far more information than the model needs simply because you can. And most developers do not realize they are doing it until the invoice arrives.

Context Window Size vs. Cost

Here is what the major models charge for input tokens, alongside their context window sizes:

| Model | Context Window | Input Cost (per 1M tokens) | Output Cost (per 1M tokens) | |-------|---------------|---------------------------|----------------------------| | Claude 3.5 Sonnet | 200K | $3.00 | $15.00 | | Claude 3 Haiku | 200K | $0.25 | $1.25 | | GPT-4o | 128K | $2.50 | $10.00 | | GPT-4o-mini | 128K | $0.15 | $0.60 | | Gemini 2.0 Flash | 1M | $0.10 | $0.40 |

The key insight: just because you can send 200K tokens does not mean you should. A 200K-token request to Claude 3.5 Sonnet costs $0.60 in input tokens alone. Do that 1,000 times a day and you are spending $18,000 a month -- just on input.

The model's context window is a maximum, not a target. Treating it as a bucket to fill is the most expensive mistake in production AI applications.

The Stuffing Problem

Context stuffing takes three common forms. Each one silently inflates your bill.

1. Sending Entire Files When You Need a Function

You have a 2,000-line service file. You need the model to review one 40-line function. But it is easier to send the whole file, so you do.

# Context stuffing: sending 15,000 tokens for a 300-token task
with open("backend/services/billing.py") as f:
    entire_file = f.read()  # 2,000 lines = ~15,000 tokens

response = client.messages.create(
    model="claude-3-5-sonnet-latest",
    max_tokens=1024,
    messages=[{
        "role": "user",
        "content": f"Review this function for bugs:\n\n{entire_file}"
    }]
)

# Targeted extraction: 300 tokens
import ast

def extract_function(filepath: str, name: str) -> str:
    source = open(filepath).read()
    tree = ast.parse(source)
    for node in ast.walk(tree):
        if isinstance(node, ast.FunctionDef) and node.name == name:
            return ast.get_source_segment(source, node)
    return ""

function_code = extract_function(
    "backend/services/billing.py", "calculate_invoice"
)

response = client.messages.create(
    model="claude-3-5-sonnet-latest",
    max_tokens=1024,
    messages=[{
        "role": "user",
        "content": f"Review this function for bugs:\n\n{function_code}"
    }]
)

That is a 98% reduction in input tokens. Same question, same quality answer, 50x cheaper.

2. Growing Conversation History Without Limits

Every message in a conversation adds to the context window. By turn 20, your conversation history might be 15,000+ tokens -- and it grows linearly with every exchange.

// Conversation history grows unbounded
const messages: ChatMessage[] = [];

async function chat(userMessage: string) {
  messages.push({ role: "user", content: userMessage });

  // By message 20, this array might contain 15,000+ tokens
  // of history, most of which is no longer relevant
  const response = await openai.chat.completions.create({
    model: "gpt-4o",
    messages: messages,
  });

  messages.push({
    role: "assistant",
    content: response.choices[0].message.content!,
  });

  return response.choices[0].message.content;
}

// Sliding window: keep only the last N messages
const MAX_HISTORY = 6; // 3 user + 3 assistant messages

async function chat(userMessage: string) {
  messages.push({ role: "user", content: userMessage });

  // Keep system prompt + last N messages
  const windowedMessages = [
    messages[0], // system prompt (always keep)
    ...messages.slice(-MAX_HISTORY),
  ];

  const response = await openai.chat.completions.create({
    model: "gpt-4o",
    messages: windowedMessages,
  });

  messages.push({
    role: "assistant",
    content: response.choices[0].message.content!,
  });

  return response.choices[0].message.content;
}

A sliding window caps your per-request cost regardless of conversation length. For long-running conversations, this can reduce costs by 70-80%.

3. RAG Pipelines That Retrieve Too Much

Retrieval-Augmented Generation is the most common source of context stuffing. Your vector search returns 20 chunks "just in case," but most of them add noise rather than signal.

# Over-retrieval: 20 chunks x 500 tokens = 10,000 tokens of context
results = vectorstore.similarity_search(query=question, k=20)
context = "\n\n".join([doc.page_content for doc in results])

# Smart retrieval: filter by relevance, cap at useful amount
results = vectorstore.similarity_search_with_score(
    query=question, k=10
)

# Keep only high-relevance results
relevant = [
    doc for doc, score in results
    if score > 0.80
][:5]  # 5 chunks x 500 tokens = 2,500 tokens

context = "\n\n".join([doc.page_content for doc in relevant])

The difference between 10,000 and 2,500 tokens of context might seem small, but multiply it by thousands of daily requests and it becomes the dominant cost in your pipeline.

The Real Cost: A Worked Example

Let us walk through a concrete scenario. You are building a code assistant that helps developers understand and refactor their codebase. It uses Claude 3.5 Sonnet with RAG.

Current architecture (context-stuffed):

  • System prompt: 1,500 tokens
  • Conversation history (average): 12,000 tokens
  • RAG context: 15,000 tokens (20 chunks)
  • Current code file: 8,000 tokens (full file)
  • User question: 200 tokens
  • Total input per request: 36,700 tokens

Your usage:

  • 10,000 requests per day
  • 30 days per month

Monthly input cost: 10,000 x 30 x 36,700 / 1,000,000 x $3.00 = $33,030/month

Now let us apply smart context management:

Optimized architecture:

  • System prompt: 400 tokens (trimmed)
  • Conversation history: 3,000 tokens (sliding window of 6 messages)
  • RAG context: 3,500 tokens (5 filtered chunks + reranking)
  • Targeted code: 800 tokens (extracted function, not full file)
  • User question: 200 tokens
  • Total input per request: 7,900 tokens

Optimized monthly cost: 10,000 x 30 x 7,900 / 1,000,000 x $3.00 = $7,110/month

Annual savings: $311,040. From one set of optimizations. No model downgrade. No feature removal. Same user experience.

Smart Context Management Patterns

Here are four patterns that keep context lean without sacrificing quality:

1. Sliding Window for Conversations. Keep the system prompt plus the last N message pairs. For most applications, 3-5 exchange pairs (6-10 messages) capture enough conversational context. Older messages rarely affect the current response.

2. Targeted Code Extraction. Use AST parsing to extract only the functions, classes, or blocks relevant to the user's question. Never send an entire file when a specific scope is sufficient.

3. Hierarchical Summarization. For long documents or extensive conversation history, summarize older content into a compact representation. A 10,000-token conversation history can be summarized into 500 tokens that capture the key decisions and context.

4. Retrieval Reranking. After your initial vector search, apply a lightweight reranker (cross-encoder or even a fast LLM call) to score chunk relevance. Send only the top-scoring results to your main model. The reranking call costs a fraction of what the wasted context would cost.

How to Audit Your Context Usage

You could manually trace every API call in your codebase, log the token counts, and calculate costs. But there is a faster way.

erabot.ai scans your codebase and identifies exactly where context stuffing is happening. It traces the token flow from data source to API call, flags oversized context windows, and generates code diffs that implement the optimizations described above.

The scan takes under five minutes. The report shows you every context-stuffed API call with its current cost, the optimized cost after applying the suggested fix, and the code diff to get there.

Start Optimizing

Your context window is not a bucket to fill. It is a budget to manage.

Every unnecessary token you send is money spent on information the model ignores. The models are powerful enough to work with focused, relevant context -- you do not need to give them everything just because you can.

Scan your codebase free at erabot.ai and see exactly how much context you are wasting.