What Is a Context Window and Why LLMs Forget

TL;DR

LLMs forget mid-conversation because every message, system prompt, document, and line of code is stored as tokens in a fixed-size short-term memory called the context window. Once that limit fills, the earliest information drops out and the model hallucinates or slows down. Raising the limit works, but on local models it consumes GPU VRAM and degrades speed.

Transcript

Sometimes when you're talking to an LLM like chat, GBT, it gets kind of dumb, right? You'll be deep into a conversation that you can't even scroll to the top of because it's so long and it starts to say weird things and hallucinate. It forgets what you're talking about. It makes stuff up and it's stinking slow. Why is this happening? Context window... Read More

Key Insights

• A context window is the maximum number of tokens an LLM can pay attention to at any one time, functioning as its short-term memory. Once a conversation exceeds that limit, the earliest information is lost and the model starts forgetting and making things up.
• Tokens are how an AI counts the text you give it, and they do not map cleanly to words. One test sentence of 133 characters and 26 words registered as 38 tokens in LM Studio but only 34 tokens in OpenAI's tokenizer.
• Token boundaries vary by model and by piece of text. A token might be an entire word, a space plus a word, or a single comma, which is why two different systems count the same sentence differently.
• In an LM Studio demo, a Gemma 3 4B model set to a 2048-token context forgot a book mentioned at the start of the chat after story generation pushed usage to 118.4 percent, or roughly 2,400 tokens.
• Raising the context length from 2048 to 4,096 tokens restored the forgotten information: after ejecting and reloading the same model, it correctly recalled the book from the beginning of the conversation.
• Conversation text is not the only thing consuming context. System prompts you never see, pasted PDFs and Excel spreadsheets, and code during vibe coding all take up tokens and fill the window.
• Advertised context windows are not always usable locally. Gemma 3 4B supports up to 131,000 tokens, listed officially as 128,000, but loading a 120,000-token context on a 4090 with 24GB of VRAM consumed all video RAM and made responses very slow.
• The research paper Lost in the Middle showed a U-shaped accuracy curve: models were more accurate with information at the beginning and the end of a long context, with a large drop-off for information in the middle.

Questions & Answers

Q: What is a context window in an LLM?

A context window is the maximum number of tokens a model can pay attention to at any one time, effectively its short-term memory. Everything in a conversation counts toward it: your messages, the model's replies, system prompts, pasted documents, and code. Once the total exceeds the window, the model can no longer hold the earliest parts of the conversation, which is why it forgets what you told it at the start, begins hallucinating, and slows down. The video compares it to a long conversation over coffee where, after several hours, both people forget the details and sometimes the point of the discussion entirely.

Q: Why does ChatGPT get dumber the longer you talk to it?

The longer a conversation runs, the more tokens accumulate in the model's short-term memory. Each thing you say and each thing the model says back has to be stored, and that memory has a hard limit set by the context window. Once the conversation pushes past that limit, the earliest information falls out of scope, so the model forgets what you were discussing, makes things up, and becomes slow to respond. Larger context windows help, but they do not eliminate the problem, because models also have trouble paying attention across very long contexts even when everything technically fits.

Q: What is a token and how is it different from a word?

Tokens are how an AI counts the text you give it, and they do not correspond one-to-one with words. In the video, a sentence measuring 133 characters and 26 words registered as 38 tokens inside LM Studio, while pasting the same text into OpenAI's tokenizer produced only 34 tokens. A single token might be an entire word, a space plus a word, or just one comma. Because not every LLM calculates tokens the same way, the same text can consume different amounts of context depending on which model is processing it.

Q: How can I see how many tokens my text uses?

Two methods are shown in the video. Inside LM Studio, the interface displays an input token count for whatever you type, plus a usage indicator showing how full the context window is as a percentage, and clicking that indicator reveals the exact token count. Separately, you can paste text into OpenAI's tokenizer website, linked in the video description, which breaks the text down and shows how it is split into individual tokens. Comparing both is instructive, since the same sentence came out as 38 tokens in one and 34 in the other.

Q: What fills up a context window besides my messages?

Several things consume tokens beyond the obvious back-and-forth of the conversation. System prompts, which are instructions given to the LLM, count toward the window, and these may be ones you explicitly provide or ones included by default that you never see. Documents also add up: pasting a PDF or an Excel spreadsheet takes more tokens. Code is another major consumer, so vibe coding sessions steadily fill the context window as more code enters the conversation. All of this competes for the same fixed token budget as your actual questions.

Q: How do I increase the context window on a local model in LM Studio?

In LM Studio you select the model, adjust the context length setting, and load the model. The video demonstrates this with Gemma 3 4B, first set to 2048 tokens, where the model forgot a book mentioned at the start of the chat. Ejecting the model, changing the context to 4,096 tokens, and reloading restored the model's ability to recall that book correctly. LM Studio will warn you when you set a high value, noting that a large context length can significantly impact your memory usage, because the context has to fit in your GPU's VRAM.

Q: Why can't I use a model's full advertised context window locally?

The limit is your hardware, specifically video RAM. Gemma 3 4B supports up to 131,000 tokens, written officially as 128,000, but running that full context locally requires enough VRAM to hold it. In the demo, LM Studio warned that setting a high context length would significantly impact memory usage on a 4090 with 24GB of VRAM. Loading a 120,000-token context consumed all available VRAM, brought the computer to a crawl, and made the previously snappy model very slow to respond. Bigger context windows require more compute and more GPU resources, not just a powerful model.

Q: What did the Lost in the Middle paper find about LLM attention?

The paper showed that models handling large contexts were more accurate with information placed at the beginning of the context and also accurate with information at the end, but accuracy dropped off sharply for information in the middle. Plotted across the board, this produces a U-shaped curve. The practical implication is that even when a conversation fits inside a large context window, the model may not reliably use everything in it, so important details buried in the middle of a long conversation or document are the most likely to be missed.

Summary & Key Takeaways

• LLMs including ChatGPT, Gemini, Claude, and local models such as Llama and DeepSeek have a short-term memory limited by their context window. Everything you say, everything the model says back, hidden system prompts, uploaded documents, and code all get stored as tokens. When that store overflows, the model forgets earlier details, hallucinates, and slows down.

• A live LM Studio demo with Gemma 3 4B at 2048 tokens showed the failure directly. Telling the model about a book consumed 14.6 percent of context, then a story, a sequel, and a prequel pushed usage to 118.4 percent, about 2,400 tokens. The model could no longer recall the book. Reloading at 4,096 tokens fixed it.

• Cloud models offer far larger windows than most local hardware can run: GPT-4o at 128,000 tokens, Claude 3.7 at 200,000, and Gemini 2.5 at 1 million, with 2 million said to be close. Meta's Llama 4 Scout, a local model, has a 10 million token context window. Local use is capped by VRAM instead.