Tokens and Context Window: What They Are and AI Costs

A token is a piece of text into which a language model splits every sentence: a short word, part of a longer word, or a punctuation mark. The context window is the maximum number of tokens a model can see at once: instructions, conversation, documents, and its own response. Anything that doesn't fit simply does not exist for the model.

How it works

The model does not read letters or words, but tokens. Common words are a single token, while rare ones are split into multiple pieces.
In Polish, the same text usually takes more tokens than in English, because models were trained mostly on English, and Polish words have many inflections.
Providers charge per token: separately for what you send (input) and what the model writes (output). Output is usually several times more expensive.
The context window is the model's working memory. A long conversation eventually overflows, causing the oldest messages to drop out or require summarization.

How our lab does it

Chat question: max 500 characters. Longer inputs are rejected immediately without calling the model. This is a cost control measure and part of defense against attacks.
Chunks up to 1,200 characters, 6 retrieved to the model. The model doesn't get entire articles, just the most relevant pieces. Fewer tokens, lower cost, less noise.
Chat response cost: approx. 0.08 grosz. At this price, 1,000 questions cost less than 1 PLN.
Free limits also count requests, not just tokens. Gemini on the free tier has a daily request limit that the lab splits across multiple processes. We plan our work to stay within the limit and have a fallback model ready when it runs out.

Common pitfalls

Dump everything into the context. A large window is tempting, but the model struggles to find details in the middle of a long text, and every token costs money.
Forgetting about the output. Asking to "write comprehensively" increases costs much more than a long question.
Counting in words. Budget in tokens and test on your own texts, especially in Polish.

Questions and answers

How much is one token?

In English, roughly 3/4 of a word on average. In Polish, usually less, so the same text has more tokens.

What happens when a conversation exceeds the context window?

The oldest parts drop out or the application summarizes them. The model remembers nothing outside the current window.

Is a larger context window always better?

No. It provides more space, but increases cost and response time, and the model doesn't always make good use of the middle of very long texts.

Sources

Liu et al.: Lost in the Middle: How Language Models Use Long Contexts, arXiv:2307.03172
Lab: chat settings (question limit, chunks), response cost (21.10.2026)

Tokens and Context Window