The short version
Before an AI model can read your text, it breaks the text into pieces. Each piece is a token. A token can be a whole word, part of a word, a single character, or a bit of punctuation. The model never sees your sentence the way you do. It sees a list of tokens.
This matters because everything about an AI model is measured in tokens. How much you pay, how much text fits in one request, and how fast you get an answer are all counted in tokens, not words.
A word is not a token
Common short words are usually one token. Longer or unusual words get split into several. Here is how a few real examples break down.
The exact split depends on the model, so treat these as close estimates, not fixed rules.
Why models split words at all
English has hundreds of thousands of words, and people invent new ones every day. A model cannot keep a slot for every possible word. So instead it keeps a fixed set of common pieces, and builds any word out of those pieces.
Think of it like building words out of a fixed box of Lego. Common words have a single ready-made brick. Rare words get assembled from smaller bricks. This is why a normal word like "the" is cheap, and a strange technical term costs more tokens.
A rule of thumb you can actually use
For everyday English, one token is about four characters. That works out to roughly:
- → 100 tokens ≈ 75 words
- → 1,000 tokens ≈ 750 words, about a page and a half
- → A short email ≈ 100 to 200 tokens
Code, other languages, and lots of numbers or symbols change the ratio. But for plain English writing, the four-characters rule gets you close enough to plan with.
The three places tokens actually show up
Tokens are not just trivia. They are the unit behind three things you deal with every time you use an AI tool, even if nobody names them.
Cost
If you pay for an AI tool or its API, you pay per token, not per word. Input tokens and output tokens are usually priced separately. A wordy prompt and a long answer both cost more, because both are more tokens.
Limits
Every model has a maximum number of tokens it can handle at once, called the context window. Your prompt, any documents you paste, the chat history, and the answer all share that budget. Go over it and something gets cut.
Memory
In a long chat, the model can only "see" the tokens that fit in the window. Once a conversation gets long enough, the oldest tokens fall out of view, which is why a model can seem to forget how a long chat began.
Tokens outside plain English
The four-characters rule holds for everyday English. It bends, sometimes a lot, once you leave that comfort zone. This matters if your work involves any of the following.
- → Code. Brackets, symbols, and indentation are token-heavy, so a page of code uses more tokens than a page of prose.
- → Other languages. Many non-English languages, especially ones that do not use spaces or use non-Latin scripts, cost noticeably more tokens for the same meaning.
- → Numbers and data. Long numbers, tables, and IDs break into small pieces and add up fast.
- → Emoji and symbols. A single emoji can be one or more tokens, and decorative symbols are not free.
None of this changes the idea, only the ratio. If you work heavily in code or another language, expect the same text to use more tokens than the English rule of thumb suggests.
Why any of this is worth knowing
You do not need to count tokens by hand to use AI well. But knowing tokens exist explains a lot of things that otherwise feel random. Why a long document gets cut off. Why one request costs more than another. Why the model sometimes forgets the start of a long chat.
All of those come back to tokens. Once you can picture your text as a list of small chunks instead of a block of words, the way AI tools behave stops feeling like magic and starts making sense.