What a token is, and why it isn't a word

The Last lesson gave you a rough rule: a token is about four characters of English. That rule is fine for quick estimates. It also hides the thing that trips developers up when the first bill or the first length error shows up.

The model doesn't see words. It doesn't see characters, either. It sees tokens, and tokens line up neatly with neither.

How text becomes tokens

Before the model reads anything, your text is split into tokens using a fixed vocabulary built during training. This step is called tokenization, and that vocabulary is decided once and then frozen.

Common words usually map to a single token. Rarer or longer ones get broken into pieces. “Writing” is likely one token. “Tokenization” might come out as “token” plus “ization.” The split follows what was frequent in the training text, not any rule about syllables or meaning.

Whitespace and punctuation count too. A leading space is often bundled into the token, so “ the” with a space and “the” without one can be different tokens. That sounds like a trivial detail. It's exactly why your hand counts never quite match the real number.

Why the word count lies

Here's the naive move: count the words in your prompt and treat that as your token count. For plain English prose, you'll land in the ballpark. Almost everywhere else, you won't.

Code, with its brackets, indentation, and symbols, runs far more tokens per line than prose. Names, URLs, and IDs split into many pieces. Numbers fragment in odd ways, so a long figure can become several tokens. Languages other than English, and emoji, cost more tokens per character than English does.

So why care about an invisible unit you never type? Because tokens are what everything downstream is counted in. What you pay, and how much the model can hold at once, are both measured in tokens, not words. The cost comes first. That's next.