A token is the unit of text a language model reads and writes, about three-quarters of an English word. Vendors bill and limit in tokens, not characters.
Category
AI
Also known as
tokens; tokenization; tokenizer
Common words are one token; rare words, code, and non-English text split into several, so the same sentence in Slovak costs more tokens than in English. Context windows and API prices are both measured in tokens, so the count decides what fits and what it costs.
In practice
When a feature feeds customer data to a model, token count is the cost line. One profile's full event history can run to thousands of tokens; send the fields the task needs, not the whole record.
Something out of date?