Tokens

The small pieces of text (word fragments, words or symbols) that models read and write, and the unit in which most AI usage is priced.

Definition

Models do not see letters or words directly; text is split into tokens by a tokeniser. In English a token averages roughly three-quarters of a word, though it varies by language and model. Context windows, rate limits and prices are all measured in tokens, usually with output tokens priced higher than input tokens.

Token counts are the basic unit of AI economics. A process that sends long documents or runs many agent steps can consume far more tokens than expected, so estimating tokens per task is the first step in any cost model.

Sources

  1. OpenAI, Tokenizer