Data
Tokenisation
What is tokenisation?
Tokenisation splits text into the discrete units a model actually processes. Token counts differ substantially between model families for identical text, which means costs, context limits and sequence lengths are not directly comparable across models.
English prose runs roughly three to four characters per token in most modern tokenisers. Code, structured data and non-Latin scripts are considerably less efficient, sometimes by a factor of two or more.
Because every per-token price and every context limit is denominated in a specific tokeniser, comparing two models on token counts alone is comparing two different units that happen to share a name.
Related terms
Start with the free tier
A magic link creates your account, your tenant and your first API key. No card until you ask for compute.