Why GPT, Claude, and Gemini Count Tokens Differently
The same sentence becomes a different number of tokens on each provider. Here's why, and why you can't reuse one provider's token count to budget another.
Short answer
Each provider trains its own tokenizer with its own vocabulary and splitting rules, so OpenAI, Anthropic and Google turn the same text into different token counts. The gap is usually small for plain English but grows for code, structured data and languages such as Japanese, Chinese or Korean. Count tokens with the provider you will actually use, and never budget Claude or Gemini with a GPT token count.
| Use this guide for | Next action |
|---|---|
| Understanding the cost decision | AI Cost Calculator → |
Paste the same paragraph into three token counters and you'll get three different numbers. It isn't a bug. GPT, Claude, and Gemini each use a different tokenizer, so "how many tokens is this text" has three answers, and that has direct consequences for your cost estimates.
A token isn't a word (or a character)
Models don't process letters or words; they process tokens, chunks of text learned from data. A rough rule is that one token is about four English characters, but that average hides a lot. Common words may be a single token, while rare words, code, emoji, and non-English scripts can fragment into many. The exact split depends on the tokenizer's vocabulary and merge rules.
Three different tokenizers
- OpenAI (GPT-5 family) uses byte-pair encoding with a large, code-friendly vocabulary that packs source code and many languages into fewer tokens than older encodings did.
- Claude (Anthropic) uses its own tokenizer, which Anthropic doesn't publish. The same text usually lands on a different count than OpenAI's, sometimes higher, sometimes lower, so you can't assume a GPT estimate transfers. Claude 4.7 and later models also use a newer tokenizer than earlier Claude models; see how to count Claude tokens for exact counts with Anthropic's counting API.
- Gemini (Google) uses a SentencePiece tokenizer with a very large vocabulary and a different segmentation method, which again produces its own counts.
The practical upshot: a 2,000-token prompt on GPT is not a 2,000-token prompt on Claude or Gemini. The gap is usually small for plain English but grows for code, structured data, and languages like Japanese, Chinese, or Korean.
Why it matters for cost
Cost is price-per-token × tokens. If you switch providers but keep the token count from your old provider, your estimate drifts, especially at scale, with code-heavy prompts, or multilingual content. Two mistakes are common:
- Estimating Claude or Gemini cost with a GPT token count. Use the target provider's own count.
- Comparing providers on price-per-token alone. A model with a lower per-token price can still cost more if its tokenizer splits your specific content into more tokens.
How to estimate honestly
- Count tokens with the provider you'll actually use; each exposes a tokenizer or returns usage metadata.
- Estimate each provider against the same real prompt, not a generic word count, in the model-specific tools: GPT, Claude, and Gemini.
- When you're choosing a default model, compare the same workload priced per provider in the AI Pricing Comparison, which folds both the per-token price and the token count into one number.
Tokenizers are an invisible variable in every AI budget. Once you know each provider counts differently, you stop trusting a single number and start estimating per provider, which is exactly where accurate cost planning begins.
AI Cost CalculatorApply this guide to your own workload.Open calculator →Common questions
Can I use a GPT token count to estimate Claude or Gemini costs?
No. Claude and Gemini tokenize text differently from GPT, so the same prompt lands on a different count, sometimes higher and sometimes lower. Count with the target provider's own tokenizer or usage metadata.
Which kinds of text show the biggest token differences between providers?
Plain English differs least. Source code, structured data such as JSON, and languages like Japanese, Chinese or Korean show the largest gaps, because each tokenizer splits them differently.
Is the model with the lower per-token price always cheaper?
Not necessarily. A model with a lower per-token price can still cost more if its tokenizer splits your content into more tokens. Compare the same real workload priced per provider rather than comparing rates alone.