Why GPT, Claude, and Gemini Count Tokens Differently

Paste the same paragraph into three token counters and you'll get three different numbers. It isn't a bug. GPT, Claude, and Gemini each use a different tokenizer, so "how many tokens is this text" has three answers, and that has direct consequences for your cost estimates.

A token isn't a word (or a character)

Models don't process letters or words; they process tokens, chunks of text learned from data. A rough rule is that one token is about four English characters, but that average hides a lot. Common words may be a single token, while rare words, code, emoji, and non-English scripts can fragment into many. The exact split depends on the tokenizer's vocabulary and merge rules.

Three different tokenizers

• OpenAI (GPT-5 family) uses byte-pair encoding with a large, code-friendly vocabulary that packs source code and many languages into fewer tokens than older encodings did.
• Claude (Anthropic) uses its own tokenizer, which Anthropic doesn't publish. The same text usually lands on a different count than OpenAI's, sometimes higher, sometimes lower, so you can't assume a GPT estimate transfers.
• Gemini (Google) uses a SentencePiece tokenizer with a very large vocabulary and a different segmentation method, which again produces its own counts.

The practical upshot: a 2,000-token prompt on GPT is not a 2,000-token prompt on Claude or Gemini. The gap is usually small for plain English but grows for code, structured data, and languages like Japanese, Chinese, or Korean.

Why it matters for cost

Cost is price-per-token × tokens. If you switch providers but keep the token count from your old provider, your estimate drifts, especially at scale, with code-heavy prompts, or multilingual content. Two mistakes are common:

1. Estimating Claude or Gemini cost with a GPT token count. Use the target provider's own count.
2. Comparing providers on price-per-token alone. A model with a lower per-token price can still cost more if its tokenizer splits your specific content into more tokens.

How to estimate honestly

• Count tokens with the provider you'll actually use; each exposes a tokenizer or returns usage metadata.
• Estimate each provider against the same real prompt, not a generic word count, in the model-specific tools: GPT, Claude, and Gemini.
• When you're choosing a default model, compare the same workload priced per provider in the AI Pricing Comparison, which folds both the per-token price and the token count into one number.

Tokenizers are an invisible variable in every AI budget. Once you know each provider counts differently, you stop trusting a single number and start estimating per provider, which is exactly where accurate cost planning begins.