If you've ever used ChatGPT, Claude, or any other AI assistant, you've interacted with tokens without even knowing it. Tokens are the fundamental unit of how large language models (LLMs) read, process, and generate text. Understanding tokens isn't just academic knowledge—it directly impacts your API costs, context limits, and how effectively you can use AI tools.
In this guide, we'll demystify tokenization, explain how different models count tokens, and show you practical techniques for estimating and optimizing token usage.
What Are Tokens?
Tokens are the atomic units that LLMs process. Rather than reading text character-by-character or word-by-word, language models break text into tokens—pieces that might be whole words, parts of words, punctuation, or even individual characters.
Consider this sentence: "Tokenization is fascinating!"
A tokenizer might break this into:
- "Token" (1 token)
- "ization" (1 token)
- " is" (1 token, note the space is included)
- " fascinating" (1 token)
- "!" (1 token)
That's 5 tokens for 3 words and 1 punctuation mark. This example illustrates a key principle: tokens don't map 1:1 to words.
Why Tokens Instead of Words?
You might wonder why AI models don't simply process whole words. There are several compelling reasons:
1. Vocabulary Size Management
English has over 170,000 words in current use, plus technical jargon, proper nouns, and foreign words. Including every possible word would create an unwieldy vocabulary. Tokenization reduces vocabulary to a manageable 32,000-100,000 tokens while still covering all possible text.
2. Handling Unknown Words
What happens when someone types "ChatGPTification" or makes a typo like "teh"? Word-based systems would fail. Subword tokenization gracefully handles any text by breaking unknown words into known pieces.
3. Multilingual Support
Tokens work across languages. Chinese characters, Arabic script, and emoji can all be tokenized without needing separate vocabularies for each language.
4. Computational Efficiency
Shorter token sequences mean faster processing. Balancing vocabulary size against sequence length optimizes the computational cost of running the model.
How Tokenization Works
Modern LLMs primarily use variations of Byte Pair Encoding (BPE), a compression algorithm adapted for natural language processing.
The BPE Process
- Start with characters: Begin with individual characters as the initial vocabulary
- Count pairs: Find the most frequent adjacent pair of tokens
- Merge: Combine that pair into a new token
- Repeat: Continue until reaching the desired vocabulary size
For example, starting with the text "low lower lowest":
- Initial: ['l', 'o', 'w', ' ', 'l', 'o', 'w', 'e', 'r', ' ', 'l', 'o', 'w', 'e', 's', 't']
- Most frequent pair: 'l' + 'o' → merge into 'lo'
- Next: 'lo' + 'w' → merge into 'low'
- Continue until vocabulary is complete
Different Tokenizers, Different Counts
Each model family uses its own tokenizer:
| Model | Tokenizer | Vocabulary Size |
|---|---|---|
| GPT-4, GPT-3.5-Turbo | cl100k_base | ~100,000 tokens (100,256 base) |
| GPT-4o, o-series, GPT-5 family | o200k_base | ~200,000 tokens |
| GPT-3 | p50k_base | ~50,000 tokens |
| Claude | proprietary (no public encoder) | not disclosed |
| Llama 2 | SentencePiece BPE | 32,000 tokens |
| Llama 3 / 3.1 / 4 | tiktoken-style BPE | 128,256 tokens |
| Gemini | proprietary SentencePiece | ~256,000 tokens |
A common misconception is that all Llama models share one tokenizer. They don't. Llama 2 used a 32K-vocab SentencePiece BPE tokenizer. Starting with Llama 3, Meta switched to a tiktoken-style byte-level BPE with a 128,256-token vocabulary—roughly 4x larger—which packs more text into each token and substantially improves efficiency on code and non-English text.
OpenAI made a similar jump. GPT-4 and GPT-3.5-Turbo use cl100k_base (~100K vocab). GPT-4o, the o-series, and the GPT-5 family moved to o200k_base (~200K vocab). The larger vocabulary means fewer tokens for the same text—biggest gains on non-English and code—which lowers the effective cost per call even at the same per-token price. Claude exposes a count-tokens API endpoint rather than a public offline encoder, and Gemini reports counts through its countTokens method.
Vocabulary sizes have grown substantially as tokenizers matured—larger vocab means fewer tokens for the same text:
This means the same text can have different token counts across models:
Text: "Artificial intelligence is transforming industries."
GPT-4 (cl100k_base): 6 tokens
GPT-4o (o200k_base): 6 tokens
Llama 3 (128K BPE): 7 tokens
When planning API costs or context usage, always count with the specific tokenizer for your chosen model. Our LLM Token Counter does this across model families in one place.
Token Counting Rules of Thumb
While exact counts require the actual tokenizer, these approximations help with quick estimates:
English Text
- 1 token ≈ 4 characters (including spaces)
- 1 token ≈ 0.75 words
- 100 tokens ≈ 75 words
- 1,000 words ≈ 1,333 tokens
Code
- Code typically uses more tokens per line than prose
- Variable names, syntax, and whitespace all consume tokens
- A 100-line Python function might be 500-800 tokens
Special Cases
- Numbers: Each digit is often a separate token ("2024" = 4 tokens)
- URLs: Very token-heavy due to punctuation and special characters
- JSON: Brackets, colons, and quotes add up quickly
- Non-English: Some languages (Chinese, Japanese) may use more tokens per character
Context Windows Explained
The context window is your token budget for a conversation—it includes everything the model can "see" at once:
Context Window = Input Tokens + Output Tokens
Current Context Window Sizes (mid-2026)
1M-token context is now standard at the frontier. Note that Llama 3.1 onward ships a 128K window—a frequent point of confusion, since the original Llama 3 launched at only 8K.
| Model | Context Window | Approximate Pages |
|---|---|---|
| GPT-5.5 | 1,000,000 tokens | ~2,500 pages |
| Claude Opus 4.8 / Sonnet 4.6 | 1,000,000 tokens | ~2,500 pages |
| Gemini 3.1 Pro | 1,000,000 tokens | ~2,500 pages |
| Llama 4 Scout | 10,000,000 tokens | ~25,000 pages |
| Llama 3.1 / 3.3 (8B–70B) | 128,000 tokens | ~300 pages |
| Llama 3 (original, 8B–70B) | 8,192 tokens | ~20 pages |
Context Window Management
When your conversation exceeds the context window, you have several options:
- Truncation: Remove older messages from the conversation
- Summarization: Condense earlier context into a summary
- RAG (Retrieval): Fetch only relevant portions of large documents
- Chunking: Process documents in segments with overlap
Practical Token Counting
Using Our Token Counter Tool
The easiest way to count tokens accurately is using a dedicated tool. Our LLM Token Counter supports multiple models and shows:
- Exact token count for your text
- Cost estimates based on current API pricing
- Context window usage percentage
- Model comparisons
Programmatic Token Counting
For developers, here's how to count tokens in code:
Python with tiktoken (OpenAI models):
import tiktoken
def count_tokens(text: str, model: str = "gpt-4") -> int:
encoding = tiktoken.encoding_for_model(model)
return len(encoding.encode(text))
text = "Hello, how are you today?"
tokens = count_tokens(text)
print(f"Token count: {tokens}") # Output: Token count: 7
For OpenAI's newer models, request the matching encoding explicitly—encoding_for_model resolves GPT-4o and the GPT-5 family to o200k_base, while GPT-4/3.5 resolve to cl100k_base:
import tiktoken
enc = tiktoken.get_encoding("o200k_base") # GPT-4o, o-series, GPT-5 family
# enc = tiktoken.get_encoding("cl100k_base") # GPT-4, GPT-3.5-Turbo
print(len(enc.encode("Hello, how are you today?")))
Python with transformers (Llama, open models):
from transformers import AutoTokenizer
# Llama 3.1+ uses a 128,256-vocab tiktoken-style BPE (not Llama 2's 32K SentencePiece)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B")
text = "Hello, how are you today?"
tokens = tokenizer.encode(text)
print(f"Token count: {len(tokens)}")
API Response Token Counts
Most LLM APIs return token usage in responses:
{
"usage": {
"prompt_tokens": 56,
"completion_tokens": 128,
"total_tokens": 184
}
}
Track these values to monitor actual usage against estimates.
Tokens and Pricing
API pricing is directly tied to tokens, typically charged per 1,000 or 1 million tokens:
Pricing Snapshot (last updated 2026-06-25)
USD per 1M tokens, standard tier, input / output. Provider pricing moves constantly—treat these as a point-in-time reference and confirm against the vendor's live pricing page before budgeting. Output reliably costs 3x–6x input across the board (Claude is exactly 5x).
| Model | Input (per 1M) | Output (per 1M) |
|---|---|---|
| GPT-5.5 | $5.00 | $30.00 |
| GPT-5.4 | $2.50 | $15.00 |
| GPT-5.4-nano | $0.20 | $1.25 |
| Claude Opus 4.8 | $5.00 | $25.00 |
| Claude Sonnet 4.6 | $3.00 | $15.00 |
| Claude Haiku 4.5 | $1.00 | $5.00 |
| Gemini 3.1 Pro | ~$2.00 | ~$12.00 |
| Gemini 3 Flash | $0.50 | $3.00 |
| Mistral Large 3 | ~$0.50 | ~$1.50 |
| DeepSeek-V3 | ~$0.14 | ~$0.28 |
| Llama 4 (Scout/Maverick) | host-dependent | host-dependent |
Two notes that matter for budgeting. First, reasoning/"thinking" tokens are billed as output tokens on every provider that exposes them (GPT-5.x reasoning_effort, Claude extended thinking, Gemini thinking levels, DeepSeek-R1)—a high-reasoning call can quietly multiply your output bill. Second, prompt caching (cached input ~10% of standard) and batch (~50% off) are near-universal levers that cut real-world cost well below the sticker price. Meta's Llama 4 has no first-party per-token price; you pay whatever a host like Together, Fireworks, Groq, or Bedrock charges.
For a deeper provider-by-provider breakdown, see LLM API Cost Comparison.
Cost Calculation Example
Processing a 10,000-word document (~13,333 tokens input) and generating a 500-word summary (~667 tokens output) with Claude Sonnet 4.6 ($3 / $15):
Input cost: 13,333 ÷ 1,000,000 × $3.00 = $0.040
Output cost: 667 ÷ 1,000,000 × $15.00 = $0.010
Total: ~$0.05 per document
At scale (10,000 documents/month): ~$500/month—and prompt caching or batch processing would cut that further. If you run this kind of volume continuously, it's worth checking the crossover point against owning hardware: our Self-Hosted LLM Cost Calculator computes the cloud-API-vs-own-hardware break-even.
Understanding token economics helps you budget accurately and choose cost-effective models.
Optimizing Token Usage
Write Concise Prompts
Every word in your prompt costs tokens. Compare:
Verbose (32 tokens):
I would really appreciate it if you could please help me by
summarizing the following article for me in a concise manner.
Concise (11 tokens):
Summarize this article in 3 bullet points:
Use System Prompts Efficiently
System prompts are included with every message. Keep them focused:
Inefficient:
You are a helpful AI assistant. You should always be polite,
professional, and thorough in your responses. You have expertise
in many areas including technology, science, business, and more.
Efficient:
You are a technical writer. Be concise and accurate.
Leverage Structured Output
Request specific formats to reduce unnecessary tokens:
Return JSON only: {"summary": "...", "key_points": [...]}
Batch Similar Requests
Instead of multiple API calls, batch related queries:
Analyze these 5 reviews and return sentiment for each:
1. [review 1]
2. [review 2]
...
Common Tokenization Pitfalls
Surprising Token Counts
Some text is surprisingly token-heavy:
- Whitespace: Multiple spaces or tabs may tokenize separately
- Special characters: Emoji can be 2-4 tokens each
- Base64/encoded data: Extremely token-inefficient
- Repetition: Repeated text isn't compressed
Language Differences
Non-English languages often require more tokens:
| Language | Tokens per 1000 characters |
|---|---|
| English | ~250 tokens |
| Spanish | ~280 tokens |
| Chinese | ~350 tokens |
| Japanese | ~400 tokens |
| Arabic | ~320 tokens |
Factor this into multilingual applications.
Code Tokenization
Code tokenizes differently than prose:
# This function might be 15-20 tokens
def calculate_total(items):
return sum(item.price for item in items)
Variable names, operators, and syntax all contribute. Minified code isn't necessarily fewer tokens—meaningful names and whitespace don't dramatically increase token count.
Conclusion
Tokens are the currency of large language models. Understanding how tokenization works empowers you to:
- Estimate costs before committing to API usage
- Optimize prompts for efficiency without sacrificing quality
- Choose appropriate models based on context needs and budget
- Debug unexpected behavior when token limits are exceeded
As AI becomes more integrated into applications, token literacy becomes a valuable skill for developers, product managers, and anyone working with LLMs.
Ready to count tokens for your specific use case? Try our LLM Token Counter to get exact counts for GPT-4, Claude, Llama, and other popular models. And if the per-token bill is the real problem, one way to flatten it is to serve the baseline on hardware you own and burst to the cloud only on overflow.