Quick Definition
AI tokens are the fundamental units of data that large language models process — not whole words but flexible chunks: a single character, part of a word, or a full word. Tokenization converts your text into numbered sequences models can compute, and those numbers determine both what the AI can “remember” and what it costs you.
If you’re here because someone told you “AI tokens are the next big crypto play” — wrong article. LLM tokens and blockchain tokens share a name and nothing else. This is about the tokens that run inside ChatGPT, Claude, and Gemini: the building blocks every language model uses to read, think, and write. Want the crypto version? CoinGecko’s got you. Want to understand what your prompts actually cost and why your AI forgets things halfway through a conversation? Keep reading.
I’ve spent more hours than I’d admit staring at tokenizer output. The first time I ran the same paragraph through GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro and got three different token counts, I thought the tools were broken. They weren’t. Each model sees your text differently, and understanding that difference is the fastest way to stop burning money on API calls you didn’t need to make.
How tokenization works: What actually happens when you type a prompt
You type a sentence. Hit enter. The model never sees your words. It sees numbers.
Before a single neuron fires, your text runs through a tokenizer — a piece of software that splits your input into pieces, maps each piece to an integer, and hands the model a list of those integers. The model does math on those numbers. When it’s done, it hands back a new list of numbers. The tokenizer decodes them into text. That’s the entire loop.
The same sentence, three different ways
Here is the sentence “The cat sat on the mat.” tokenized by three major models:
| Model | Token Count | Token Breakdown |
|---|---|---|
| GPT-4o | 7 tokens | The cat sat on the mat . |
| Claude 3.5 Sonnet | 6 tokens | The cat sat on the mat. |
| Gemini 1.5 Pro | 7 tokens | The cat sat on the mat . |
A trivial sentence. Still different counts. Claude merges the period — everyone else splits it. Run a 500-word article and the gap widens: GPT-4o might see 680 tokens, Claude sees 660, Gemini sees 710. Over thousands of API calls, those differences compound into real money.
Now try something messier:
“I can’t believe the tokenizer splits ‘unbelievable’ and ‘darkness’ differently — this is wild!”
| Model | Token Count | Noteworthy Splits |
|---|---|---|
| GPT-4o | 28 tokens | unbeliev able |
| Claude 3.5 Sonnet | 27 tokens | un believ able |
| Gemini 1.5 Pro | 26 tokens | unbelievable |
Gemini keeps “unbelievable” whole. GPT-4o slices it into two. Claude slices into three. Same word, same meaning — three different approaches. This is why “just count words” doesn’t work.
What is Byte Pair Encoding?
Most modern tokenizers — including GPT-4o’s, Claude’s, and Gemini’s — use a method called Byte Pair Encoding, or BPE. The name sounds arcane. The idea is simple.
Start with every single character as its own token. Scan an enormous corpus of text. Find the most common pair of adjacent tokens. Merge them into one new token. Repeat. Do this thousands of times until you reach your target vocabulary size — usually around 50,000 to 100,000 tokens.
The result? Common words like “the” stay as single tokens because they appear constantly and merge early. Rare words like “defenestration” get split into pieces because the pair never quite becomes common enough to merge. The model gets a manageable vocabulary and can still handle any word by assembling it from parts it already knows.
BPE also explains why code and JSON can be expensive. A GPT-4o tokenizer sees { and "name" and : and "Alice" and } — each a separate token. A 50-character JSON object can eat 30 tokens. Ten messages of structured API output? Hundreds of tokens you didn’t think about.
Types of tokens: It’s not just text anymore
Text Tokens
A text token can be:
- A whole word:
cat - A subword piece:
ness(the shared ending of “darkness” and “brightness”) - A single character:
a - Punctuation:
.or, - Whitespace:
␣(some tokenizers encode spaces explicitly)
The tokenizer assigns each a numerical ID. When “darkness” and “brightness” both end in token 655 (“ness”), the model learns that words sharing this suffix share morphological patterns without anyone telling it about English prefixes and suffixes. The pattern emerges from the data.
Multimodal tokens
Models now process images, audio, and video. Each gets tokenized differently:
- Images: For Gemini, a small image (under 384px on both sides) costs 258 tokens. Larger images get sliced into 768×768-pixel tiles — each tile costs 258 tokens. A high-res photo might eat 1,032 tokens before the model even reads your question about it.
- Video: 263 tokens per second. A 30-second clip? 7,890 tokens. Uploading video adds up fast.
- Audio: 32 tokens per second. A 5-minute voice memo costs 9,600 tokens just to get through the door.
Most people don’t realize they’re burning tokens on media. They upload a screenshot, ask a question, and wonder why the context window filled up so fast.
What is a context window?
A context window is the maximum number of tokens a model can “hold in its head” at once. It includes your prompt, the conversation history, any uploaded documents, system instructions, and the space needed for the model’s response.
Think of it as short-term memory with a hard ceiling.
| Model | Context window | Rough page equivalent |
|---|---|---|
| GPT-3.5 Turbo | 16K tokens | ~50 pages |
| GPT-4o | 128K tokens | ~400 pages |
| Claude 3.5 Sonnet | 200K tokens | ~600 pages |
| Gemini 1.5 Pro | 2M tokens | ~6,000 pages |
A 2-million-token window sounds infinite. It isn’t. Models also suffer from a well-documented problem called “lost in the middle”, meaning they pay attention to the beginning and end of a long context but lose track of information buried in the center. A wider window doesn’t mean the model reads everything equally.
What happens when you hit the limit of the context window
There isn’t a dramatic error message. The model just silently drops the oldest tokens to make room for new ones. Your carefully written system prompt from message one? Gone. The instruction to “always respond in Spanish”? Vanished. The model didn’t ignore you — it literally can’t see those instructions anymore.
I’ve debugged this exact problem for a client who couldn’t figure out why their chatbot started giving English answers after ten messages. They had a Spanish-only system instruction. It worked perfectly for the first eight turns. By message ten, the instruction had been pushed out of the context window. The model wasn’t broken, the context was.
Why tokens matter: follow the money
Most people treat AI like a free utility. Search a question. Get an answer. Move on. But every word costs something. And the people building on top of these APIs are the ones who feel it.
How token pricing works
Every AI provider charges per token. You pay for input tokens (your prompt + conversation history + any uploaded files) and output tokens (what the model writes back). Output tokens almost always cost more — generating text is more computationally expensive than reading it.
| Model | Input (per 1M tokens) | Output (per 1M tokens) |
|---|---|---|
| GPT-4o Mini | $0.15 | $0.60 |
| Gemini 2.0 Flash | $0.10 | $0.40 |
| GPT-4o | $2.50 | $10.00 |
| Claude 3.5 Sonnet | $3.00 | $15.00 |
| Claude Opus 4.6 | $5.00 | $25.00 |
A million tokens sounds like a lot. It’s roughly 750,000 English words (about three copies of Moby Dick). Run a chatbot handling 10,000 daily conversations, each averaging 2,000 tokens of input and 500 tokens of output, and you’re looking at roughly $3,300 per month on a mid-tier model. Reasoning models, which generate internal “thinking tokens” you never even see, can push that to 100× more.
The mistake that costs the most
The single most expensive error I see: using the wrong model for the job.
A developer builds a simple text classifier. Classify support tickets into “urgent” or “not urgent.” They wire it up to GPT-4o. Every ticket — thousands a day — runs through a model priced at $2.50 per million input tokens. The classification is 99% accurate.
The same classifier on GPT-4o Mini costs $0.15 per million input tokens. Accuracy is 98.5%. That half-percent accuracy gain costs roughly 16× more. Over a year, the difference is thousands of dollars — for a task where the cheaper model performs essentially identically.
The rule: if the task is classification, extraction, summarization, or simple Q&A, try the cheapest model first. Reserve the expensive models for tasks where the extra reasoning actually produces different output.
How tokens are used during training and inference
Training
Pretraining an LLM starts with tokenizing the entire training dataset and uses billions or trillions of tokens. The model then plays a massive game of “predict the next token.” It sees a sequence, guesses what comes next, checks the answer, and adjusts its internal weights when it’s wrong. Repeat until the error rate drops below an acceptable threshold. This is model convergence.
After pretraining comes post-training: fine-tuning on a smaller, curated set of tokens for a specific domain or task. Legal documents. Medical records. Conversational formats. The goal is a model that generates the right tokens for the right context.
Inference
Inference is where you meet the model. Your prompt gets tokenized. The model processes those input tokens and starts generating output tokens one at a time — each new token conditioned on everything that came before it. When the model generates a special “stop” token, the response ends. The tokenizer converts the output numbers back to text and hands you the result.
Why reasoning models cost more
The newest generation of models — GPT-5, Claude with extended thinking, Gemini with reasoning — generate hidden tokens you never see. The model works through a problem internally, producing “thinking tokens” that occupy context window space, consume compute, and get billed as output tokens. A single complex query might generate 10,000 reasoning tokens before producing a 200-token final answer. You pay for all 10,200.
How to optimize your token usage
Most token waste isn’t from bad code. It’s from habits you don’t notice — sending the full chat history with every message, being polite to a machine that doesn’t care, routing every query to the most expensive model because it’s the default.
None of these will break your app. They’ll just quietly double your bill. Here are six changes that take minutes to implement and pay for themselves by the end of the week.
1. Stop sending full conversation history
The default behavior in many API integrations: include the entire conversation history with every new message. Message 20 includes messages 1 through 19. By message 30, you’re sending 29 previous messages — most of which the model already saw — doubling or tripling your input tokens per turn. Summarize the conversation periodically. Replace the full history with a 200-token summary.
2. Concise prompts beat polite prompts
Every “please” and “could you possibly” and “I was wondering if you might be able to” burns tokens. The model doesn’t need them. Neither does your budget. Write prompts like you’re texting a colleague: direct, specific, minimal.
3. Use prompt caching
Claude and Gemini both offer prompt caching: mark repeated content like system instructions and have it processed at a steep discount. Claude Opus 4.6 charges $5 per million input tokens at standard rates — but only $0.50 per million for cached reads. A 90% discount on the part of your prompt that never changes.
4. Route tasks by complexity
Not every prompt needs GPT-4o or Claude Sonnet. Route classification, extraction, and simple Q&A to a fast cheap model. Route complex reasoning, code generation, and nuanced analysis to the expensive model. A 70/30 split — 70% cheap, 30% expensive — produces near-identical results at a fraction of the cost.
5. Set a max_tokens limit
Most APIs let you cap the response length. If you’re extracting a single answer from a document, you don’t need the model to write 500 words about it. Set max_tokens to 50. You’ll pay for 50 tokens of output instead of 500.
6. Pre-process long documents
Don’t send a 50-page PDF and ask “summarize this.” Chunk the document first. Send each chunk for a summary of its own. Then send the collected summaries to the model for a final synthesis. You’ll use fewer total tokens than cramming the full document into the context window.
Common token problems and how to fix them
“My AI keeps forgetting the start of our conversation”
Your context window is full. The model is silently dropping old tokens. Fix: summarize the conversation so far, start a new session with the summary as context, or upgrade to a model with a larger window.
“My API bill Is higher than I expected”
Check your per-message input token count. If you’re sending full conversation history with every turn, that’s the culprit. Also check if you’re accidentally running classification or extraction through an expensive model — the most common silent cost multiplier.
“My prompt got truncated”
The model hit its output token limit before finishing. Either your max_tokens setting is too low, or your prompt + expected response exceeds the context window. Shorten your prompt or request shorter output.
“The same word uses different token counts in different models”
Expected behavior. Each model has its own tokenizer. A word that’s one token in Claude might be three tokens in GPT-4o. This isn’t a bug — it’s why “1,000 tokens ≈ 750 words” is always approximate.
“Code and JSON eat way more tokens than I expected”
Code is token-dense. Every brace, bracket, colon, and indentation gets its own token. A 200-character JSON object can consume 50-80 tokens — far more than equivalent natural language. If you’re sending structured data, expect higher-than-expected token counts.
FAQ
Why do AI models use tokens instead of words?
Because neural networks work on numbers, not text. Tokenization converts human language into sequences of integers the model can compute. Using subword tokens keeps the vocabulary manageable — 50,000 tokens can represent millions of words — while enabling the model to handle unfamiliar words by assembling them from known pieces.
Is a larger context window always better?
No. Larger windows cost more to fill and process. Models also suffer from “lost in the middle” — they pay less attention to information buried deep in a long context. A 200K-token window doesn’t help if the model ignores the middle 100K tokens.
How many tokens are in a word?
Roughly 0.75 words per token, or 1,000 tokens ≈ 750 English words. But this varies by model, by language, and by content type. Code and JSON produce significantly more tokens per “word” than natural English prose.
Do tokens work the same across languages?
No. BPE tokenizers are trained primarily on English text. Languages with different scripts — Japanese, Arabic, Hindi — often require more tokens per semantic unit because the common character pairs in English don’t appear in those languages. Expect higher token counts and higher costs for non-English content.
How do I check how many tokens my text will use?
- GPT models: Use OpenAI’s tokenizer tool at platform.openai.com/tokenizer or the
tiktokenPython library - Claude: Anthropic provides a token counting endpoint in their API
- Gemini: Call the
count_tokensmethod in the Gemini API before sending your request
Updated April 2026. Token pricing and context window sizes change frequently. Verify current rates with your provider before budgeting.
Related topics
- LLM context windows explained: limits, costs, and developer workarounds. How context size affects your costs, the lost-in-the-middle problem, and 5 strategies for working within limits.
- AI token pricing guide: What LLMs actually cost per model. Full pricing comparison across 9 models, hidden costs like reasoning tokens and system prompt tax, and a Python cost estimator.
- How to stop wasting tokens in Claude Code: 7 data-backed fixes. Optimization for the most common developer environment.