🤖 AI Token Counter

Complete Guide: Token Counter

As large language models (LLMs) like GPT-4 and Claude become integral to modern development workflows, understanding how they measure and charge for text has become a practical skill. LLMs do not process text character-by-character or word-by-word — they process tokens, which are chunks of text produced by a subword tokenization algorithm. Knowing your token count before sending a prompt can save money, prevent context overflow errors, and help you optimize prompts for better performance.

What Is a Token?

A token is a unit of text produced by a Byte-Pair Encoding (BPE) tokenizer. BPE starts with individual characters and iteratively merges the most frequent adjacent pairs until it reaches a target vocabulary size (typically 50,000–100,000 tokens). The result is that common words become single tokens (the, is, cat), while rare words are split into subword units (un + believ + able).

A useful rule of thumb: 1 token ≈ 4 characters or roughly 0.75 words for English prose. This means 1,000 words is approximately 1,333 tokens.

Tokenizer Differences: GPT vs Claude

OpenAI's tiktoken library implements the exact tokenizers used by GPT-3.5 and GPT-4 (cl100k_base encoding). You can install it with pip install tiktoken and use it programmatically to get exact counts. Claude uses Anthropic's own tokenizer, which produces similar but not identical token counts — the same text will typically differ by a few percent between tokenizers. For rough estimation, both tokenizers yield approximately the same 4-characters-per-token average.

Context Window Limits

Every LLM has a maximum context window — the total number of tokens it can process at once, including both your input (prompt + conversation history) and its output (completion). Key limits as of mid-2025:

• GPT-4o: 128,000 tokens
• Claude 3.5 Sonnet / Claude 3.5 Haiku: 200,000 tokens
• Gemini 1.5 Pro: 1,000,000 tokens

Exceeding the context limit causes an API error. In chat applications, this means you must summarize or truncate older messages to stay within the window.

Why Token Count Affects Cost

All major LLM APIs charge per token, with separate rates for input tokens (your prompt) and output tokens (the model's response). Output tokens are typically 3–5× more expensive than input tokens. For a production application making 100,000 API calls per day, cutting average prompt length from 500 tokens to 300 tokens can reduce monthly costs by thousands of dollars.

Prompt Optimization Tips

• Remove boilerplate preambles — most models do not need long explanations of their role before each call
• Use structured formats (JSON, bullet lists) instead of verbose prose instructions; they convey the same information in fewer tokens
• For repeated context (e.g., a system prompt used in every call), investigate prompt caching features offered by Anthropic and OpenAI, which significantly reduce cost for cached tokens
• Truncate or summarize long conversation histories instead of including every prior exchange

Code vs Prose Tokenization

Code tokenizes differently from prose. Programming keywords, operators, and identifiers are often split into multiple tokens because they are less frequent in training data. Indentation-heavy code (Python) produces extra whitespace tokens. As a rough estimate, code is typically 1.5–2× more tokens per character than English prose. This matters when using LLMs for code review or generation with large files.

To check the raw word and character count of text, use the Word Counter. For counting tokens in large JSON payloads, format the JSON first with JSON Formatter.