Counting tokens for SmolLM3
Hugging Face publishes SmolLM3's tokenizer with the Apache-licensed weights, so the count on this page is exact. SmolLM3 3B loads by default. It shows as free: at 3B parameters it runs on almost any Mac and there is no hosted API to pay for, so the number to watch is the context bar. SmolLM3 was trained to 64K tokens and reaches 128K with YaRN scaling; the bar here measures against the 64K window that works out of the box.
Pricing and context windows
SmolLM3 with its tokenizer and context window. The weights are open under Apache 2.0, so it shows as free.
| Model | Tokenizer | Price / 1M input | Context window |
|---|
| SmolLM3 3B | smollm3 (llama-3) | Free | 66K |
Prices and context windows are curated metadata refreshed periodically from the providers and OpenRouter, so treat them as guidance rather than a quote. Token counts are always produced by the real tokenizer.
How SmolLM3 tokenization works
SmolLM3 uses the Llama 3.2 tokenizer, so it counts exactly like Llama 3 on ordinary text. Hugging Face's release notes say so directly, and the files confirm it: the 128,000-entry vocabulary and the merge rules are byte-identical to Llama 3.2's, with 100,000 entries from OpenAI's tiktoken encoding and 28,000 added for the six languages SmolLM3 supports (English, French, Spanish, German, Italian, and Portuguese). SmolLM3 repurposes ten reserved slots for its chat, thinking, and tool-call markers and drops the beginning-of-text token, none of which affects pasted text. On our fixed 500-word English sample it counts 574 tokens, the same as GPT-4 Turbo's cl100k_base.
SmolLM2 used a different, 49,000-entry vocabulary, so SmolLM2 counts do not carry over. llama.cpp, Ollama, and transformers.js all load this same BPE, so the number here matches what a local run reports.
Frequently asked questions
Is this SmolLM3's official tokenizer?
Yes. The tool loads the tokenizer file from the SmolLM3-3B repository. The text is tokenized in your browser, so the count is exact and nothing is uploaded. The GGUF builds carry the same vocabulary and splitting rule, so llama.cpp and Ollama report the same number.
Does SmolLM3 count the same as Llama 3?
Yes, for any ordinary text. The vocabulary and merges are Llama 3.2's. The one difference is that SmolLM3 adds no beginning-of-text token, so a full prompt comes out one token shorter than the same prompt on Llama 3.
Is SmolLM3's context 64K or 128K?
64K as trained; 128K with YaRN scaling enabled in your runtime. The model configuration sets 65,536 tokens. Extending to 131,072 needs a YaRN setting in llama.cpp or transformers, and a bigger KV cache. The context bar here uses the 64K window, so a bar past halfway means you are already into YaRN territory.
What does the chat template add on top of my text?
A few tokens per message, plus a system prompt the template inserts on its own. SmolLM3 wraps each turn in ChatML markers and prepends a default system prompt with metadata such as the date and the reasoning mode, which costs a few hundred tokens before your message. In thinking mode the reply also carries a reasoning block, which lives in the same context.
