Research from 2022 to September 2026 on two questions for our 1.1B English–Chinese model: does the tokenizer change model quality, and what text should the tokenizer be trained on. Every venue below was checked on its official page; papers found only on arXiv are marked.
We choose the tokenizer once, before 50B tokens of 训练。
Highlight shows one token per color, the way tokenizer viewers draw boundaries.Question one
Does the tokenizer change model quality?
Yes, and the best vocabulary grows with compute. At 3B parameters and equal FLOPs, going from 32K to 43K entries raised ARC-Challenge from 29.1 to 32.0. NeurIPS 2024
It matters most outside English. Tokenizer choice barely moved English tasks but changed multilingual results consistently, and small models predicted the effect on large ones. An English-centric tokenizer cost multilingual models up to 68% more training compute. ACL 2025Findings NAACL 2024
Fewer tokens is not the same as better. Across 64 models from 350M to 2.4B parameters, compression did not predict quality; the pre-tokenizer had the biggest effect. EMNLP 2024Findings ACL 2025
Letting tokens cross spaces helps English: +4.0% averaged over 30 tasks and +8.2 on MMLU, at the same model and vocabulary size. COLM 2025 It may not carry over to Chinese: a later study found SuperBPE matched BPE on English but was 0.01–0.06 bits per byte worse on Mandarin. ACL 2026 workshop
For numbers, one digit per token was more data-efficient than 2- or 3-digit tokens when training from scratch. Findings EMNLP 2024
The cleanest test so far trains 14 models that differ only in their tokenizer, with a Chinese robustness benchmark. ICML 2026
Question two
What should the tokenizer be trained on?
The same cleaned text the model trains on. When tokenizer data and model data differ, the vocabulary keeps tokens the model never learns. EMNLP 2024
Filter Chinese web text first: more than 23% of long Chinese tokens in GPT vocabularies are porn or gambling terms. EMNLP 2025
Normalize with care. Merging near-duplicate tokens hurt model quality, though the loss was easy to reduce, and always applying Unicode normalization removed signal that language classification needed. Findings ACL 2024Findings ACL 2026
Enough, not more: gains flatten near 150 GB of English and 200 GB of Russian text. arXiv 2025
Choose the language mix with cheap proxy tokenizers; this beat a LLaMA-3-style mix by up to 12% in compression. EACL 2026
Decide before pretraining. Switching a trained model to a new tokenizer took more than 50B tokens of further training for code models, though embedding-transfer methods now close most of the gap far sooner. ICML 2024NeurIPS 2024
Our own measurement, September 12, 2026
Tokenizer
Vocab
English
Chinese
Code
Math
Our mix
Embedding share
Text in 5 days
GPT-NeoX (Pile)
50K
4.59
2.10
3.59
3.61
3.27
10%
162 GB
cl100k_base
100K
4.67
2.35
4.31
3.69
3.53
17%
161 GB
Llama 3
128K
4.67
3.43
4.31
3.70
4.09
21%
179 GB
DeepSeek-V3
129K
4.72
4.51
4.05
3.77
4.46
21%
195 GB
Qwen3
152K
4.55
4.14
4.26
3.51
4.26
24%
180 GB
o200k_base
200K
4.73
3.57
4.27
3.72
4.16
30%
165 GB
Gemma 3
262K
4.58
3.98
3.66
3.45
4.14
36%
151 GB
UTF-8 bytes per token on about 1 MB per domain (FineWeb-Edu, FineWeb-2 Mandarin, FineMath, Python standard library); higher is better compression. "Our mix" weights English 49%, Chinese 29%, code 11%, math 11%. Embedding share and text read in 5 days are for a Llama-3.2-1B-shaped model on 8 cards at the measured 13,902 tokens/s per card. Compression alone does not rank model quality; see the papers under "Compression, theory, evaluation" below.