CantoneseLLM v2 Technical Report

This is an English translation of the Chinese original.

More than two years after our first CantoneseLLM technical report, we have finally released CantoneseLLM v2, together with a full technical report on arXiv. This generation is built on Qwen3 8B and Qwen3 30B-A3B (MoE), and to our knowledge it is the first systematic attempt to make a model reason in Cantonese — not merely answer in Cantonese, but carry out its entire chain-of-thought in the language. All model weights, training environments and datasets are open-sourced and available in the HuggingFace Collection.

¶Why the Standard Recipe Does Not Transfer

Language adaptation of LLMs now has a well-established recipe: large-scale continuous pre-training, followed by post-training data distilled from a teacher model. Taiwan-LLM used 35.1B tokens; Japan's Swallow and Southeast Asia's SEA-LION each used 200B. For Cantonese, this recipe simply does not work:

1. Corpus scale: everything we could collect amounts to 784M tokens, of which only 5.4% is actually Cantonese.
2. No teacher model: no open-weight model produces natural colloquial Cantonese, so there is nothing to distil from.
3. Zero reasoning traces: native Cantonese chain-of-thought data does not exist; traces can only be machine translated.
4. Evaluation blind spots: the existing Cantonese benchmarks are all multiple-choice — a model can reason in entirely the wrong language and still post excellent scores.

None of these constraints is unique to Cantonese; they are the shared condition of most low-resource languages. In that sense the report doubles as an experiment log for building reasoning models in any low-resource language.

¶Five Training Stages, Each Fixing the Previous One's Failure

The v2 pipeline has five stages: continuous pre-training (CPT) → chat-vector merging → supervised fine-tuning (SFT) → direct preference optimisation (DPO) → reinforcement learning with verifiable rewards (RLVR). Rather than a recipe designed up front, each stage was motivated by a failure diagnosed in the checkpoint before it. After every stage, the models were evaluated on HKCanto-Eval plus a set of generation probes checking the presence, length, language and script of the reasoning block.

Continuous Pre-Training (CPT)

The CPT corpus totals 784M tokens across 568K rows, trained on 64 TPU v6e chips with MaxText. By composition: Common Crawl written Chinese accounts for 27.5%, English replay data (Nemotron) 20.9%, web fiction and creative writing 17.1%, encyclopaedic text 12.5%, news and synthetic commentary 8.4% — and Common Crawl Cantonese just 5.4%, once again reflecting how low-resource the language is. Along the way we processed thirteen years of Common Crawl snapshots (2013-20 through 2025-38), yielding 477,298 deduplicated Cantonese documents; this dataset has also been released.

This stage produced a finding worth highlighting: in the learning-rate sweep on the 30B-A3B model, final training loss anti-correlated with downstream benchmark scores — the run with the lowest loss (1.52) actually scored below the untrained base model. This is the signature of domain overfitting: with a corpus this small, the closer the model fits the data, the more general capability it gives up. In other words, in low-resource CPT, training loss cannot be used as a model-selection signal.

Chat Vector Merging

After CPT the models had Hong Kong knowledge but no instruction-following behaviour. The conventional fixes are SFT or distillation, but the former needs large volumes of well-labelled data and the latter needs a teacher model — we had neither. The Chat Vector method offers a shortcut: subtract the base model's weights from the official chat model's weights to isolate an "instruction vector", then add it directly onto our CPT checkpoint:

Δ_chat = θ_instruct − θ_base
θ_cv   = θ_cpt + Δ_chat

The whole operation costs zero training. The merged 8B model actually beat the official Qwen3 8B on HKCanto-Eval by 3.18%; the 30B-A3B regressed slightly, by 2.55%.

But the multiple-choice scores concealed a problem: the language of the reasoning trace is inherited wholesale from the donor model. Probed with machine-translated GSM8K questions, the merged models produced correct answers while reasoning in Simplified Chinese — using 只 (zi2) as the classifier for eggs where idiomatic Cantonese requires 隻 (zek3), or 現在 where a Cantonese speaker would write 而家. This is exactly the failure mode that multiple-choice benchmarks cannot see.

Supervised Fine-Tuning (SFT)

SFT was meant to install the capabilities that merging cannot supply: translation between written Chinese and Cantonese, data curation, and LLM-as-a-judge skills — the abilities needed for the models to help produce the next iteration of the training corpus. The mixture comprised 74,865 rows and 177.4M tokens, 52.2% of which were reasoning tokens.

The cost was severe: SFT all but destroyed the models' reasoning. Because Qwen3 8B is a hybrid model, we stripped the reasoning field from 25% of the training rows to preserve its non-reasoning mode — and the empty <think></think> pair became a low-loss attractor. The 8B checkpoint emitted an empty reasoning block on 64.5% of generations, and the 30B-A3B's reasoning length collapsed to roughly a tenth of the original (the official model averages 1,042 tokens per probe; after SFT, 158). Benchmark averages fell by 20.52 points at 8B and 12.46 points at 30B-A3B.

Direct Preference Optimisation (DPO)

DPO was applied as a targeted repair for the defect diagnosed above. With no resources for human annotation, all preference pairs were generated by the SFT checkpoints themselves and scored by Gemini 3.5 Flash, yielding 12,204 pairs for the 8B model and 14,679 for the 30B-A3B.

DPO successfully restored the format of the reasoning block: the 8B average recovered by 10.19 points, and the 30B-A3B's runaway repetition loops (endlessly cycling "但係等等⋯⋯可能問題係⋯⋯" until hitting the token limit) largely disappeared. But only about half of the lost capability came back — a format can be repaired with preference pairs; problem-solving ability cannot.

One incidental finding: we had assumed that a verbose system prompt demanding detailed, comprehensive answers would produce better responses to serve as the chosen side of each pair. The judge scores said the opposite — the plain prompt beat the verbose one at both model sizes. A prompt perturbation label is not a quality label.

Reinforcement Learning with Verifiable Rewards (RLVR)

This is the core contribution of the report. None of the preceding stages could make the models reason in Cantonese; RLVR did, using GRPO with a multiplicative language-and-script reward:

R = m_lang × g_fmt × (w_t·r_t + w_a·r̄_a)

Here g_fmt is a hard format gate (exactly one non-empty reasoning block required), r_t is the task reward (e.g. exact numeric verification for maths), and the crucial term is the language multiplier m_lang:

Condition	Multiplier
Reasoning dominated by Latin script	0
Cantonese or mixed Chinese reasoning	1.0
Written (Mandarin-style) Chinese reasoning	0.1
Simplified-character share ≥ 20%	0

Making the language signal a multiplier rather than an additive term is deliberate: as an additive term, the penalty for the wrong language would be smaller than the penalty for a wrong answer, and the policy would happily trade language for correctness. As a multiplier, the target language becomes non-negotiable — no amount of verified correctness can recover the loss.

Training ran in two stages. Stage one used GSM8K arithmetic (50% Cantonese, 25% written Chinese, 25% English translations) to install the format and language behaviour. Stage two broadened the same reward across six environments — mathematics, code generation, STEM multiple-choice, workplace assistant, plus two newly built specifically for Cantonese: rule-based instruction following (52 verifiable constraint types covering traditional-script purity, colloquial classifiers, sentence-final particles, four-character idioms, and even Jyutping rhyme) and structured outputs (JSON/YAML/TOML conversion tasks grounded in Hong Kong news articles). These environments are open-sourced on GitHub.

The difficulty-profiling pass before stage two also produced the clearest evidence in the whole report: given the same questions in English and in Cantonese, the maths pass rate drops by 0.247 and code generation by 0.105, while structured outputs (−0.012) and instruction following (−0.003) transfer almost losslessly. The cross-language reasoning gap is real and quantifiable, and it concentrates in reasoning-heavy tasks like maths and code.

¶Benchmark Results

The table below shows every checkpoint in the pipeline on HKCanto-Eval (Δ is relative to the official chat model):

Model	MMLU	CantoMMLU	Cultural	Linguistic	Academic & Prof.	Avg	Δ
Qwen3 8B	81.08	76.68	57.94	39.50	81.61	67.36	-
8B Chat Vector	80.04	76.96	66.67	41.50	82.36	69.51	+3.18%
8B SFT	64.23	56.15	47.62	23.00	53.97	48.99	-27.27%
8B DPO	69.19	61.85	59.52	34.00	71.35	59.18	-12.14%
8B GRPO	73.86	69.73	58.33	36.00	76.08	62.80	-6.77%
Qwen3 30B-A3B Thinking 2507	86.65	82.52	68.65	57.00	86.70	76.30	-
30B-A3B Chat Vector	80.71	80.26	70.24	55.00	85.59	74.36	-2.55%
30B-A3B SFT	57.78	64.77	69.05	47.50	70.29	61.90	-18.87%
30B-A3B DPO	73.71	62.47	67.86	48.00	68.82	64.17	-15.90%
30B-A3B GRPO	84.26	76.24	65.06	56.50	83.70	73.16	-4.13%

The final 30B-A3B model climbs back to 73.16, within 1.20 points of the chat-vector merged checkpoint it started from (74.36) — but with one crucial difference: the merged checkpoint never reasoned in Cantonese, whereas the final model thinks and answers in Cantonese, and has additionally learned translation and data-curation skills that multiple-choice benchmarks cannot measure. The 8B model's recovery is only partial (still 6.71 points below its merged checkpoint), suggesting that teaching a smaller model to reason in a new language costs more.

The compute breakdown is telling: SFT and DPO together used just 283 GPU-hours, while RLVR consumed 1,697 GPU-hours (not counting a further 2,472 spent on three failed 30B-A3B attempts). Getting a model to reason in Cantonese is, overwhelmingly, a reinforcement-learning bill.

¶Released Resources

This work releases more than model weights:

• Four model checkpoints (two per size: the chat-vector merged models and the final Thinking models) — HuggingFace Collection
• A thirteen-year Traditional Chinese Common Crawl dataset, released per snapshot, containing Cantonese and some written Chinese material
• The Cantonese RL training environments (cantonese-nemo-gym-environments)
• The machine-translated Cantonese and written Chinese RL data (Nemotron-3-Nano-RL-Training-Blend-STEM-Yue-Translated)

¶Closing Thoughts

Back in the v1 days, we trained a model that could speak Cantonese on 200 million tokens; two years later, with 784M tokens and five stages of post-training, v2 is a model that thinks in Cantonese. The deepest lesson along the way: the problems of a low-resource language resurface at every link of the pipeline in a different guise — loss cannot be trusted, teacher models do not exist, reasoning data must be translated, and benchmarks are blind to language failures — and each one demands its own diagnosis and design.

The fundamental constraint still stands: there is no long-form native Cantonese reasoning corpus anywhere in the world. But the chicken-and-egg deadlock now has an exit — v2 is the first model capable of generating and translating Cantonese reasoning traces, and those traces will become the training material for the next generation. As we wrote in the v1 report, we still believe the future of large language models should be driven by the open-source community; and Cantonese, on that road, has just taken one more step forward.