How I Read a 389B MoE Launch Without Believing the Parameter Count

By Elena Volkov, Machine Learning Research Editor

Every few months a lab announces the “largest open-source MoE.” Tencent’s Hunyuan-Large—about 389B total parameters with roughly 52B activated, long context claims up to 256k, commercial-friendly licensing talk, and slides comparing favorably to Llama 3.1-class dense giants—is a clean specimen of that genre. I read architecture papers for a living. My job is not to cheer the integer. It is to ask whether the release changes what I can train, serve, and reproduce.

I have not personally served Hunyuan-Large at production QPS for this article. Claims below about benchmarks and scaling formulas are as stated by the release materials unless I mark them as my interpretation.

What I extract from the press kit in five minutes

1. Sparsity shape: total vs activated params (389B / ~52B). Activated count is the inference story; total is the capacity story.
2. Artifacts shipped: pretrain, SFT, and FP8-quantized SFT—if real, that is more useful than a single “demo” checkpoint.
3. Lineage claim: “same family as internal apps” (e.g., long-doc features). I treat that as a hint about data recipes, not a guarantee of quality.
4. MoE methodology hooks: shared + routed experts, load-balancing / “recycling” ideas, CLA-style KV savings, a published MoE scaling relation.

If item 2 is missing, I stop. Parameter count without runnable artifacts is theater.

My MoE reading checklist

Activated FLOPs vs marketing FLOPs

Dense comparisons that ignore activation sparsity lie by omission. When a slide says “beats Llama 3.1 405B,” I ask: same eval harness? same contamination controls? same decoding settings? Until I see a reproducible table, I park the claim under “vendor-reported.”

Routing tax

Tencent’s write-up sketches a MoE cost form in the spirit of (C \approx 9.59ND + 2.3\times10^8 D)—a higher coefficient than the classic dense (6ND) style rule-of-thumb, plus a long-context attention term. My take: routing and long sequences are not free. If your serving stack cannot absorb expert imbalance, the “52B activated” sticker price becomes fiction at tail latency.

Shared expert + specialized expert

Hybrid routing (one shared expert always on, plus routed specialists) is a familiar stability bet. I look for evidence of load metrics and failed-expert behavior—not just a diagram. “Expert recycling” language is promising if it reduces dropped tokens; I still want ablations.

Why ~52B activated?

Their isoFLOP-style exploration reportedly pointed near ~58B optimal activation under their assumptions; 52B is a practical settle. That is the kind of detail I respect: an engineering compromise, not a round marketing number. I still want the curves, not the bedtime story.

Synthetic data honesty

Heavy synthetic data can boost math and tool-like behavior. It can also overfit to generator quirks. My rule: show generator family, filters, and a contamination check against the eval set—or I assume the leaderboard is optimistic.

Quantized release as a seriousness test

Shipping FP8 SFT weights signals they expect real deployers, not just download tourists. My next question is always: quality drop table vs BF16 on the same tasks.

What would change my mind in either direction

Upgrade trust: independent eval reproduction; clear commercial license text; serving recipes with measured tokens/sec and expert imbalance; training code that matches the report.

Downgrade trust: benchmark-only launch; missing quant quality tables; silent license caveats; MoE that collapses under realistic batch sizes.

Bottom line

I do not “believe” 389B. I believe activated compute, artifacts, routing behavior, and eval hygiene. Hunyuan-Large is worth a lab weekend if you already operate MoE serving. It is not worth rewriting your stack because a slide outscored Llama on a vendor chart. Read the sparsity, demand the artifacts, and keep a written “what I have not verified” list—mine starts with full independent benchmark reproduction.