18 of the 20 best open models today are Mixture-of-Experts. Dense is basically over.
This is a AI post classified by AGK Intelligence as AI infra & evals (a free resource), kept by the AI Radar because it carries real work, not commentary.
18 of the 20 best open models today are Mixture-of-Experts. Dense is basically over. And yet the know-how to train a good MoE still lives mostly in the heads of a handful of teams. So we spent 800k GPU hours running the experiments ourselves and wrote everything down. Super happy to finally release "The Ultimate MoE Training Handbook", free and open, organized around the questions training teams hit: - how many experts, how big, shared or not? - why is half my step spent in all-to-all, and how do I hide it? - why is my run 25x slower with no error in the logs? - softmax, sigmoid or sqrt(soft
Posted by Thomas Wolf (137.3k followers) 19 h ago · 3.1k likes · 149k views · view the original post on X. Kept by the AI Radar as AI infra & evals. Tools mentioned: moe-blogpost.
More AI work like this
- A small model coaching a giant? Aiona checks the held-out tie and SMF’s memory-card… — @MichaelGannotti
- Ranking inference providers six ways — @AdamHoltererer
- How are Decision Models Trained? — @NaceAI
- Nemotron 3.5 Transcribe from @NVIDIAAI is very fast with @vllm_project! — @vanstriendaniel
- The new GLM-5.3-Flash recipe on 2× DGX Spark — now with the cache fix. — @jmurillocode
- You would think that there's a trade-off between speed and accuracy when it comes to… — @AdamHoltererer
- 太卷了吧！OpenDesign 这波是不是疯了？ — @NFT_Chen
- Guess the model that delivers this cloud TPS? — @wrldsuksgo2mars
Every post is read and classified by AGK Intelligence: what it is, which market it belongs to, and whether the link is a real tool. 65.6k posts from 5k X accounts over the last 14 days, 10.4k tools, 19 markets. New posts are collected several times an hour and the full ranking is recomputed every hour — last update 2026-11-06 15:10 UTC. Full method.