thoughtful work in pre-training. they test every recipe change on a ladder of small…
This is a AI post classified by AGK Intelligence as AI research (research), kept by the AI Radar because it carries real work, not commentary.
thoughtful work in pre-training. they test every recipe change on a ladder of small models (about 300M to 6B active params) before trusting it for the final run but the ladder is only useful if the small models face the same conditions as the big one. like this approach differs for data. if taken a small ladder model trained on (lets say 238B tokens which is 1% of the final run). if you sample its data from the full 13.7T pool, it never sees anything twice and the pool is about 58 times bigger than its budget. so the small model is always training on fresh data and the big one is not. they f
Posted by himanshu (29.4k followers) 19 h ago · 124 likes · 7.5k views · view the original post on X. Kept by the AI Radar as AI research. Tools mentioned: reflection.ai.
More AI work like this
- 3rd place ARC-AGI-2 Public Leaderboard submission open sourced by @gonedarragh and… — @GregKamradt
- 按 token 计费的前提是 token 一样贵，但真实情况是：前面一个 token 会抬高后面所有 token 的能耗。 — @qingke_ai
- this is free f*cking gold — @Mr_Buzzoni
- Zyphra's new paper proposes Zyphra Hierarchical Attention (ZHA), which organizes the… — @askalphaxiv
- New AI rigger that handles hard stuff like skirts, capes, hair and sleeves. — @Stefan_3D_AI
- How transformers work. — @pmddomingos
- Sparse attention reduces computation by selecting relevant tokens, but finding those… — @askalphaxiv
- Hello to everyone from all over the world, — @rlaope
Every post is read and classified by AGK Intelligence: what it is, which market it belongs to, and whether the link is a real tool. 65.2k posts from 5k X accounts over the last 14 days, 10.6k tools, 19 markets. New posts are collected several times an hour and the full ranking is recomputed every hour — last update 2026-11-07 16:45 UTC. Full method.