Qwen3.8-2.4T on vLLM: a Pareto frontier spanning 5K total tokens/s/GPU at high…
This is a AI post classified by Jev as AI infra & evals (a tutorial), kept by the AI Radar because it carries real work, not commentary.
Qwen3.8-2.4T on vLLM: a Pareto frontier spanning 5K total tokens/s/GPU at high throughput and 180 output tokens/s/user at low latency, across tuned PD configurations on @nvidia GB300 NVL72. Workload: 8K input / 1K output. Drawing on lessons from trial and error, we walk through the tuning decisions step by step: budget KV cache, benchmark prefill and decode separately, then choose topologies and MTP settings for each serving target. Deployment configs are included so you can reproduce the results. Great work from the @NVIDIAAI contributors and the vLLM community! Explore the frontier: https
Posted by vLLM (49.8k followers) 1 h ago · 14 likes · 1.2k views · view the original post on X. Kept by the AI Radar as AI infra & evals.
More AI work like this
- Stay on top of model changes, low credits, workspace budgets, and API key limits with… — @OpenRouter
- BeeLlama v0.4.7 Preview just made DFlash 2 by @inco_ai much more efficient in terms of… — @Anbeeld
- Me looking at "Apple's M5 Ultra Mac Studio Shines in Local AI Tests" trending on X with… — @ivanfioravanti
- Your users have already generated a better eval dataset than you'd write yourself. — @sentry
- 🚀 Introducing Luminary, our AI wind tunnel for enterprise model selection. — @MultiverseCompu
- cool how Hugging Face made tokenizers 15.9× faster without changing a single token ID: — @victormustar
- i gave Apodex 1.1 a messy synthetic benchmark, then changed the workload mid-run. — @chetaslua
- GLM 5.3 Flash EXL3 on 2x DGX Spark: ~15% faster long-prompt prefill! — @plotarmordev
Every post is read and classified by Jev (TypeSafe): what it is, which market it belongs to, and whether the link is a real tool. 38.9k posts from 5k X accounts over the last 14 days, 1.6k tools, 19 markets. Collected every 5 minutes, fully re-ranked every hour — last update 2026-09-21 15:58 UTC. Full method.