cool how Hugging Face made tokenizers 15.9× faster without changing a single token ID:
This is a AI post classified by Jev as AI infra & evals (a launch), kept by the AI Radar because it carries real work, not commentary.
cool how Hugging Face made tokenizers 15.9× faster without changing a single token ID: → SIMD splitter instead of a regex engine → cache for repeated words → merge loop that never allocates → batched model calls Same API. Rust RC out now 🚀
Posted by Victor M (30.7k followers) 1 h ago · 26 likes · 1.1k views · view the original post on X. Kept by the AI Radar as AI infra & evals. Tools mentioned: tokenizers-v1.
More AI work like this
- Stay on top of model changes, low credits, workspace budgets, and API key limits with… — @OpenRouter
- BeeLlama v0.4.7 Preview just made DFlash 2 by @inco_ai much more efficient in terms of… — @Anbeeld
- Me looking at "Apple's M5 Ultra Mac Studio Shines in Local AI Tests" trending on X with… — @ivanfioravanti
- Your users have already generated a better eval dataset than you'd write yourself. — @sentry
- Qwen3.8-2.4T on vLLM: a Pareto frontier spanning 5K total tokens/s/GPU at high… — @vllm_project
- 🚀 Introducing Luminary, our AI wind tunnel for enterprise model selection. — @MultiverseCompu
- i gave Apodex 1.1 a messy synthetic benchmark, then changed the workload mid-run. — @chetaslua
- GLM 5.3 Flash EXL3 on 2x DGX Spark: ~15% faster long-prompt prefill! — @plotarmordev
Every post is read and classified by Jev (TypeSafe): what it is, which market it belongs to, and whether the link is a real tool. 38.9k posts from 5k X accounts over the last 14 days, 1.6k tools, 19 markets. Collected every 5 minutes, fully re-ranked every hour — last update 2026-09-21 15:58 UTC. Full method.