Got Arete 27B running blazing fast on my 10GB RTX 3080!
This is a AI post classified by AGK Intelligence as Open & local models (a tutorial), kept by the AI Radar because it carries real work, not commentary.
Got Arete 27B running blazing fast on my 10GB RTX 3080! The setup processed a 50,000-token prompt at about 1,042 tokens/s and generated at about 32 tokens/s. The main choices were the small IQ2_XXS model from Bartowski, 51k context, a Q4_0 KV cache, Flash Attention, and keeping all 65 transformer layers on the GPU. To fit the working buffers, llama.cpp left the roughly 521 MB input embedding table in system RAM. This put put GPU use at about 9.9 GiB. A fully GPU-resident attempt ran much slower, around 22–24 tokens/s, with shared GPU memory in use. Example command to use: llama-server \
Posted by GooGZ AI (1.6k followers) 2 h ago · 9 likes · 349 views · view the original post on X. Kept by the AI Radar as Open & local models.
More AI work like this
- The Hardware Cup final! 🏆 — @niklaslenz_ai
- > GameFeel Bench has about 60,000 tasks about game balancing and understanding features — @teortaxesTex
- Another magic by DeepSeek v4.1 + Blender MCP + DeepSeek Harness running locally! — @ivanfioravanti
- ok this one's nerdy but I love it. Josef Gattermayer wrote AMD kernels for EXL3, the… — @MichaelGannotti
- 🥱 Tired of your #ai open-weight #LLM answering free speech questions like it's… — @michabbb
- You can now Run Uncensored DeepSeek-V4-Pro-Qwen3.5-9B locally on a 5 GB Laptop with zero… — @0x0SojalSec
- You can run locally Uncensored "DeepSeek-V4-Pro-Qwen3.5-9B" 5.6GB. — @0x0SojalSec
- Meet the Qwen3.8-27B Ultimate Heretic Uncensored Turbo Brilliance 50x AGI NEO GGUF. It's… — @HuggingModels
Every post is read and classified by AGK Intelligence: what it is, which market it belongs to, and whether the link is a real tool. 65.2k posts from 5k X accounts over the last 14 days, 10.6k tools, 19 markets. New posts are collected several times an hour and the full ranking is recomputed every hour — last update 2026-11-08 00:39 UTC. Full method.