colibri Hit 30,600 Stars by Streaming a 2.8 Trillion Parameter Model Off an SSD
A pure C engine with zero dependencies that runs 744B to 2.8T parameter MoE models by treating your disk as the bottom tier of a memory hierarchy. It gained 30,600 stars in 75 days.
The WJS Desk
Sep 14, 2026 · updated 10 days ago · 5 min read
The number that made us look
colibri went from zero to 30,600 stars between July 1 and September 14, 2026, averaging about 408 stars per day. On the day we checked, it was gaining 868 stars and sitting at #1 on GitHub Trending. The pitch: run frontier Mixture-of-Experts models on hardware you already own, in pure C, with zero engine dependencies.
That is not a typo. Nine model families are supported, from the 7B OLMoE to the 2.8 trillion parameter Kimi K3. The trick is that MoE models only activate a fraction of their parameters per token. GLM-5.2 has 744 billion parameters but activates about 40 billion per forward pass, a 13:1 sparsity ratio. colibri exploits that by keeping the shared backbone in RAM and streaming experts from disk on demand.
What it actually is
colibri is a single-binary inference engine written in C with no external dependencies. It reads model checkpoints natively (no conversion to GGUF or other intermediate formats) and treats storage, RAM, and VRAM as a unified memory hierarchy. Experts that fit in VRAM stay there; overflow goes to RAM; overflow from RAM streams from NVMe.
The core API is minimal. You point it at a model directory and it runs:
./colibri chat --model ~/models/glm-5.2 --ctx 8192There is also a cluster mode (via a companion project called lumabri) for distributing expert execution across multiple machines on a LAN. The web UI supports multi-turn chat, a reasoning depth selector for GLM-5.3, and session persistence.
Nine model families run today: GLM-5.2 and 5.3 (744B), GLM-5.3-Flash (321B with vision), Inkling (975B), Kimi K3 (2.8T), DeepSeek V4 Flash (284B), DeepSeek V4.1 Flash (552B with vision), Qwen3.8-Flash-Next (125B), Qwen3.6 (35B), and OLMoE (7B).
The speed question, answered honestly
colibri does not compete on speed and does not pretend to. The project's own benchmarks are unusually transparent about this:
| Setup | Model | Decode (tok/s) | Notes |
|---|---|---|---|
| 6x RTX 5090, full residency | GLM-5.2 | 5.8 to 6.8 | All experts in VRAM |
| Apple M5 Max | GLM-5.2 | ~1.06 | Unified memory helps |
| 128 GB CPU desktop, warm | GLM-5.2 | ~1.8 | Experts cached in RAM |
| RTX 5070 Ti laptop | GLM-5.2 | 1.07 | Mixed disk/VRAM |
| 25 GB developer box (cold) | GLM-5.2 | 0.05 to 0.1 | Streaming from NVMe |
For comparison, llama.cpp running a quantized 70B dense model on equivalent hardware typically produces 8 to 15 tok/s. colibri is running a model ten times larger, so the comparison is unfair in both directions: colibri is slower per token, but it is running a model that llama.cpp simply cannot load.
The v1.11.0 release (September 13) pushed DeepSeek V4.1 Flash inference from 0.305 to 0.957 tok/s through batched expert reads. That is a 3x improvement in a single release cycle.
What it does not do
- Fast interactive chat on cheap hardware. At 0.05 to 0.1 tok/s on a 25 GB box, a 500-token response takes 80 to 170 minutes. This is a research and batch tool at that tier, not a conversation partner.
- Dense models. The entire architecture assumes MoE sparsity. If your model is dense, llama.cpp, vLLM, or SGLang are better choices.
- AMD GPUs natively. CUDA support exists; ROCm does not. Metal acceleration landed in v1.11.0 for Apple Silicon, but AMD discrete GPU users are out.
- Training or fine-tuning. Inference only.
- Quantization. It reads native checkpoint formats (fp8 dense, fp4 experts). There is no GGUF conversion, which saves a step but also means you cannot trade quality for speed the way you can with llama.cpp's quantization tiers.
Hidden cost: The disk footprint is enormous. GLM-5.2 needs roughly 372 GB of disk space for the checkpoint. Kimi K3 is larger. If you are running this on a laptop with a 512 GB SSD, you are giving up most of your storage to one model. NVMe read speed directly limits throughput, so a cheap SATA SSD will make the already-slow streaming even slower.
Who made it, and the bus factor
JustVugg is the primary author with 1,195 commits, which is 73% of total contributions. Nine other contributors have landed meaningful work: ZacharyZcR (164 commits), monotophic (114), steve-m (60), woolcoxm (55), and five more in the 20 to 40 range. That is a healthier contributor distribution than most trending repos we cover, though it is still firmly a single-maintainer project in practice.
The release cadence is aggressive: five releases in the last 17 days (v1.9.0 through v1.11.0). The codebase is roughly 15 MB and written in C, which means it is auditable by anyone who reads C and small enough to vendor if the project stalls. The Apache 2.0 license makes vendoring straightforward.
The project is 75 days old and has 127 open issues, which is a normal ratio for a fast-growing repo but worth monitoring. Active maintenance is happening: the v1.11.0 release closed 56 merged PRs including memory leak fixes, VRAM accounting corrections, and a Metal crash.
Verdict
colibri is not a replacement for llama.cpp. It is a new category: an engine for models that are too large to fit in memory at any quantization level. If you have a use case where model quality matters more than latency (offline analysis, batch processing, research), and you have the disk space, colibri lets you run models that previously required a multi-GPU server.
If you need interactive chat, you need either the 6x RTX 5090 setup (which defeats the "hardware you already own" pitch) or a model small enough for llama.cpp. The sweet spot right now is the 128 GB RAM desktop tier, where warm inference hits 1.8 tok/s on a 744B model. That is slow but usable for many workflows.
We would adopt today for batch workloads on machines with 64+ GB RAM and fast NVMe. We would wait for AMD GPU support and better disk-tier performance before recommending it for anything interactive.
Your turn
If you have tried running any MoE model locally, what hardware are you on and what throughput did you actually see? The README benchmarks are specific, and we are curious whether they hold outside the maintainer's test rigs.
For another repo that promised to replace cloud infrastructure with local hardware, see our piece on Fastpotify going from 2,848 to 3,801 stars and what we found when we tried to build it.
