local-inference

colibri Hit 30,600 Stars by Streaming a 2.8 Trillion Parameter Model Off an SSD

A pure C engine with zero dependencies that runs 744B to 2.8T parameter MoE models by treating your disk as the bottom tier of a memory hierarchy. It gained 30,600 stars in 75 days.