Kimi K3 (2.8T) runs on a single card in 3.72GB
Kimi K3 is the largest open-source model released to date. AirLLM runs it on one consumer-class GPU.
Measured end to end on a single RTX 6000 Ada (48GB) against the full 1.56TB checkpoint, generating real tokens:
|
|
| Peak VRAM during generation |
3.72 GB |
| Peak VRAM after init |
0.83 GB |
| Init (one-time per process) |
900 s |
| Generation |
292 s/token, disk-bound |
The reason a 2.8T model needs less VRAM than a 671B one is that sparse MoE checkpoints stream one expert at a time rather than a whole layer. K3 holds 896 experts per layer and routes each token to 16 of them — expanded, a layer's experts are ~55GB, but a token only needs ~1GB. AirLLM loads just those. MXFP4 weights also cross PCIe packed and expand on the GPU, moving 4x less data.
Fitting the checkpoint on disk needed the same kind of trick: a naive split would want 3.12TB for a 1.56TB model. K3's shards turn out to be pure, one module each, so split layers are hardlinked to the originals instead of copied.
Before you run K3
K3 brings three requirements of its own, none of them optional:
pip install airllm compressed-tensors flash-attn
- flash-attn is mandatory. K3's model code overwrites whatever attention implementation you request.
- A CUDA 12 build of torch, because no prebuilt flash-attn wheel exists for CUDA 13 yet.
- transformers 4.56.x, as K3's remote code does not load on 5.x.
Also in this release
Four fixes that came out of the K3 work but are not K3-specific, and should help other models:
- Adopt the checkpoint's shape when a model class builds a different one for a parameter it never materialises.
- Expand packed weights for modules whose forward reads a plain
weight rather than the packed payload.
- Read
scheme.format across compressed-tensors versions, where it is an enum in some and a plain string in others.
- Propagate the attention implementation into nested sub-configs, which multimodal wrappers keep their decoder under.
Full details in #316.
详细ChangeLog
AirLLM v3.0.0
Big update: AirLLM now runs today's largest open models on tiny GPUs, with full support for the latest model families and Hugging Face versions — still no quantization, distillation, or pruning required.
Highlights
- Run the biggest open models on a single small GPU. Stream 70B models on 4GB, 405B Llama 3.1 on 8GB, and even DeepSeek-V3 (671B) on ~12GB.
- Native FP8 support. Pre-quantized FP8 (block-FP8) checkpoints now load and run correctly — including DeepSeek-V3 and the Qwen3-FP8 family.
- Latest models supported, including Qwen3 (dense + MoE, e.g. Qwen3-32B, Qwen3-30B-A3B, Qwen3-235B-A22B-FP8), DeepSeek-V3, Phi-4, Mixtral-8x7B, and DeepSeek-V2-Lite.
- Up to date with modern Hugging Face. Works with current
transformers / accelerate releases, so a plain pip install airllm just works — no manual dependency juggling.
Improvements & fixes
- Reworked layer streaming to build on the standard Transformers model path for better model compatibility and
generate() behavior.
- Runtime precision now follows each model's native dtype (e.g. bfloat16) instead of being forced to fp16, fixing garbled output on very deep models.
- Fixed weight loading for layers whose tensors span multiple checkpoint shards (affected large FP8/MoE models).
- More robust shard naming and attention-implementation fallback.
Install / upgrade
pip install --upgrade airllm
See the README for quickstart and the full list of supported models.
详细ChangeLog
