Watch

mirror of https://github.com/ggml-org/llama.cpp.git synced 2026-11-10 05:18:46 +00:00

	CPU (AVX/AVX2)	CPU (ARM NEON)	Metal	CUDA	ROCm	SYCL	Vulkan
K-quants	✅	✅	✅	✅	✅	✅	✅⁵
I-quants	✅ 🐢⁴	✅ 🐢⁴	✅ 🐢⁴	✅	✅	Partial¹	✅ 🐢⁴
Parallel Multi-GPU	N/A	N/A	N/A	✅	✅	Sequential only	✅
K cache quants	✅	✅	✅	✅	✅	❓	✅
MoE architecture	✅	✅	✅	✅	✅	❓	✅
Flash Attention	✅	✅	✅	✅	✅	❓	✅

✅: feature works
🚫: feature does not work
❓: unknown, please contribute if you can test it yourself
🐢: feature is slow
¹: IQ3_S and IQ1_S, see #5886
⁴: Slower than K-quants of comparable size
⁵: Generally the CUDA or ROCM backends are faster than Vulkan in most cases. See #10879 for benchmarks.

Users Guide

Useful information for users that doesn't fit into Readme.

Technical Details

These are information useful for Maintainers and Developers which does not fit into code comments

Github Actions Main Branch Status

Click on a badge to jump to workflow. This is here as a useful general view of all the actions so that we may notice quicker if main branch automation is broken and where.