Hardware Requirements
🤖 Text-based AI LLM
4-bit System Requirements
| Model | Minimum Total VRAM | Card Examples | RAM/Swap to Load* |
|---|---|---|---|
| 7B | 6GB | GTX 1660, 2060, AMD 5700 XT, RTX 3050, 3060 | 6 GB |
| 13B | 10GB | AMD 6900 XT, RTX 2060 12GB, 3060 12GB, 3080, A2000 | 12 GB |
| 30B | 20GB | RTX 3080 20GB, A4500, A5000, 3090, 4090, 6000, Tesla V100 | 32 GB |
| 65B | 40GB | A100 40GB, 2x3090, 2x4090, A40, RTX A6000, 8000 | 64 GB |
8-bit System Requirements
| Model | VRAM Used | Minimum Total VRAM | Card Examples | RAM/Swap to Load* |
|---|---|---|---|---|
| 7B | 9.2GB | 10GB | 3060 12GB, 3080 10GB | 24 GB |
| 13B | 16.3GB | 20GB | 3090, 3090 Ti, 4090 | 32 GB |
| 30B | 36GB | 40GB | A6000 48GB, A100 40GB | 64 GB |
| 65B | 74GB | 80GB | A100 80GB | 128 GB |