AQLM+PV
Collection
Official AQLM quantizations for "PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression": https://arxiv.org/abs/2405.14852
•
21 items
•
Updated
•
15
Official AQLM quantization of meta-llama/Meta-Llama-3.1-8B-Instruct finetuned with PV-Tuning.
For this quantization, we used 1 codebook of 16 bits and groupsize of 8.
Results:
| Model | Quantization | MMLU (5-shot) | ArcC | ArcE | Hellaswag | PiQA | Winogrande | Model size, Gb |
|---|---|---|---|---|---|---|---|---|
| meta-llama/Meta-Llama-3.1-8B-Instruct | None | 0.6817 | 0.5162 | 0.8186 | 0.5909 | 0.8014 | 0.7364 | 16.1 |
| 1x16g8 | 0.5966 | 0.4667 | 0.7790 | 0.5582 | 0.7878 | 0.7001 | 4.1 |
Note
We used lm-eval=0.4.0 for evaluation.
UPD (09.08.2024)
Uploaded new version finetuned on more data for longer with better quality.
- Downloads last month
- 319
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.
Model tree for ISTA-DASLab/Meta-Llama-3.1-8B-Instruct-AQLM-PV-2Bit-1x16-hf
Base model
meta-llama/Llama-3.1-8B
Finetuned
meta-llama/Llama-3.1-8B-Instruct