Hugging Face's logo

bartowski
/
Mistral-quiet-star-demo-GGUF

Text Generation
Transformers
GGUF
English
text-generation-inference
unsloth
mistral
trl
Inference Endpoints
Model card Files Community
1
bartowski commited on
Commit
4cf9564
•
1 Parent(s): 418a735

Llamacpp quants

Browse files
Files changed (18) hide show
  1. .gitattributes +16 -0
  2. Mistral-quiet-star-IQ3_M.gguf +3 -0
  3. Mistral-quiet-star-IQ3_S.gguf +3 -0
  4. Mistral-quiet-star-IQ4_NL.gguf +3 -0
  5. Mistral-quiet-star-IQ4_XS.gguf +3 -0
  6. Mistral-quiet-star-Q2_K.gguf +3 -0
  7. Mistral-quiet-star-Q3_K_L.gguf +3 -0
  8. Mistral-quiet-star-Q3_K_M.gguf +3 -0
  9. Mistral-quiet-star-Q3_K_S.gguf +3 -0
  10. Mistral-quiet-star-Q4_0.gguf +3 -0
  11. Mistral-quiet-star-Q4_K_M.gguf +3 -0
  12. Mistral-quiet-star-Q4_K_S.gguf +3 -0
  13. Mistral-quiet-star-Q5_0.gguf +3 -0
  14. Mistral-quiet-star-Q5_K_M.gguf +3 -0
  15. Mistral-quiet-star-Q5_K_S.gguf +3 -0
  16. Mistral-quiet-star-Q6_K.gguf +3 -0
  17. Mistral-quiet-star-Q8_0.gguf +3 -0
  18. README.md +45 -0
.gitattributes CHANGED
@@ -33,3 +33,19 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ Mistral-quiet-star-IQ3_M.gguf filter=lfs diff=lfs merge=lfs -text
37
+ Mistral-quiet-star-IQ3_S.gguf filter=lfs diff=lfs merge=lfs -text
38
+ Mistral-quiet-star-IQ4_NL.gguf filter=lfs diff=lfs merge=lfs -text
39
+ Mistral-quiet-star-IQ4_XS.gguf filter=lfs diff=lfs merge=lfs -text
40
+ Mistral-quiet-star-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
41
+ Mistral-quiet-star-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
42
+ Mistral-quiet-star-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
43
+ Mistral-quiet-star-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
44
+ Mistral-quiet-star-Q4_0.gguf filter=lfs diff=lfs merge=lfs -text
45
+ Mistral-quiet-star-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
46
+ Mistral-quiet-star-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
47
+ Mistral-quiet-star-Q5_0.gguf filter=lfs diff=lfs merge=lfs -text
48
+ Mistral-quiet-star-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
49
+ Mistral-quiet-star-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
50
+ Mistral-quiet-star-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
51
+ Mistral-quiet-star-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
Mistral-quiet-star-IQ3_M.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5a69a593fa1b41efb7d8bf7ddfe3ab07ca4d077c89e0ca983efd0be3847084f8
3
+ size 3284891424
Mistral-quiet-star-IQ3_S.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:641ac189cd23acd6c5e3a487108ce70184ebccaeee3c0104126373e0e977c0b1
3
+ size 3182393120
Mistral-quiet-star-IQ4_NL.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:83087f0b07872e9875cd3dcd30b335cf75fd1cb9ec2845525b72b4b1a077ea21
3
+ size 4155053856
Mistral-quiet-star-IQ4_XS.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c67d0d1d395aa6adb2d38a331597ef36a87a79fabad9653c530190f00663ab45
3
+ size 3944388384
Mistral-quiet-star-Q2_K.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e426e44fca9a69f502bd0a4f9b894f68418e753a2ceae3ca52a262f6607fba39
3
+ size 2719242016
Mistral-quiet-star-Q3_K_L.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:65f69592dcbf74080b3e8cb3198d885c08a215e5430921702d6d7eb07629f8df
3
+ size 3822024480
Mistral-quiet-star-Q3_K_M.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c4bcfb40e99ad3be130ed0bd11725f7fcb55bf243a17a6e66dd189e4a368b101
3
+ size 3518986016
Mistral-quiet-star-Q3_K_S.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2e7794c60c0f1c888e50bf6441497432efd5fdd9577025de3b9f88df9df48878
3
+ size 3164567328
Mistral-quiet-star-Q4_0.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:58ac0ceaef4ed8722ff2a2fec00f6cf33083aacf64590af378ccd14f0c075958
3
+ size 4108916512
Mistral-quiet-star-Q4_K_M.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c2d29dc3ccf85d0b3fe933d0098d682c3d65b0b7081cc01711dfc3216d90e147
3
+ size 4368439072
Mistral-quiet-star-Q4_K_S.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:476de724e4c6b988c373fd78662361ba9306350e1867163e50aed48c822028d8
3
+ size 4140373792
Mistral-quiet-star-Q5_0.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:93fbd5fce10fdfaa768b373f0026084819120224bfce2327844d3635f30eaa80
3
+ size 4997715744
Mistral-quiet-star-Q5_K_M.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c9319ee735c3af54015b7f5854b0611cbf696a6f94e91cf8b4a2e6beaae8c7e7
3
+ size 5131409184
Mistral-quiet-star-Q5_K_S.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:47b3e923dc4da194ba5d11c7437c9bab5437a5750e0ef31ea6883609b3eeb2a5
3
+ size 4997715744
Mistral-quiet-star-Q6_K.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9f2730c6965fb2979b1f11c6133c8e5c3f3f4a4b9b59e68e5c433230d80e46e9
3
+ size 5942064928
Mistral-quiet-star-Q8_0.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:da3530ddebc3ed5cb4741fac16fde281aa3fc621a2ebc2682697e4db4fbc768f
3
+ size 7695857440
README.md ADDED
@@ -0,0 +1,45 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - en
4
+ license: apache-2.0
5
+ tags:
6
+ - text-generation-inference
7
+ - transformers
8
+ - unsloth
9
+ - mistral
10
+ - trl
11
+ base_model: unsloth/mistral-7b-bnb-4bit
12
+ datasets:
13
+ - gate369/Alpaca-Star
14
+ quantized_by: bartowski
15
+ pipeline_tag: text-generation
16
+ ---
17
+
18
+ ## Llamacpp Quantizations of Mistral-quiet-star
19
+
20
+ Using <a href="https://github.com/ggerganov/llama.cpp/">llama.cpp</a> release <a href="https://github.com/ggerganov/llama.cpp/releases/tag/b2536">b2536</a> for quantization.
21
+
22
+ Original model: https://huggingface.co/liminerity/Mistral-quiet-star
23
+
24
+ Download a file (not the whole branch) from below:
25
+
26
+ | Filename | Quant type | File Size | Description |
27
+ | -------- | ---------- | --------- | ----------- |
28
+ | [Mistral-quiet-star-Q8_0.gguf](https://huggingface.co/bartowski/Mistral-quiet-star-GGUF/blob/main/Mistral-quiet-star-Q8_0.gguf) | Q8_0 | 7.69GB | Extremely high quality, generally unneeded but max available quant. |
29
+ | [Mistral-quiet-star-Q6_K.gguf](https://huggingface.co/bartowski/Mistral-quiet-star-GGUF/blob/main/Mistral-quiet-star-Q6_K.gguf) | Q6_K | 5.94GB | Very high quality, near perfect, *recommended*. |
30
+ | [Mistral-quiet-star-Q5_K_M.gguf](https://huggingface.co/bartowski/Mistral-quiet-star-GGUF/blob/main/Mistral-quiet-star-Q5_K_M.gguf) | Q5_K_M | 5.13GB | High quality, very usable. |
31
+ | [Mistral-quiet-star-Q5_K_S.gguf](https://huggingface.co/bartowski/Mistral-quiet-star-GGUF/blob/main/Mistral-quiet-star-Q5_K_S.gguf) | Q5_K_S | 4.99GB | High quality, very usable. |
32
+ | [Mistral-quiet-star-Q5_0.gguf](https://huggingface.co/bartowski/Mistral-quiet-star-GGUF/blob/main/Mistral-quiet-star-Q5_0.gguf) | Q5_0 | 4.99GB | High quality, older format, generally not recommended. |
33
+ | [Mistral-quiet-star-Q4_K_M.gguf](https://huggingface.co/bartowski/Mistral-quiet-star-GGUF/blob/main/Mistral-quiet-star-Q4_K_M.gguf) | Q4_K_M | 4.36GB | Good quality, uses about 4.83 bits per weight. |
34
+ | [Mistral-quiet-star-Q4_K_S.gguf](https://huggingface.co/bartowski/Mistral-quiet-star-GGUF/blob/main/Mistral-quiet-star-Q4_K_S.gguf) | Q4_K_S | 4.14GB | Slightly lower quality with small space savings. |
35
+ | [Mistral-quiet-star-IQ4_NL.gguf](https://huggingface.co/bartowski/Mistral-quiet-star-GGUF/blob/main/Mistral-quiet-star-IQ4_NL.gguf) | IQ4_NL | 4.15GB | Decent quality, similar to Q4_K_S, new method of quanting, |
36
+ | [Mistral-quiet-star-IQ4_XS.gguf](https://huggingface.co/bartowski/Mistral-quiet-star-GGUF/blob/main/Mistral-quiet-star-IQ4_XS.gguf) | IQ4_XS | 3.94GB | Decent quality, new method with similar performance to Q4. |
37
+ | [Mistral-quiet-star-Q4_0.gguf](https://huggingface.co/bartowski/Mistral-quiet-star-GGUF/blob/main/Mistral-quiet-star-Q4_0.gguf) | Q4_0 | 4.10GB | Decent quality, older format, generally not recommended. |
38
+ | [Mistral-quiet-star-Q3_K_L.gguf](https://huggingface.co/bartowski/Mistral-quiet-star-GGUF/blob/main/Mistral-quiet-star-Q3_K_L.gguf) | Q3_K_L | 3.82GB | Lower quality but usable, good for low RAM availability. |
39
+ | [Mistral-quiet-star-Q3_K_M.gguf](https://huggingface.co/bartowski/Mistral-quiet-star-GGUF/blob/main/Mistral-quiet-star-Q3_K_M.gguf) | Q3_K_M | 3.51GB | Even lower quality. |
40
+ | [Mistral-quiet-star-IQ3_M.gguf](https://huggingface.co/bartowski/Mistral-quiet-star-GGUF/blob/main/Mistral-quiet-star-IQ3_M.gguf) | IQ3_M | 3.28GB | Medium-low quality, new method with decent performance. |
41
+ | [Mistral-quiet-star-IQ3_S.gguf](https://huggingface.co/bartowski/Mistral-quiet-star-GGUF/blob/main/Mistral-quiet-star-IQ3_S.gguf) | IQ3_S | 3.18GB | Lower quality, new method with decent performance, recommended over Q3 quants. |
42
+ | [Mistral-quiet-star-Q3_K_S.gguf](https://huggingface.co/bartowski/Mistral-quiet-star-GGUF/blob/main/Mistral-quiet-star-Q3_K_S.gguf) | Q3_K_S | 3.16GB | Low quality, not recommended. |
43
+ | [Mistral-quiet-star-Q2_K.gguf](https://huggingface.co/bartowski/Mistral-quiet-star-GGUF/blob/main/Mistral-quiet-star-Q2_K.gguf) | Q2_K | 2.71GB | Extremely low quality, *not* recommended.
44
+
45
+ Want to support my work? Visit my ko-fi page here: https://ko-fi.com/bartowski