RichardErkhov
commited on
Commit
•
a082aae
1
Parent(s):
325946e
uploaded readme
Browse files
README.md
ADDED
@@ -0,0 +1,313 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
Quantization made by Richard Erkhov.
|
2 |
+
|
3 |
+
[Github](https://github.com/RichardErkhov)
|
4 |
+
|
5 |
+
[Discord](https://discord.gg/pvy7H8DZMG)
|
6 |
+
|
7 |
+
[Request more models](https://github.com/RichardErkhov/quant_request)
|
8 |
+
|
9 |
+
|
10 |
+
NeuralHermes-2.5-Mistral-7B-laser - GGUF
|
11 |
+
- Model creator: https://huggingface.co/mlabonne/
|
12 |
+
- Original model: https://huggingface.co/mlabonne/NeuralHermes-2.5-Mistral-7B-laser/
|
13 |
+
|
14 |
+
|
15 |
+
| Name | Quant method | Size |
|
16 |
+
| ---- | ---- | ---- |
|
17 |
+
| [NeuralHermes-2.5-Mistral-7B-laser.Q2_K.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.Q2_K.gguf) | Q2_K | 2.53GB |
|
18 |
+
| [NeuralHermes-2.5-Mistral-7B-laser.IQ3_XS.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.IQ3_XS.gguf) | IQ3_XS | 2.81GB |
|
19 |
+
| [NeuralHermes-2.5-Mistral-7B-laser.IQ3_S.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.IQ3_S.gguf) | IQ3_S | 2.96GB |
|
20 |
+
| [NeuralHermes-2.5-Mistral-7B-laser.Q3_K_S.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.Q3_K_S.gguf) | Q3_K_S | 2.95GB |
|
21 |
+
| [NeuralHermes-2.5-Mistral-7B-laser.IQ3_M.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.IQ3_M.gguf) | IQ3_M | 3.06GB |
|
22 |
+
| [NeuralHermes-2.5-Mistral-7B-laser.Q3_K.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.Q3_K.gguf) | Q3_K | 3.28GB |
|
23 |
+
| [NeuralHermes-2.5-Mistral-7B-laser.Q3_K_M.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.Q3_K_M.gguf) | Q3_K_M | 3.28GB |
|
24 |
+
| [NeuralHermes-2.5-Mistral-7B-laser.Q3_K_L.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.Q3_K_L.gguf) | Q3_K_L | 3.56GB |
|
25 |
+
| [NeuralHermes-2.5-Mistral-7B-laser.IQ4_XS.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.IQ4_XS.gguf) | IQ4_XS | 3.67GB |
|
26 |
+
| [NeuralHermes-2.5-Mistral-7B-laser.Q4_0.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.Q4_0.gguf) | Q4_0 | 3.83GB |
|
27 |
+
| [NeuralHermes-2.5-Mistral-7B-laser.IQ4_NL.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.IQ4_NL.gguf) | IQ4_NL | 3.87GB |
|
28 |
+
| [NeuralHermes-2.5-Mistral-7B-laser.Q4_K_S.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.Q4_K_S.gguf) | Q4_K_S | 3.86GB |
|
29 |
+
| [NeuralHermes-2.5-Mistral-7B-laser.Q4_K.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.Q4_K.gguf) | Q4_K | 4.07GB |
|
30 |
+
| [NeuralHermes-2.5-Mistral-7B-laser.Q4_K_M.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.Q4_K_M.gguf) | Q4_K_M | 4.07GB |
|
31 |
+
| [NeuralHermes-2.5-Mistral-7B-laser.Q4_1.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.Q4_1.gguf) | Q4_1 | 4.24GB |
|
32 |
+
| [NeuralHermes-2.5-Mistral-7B-laser.Q5_0.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.Q5_0.gguf) | Q5_0 | 4.65GB |
|
33 |
+
| [NeuralHermes-2.5-Mistral-7B-laser.Q5_K_S.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.Q5_K_S.gguf) | Q5_K_S | 4.65GB |
|
34 |
+
| [NeuralHermes-2.5-Mistral-7B-laser.Q5_K.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.Q5_K.gguf) | Q5_K | 4.78GB |
|
35 |
+
| [NeuralHermes-2.5-Mistral-7B-laser.Q5_K_M.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.Q5_K_M.gguf) | Q5_K_M | 4.78GB |
|
36 |
+
| [NeuralHermes-2.5-Mistral-7B-laser.Q5_1.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.Q5_1.gguf) | Q5_1 | 5.07GB |
|
37 |
+
| [NeuralHermes-2.5-Mistral-7B-laser.Q6_K.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.Q6_K.gguf) | Q6_K | 5.53GB |
|
38 |
+
| [NeuralHermes-2.5-Mistral-7B-laser.Q8_0.gguf](https://huggingface.co/RichardErkhov/mlabonne_-_NeuralHermes-2.5-Mistral-7B-laser-gguf/blob/main/NeuralHermes-2.5-Mistral-7B-laser.Q8_0.gguf) | Q8_0 | 7.17GB |
|
39 |
+
|
40 |
+
|
41 |
+
|
42 |
+
|
43 |
+
Original model description:
|
44 |
+
---
|
45 |
+
language:
|
46 |
+
- en
|
47 |
+
license: apache-2.0
|
48 |
+
tags:
|
49 |
+
- mistral
|
50 |
+
- instruct
|
51 |
+
- finetune
|
52 |
+
- chatml
|
53 |
+
- gpt4
|
54 |
+
- synthetic data
|
55 |
+
- distillation
|
56 |
+
- dpo
|
57 |
+
- rlhf
|
58 |
+
- laser
|
59 |
+
datasets:
|
60 |
+
- mlabonne/chatml_dpo_pairs
|
61 |
+
base_model: teknium/OpenHermes-2.5-Mistral-7B
|
62 |
+
model-index:
|
63 |
+
- name: NeuralHermes-2.5-Mistral-7B-laser
|
64 |
+
results:
|
65 |
+
- task:
|
66 |
+
type: text-generation
|
67 |
+
name: Text Generation
|
68 |
+
dataset:
|
69 |
+
name: AI2 Reasoning Challenge (25-Shot)
|
70 |
+
type: ai2_arc
|
71 |
+
config: ARC-Challenge
|
72 |
+
split: test
|
73 |
+
args:
|
74 |
+
num_few_shot: 25
|
75 |
+
metrics:
|
76 |
+
- type: acc_norm
|
77 |
+
value: 66.38
|
78 |
+
name: normalized accuracy
|
79 |
+
source:
|
80 |
+
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mlabonne/NeuralHermes-2.5-Mistral-7B-laser
|
81 |
+
name: Open LLM Leaderboard
|
82 |
+
- task:
|
83 |
+
type: text-generation
|
84 |
+
name: Text Generation
|
85 |
+
dataset:
|
86 |
+
name: HellaSwag (10-Shot)
|
87 |
+
type: hellaswag
|
88 |
+
split: validation
|
89 |
+
args:
|
90 |
+
num_few_shot: 10
|
91 |
+
metrics:
|
92 |
+
- type: acc_norm
|
93 |
+
value: 85.09
|
94 |
+
name: normalized accuracy
|
95 |
+
source:
|
96 |
+
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mlabonne/NeuralHermes-2.5-Mistral-7B-laser
|
97 |
+
name: Open LLM Leaderboard
|
98 |
+
- task:
|
99 |
+
type: text-generation
|
100 |
+
name: Text Generation
|
101 |
+
dataset:
|
102 |
+
name: MMLU (5-Shot)
|
103 |
+
type: cais/mmlu
|
104 |
+
config: all
|
105 |
+
split: test
|
106 |
+
args:
|
107 |
+
num_few_shot: 5
|
108 |
+
metrics:
|
109 |
+
- type: acc
|
110 |
+
value: 63.43
|
111 |
+
name: accuracy
|
112 |
+
source:
|
113 |
+
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mlabonne/NeuralHermes-2.5-Mistral-7B-laser
|
114 |
+
name: Open LLM Leaderboard
|
115 |
+
- task:
|
116 |
+
type: text-generation
|
117 |
+
name: Text Generation
|
118 |
+
dataset:
|
119 |
+
name: TruthfulQA (0-shot)
|
120 |
+
type: truthful_qa
|
121 |
+
config: multiple_choice
|
122 |
+
split: validation
|
123 |
+
args:
|
124 |
+
num_few_shot: 0
|
125 |
+
metrics:
|
126 |
+
- type: mc2
|
127 |
+
value: 54.95
|
128 |
+
source:
|
129 |
+
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mlabonne/NeuralHermes-2.5-Mistral-7B-laser
|
130 |
+
name: Open LLM Leaderboard
|
131 |
+
- task:
|
132 |
+
type: text-generation
|
133 |
+
name: Text Generation
|
134 |
+
dataset:
|
135 |
+
name: Winogrande (5-shot)
|
136 |
+
type: winogrande
|
137 |
+
config: winogrande_xl
|
138 |
+
split: validation
|
139 |
+
args:
|
140 |
+
num_few_shot: 5
|
141 |
+
metrics:
|
142 |
+
- type: acc
|
143 |
+
value: 78.14
|
144 |
+
name: accuracy
|
145 |
+
source:
|
146 |
+
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mlabonne/NeuralHermes-2.5-Mistral-7B-laser
|
147 |
+
name: Open LLM Leaderboard
|
148 |
+
- task:
|
149 |
+
type: text-generation
|
150 |
+
name: Text Generation
|
151 |
+
dataset:
|
152 |
+
name: GSM8k (5-shot)
|
153 |
+
type: gsm8k
|
154 |
+
config: main
|
155 |
+
split: test
|
156 |
+
args:
|
157 |
+
num_few_shot: 5
|
158 |
+
metrics:
|
159 |
+
- type: acc
|
160 |
+
value: 55.72
|
161 |
+
name: accuracy
|
162 |
+
source:
|
163 |
+
url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mlabonne/NeuralHermes-2.5-Mistral-7B-laser
|
164 |
+
name: Open LLM Leaderboard
|
165 |
+
---
|
166 |
+
|
167 |
+
<center><img src="https://i.imgur.com/gUlEJuU.jpeg"></center>
|
168 |
+
|
169 |
+
# NeuralHermes 2.5 - Mistral 7B - LASER
|
170 |
+
|
171 |
+
This is an experimental LASER version of NeuralHermes using [laserRMT](https://github.com/cognitivecomputations/laserRMT), based on [this paper](https://arxiv.org/pdf/2312.13558.pdf).
|
172 |
+
|
173 |
+
| Model |AGIEval|GPT4All|TruthfulQA|Bigbench|Average|
|
174 |
+
|------------------------------------------------------------------------------------------------------|------:|------:|---------:|-------:|------:|
|
175 |
+
|[NeuralHermes-2.5-Mistral-7B-laser](https://huggingface.co/mlabonne/NeuralHermes-2.5-Mistral-7B-laser)| 43.54| 73.44| 55.26| 42.24| 53.62|
|
176 |
+
|[NeuralHermes-2.5-Mistral-7B](https://huggingface.co/mlabonne/NeuralHermes-2.5-Mistral-7B) | 43.67| 73.24| 55.37| 41.76| 53.51|
|
177 |
+
|
178 |
+
Fernando Fernandes Neto and Eric Hartford. "Optimizing Large Language Models Using Layer-Selective Rank Reduction and Random Matrix Theory." 2024.
|
179 |
+
|
180 |
+
NeuralHermes is an [teknium/OpenHermes-2.5-Mistral-7B](https://huggingface.co/teknium/OpenHermes-2.5-Mistral-7B) model that has been further fine-tuned with Direct Preference Optimization (DPO) using the [mlabonne/chatml_dpo_pairs](https://huggingface.co/datasets/mlabonne/chatml_dpo_pairs) dataset. It surpasses the original model on several benchmarks (see results).
|
181 |
+
|
182 |
+
It is directly inspired by the RLHF process described by [Intel/neural-chat-7b-v3-1](https://huggingface.co/Intel/neural-chat-7b-v3-1)'s authors to improve performance. I used the same dataset and reformatted it to apply the ChatML template.
|
183 |
+
|
184 |
+
The code to train this model is available on [Google Colab](https://colab.research.google.com/drive/15iFBr1xWgztXvhrj5I9fBv20c7CFOPBE?usp=sharing) and [GitHub](https://github.com/mlabonne/llm-course/tree/main). It required an A100 GPU for about an hour.
|
185 |
+
|
186 |
+
## Results
|
187 |
+
|
188 |
+
### AGIEval
|
189 |
+
| Task |Version| Metric |Value| |Stderr|
|
190 |
+
|------------------------------|------:|--------|----:|---|-----:|
|
191 |
+
|agieval_aqua_rat | 0|acc |21.26|± | 2.57|
|
192 |
+
| | |acc_norm|22.83|± | 2.64|
|
193 |
+
|agieval_logiqa_en | 0|acc |39.32|± | 1.92|
|
194 |
+
| | |acc_norm|40.71|± | 1.93|
|
195 |
+
|agieval_lsat_ar | 0|acc |25.65|± | 2.89|
|
196 |
+
| | |acc_norm|25.65|± | 2.89|
|
197 |
+
|agieval_lsat_lr | 0|acc |48.82|± | 2.22|
|
198 |
+
| | |acc_norm|50.00|± | 2.22|
|
199 |
+
|agieval_lsat_rc | 0|acc |58.36|± | 3.01|
|
200 |
+
| | |acc_norm|57.25|± | 3.02|
|
201 |
+
|agieval_sat_en | 0|acc |74.27|± | 3.05|
|
202 |
+
| | |acc_norm|73.30|± | 3.09|
|
203 |
+
|agieval_sat_en_without_passage| 0|acc |43.69|± | 3.46|
|
204 |
+
| | |acc_norm|42.23|± | 3.45|
|
205 |
+
|agieval_sat_math | 0|acc |37.27|± | 3.27|
|
206 |
+
| | |acc_norm|36.36|± | 3.25|
|
207 |
+
|
208 |
+
Average: 43.54%
|
209 |
+
|
210 |
+
### GPT4All
|
211 |
+
| Task |Version| Metric |Value| |Stderr|
|
212 |
+
|-------------|------:|--------|----:|---|-----:|
|
213 |
+
|arc_challenge| 0|acc |57.76|± | 1.44|
|
214 |
+
| | |acc_norm|60.32|± | 1.43|
|
215 |
+
|arc_easy | 0|acc |83.84|± | 0.76|
|
216 |
+
| | |acc_norm|81.10|± | 0.80|
|
217 |
+
|boolq | 1|acc |86.70|± | 0.59|
|
218 |
+
|hellaswag | 0|acc |63.15|± | 0.48|
|
219 |
+
| | |acc_norm|82.55|± | 0.38|
|
220 |
+
|openbookqa | 0|acc |34.40|± | 2.13|
|
221 |
+
| | |acc_norm|45.20|± | 2.23|
|
222 |
+
|piqa | 0|acc |81.94|± | 0.90|
|
223 |
+
| | |acc_norm|82.97|± | 0.88|
|
224 |
+
|winogrande | 0|acc |75.22|± | 1.21|
|
225 |
+
|
226 |
+
Average: 73.44%
|
227 |
+
|
228 |
+
### TruthfulQA
|
229 |
+
| Task |Version|Metric|Value| |Stderr|
|
230 |
+
|-------------|------:|------|----:|---|-----:|
|
231 |
+
|truthfulqa_mc| 1|mc1 |37.70|± | 1.70|
|
232 |
+
| | |mc2 |55.26|± | 1.52|
|
233 |
+
|
234 |
+
Average: 55.26%
|
235 |
+
|
236 |
+
### Bigbench
|
237 |
+
| Task |Version| Metric |Value| |Stderr|
|
238 |
+
|------------------------------------------------|------:|---------------------|----:|---|-----:|
|
239 |
+
|bigbench_causal_judgement | 0|multiple_choice_grade|53.16|± | 3.63|
|
240 |
+
|bigbench_date_understanding | 0|multiple_choice_grade|65.31|± | 2.48|
|
241 |
+
|bigbench_disambiguation_qa | 0|multiple_choice_grade|34.11|± | 2.96|
|
242 |
+
|bigbench_geometric_shapes | 0|multiple_choice_grade|27.02|± | 2.35|
|
243 |
+
| | |exact_str_match | 0.28|± | 0.28|
|
244 |
+
|bigbench_logical_deduction_five_objects | 0|multiple_choice_grade|27.80|± | 2.01|
|
245 |
+
|bigbench_logical_deduction_seven_objects | 0|multiple_choice_grade|19.86|± | 1.51|
|
246 |
+
|bigbench_logical_deduction_three_objects | 0|multiple_choice_grade|48.33|± | 2.89|
|
247 |
+
|bigbench_movie_recommendation | 0|multiple_choice_grade|41.40|± | 2.20|
|
248 |
+
|bigbench_navigate | 0|multiple_choice_grade|50.00|± | 1.58|
|
249 |
+
|bigbench_reasoning_about_colored_objects | 0|multiple_choice_grade|65.00|± | 1.07|
|
250 |
+
|bigbench_ruin_names | 0|multiple_choice_grade|46.21|± | 2.36|
|
251 |
+
|bigbench_salient_translation_error_detection | 0|multiple_choice_grade|27.25|± | 1.41|
|
252 |
+
|bigbench_snarks | 0|multiple_choice_grade|70.72|± | 3.39|
|
253 |
+
|bigbench_sports_understanding | 0|multiple_choice_grade|65.72|± | 1.51|
|
254 |
+
|bigbench_temporal_sequences | 0|multiple_choice_grade|30.40|± | 1.46|
|
255 |
+
|bigbench_tracking_shuffled_objects_five_objects | 0|multiple_choice_grade|22.56|± | 1.18|
|
256 |
+
|bigbench_tracking_shuffled_objects_seven_objects| 0|multiple_choice_grade|17.09|± | 0.90|
|
257 |
+
|bigbench_tracking_shuffled_objects_three_objects| 0|multiple_choice_grade|48.33|± | 2.89|
|
258 |
+
|
259 |
+
Average: 42.24%
|
260 |
+
|
261 |
+
Average score: 53.62%
|
262 |
+
|
263 |
+
## Usage
|
264 |
+
|
265 |
+
You can run this model using [LM Studio](https://lmstudio.ai/) or any other frontend.
|
266 |
+
|
267 |
+
You can also run this model using the following code:
|
268 |
+
|
269 |
+
```python
|
270 |
+
import transformers
|
271 |
+
from transformers import AutoTokenizer
|
272 |
+
|
273 |
+
# Format prompt
|
274 |
+
message = [
|
275 |
+
{"role": "system", "content": "You are a helpful assistant chatbot."},
|
276 |
+
{"role": "user", "content": "What is a Large Language Model?"}
|
277 |
+
]
|
278 |
+
tokenizer = AutoTokenizer.from_pretrained(new_model)
|
279 |
+
prompt = tokenizer.apply_chat_template(message, add_generation_prompt=True, tokenize=False)
|
280 |
+
|
281 |
+
# Create pipeline
|
282 |
+
pipeline = transformers.pipeline(
|
283 |
+
"text-generation",
|
284 |
+
model="mlabonne/NeuralHermes-2.5-Mistral-7B-laser",
|
285 |
+
tokenizer=tokenizer
|
286 |
+
)
|
287 |
+
|
288 |
+
# Generate text
|
289 |
+
sequences = pipeline(
|
290 |
+
prompt,
|
291 |
+
do_sample=True,
|
292 |
+
temperature=0.7,
|
293 |
+
top_p=0.9,
|
294 |
+
num_return_sequences=1,
|
295 |
+
max_length=200,
|
296 |
+
)
|
297 |
+
print(sequences[0]['generated_text'])
|
298 |
+
```
|
299 |
+
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
|
300 |
+
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_mlabonne__NeuralHermes-2.5-Mistral-7B-laser)
|
301 |
+
|
302 |
+
| Metric |Value|
|
303 |
+
|---------------------------------|----:|
|
304 |
+
|Avg. |67.29|
|
305 |
+
|AI2 Reasoning Challenge (25-Shot)|66.38|
|
306 |
+
|HellaSwag (10-Shot) |85.09|
|
307 |
+
|MMLU (5-Shot) |63.43|
|
308 |
+
|TruthfulQA (0-shot) |54.95|
|
309 |
+
|Winogrande (5-shot) |78.14|
|
310 |
+
|GSM8k (5-shot) |55.72|
|
311 |
+
|
312 |
+
|
313 |
+
|