File size: 2,916 Bytes
93cadd1 43ca93f 93cadd1 43ca93f 93cadd1 43ca93f 93cadd1 43ca93f 93cadd1 566dd40 3a612e6 5869676 25abefe a3da97a 93cadd1 a3da97a 1b748b1 d3a4ff5 1f48029 93cadd1 5869676 9c4574d 93cadd1 1305186 51efdf2 1305186 51efdf2 3fe8bf9 93cadd1 1305186 93cadd1 23a2db5 eb23884 23a2db5 93cadd1 9e7b43c 6358f4d ed9feeb 79df29a cd0c5ad 058d82f 1a457fd cd0c5ad 79df29a ed9feeb 1a457fd c904c38 1a457fd |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 |
---
license: apache-2.0
datasets:
- wikitext
- ptb_text_only
language:
- en
metrics:
- perplexity
pipeline_tag: text-generation
model-index:
- name: distilgpt2
results:
- task:
type: text-generation
dataset:
name: penn_treebank
type: ptb_text_only
metrics:
- name: perlexity@distilgpt2:BASELINE
type: dmx-perlexity
value: 63.45857238769531
- name: perlexity@distilgpt2:BASIC
type: dmx-perlexity
value: 64.36720275878906
- task:
type: text-generation
dataset:
name: wikitext2
type: wikitext-2-raw-v1
metrics:
- name: perlexity@distilgpt2:BASELINE
type: dmx-perlexity
value: 46.05925369262695
- name: perlexity@distilgpt2:BASIC
type: dmx-perlexity
value: 46.570838928222656
---
This is a d-Matrix functional reference of the GPT2 model family, with the following *revisions*:
- [`distilgpt2`](https://huggingface.co./distilbert/distilgpt2)
- [`gpt2`](https://huggingface.co./openai-community/gpt2)
- [`gpt2-medium`](https://huggingface.co./openai-community/gpt2-medium)
- [`gpt2-large`](https://huggingface.co./openai-community/gpt2-large)
- [`gpt2-xl`](https://huggingface.co./openai-community/gpt2-xl)
The reference provides the following functional *configurations*:
Configuration | Explanation
:-- | :--
**`BASELINE`** | a reference functionally equivalent to the original model
**`BASIC`** | all linear algebraic operands quantized to `BFP16-64`, and all other operations transformed to approximated kernel simulations
### Usage
Install d-Matrix [Dmx_Compressor](https://github.com/d-matrix-ai/dmx-compressor) first.
```sh
pip install dmx_compressor
```
The following is an example model and its evaluation.
```python
from dmx.compressor.dmx import pipeline
pipe = pipeline(
task="text-generation",
model="d-matrix/gpt2",
revision="gpt2-xl", # see above for other variants
dmx_config="BASELINE", # see above for other variants
)
results = pipe.evaluate(
metric="d-matrix/dmx_perplexity",
dataset="wikitext",
dataset_version="wikitext-2-raw-v1",
)
```
### Evaluation results
- `perplexity` on `penn_treebank`
Revision \ Configuration | **`BASELINE`** | **`BASIC`**
:-- | --: | --:
`distilgpt2` | 63.46 | 64.13
`gpt2` | 35.77 | 35.93
`gpt2-medium` | 27.06 | 27.10
`gpt2-large` | 23.03 | 23.04
`gpt2-xl` | 21.01 | 21.02
- `perplexity` on `wikitext2`
Revision \ Configuration | **`BASELINE`** | **`BASIC`**
:-- | --: | --:
`distilgpt2` | 46.06 | 46.44
`gpt2` | 29.94 | 30.08
`gpt2-medium` | 21.71 | 21.73
`gpt2-large` | 19.42| 19.43
`gpt2-xl` | 17.40| 17.40
- `perplexity` on `wikitext103`
Revision \ Configuration | **`BASELINE`** | **`BASIC`**
:-- | --: | --:
`distilgpt2` | 46.06 | 46.44
`gpt2` | 29.94 |30.08
`gpt2-medium` | 21.71 | 21.73
`gpt2-large` | 19.43 | 19.43
`gpt2-xl` | 17.40 | 17.40 |