Commit
•
37684bf
1
Parent(s):
9bb1142
Update README.md
Browse files
README.md
CHANGED
|
@@ -59,5 +59,26 @@ print(output)
|
|
| 59 |
"""
|
| 60 |
```
|
| 61 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 62 |
### Author
|
| 63 |
[Rakuto Suda](https://huggingface.co/sudy-super)
|
|
|
|
| 59 |
"""
|
| 60 |
```
|
| 61 |
|
| 62 |
+
### Hyperparameter
|
| 63 |
+
```
|
| 64 |
+
num_train_epochs: 5
|
| 65 |
+
per_device_train_batch_size: 4
|
| 66 |
+
per_device_eval_batch_size: 4
|
| 67 |
+
gradient_accumulation_steps: 64
|
| 68 |
+
learning_rate: 2.5e-5
|
| 69 |
+
lr_scheduler_kwargs={"min_lr": 2.5e-6}
|
| 70 |
+
lr_scheduler_type: "cosine_with_min_lr"
|
| 71 |
+
warmup_ratio: 0.1
|
| 72 |
+
dataloader_pin_memory: True
|
| 73 |
+
gradient_checkpointing: True
|
| 74 |
+
bf16: True
|
| 75 |
+
optim: "adamw_torch_fused"
|
| 76 |
+
weight_decay: 0.0
|
| 77 |
+
max_grad_norm: 1.0
|
| 78 |
+
adam_beta2: 0.99
|
| 79 |
+
label_smoothing_factor: 0.0
|
| 80 |
+
seed: 42
|
| 81 |
+
```
|
| 82 |
+
|
| 83 |
### Author
|
| 84 |
[Rakuto Suda](https://huggingface.co/sudy-super)
|