DeepSeek/DeepSeek-MoE
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepSeek-MoE synced 2025-01-22 10:35:57 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

update readme

Browse Source
This commit is contained in:
zwd973-deepseek 2024-01-15 15:42:20 +08:00
parent 4b411c7d2b
commit 80af6ea9ac
1 changed files with 3 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
README.md
View File

@ -193,6 +193,8 @@ Each item has two required fields `instruction` and `output`.
After data preparation, you can use the sample shell script to finetune the DeepSeekMoE model.
Remember to specify `DATA_PATH`, `OUTPUT_PATH`.
And please choose appropriate hyper-parameters(e.g., `learning_rate`, `per_device_train_batch_size`) according to your scenario.
We have used flash_attention2 by default. For devices supported by flash_attention, you can refer [here](https://github.com/Dao-AILab/flash-attention).
For this configuration, zero_stage needs to be set to 3, and we run it on eight A100 40 GPUs.
```bash
DATA_PATH="<your_data_path>"
@ -224,7 +226,7 @@ deepspeed finetune.py \
--use_lora False
```
You can also finetune the model with 4/8-bits qlora, feel free to try it.
You can also finetune the model with 4/8-bits qlora, feel free to try it. For this configuration, it is possible to run on a single A100 80G GPU, and adjustments can be made according to your resources.
```bash
DATA_PATH="<your_data_path>"
OUTPUT_PATH="<your_output_path>"