You signed in with another tab or window.Reload to refresh your session.You signed out in another tab or window.Reload to refresh your session.You switched accounts on another tab or window.Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+23-17Lines changed: 23 additions & 17 deletions
Original file line number
Diff line number
Diff line change
@@ -46,14 +46,19 @@
46
46
47
47
48
48
##News
49
+
🔥🔥🔥[2024/05/20] We released**MFTCoder v0.4.2**, mainly for MFTCoder-accelerate. It supports**QLoRA + DeepSpeed Zero3** and**QLoRA + FSDP** as options allowing you training very large models. It now supports new models like Qwen2, Qwen2-MoE, Starcoder2, Gemma, etc.
49
50
50
-
🔥🔥🔥[2024/01/30] The model[CodeFuse-DeepSeek-33B](https://huggingface.co/codefuse-ai/CodeFuse-DeepSeek-33B) fine-tuned with MFTCoder ranks first in HuggingFace[Big Code Models LeaderBoard](https://huggingface.co/spaces/bigcode/bigcode-models-leaderboard)
51
+
🔥🔥🔥[2024/05/20] Our paper[MFTCoder: Boosting Code LLMs with Multitask Fine-Tuning](https://arxiv.org/abs/2311.02303) has been accepted by KDD2024.
51
52
52
-
🔥🔥🔥[2024/01/17] We released MFTCoder v0.3.0, mainly for MFTCoder-accelerate. It now supports new models like Mixtral(MoE), DeepSeek-coder, chatglm3. It supports FSDP as an option. It also supports Self-paced Loss as a solution for convergence balance in Multitask Fine-tuning.
53
+
🔥🔥🔥[2024/05/20][CodeFuse-StarCoder2-15B](https://huggingface.co/codefuse-ai/CodeFuse-StarCoder2-15B) has been released, achieving a pass@1 (greedy decoding) score of 73.2% on HumanEval.
53
54
54
-
🔥🔥🔥[2024/01/17][CodeFuse-DeepSeek-33B](https://huggingface.co/codefuse-ai/CodeFuse-DeepSeek-33B)has been released, achieving a pass@1 (greedy decoding) score of 78.7% on HumanEval. It lists as top-1 LLM on Bigcode Leardboard interms of win-rate, the official result is going to be published later.
55
+
🔥🔥[2024/01/30] The model[CodeFuse-DeepSeek-33B](https://huggingface.co/codefuse-ai/CodeFuse-DeepSeek-33B)fine-tuned with MFTCoder ranks first inHuggingFace[Big Code Models LeaderBoard](https://huggingface.co/spaces/bigcode/bigcode-models-leaderboard)
55
56
56
-
🔥🔥🔥[2024/01/17][CodeFuse-Mixtral-8x7B](https://huggingface.co/codefuse-ai/CodeFuse-Mixtral-8X7B) has been released, achieving a pass@1 (greedy decoding) score of 56.1% on HumanEval.
57
+
🔥🔥[2024/01/17] We released MFTCoder v0.3.0, mainly for MFTCoder-accelerate. It now supports new models like Mixtral(MoE), DeepSeek-coder, chatglm3. It supports FSDP as an option. It also supports Self-paced Loss as a solution for convergence balance in Multitask Fine-tuning.
58
+
59
+
🔥🔥[2024/01/17][CodeFuse-DeepSeek-33B](https://huggingface.co/codefuse-ai/CodeFuse-DeepSeek-33B) has been released, achieving a pass@1 (greedy decoding) score of 78.7% on HumanEval. It lists as top-1 LLM on Bigcode Leardboard in terms of win-rate, the official result is going to be published later.
60
+
61
+
🔥🔥[2024/01/17][CodeFuse-Mixtral-8x7B](https://huggingface.co/codefuse-ai/CodeFuse-Mixtral-8X7B) has been released, achieving a pass@1 (greedy decoding) score of 56.1% on HumanEval.
57
62
58
63
🔥🔥[2023/11/07][MFTCoder Paper](https://arxiv.org/abs/2311.02303) has been released on Arxiv, which discloses technique details of multi-task-fine-tuning.
[MFTCoder: Boosting Code LLMs with Multitask Fine-Tuning (KDD2024)](https://arxiv.org/abs/2311.02303)
92
98
93
99
##Introduction
94
100
@@ -125,13 +131,13 @@ The main components of this project include:
125
131
126
132
127
133
##Requirements
128
-
To begin, ensure that you have successfully installed CUDA (version >= 11.4, preferably11.7) along with the necessary drivers. Additionally, make sure you have installed torch (version2.0.1).
134
+
To begin, ensure that you have successfully installed CUDA (version >= 11.4, preferably12.1) along with the necessary drivers. Additionally, make sure you have installed torch (version>= 2.1.0).
129
135
130
136
Next, we have provided an init_env.sh script to simplify the installation of required packages. Execute the following command to run the script:
131
137
```bash
132
138
sh init_env.sh
133
139
```
134
-
We highly recommend training with flash attention(version >= 2.1.0, preferably 2.3.6), please refer to the following link for installation instructions:https://github.com/Dao-AILab/flash-attention
140
+
We highly recommend training with flash attention(version >= 2.3.0), please refer to the following link for installation instructions:https://github.com/Dao-AILab/flash-attention
135
141
136
142
137
143
##Training
@@ -152,16 +158,16 @@ If you want to explore some new framework like atorch, you could check:
152
158
We are excited to release the following two CodeLLMs trained by MFTCoder, now available on both HuggingFace and ModelScope:
153
159
154
160
155
-
| Model| HuggingFace Links| ModelScope Links| Base Model| Num of examples trained| Batch Size| Seq Length|
We are also pleased to release two code-related instruction datasets, meticulously selected from a range of datasets to facilitate multitask training. Moving forward, we are committed to releasing additional instruction datasets covering various code-related tasks.