Qwen3-235B-A22B-Instruct-250718 单机8卡H20 LoRA训练的最佳实践参考：[https://github.com/modelscope/ms-swift/pull/5033](https://github.com/modelscope/ms-swift/pull/5033)。

ms-swift 引入了 Megatron 并行技术以加速大模型的CPT/SFT/DPO/KTO/RM。支持的模型可以在[支持的模型文档](../Instruction/Supported-models-and-datasets.md)中找到。

ms-swift 引入了 Megatron 并行技术以加速大模型的CPT/SFT/DPO/GRPO。支持的模型可以在[支持的模型文档](../Instruction/Supported-models-and-datasets.md)中找到。

关于环境准备以及 HF 和 MCore 模型权重的转换，可以参考[Megatron-SWIFT训练文档](../Megatron-SWIFT/Quick-start.md)。

ms-swift引入了Megatron的并行技术来加速大模型的训练，包括数据并行、张量并行、流水线并行、序列并行，上下文并行，专家并行。支持Qwen3、[Qwen3-MoE](https://github.com/modelscope/ms-swift/blob/main/examples/megatron/mcore_bridge/full/moe.sh)、Qwen2.5、Llama3、Deepseek-R1、GLM4.5等模型的CPT/SFT/DPO。完整支持的模型可以参考[支持的模型与数据集文档](../Instruction/Supported-models-and-datasets.md)。推荐在MoE训练时使用Megatron-SWIFT，这通常可以获得10倍的训练速度提升。

ms-swift引入了Megatron的并行技术来加速大模型的训练，包括数据并行、张量并行、流水线并行、序列并行，上下文并行，专家并行。支持Qwen3、[Qwen3-MoE](https://github.com/modelscope/ms-swift/blob/main/examples/megatron/mcore_bridge/full/moe.sh)、Qwen2.5、Llama3、Deepseek-R1、GLM4.5等模型的CPT/SFT/DPO/GRPO。完整支持的模型可以参考[支持的模型与数据集文档](../Instruction/Supported-models-and-datasets.md)。推荐在MoE训练时使用Megatron-SWIFT，这通常可以获得10倍的训练速度提升。

| 方法| 全参数| LoRA| MoE| 多模态| FP8|

Best practice reference for single-node 8xH20 LoRA training with Qwen3-235B-A22B-Instruct-250718:https://github.com/modelscope/ms-swift/pull/5033.

ms-swift introduces Megatron parallelism techniques to accelerate CPT/SFT/DPO/KTO/RM for large models. Supported models can be found in the[Supported Models and Datasets Document](../Instruction/Supported-models-and-datasets.md).

ms-swift introduces Megatron parallelism techniques to accelerate CPT/SFT/DPO/GRPO for large models. Supported models can be found in the[Supported Models and Datasets Document](../Instruction/Supported-models-and-datasets.md).

For environment setup and conversion between HF and MCore model weights, refer to the[Megatron-SWIFT Training Documentation](../Megatron-SWIFT/Quick-start.md).

ms-swift incorporates Megatron's parallelization techniques to accelerate the training of large models, including data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism, context parallelism, and expert parallelism. It supports CPT/SFT/DPO for models such as Qwen3,[Qwen3-MoE](https://github.com/modelscope/ms-swift/blob/main/examples/megatron/mcore_bridge/full/moe.sh), Qwen2.5, Llama3, Deepseek-R1 and GLM4.5 series. For a complete list of supported models, please refer to the[Supported Models and Datasets documentation](../Instruction/Supported-models-and-datasets.md). We recommend using Megatron-SWIFT for MoE training; it can typically achieve a 10x speedup in training.

ms-swift incorporates Megatron's parallelization techniques to accelerate the training of large models, including data parallelism, tensor parallelism, pipeline parallelism, sequence parallelism, context parallelism, and expert parallelism. It supports CPT/SFT/DPO/GRPO for models such as Qwen3,[Qwen3-MoE](https://github.com/modelscope/ms-swift/blob/main/examples/megatron/mcore_bridge/full/moe.sh), Qwen2.5, Llama3, Deepseek-R1 and GLM4.5 series. For a complete list of supported models, please refer to the[Supported Models and Datasets documentation](../Instruction/Supported-models-and-datasets.md). We recommend using Megatron-SWIFT for MoE training; it can typically achieve a 10x speedup in training.

| Method| Full-Parameter| LoRA| MoE| Multimodal| FP8|

use_rslora:bool=False

def__post_init__(self):

ifself.freeze_parameters_ratio>0andself.pipeline_model_parallel_size>1:

if0<self.freeze_parameters_ratio<1andself.pipeline_model_parallel_size>1:

raiseValueError('`freeze_parameters_ratio` is not supported when `pipeline_model_parallel_size` > 1')

ifself.target_regex:

self.target_modules=self.target_regex

if (notto_mcoreoris_pp_last_stage)andself.args.mtp_num_layers:

lm_model=getattr(mg_model,'language_model')ifself.args.is_multimodalelsemg_model

ifto_mcoreandself.pp_rank>0:

self._set_state_dict(lm_model,'embedding.word_embeddings.weight',hf_state_dict,self.hf_embed_key,

to_mcore)

whilelayer_idx<self.args.mtp_num_layers:

res=self._convert_mtp_layer(lm_model,hf_state_dict,f'{self.hf_mtp_prefix}.',layer_idx,to_mcore)

rotary_pos_emb=rotary_pos_emb,

rotary_pos_cos=rotary_pos_cos,

rotary_pos_sin=rotary_pos_sin,

mtp_in_postprocess=self.mtp_process,

loss_mask=loss_mask,

decoder_input=decoder_input,

attention_mask=attention_mask,

rotary_pos_emb,

rotary_pos_cos,

rotary_pos_sin,

mtp_in_postprocess=None,

loss_mask=None,

decoder_input=None,

attention_mask=None,

ifnotself.post_process:

args=get_args()

labels=labelsifargs.task_type=='causal_lm'elseNone

in_inference_mode=inference_contextisnotNoneandnotself.training

ifself.share_embeddings_and_output_weights:

output_weight=self.shared_embedding_or_output_weight()

ifmtp_in_postprocess:

ifself.mtp_process:

hidden_states=self.mtp(

input_ids=input_ids,

position_ids=position_ids,

embedding=self.embedding,

**(extra_block_kwargsor {}),

)

ifnotself.post_process:

ifself.mtp_process:

mtp_labels=labels.clone()

from ..trainers.utilsimportsplit_cp_inputs

hidden_states_list=torch.chunk(hidden_states,1+self.config.mtp_num_layers,dim=0)

hidden_states=hidden_states_list[0]

ifloss_maskisNone:

loss_mask=mtp_labels.new_ones((1,packed_seq_params.cu_seqlens_q[-1]))

cu_seqlens=packed_seq_params.cu_seqlens_q

formtp_layer_numberinrange(self.config.mtp_num_layers):

mtp_logits,_=self.output_layer(

hidden_states_list[mtp_layer_number+1],

weight=output_weight,

runtime_gather_output=runtime_gather_output,

)

mtp_labels,_=roll_tensor(mtp_labels,shifts=-1,dims=-1,cp_group=self.cp_group)

loss_mask[:,cu_seqlens[:-1]]=0

loss_mask,_=roll_tensor(loss_mask,shifts=-1,dims=-1)

ifargs.context_parallel_size>1:

loss_mask_=split_cp_inputs(loss_mask,cu_seqlens,dim=1)

else:

loss_mask_=loss_mask.clone()

mtp_loss=self.compute_language_model_loss(mtp_labels,mtp_logits)

num_tokens=loss_mask_.sum()

ifself.training:

MTPLossLoggingHelper.save_loss_to_tracker(

torch.sum(mtp_loss)/num_tokens,

mtp_layer_number,

self.config.mtp_num_layers,

avg_group=parallel_state.get_data_parallel_group(with_context_parallel=True),

iflabelsisnotNone:

from ..trainers.utilsimportsplit_cp_inputs

mtp_labels=labels.clone()

ifloss_maskisNone:

loss_mask=mtp_labels.new_ones((1,packed_seq_params.cu_seqlens_q[-1]))

cu_seqlens=packed_seq_params.cu_seqlens_q

formtp_layer_numberinrange(self.config.mtp_num_layers):

mtp_logits,_=self.output_layer(

hidden_states_list[mtp_layer_number+1],

weight=output_weight,

runtime_gather_output=runtime_gather_output,

)

mtp_loss_scale=self.config.mtp_loss_scaling_factor/self.config.mtp_num_layers

ifself.config.calculate_per_token_loss:

hidden_states=MTPLossAutoScaler.apply(hidden_states,mtp_loss_scale*mtp_loss)

else:

hidden_states=MTPLossAutoScaler.apply(hidden_states,mtp_loss_scale*mtp_loss/num_tokens)

mtp_labels,_=roll_tensor(mtp_labels,shifts=-1,dims=-1,cp_group=self.cp_group)

loss_mask[:,cu_seqlens[:-1]]=0

loss_mask,_=roll_tensor(loss_mask,shifts=-1,dims=-1)

ifargs.context_parallel_size>1:

loss_mask_=split_cp_inputs(loss_mask,cu_seqlens,dim=1)

else:

loss_mask_=loss_mask.clone()

mtp_loss=self.compute_language_model_loss(mtp_labels,mtp_logits)

num_tokens=loss_mask_.sum()

ifself.training:

MTPLossLoggingHelper.save_loss_to_tracker(

torch.sum(mtp_loss)/num_tokens,

mtp_layer_number,

self.config.mtp_num_layers,

avg_group=parallel_state.get_data_parallel_group(with_context_parallel=True),

)

mtp_loss_scale=self.config.mtp_loss_scaling_factor/self.config.mtp_num_layers

ifself.config.calculate_per_token_loss:

hidden_states=MTPLossAutoScaler.apply(hidden_states,mtp_loss_scale*mtp_loss)

else:

hidden_states=MTPLossAutoScaler.apply(hidden_states,mtp_loss_scale*mtp_loss/num_tokens)

ifin_inference_modeandinference_context.materialize_only_last_token_logits:

ifinference_context.is_static_batching():