- dataset_shuffle: 是否对dataset进行随机操作，默认为True。

- truncation_strategy: 对输入长度超过`max_length`的处理方式，支持`delete`和`left`，代表删除、左侧裁剪，默认为`left`, 注意对于多模态模型，

左裁剪可能会裁剪掉多模态token导致模型前向报错shape mismatch。使用`delete`方式，对于超长数据和编码失败的样例会在原数据集中重采样其他数据作为补充。

- loss_type: loss 归一化的类型，可选项为['grpo', 'bnpo', 'dr_grpo', 'dapo', 'cispo'], 默认为'grpo', 具体参考[文档](./GRPO/DeveloperGuide/loss_types.md)

- loss_type: loss 归一化的类型，可选项为['grpo', 'bnpo', 'dr_grpo', 'dapo', 'cispo', 'sapo'], 默认为'grpo', 具体参考[文档](./GRPO/DeveloperGuide/loss_types.md)

- log_completions: 是否记录训练中的模型生成内容，搭配`--report_to wandb/swanlab` 使用。默认为False。

- 提示：若没有设置`--report_to wandb/swanlab`，则会在checkpoint中创建`completions.jsonl`来存储生成内容。

- use_vllm: 是否使用 vLLM 作为 GRPO 生成的 infer_backend，默认为False。

- num_iterations: 每条数据的更新次数，[GRPO论文](https://arxiv.org/abs/2402.03300)中的 $\mu$ 值，默认为1。

- epsilon: clip 系数，默认为0.2。

- epsilon_high: upper clip 系数，默认为None，设置后与epsilon共同构成[epsilon, epsilon_high]裁剪范围。

- tau_pos:[SAPO](https://arxiv.org/abs/2511.20347)算法中正优势的温度参数，控制软门控函数的锐度。较大值使门控更锐利（接近硬裁剪），较小值使门控更平滑。默认为1.0。

- tau_neg: SAPO算法中负优势的温度参数，控制软门控函数的锐度。通常设置`tau_neg > tau_pos`以对负优势施加更强约束。默认为1.05。

- dynamic_sample：筛除group内奖励标准差为0的数据，额外采样新数据，默认为False。

- max_resample_times：dynamic_sample设置下限制重采样次数，默认3次。

- overlong_filter：跳过超长截断的样本，不参与loss计算，默认为False。

**版本依赖**：ms-swift>=3.11

[Soft Adaptive Policy Optimization (SAPO)](https://arxiv.org/abs/2511.20347) 针对 GRPO 中硬裁剪（hard clipping）带来的问题，提出了一种基于温度控制的软门控（soft gate）机制，用于平滑地衰减离策略更新，同时保留有用的学习信号。

在强化学习训练 LLM 时，GRPO 通过计算 token 级别的重要性采样比（Importance Sampling Ratio）来处理 off-policy 训练：

然而，token 级别的重要性采样比往往表现出高方差，这一现象在以下情况下可能更严重：

-**长文本生成**

-**MoE 模型的路由异质性**：采样时的 old-policy 模型与训练模型可能使用不同的专家路由，导致 logps 差异显著放大

为此，GRPO 通过硬裁剪来限制策略更新的幅度：

**硬裁剪的困境**：硬裁剪难以在稳定性和学习效率之间取得平衡——裁剪范围过严格会限制有效样本的数量，而过宽松则会引入离策略样本的噪声梯度，导致训练不稳定。

SAPO 使用温度控制的 sigmoid 软门控函数替代硬裁剪，实现平滑的梯度衰减。

SAPO 的核心是使用 sigmoid 函数对重要性采样比进行软门控：

对于正向优势（$A > 0$），使用正向门控：

对于负向优势（$A < 0$），使用负向门控：

其中：

- $\sigma(\cdot)$ 是 sigmoid 函数

- $\tau_{\mathrm{pos}}$ 和 $\tau_{\mathrm{neg}}$ 是温度参数，控制门控函数的斜率

- $r_t$ 是重要性采样比

其中 $g_t = g^{+}_t$ 当 $A > 0$，$g_t = g^{-}_t$ 当 $A < 0$。

温度参数 $\tau$ 控制软门控函数的衰减速率，数值越大，衰减越快。


论文指出正向优势会提升采样token的logit，并降低所有未采样token的logit；负向优势相反，提高许多未采样token的logit，可能会扩散到大量无关token上，带来一定的不稳定性。所以论文推荐设置温度 $\tau_\text{neg} > \tau_\text{pos}$，来使负向奖励的token梯度衰减更快，提升训练的稳定性和性能。

论文默认推荐 $\tau_{\mathrm{pos}} = 1.0$，$\tau_{\mathrm{neg}} = 1.05$。

| 参数| 类型| 默认值| 说明|

|------|------|--------|------|

|`--loss_type`|`str`| -| 设置为`sapo`|

|`--tau_pos`|`float`|`1.0`| 正向优势的温度参数，控制门控斜率|

|`--tau_neg`|`float`|`1.05`| 负向优势的温度参数，控制门控斜率|

swift rlhf \

--rlhf_type grpo \

--loss_type sapo \

--tau_pos 1.0 \

--tau_neg 1.05 \

训练脚本参考

-[swift](https://github.com/modelscope/ms-swift/blob/main/examples/train/grpo/internal/sapo.sh)

-[megatron swift](https://github.com/modelscope/ms-swift/blob/main/examples/megatron/grpo/sapo.sh)

GRPO训练支持五种不同的loss类型，主要区别在于归一化的维度上有所不同。

GRPO训练支持多种不同的loss类型，主要区别在于归一化的维度和梯度处理方式上有所不同。

当设置`loss_type sapo`时，使用软门控替代硬裁剪，详见[SAPO](../AdvancedResearch/SAPO.md)

其中 $g_{i,t} = \sigma(\tau \cdot (\rho_{i,t} - 1))$ 是温度控制的软门控函数。

其中：

- $\rho_{i,t} = \frac{\pi_\theta(y_{i,t}|y_{i,<t})}{\pi_{\theta_{\text{old}}}(y_{i,t}|y_{i,<t})}$ 是重要性采样权重

- $A_{i,t}$ 是优势函数

- $\epsilon$ 和 $\epsilon_{\text{high}}$ 是clipping参数

- $\text{detach}(\cdot)$ 表示该项不参与梯度计算

- $\sigma(\cdot)$ 是 sigmoid 函数，$\tau$ 是温度参数

- $N_p$ 是第$p$个进程的样本数量

**归一化维度：** 全局token维度（跨所有进程的completion token总数）

SAPO使用温度控制的软门控替代硬裁剪，实现平滑的梯度衰减。归一化方式与GRPO相同。

详细说明请参考[SAPO](../AdvancedResearch/SAPO.md)

- reward_model_plugin: The logic for the reward model, which defaults to ORM logic. For more information, please refer to[Customized Reward Models](./GRPO/DeveloperGuide/reward_model.md#custom-reward-model).

- dataset_shuffle: Whether to shuffle the dataset randomly. Default is True.

- truncation_strategy: The method to handle inputs exceeding`max_length`. Supported values are`delete` and`left`, representing deletion and left-side truncation respectively. The default is`left`. Note that for multi-modal models, left-side truncation may remove multi-modal tokens and cause a shape mismatch error during model forward. With the delete strategy, over-long or encoding-failed samples are discarded, and new samples are resampled from the original dataset to maintain the intended batch size.

- loss_type: The type of loss normalization. Options are['grpo', 'bnpo', 'dr_grpo', 'dapo', 'cispo'], default is 'grpo'. For details, refer to this[doc](./GRPO/DeveloperGuide/loss_types.md)

- loss_type: The type of loss normalization. Options are['grpo', 'bnpo', 'dr_grpo', 'dapo', 'cispo', 'sapo'], default is 'grpo'. For details, refer to this[doc](./GRPO/DeveloperGuide/loss_types.md)

- log_completions: Whether to log the model-generated content during training, to be used in conjunction with`--report_to wandb/swanlab`, default is False.

- Note: If`--report_to wandb/swanlab` is not set, a`completions.jsonl` will be created in the checkpoint to store the generated content.

- use_vllm: Whether to use vLLM as the infer_backend for GRPO generation, default is False.

- num_iterations: The number of updates per data sample, corresponding to the $\mu$ value in the GRPO paper. Default is 1.

- epsilon: epsilon value for clipping. Default is 0.2.

- epsilon_high: Upper clip coefficient, default is None. When set, it forms a clipping range of[epsilon, epsilon_high] together with epsilon.

- tau_pos: Temperature parameter for positive advantages in[SAPO](https://arxiv.org/abs/2511.20347) algorithm, controlling the sharpness of the soft gating function. Larger values make the gate sharper (closer to hard clipping), smaller values make it smoother. Default is 1.0.

- tau_neg: Temperature parameter for negative advantages in SAPO algorithm, controlling the sharpness of the soft gating function. Typically set`tau_neg > tau_pos` to apply stronger constraints on negative advantages. Default is 1.05.

- dynamic_sample: Exclude data within the group where the reward standard deviation is 0, and additionally sample new data. Default is False.

- max_resample_times: Under the dynamic_sample setting, limit the number of resampling attempts to a maximum of 3. Default is 3 times.

- overlong_filter: Skip overlong truncated samples, which will not be included in loss calculation. Default is False.

**Version Requirement**: ms-swift>=3.11

[Soft Adaptive Policy Optimization (SAPO)](https://arxiv.org/abs/2511.20347) addresses the issues caused by hard clipping in GRPO by proposing a temperature-controlled soft gate mechanism that smoothly attenuates off-policy updates while preserving useful learning signals.

When training LLMs with reinforcement learning, GRPO handles off-policy training by computing token-level importance sampling ratios:

However, token-level importance sampling ratios often exhibit high variance, which can be exacerbated in the following cases:

-**Long text generation**

-**MoE model routing heterogeneity**: The old-policy model during sampling and the training model may use different expert routing, significantly amplifying logps differences

To address this, GRPO uses hard clipping to limit the magnitude of policy updates:

**The Dilemma of Hard Clipping**: Hard clipping struggles to balance stability and learning efficiency—too strict clipping limits the number of effective samples, while too loose clipping introduces noisy gradients from off-policy samples, leading to training instability.

SAPO uses a temperature-controlled sigmoid soft gate function to replace hard clipping, achieving smooth gradient attenuation.

The core of SAPO is using the sigmoid function to apply soft gating on the importance sampling ratio:

For positive advantages ($A > 0$), use positive gating:

For negative advantages ($A < 0$), use negative gating:

where:

- $\sigma(\cdot)$ is the sigmoid function

- $\tau_{\mathrm{pos}}$ and $\tau_{\mathrm{neg}}$ are temperature parameters that control the gate function slope

- $r_t$ is the importance sampling ratio

where $g_t = g^{+}_t$ when $A > 0$, $g_t = g^{-}_t$ when $A < 0$.

The temperature parameter $\tau$ controls the decay rate of the soft gate function—larger values result in faster decay.


The paper points out that positive advantages increase the logit of sampled tokens while decreasing the logits of all unsampled tokens; negative advantages do the opposite, increasing the logits of many unsampled tokens, which may spread to a large number of irrelevant tokens and introduce instability. Therefore, the paper recommends setting $\tau_\text{neg} > \tau_\text{pos}$ to make the gradient decay faster for tokens with negative rewards, improving training stability and performance.

The paper recommends default values of $\tau_{\mathrm{pos}} = 1.0$ and $\tau_{\mathrm{neg}} = 1.05$.

| Parameter| Type| Default| Description|

|-----------|------|---------|-------------|

|`--loss_type`|`str`| -| Set to`sapo`|

|`--tau_pos`|`float`|`1.0`| Temperature parameter for positive advantages, controls gate slope|

|`--tau_neg`|`float`|`1.05`| Temperature parameter for negative advantages, controls gate slope|

swift rlhf \

--rlhf_type grpo \

--loss_type sapo \

--tau_pos 1.0 \

--tau_neg 1.05 \

Example training scripts:

-[swift](https://github.com/modelscope/ms-swift/blob/main/examples/train/grpo/internal/sapo.sh)

-[megatron swift](https://github.com/modelscope/ms-swift/blob/main/examples/megatron/grpo/sapo.sh)

GRPO training supportsfive differentloss types, with the maindifference being the normalization dimension.

GRPO training supportsmultipleloss types, with the maindifferences being the normalization dimension and gradient handling.

When setting`loss_type sapo`, soft gating replaces hard clipping, see[SAPO](../AdvancedResearch/SAPO.md)

where $g_{i,t} = \sigma(\tau \cdot (\rho_{i,t} - 1))$ is the temperature-controlled soft gate function.

where:

- $\rho_{i,t} = \frac{\pi_\theta(y_{i,t}|y_{i,<t})}{\pi_{\theta_{\text{old}}}(y_{i,t}|y_{i,<t})}$ is the importance sampling weight

- $A_{i,t}$ is the advantage function

- $\epsilon$ and $\epsilon_{\text{high}}$ are the clipping parameters

- $\text{detach}(\cdot)$ indicates that this term does not participate in gradient computation

- $\sigma(\cdot)$ is the sigmoid function, $\tau$ is the temperature parameter

- $N_p$ is the number of samples for the $p$-th process

**Normalization Dimension:** Global token dimension (total completion tokens across all processes)

SAPO uses temperature-controlled soft gating instead of hard clipping to achieve smooth gradient attenuation. The normalization method is the same as GRPO.

For details, please refer to[SAPO](../AdvancedResearch/SAPO.md)

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \

NPROC_PER_NODE=8 \

MAX_PIXELS=602112 \

megatron rlhf \

--rlhf_type grpo \

--loss_type sapo \

--tau_pos 1 \

--tau_neg 1.05 \

--model Qwen/Qwen2.5-VL-3B-Instruct \

--context_parallel_size 1 \

--tensor_model_parallel_size 1 \

--pipeline_model_parallel_size 1 \

--dataset AI-ModelScope/clevr_cogen_a_train \

--load_safetensorstrue \

--save_safetensorstrue \

--external_plugins examples/train/grpo/plugin/plugin.py \

--reward_funcs external_r1v_acc format \

--dynamic_samplefalse \

--steps_per_generation 4 \

--micro_batch_size 2 \

--global_batch_size 128 \

--num_generations 8 \

--use_vllmtrue \

--vllm_mode colocate \

--vllm_gpu_memory_utilization 0.7 \

--vllm_max_model_len 8192 \

--max_length 4096 \

--max_completion_length 4096 \

--train_type full \

--bf16true \

--importance_sampling_level token \

--epsilon 0.2 \

--epsilon_high 0.2 \

--overlong_filtertrue \

--max_epochs 1 \

--eval_interval 1000 \

--save_interval 1000 \

--sleep_level 2 \

--offload_modeltrue \

--offload_optimizertrue \

--log_interval 1 \

--recompute_granularity selective \

--finetune \

--lr 1e-6 \

--num_workers 8 \

--dataset_num_proc 8 \

--no_save_optim \

--no_save_rng \

--attention_backend flash \

--temperature 1.0 \

--system examples/train/grpo/prompt.txt \

--beta 0.001 \

--padding_freetrue \

--wandb_project swift-megatron \

--wandb_exp_name xxx