Repository files navigation

|Paper |Documentation |Ask DeepWiki |🤗 Models & Data |WeChat (微信) Group |

AReaL is an open-sourcefully asynchronous reinforcement learning training systemfor largereasoning and agentic models, developed by the AReaL Team at Ant Group.Built upon the open-source projectReaLHF,we are fully committed to open-source principles by providing training details, data,and infrastructure required to reproduce our results along with the models themselves.AReaL aims to help everyone build their own AI agents easily and affordably. Our teamloves milk tea because it's delicious, customizable, and affordable. We hope you enjoyour project just as you enjoy real-world milk tea (cheers).

AReaL Highlights

• ⚡Flexibility: Seamless customization formulti-turn agentic rolloutworkflows within a single file, and smooth integration withother agentic tooling frameworks.
• 🚀Scalability: Through algorithm-system co-design, AReaL deliversstable fullyasynchronous RL training withindustry-leading speed. AReaL seamlessly adapts todiverse computational environments, scaling from a single node to 1,000+ GPUs.
• 🔪Cutting-Edge Performance: AReaL produces state-of-the-artmath,coding, andsearch agents with exceptionalcapabilities.

[2025/08/30] Introducing ASearcher, a state-of-the-art search agent built withAReaL's end-to-end asynchronous RL training. Check out thepaper andtheopen-source repository!

[2025/07/31] (AReaL-lite) We introduce AReaL-lite, alightweight version ofAReaL designed specifically for AI researchers and rapid prototyping. AReaL-litefeatures analgorithm-first API design that prioritizes ease of use and algorithmdevelopment, while natively supportingfully asynchronous agentic RL. With 80% fewerlines of code, AReaL-lite maintains 90% of AReaL's performance and core functionality.Check outour AReaL-lite design documentation andthe quickstart guide tobegin your journey withAReaL-lite!

Our training scripts automatically download the required dataset (openai/gsm8k) andmodel (Qwen/Qwen2-1.5B-Instruct). To run on a single node:

python3 -m areal.launcher.local \  examples/math/gsm8k_grpo.py \  --config examples/math/gsm8k_grpo.yaml

To run on a Ray cluster with 2 nodes and 8 GPUs per node (remember to update paths inthe YAML file to point to your shared storage):

python3 -m areal.launcher.ray \  examples/math/gsm8k_grpo.py \  --config examples/math/gsm8k_grpo.yaml \  cluster.n_nodes=2 \  cluster.n_gpus_per_node=8

For comprehensive setup instructions, seeour quickstart guide.

• Installation
• Quickstart
• CLI Configurations
• Asynchronous RL Explained
• Fine-Tuning Large MoE
• Agentic RL
• Debugging Best Practices
• Handling OOM Issues

• Running GRPO on GSM8K dataset with AReaL-lite

• Customize dataset with AReaL-lite
• Customize Agentic/RVLR rollout workflows with AReaL-lite
• Customize algorithms with AReaL-lite

We warmly welcome contributions from the community! Whether you're fixing bugs, addingfeatures, improving documentation, or helping others, your contribution is valued.Please check ourContributing Guide for detailed information.

# Fork and clone the repositorygit clone https://github.com/YOUR-USERNAME/AReaLcd AReaL# Install in development modepip install -e".[dev]"# Set up pre-commit hooks for automatic formattingpip install pre-commitpre-commit install# Make changesgit checkout -b feat/gpt-o5git add.# `git commit` will automatically format your filegit commit -m"Implement gpt-o5 training loop"git push

• GitHub Discussions - Askquestions, share ideas, and connect with the community
• WeChat Group - Join our WeChat community (微信群)
• Project Roadmap - See what we're working on and what's planned

• Full Roadmap
• 2025 Q4 Roadmap

AReaL is under active development with planned minor releases weekly and major releasesmonthly. We warmly welcome community engagement and contributions. We are alsoactively hiring interns and full-time employees with open positions in both the USand China.

We gratefully acknowledge that major contributors are from the AReaL Team at Ant Groupand the Institute for Interdisciplinary Information Sciences, Tsinghua University.

We have also received invaluable assistance from the following groups (listedalphabetically):

• The Data Intelligence Lab at Ant Research for their data support

• TheRelaxed System Lab from HKUST forseamless collaboration on numerous system-related aspects

• TheSGLang team for supporting custom weightupdate features and their contributions during AReaL-lite development

• The Super Computing Technology (SCT) team at Ant Group for their expertise inlarge-scale cluster operations and maintenance

• Special thanks to @Lyken17 for providing valuable suggestions throughout ourdevelopment process

We also deeply appreciate all pioneering work from the community, particularly theReaLHF project from OpenPsi Inc. and otheroutstanding projects, including but not limited toDeepScaleR,Open-Reasoner-Zero,OpenRLHF,VeRL,SGLang,QwQ,Light-R1, andDAPO.

@inproceedings{mei2025real,author       ={Mei, Zhiyu and Fu, Wei and Li, Kaiwei and Wang, Guangju and Zhang, Huanchen and Wu, Yi},title        ={ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocation},booktitle    ={Proceedings of the Eighth Conference on Machine Learning and Systems,                  MLSys 2025, Santa Clara, CA, USA, May 12-15, 2025},publisher    ={mlsys.org},year         ={2025},}

@misc{fu2025areal,title={AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning},author={Wei Fu and Jiaxuan Gao and Xujie Shen and Chen Zhu and Zhiyu Mei and Chuyi He and Shusheng Xu and Guo Wei and Jun Mei and Jiashu Wang and Tongkai Yang and Binhang Yuan and Yi Wu},year={2025},eprint={2505.24298},archivePrefix={arXiv},primaryClass={cs.LG},url={https://arxiv.org/abs/2505.24298},}