CleanRL - Overview
CleanRL is a Deep Reinforcement Learning library that provides high-quality single-file implementation with research-friendly features. The implementation is clean and simple, yet we can scale it to run thousands of experiments using AWS Batch. The highlight features of CleanRL are:
- 📜 Single-file implementation
- Every detail about an algorithm variant is put into a single standalone file.
- For example, our
ppo_atari.pyonly has 340 lines of code but contains all implementation details on how PPO works with Atari games,so it is a great reference implementation to read for folks who do not wish to read an entire modular library. - 📊 Benchmarked Implementation (7+ algorithms and 34+ games at https://benchmark.cleanrl.dev)
- 📈 Tensorboard Logging
- 🪛 Local Reproducibility via Seeding
- 🎮 Videos of Gameplay Capturing
- 🧫 Experiment Management withWeights and Biases
- 💸 Cloud Integration with docker and AWS
You can read more about CleanRL in ourtechnical paper anddocumentation.
CleanRL only contains implementations ofonline deep reinforcement learning algorithms. If you are looking foroffline algorithms, please check outcorl-team/CORL, which shares a similar design philosophy as CleanRL.
Info
Support for Gymnasium:Farama-Foundation/Gymnasium is the next generation ofopenai/gym that will continue to be maintained and introduce new features. Please see theirannouncement for further detail. We are migrating togymnasium and the progress can be tracked invwxyzjn/cleanrl#277.
Warning
CleanRL isnot a modular library and therefore it is not meant to be imported. At the cost of duplicate code, we make all implementation details of a DRL algorithm variant easy to understand, so CleanRL comes with its own pros and cons. You should consider using CleanRL if you want to 1) understand all implementation details of an algorithm's variant or 2) prototype advanced features that other modular DRL libraries do not support (CleanRL has minimal lines of code so it gives you great debugging experience and you don't have do a lot of subclassing like sometimes in modular DRL libraries).
Citing CleanRL
If you use CleanRL in your work, please cite our technicalpaper:
@article{huang2022cleanrl,author={Shengyi Huang and Rousslan Fernand Julien Dossa and Chang Ye and Jeff Braga and Dipam Chakraborty and Kinal Mehta and João G.M. Araújo},title={CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms},journal={Journal of Machine Learning Research},year={2022},volume={23},number={274},pages={1--18},url={http://jmlr.org/papers/v23/21-1342.html}}