Repository files navigation

Tianshou (天授) is a reinforcement learning (RL) library based on pure PyTorch andGymnasium. Tianshou's main features at a glance are:

1. Modular low-level interfaces for algorithm developers (RL researchers) that are both flexible, hackable and type-safe.
2. Convenient high-level interfaces for applications of RL (training an implemented algorithm on a custom environment).
3. Large scope: online (on- and off-policy) and offline RL, experimental support for multi-agent RL (MARL), experimental support for model-based RL, and more

Unlike other reinforcement learning libraries, which may have complex codebases,unfriendly high-level APIs, or are not optimized for speed, Tianshou provides a high-performance, modularized frameworkand user-friendly interfaces for building deep reinforcement learning agents. One more aspect that sets Tianshou apart is itsgenerality: it supports online and offline RL, multi-agent RL, and model-based algorithms.

Tianshou aims at enabling concise implementations, both for researchers and practitioners, without sacrificing flexibility.

Supported algorithms include:

• Deep Q-Network (DQN)
• Double DQN
• Dueling DQN
• Branching DQN
• Categorical DQN (C51)
• Rainbow DQN (Rainbow)
• Quantile Regression DQN (QRDQN)
• Implicit Quantile Network (IQN)
• Fully-parameterized Quantile Function (FQF)
• Policy Gradient (PG)
• Natural Policy Gradient (NPG)
• Advantage Actor-Critic (A2C)
• Trust Region Policy Optimization (TRPO)
• Proximal Policy Optimization (PPO)
• Deep Deterministic Policy Gradient (DDPG)
• Twin Delayed DDPG (TD3)
• Soft Actor-Critic (SAC)
• Randomized Ensembled Double Q-Learning (REDQ)
• Discrete Soft Actor-Critic (SAC-Discrete)
• Vanilla Imitation Learning
• Batch-Constrained deep Q-Learning (BCQ)
• Conservative Q-Learning (CQL)
• Twin Delayed DDPG with Behavior Cloning (TD3+BC)
• Discrete Batch-Constrained deep Q-Learning (BCQ-Discrete)
• Discrete Conservative Q-Learning (CQL-Discrete)
• Discrete Critic Regularized Regression (CRR-Discrete)
• Generative Adversarial Imitation Learning (GAIL)
• Prioritized Experience Replay (PER)
• Generalized Advantage Estimator (GAE)
• Posterior Sampling Reinforcement Learning (PSRL)
• Intrinsic Curiosity Module (ICM)
• Hindsight Experience Replay (HER)

Other noteworthy features:

• Elegant framework with dual APIs:
  • Tianshou's high-level API maximizes ease of use for application development while still retaining a high degreeof flexibility.
  • The fundamental procedural API provides a maximum of flexibility for algorithm development without beingoverly verbose.
• State-of-the-art results inMuJoCo benchmarks for REINFORCE/A2C/TRPO/PPO/DDPG/TD3/SAC algorithms
• Support for vectorized environments (synchronous or asynchronous) for all algorithms (seeusage)
• Support for super-fast vectorized environments based onEnvPool for all algorithms (seeusage)
• Support for recurrent state representations in actor networks and critic networks (RNN-style training for POMDPs) (seeusage)
• Support any type of environment state/action (e.g. a dict, a self-defined class, ...)Usage
• Support for customized training processes (seeusage)
• Support n-step returns estimation and prioritized experience replay for all Q-learning based algorithms; GAE, nstep and PER are highly optimized thanks to numba's just-in-time compilation and vectorized numpy operations
• Support for multi-agent RL (seeusage)
• Support for logging based on bothTensorBoard andW&B
• Support for multi-GPU training (seeusage)
• Comprehensive documentation, PEP8 code-style checking, type checking and thoroughtests

In Chinese, Tianshou means divinely ordained, being derived to the gift of being born.Tianshou is a reinforcement learning platform, and the nature of RL is not learn from humans.So taking "Tianshou" means that there is no teacher to learn from, but rather to learn by oneself through constant interaction with the environment.

“天授”意指上天所授，引申为与生具有的天赋。天授是强化学习平台，而强化学习算法并不是向人类学习的，所以取“天授”意思是没有老师来教，而是自己通过跟环境不断交互来进行学习。

Tianshou is currently hosted onPyPI andconda-forge. It requires Python >= 3.11.

For installing the most recent version of Tianshou, the best way is clone the repository and install it withpoetry(which you need to install on your system first)

git clone git@github.com:thu-ml/tianshou.gitcd tianshoupoetry install

You can also install the dev requirements by adding--with dev or the extrasfor say mujoco and acceleration byenvpoolby adding--extras "mujoco envpool"

If you wish to install multiple extras, ensure that you include them in a single command. Sequential calls topoetry install --extras xxx will overwrite prior installations, leaving only the last specified extras installed.Or you may install all the following extras by adding--all-extras.

Available extras are:

• atari (for Atari environments)
• box2d (for Box2D environments)
• classic_control (for classic control (discrete) environments)
• mujoco (for MuJoCo environments)
• mujoco-py (for legacy mujoco-py environments¹)
• pybullet (for pybullet environments)
• robotics (for gymnasium-robotics environments)
• vizdoom (for ViZDoom environments)
• envpool (forenvpool integration)
• argparse (in order to be able to run the high level API examples)

Otherwise, you can install the latest release from PyPI (currentlyfar behind the master) with the following command:

$ pip install tianshou

If you are using Anaconda or Miniconda, you can install Tianshou from conda-forge:

$ conda install tianshou -c conda-forge

Alternatively to the poetry install, you can also install the latest source version through GitHub:

$ pip install git+https://github.com/thu-ml/tianshou.git@master --upgrade

Finally, you may check the installation via your Python console as follows:

importtianshouprint(tianshou.__version__)

If no errors are reported, you have successfully installed Tianshou.

Tutorials and API documentation are hosted ontianshou.readthedocs.io.

Find example scripts in thetest/ andexamples/ folders.

RL Platform	Documentation	Code Coverage	Type Hints	Last Update
Stable-Baselines3			✔️
Ray/RLlib		➖(1)	✔️
SpinningUp		❌	❌
Dopamine		❌	❌
ACME		➖(1)	✔️
Sample Factory	➖		❌
Tianshou			✔️

^{(1): it has continuous integration but the coverage rate is not available}

Tianshou is rigorously tested. In contrast to other RL platforms,our tests include the full agent training procedure for all of the implemented algorithms. Our tests would fail once if any of the agents failed to achieve a consistent level of performance on limited epochs.Our tests thus ensure reproducibility.Check out theGitHub Actions page for more detail.

Atari and MuJoCo benchmark results can be found in theexamples/atari/ andexamples/mujoco/ folders respectively.Our MuJoCo results reach or exceed the level of performance of most existing benchmarks.

All algorithms implement the following, highly general API:

• __init__: initialize the policy;
• forward: compute actions based on given observations;
• process_buffer: process initial buffer, which is useful for some offline learning algorithms
• process_fn: preprocess data from the replay buffer (since we have reformulatedall algorithms to replay buffer-based algorithms);
• learn: learn from a given batch of data;
• post_process_fn: update the replay buffer from the learning process (e.g., prioritized replay buffer needs to update the weight);
• update: the main interface for training, i.e.,process_fn -> learn -> post_process_fn.

The implementation of this API suffices for a new algorithm to be applicable within Tianshou,making experimenation with new approaches particularly straightforward.

Tianshou provides two API levels:

• the high-level interface, which provides ease of use for end users seeking to run deep reinforcement learning applications
• the procedural interface, which provides a maximum of control, especially for very advanced users and developers of reinforcement learning algorithms.

In the following, let us consider an example application using theCartPole gymnasium environment.We shall apply the deep Q network (DQN) learning algorithm using both APIs.

To get started, we need some imports.

fromtianshou.highlevel.configimportSamplingConfigfromtianshou.highlevel.envimport (EnvFactoryRegistered,VectorEnvType,)fromtianshou.highlevel.experimentimportDQNExperimentBuilder,ExperimentConfigfromtianshou.highlevel.params.policy_paramsimportDQNParamsfromtianshou.highlevel.trainerimport (EpochTestCallbackDQNSetEps,EpochTrainCallbackDQNSetEps,EpochStopCallbackRewardThreshold)

In the high-level API, the basis for an RL experiment is anExperimentBuilderwith which we can build the experiment we then seek to run.Since we want to use DQN, we use the specializationDQNExperimentBuilder.The other imports serve to provide configuration options for our experiment.

The high-level API provides largely declarative semantics, i.e. the code isalmost exclusively concerned with configuration that controls what to do(rather than how to do it).

experiment= (DQNExperimentBuilder(EnvFactoryRegistered(task="CartPole-v1",train_seed=0,test_seed=0,venv_type=VectorEnvType.DUMMY),ExperimentConfig(persistence_enabled=False,watch=True,watch_render=1/35,watch_num_episodes=100,        ),SamplingConfig(num_epochs=10,step_per_epoch=10000,batch_size=64,num_train_envs=10,num_test_envs=100,buffer_size=20000,step_per_collect=10,update_per_step=1/10,        ),    )    .with_dqn_params(DQNParams(lr=1e-3,discount_factor=0.9,estimation_step=3,target_update_freq=320,        ),    )    .with_model_factory_default(hidden_sizes=(64,64))    .with_epoch_train_callback(EpochTrainCallbackDQNSetEps(0.3))    .with_epoch_test_callback(EpochTestCallbackDQNSetEps(0.0))    .with_epoch_stop_callback(EpochStopCallbackRewardThreshold(195))    .build())experiment.run()

The experiment builder takes three arguments:

• the environment factory for the creation of environments. In this case,we use an existing factory implementation for gymnasium environments.
• the experiment configuration, which controls persistence and the overallexperiment flow. In this case, we have configured that we want to observethe agent's behavior after it is trained (watch=True) for a number ofepisodes (watch_num_episodes=100). We have disabled persistence, becausewe do not want to save training logs, the agent or its configuration forfuture use.
• the sampling configuration, which controls fundamental training parameters,such as the total number of epochs we run the experiment for (num_epochs=10)
  and the number of environment steps each epoch shall consist of(step_per_epoch=10000).Every epoch consists of a series of data collection (rollout) steps andtraining steps.The parameterstep_per_collect controls the amount of data that iscollected in each collection step and after each collection step, weperform a training step, applying a gradient-based update based on a sampleof data (batch_size=64) taken from the buffer of data that has beencollected. For further details, see the documentation ofSamplingConfig.

We then proceed to configure some of the parameters of the DQN algorithm itselfand of the neural network model we want to use.A DQN-specific detail is the use of callbacks to configure the algorithm'sepsilon parameter for exploration. We want to use random exploration during rollouts(train callback), but we don't when evaluating the agent's performance in the testenvironments (test callback).

Find the script inexamples/discrete/discrete_dqn_hl.py.Here's a run (with the training time cut short):

Find many further applications of the high-level API in theexamples/ folder;look for scripts ending with_hl.py.Note that most of these examples require the extra packageargparse(install it by adding--extras argparse when invoking poetry).

Let us now consider an analogous example in the procedural API.Find the full script inexamples/discrete/discrete_dqn.py.

First, import some relevant packages:

importgymnasiumasgymimporttorchfromtorch.utils.tensorboardimportSummaryWriterimporttianshouasts

Define some hyper-parameters:

task='CartPole-v1'lr,epoch,batch_size=1e-3,10,64train_num,test_num=10,100gamma,n_step,target_freq=0.9,3,320buffer_size=20000eps_train,eps_test=0.1,0.05step_per_epoch,step_per_collect=10000,10

Initialize the logger:

logger=ts.utils.TensorboardLogger(SummaryWriter('log/dqn'))# For other loggers, see https://tianshou.readthedocs.io/en/master/01_tutorials/05_logger.html

Make environments:

# You can also try SubprocVectorEnv, which will use parallelizationtrain_envs=ts.env.DummyVectorEnv([lambda:gym.make(task)for_inrange(train_num)])test_envs=ts.env.DummyVectorEnv([lambda:gym.make(task)for_inrange(test_num)])

Create the network as well as its optimizer:

fromtianshou.utils.net.commonimportNet# Note: You can easily define other networks.# See https://tianshou.readthedocs.io/en/master/01_tutorials/00_dqn.html#build-the-networkenv=gym.make(task,render_mode="human")state_shape=env.observation_space.shapeorenv.observation_space.naction_shape=env.action_space.shapeorenv.action_space.nnet=Net(state_shape=state_shape,action_shape=action_shape,hidden_sizes=[128,128,128])optim=torch.optim.Adam(net.parameters(),lr=lr)

Set up the policy and collectors:

policy=ts.policy.DQNPolicy(model=net,optim=optim,discount_factor=gamma,action_space=env.action_space,estimation_step=n_step,target_update_freq=target_freq)train_collector=ts.data.Collector(policy,train_envs,ts.data.VectorReplayBuffer(buffer_size,train_num),exploration_noise=True)test_collector=ts.data.Collector(policy,test_envs,exploration_noise=True)# because DQN uses epsilon-greedy method

Let's train it:

result=ts.trainer.OffpolicyTrainer(policy=policy,train_collector=train_collector,test_collector=test_collector,max_epoch=epoch,step_per_epoch=step_per_epoch,step_per_collect=step_per_collect,episode_per_test=test_num,batch_size=batch_size,update_per_step=1/step_per_collect,train_fn=lambdaepoch,env_step:policy.set_eps(eps_train),test_fn=lambdaepoch,env_step:policy.set_eps(eps_test),stop_fn=lambdamean_rewards:mean_rewards>=env.spec.reward_threshold,logger=logger,).run()print(f"Finished training in{result.timing.total_time} seconds")

Save/load the trained policy (it's exactly the same as loading atorch.nn.module):

torch.save(policy.state_dict(),'dqn.pth')policy.load_state_dict(torch.load('dqn.pth'))

Watch the agent with 35 FPS:

policy.eval()policy.set_eps(eps_test)collector=ts.data.Collector(policy,env,exploration_noise=True)collector.collect(n_episode=1,render=1/35)

Inspect the data saved in TensorBoard:

$ tensorboard --logdir log/dqn

Please read thedocumentation for advanced usage.

Tianshou is still under development.Further algorithms and features are continuously being added, and we always welcome contributions to help make Tianshou better.If you would like to contribute, please check outthis link.

If you find Tianshou useful, please cite it in your publications.

@article{tianshou,  author  = {Jiayi Weng and Huayu Chen and Dong Yan and Kaichao You and Alexis Duburcq and Minghao Zhang and Yi Su and Hang Su and Jun Zhu},  title   = {Tianshou: A Highly Modularized Deep Reinforcement Learning Library},  journal = {Journal of Machine Learning Research},  year    = {2022},  volume  = {23},  number  = {267},  pages   = {1--6},  url     = {http://jmlr.org/papers/v23/21-1127.html}}

Tianshou is supported byappliedAI Institute for Europe,who is committed to providing long-term support and development.

Tianshou was previously a reinforcement learning platform based on TensorFlow. You can check out the branchpriv for more detail. Many thanks toHaosheng Zou's pioneering work for Tianshou before version 0.1.1.

We would like to thankTSAIL andInstitute for Artificial Intelligence, Tsinghua University for providing such an excellent AI research platform.