Repository files navigation

This repository provides the implementation of several reinforcement learning tasks with multiple real-world robots.These tasks come with an interface similar toOpenAI-Gym so that learning algorithms can be plugged in easily and in a uniform manner across tasks.All the tasks here are implemented based on a computational framework of robot-agent communication proposed by Mahmood et al. (2018a), which we callSenseAct.In this computational framework, agent and environment-related computations are ordered and distributed among multiple concurrent processes in a specific way. By doing so, SenseAct enables the following:

• Timely communication between the learning agent and multiple robotic devices with reduced latency,
• Easy and systematic design of robotic tasks for reinforcement learning agents,
• Facilitate reproducible real-world reinforcement learning.

This repository provides the following real-world robotic tasks, which are proposed by Mahmood et al. (2018b) as benchmark tasks for reinforcement learning algorithms:

Tested on UR Software v. 3.3.4.310

• UR-Reacher (both 2 joint and 6 joint control)

UR-Reacher-2	UR-Reacher-6

Currently we only support MX-64AT.

• DXL-Reacher
• DXL-Tracker

DXL-Reacher	DXL-Tracker

• Create-Mover
• Create-Docker

Create-Mover	Create-Docker

Mahmood et al. (2018b) provide extensive results comparing multiple reinforcement learning algorithms on the above tasks, and Mahmood et al. (2018a) show the effect of different task-setup elements in learning. Their results can be reproduced by using this repository (seedocumentation for more information).

The branchmaster is the latest official release anddev is current development branch.

SenseAct uses Python3 (>=3.5), and all other requirements are automatically installed via pip.

On Linux and Mac OS X, run the following:

1. git clone https://github.com/kindredresearch/SenseAct.git
2. cd SenseAct
3. pip install -e . orpip3 install -e . depends on your setup

To replicate experimental results from the paper please install the tag v0.1.1 from the git repo.

1. git fetch --all --tags
2. git checkout tags/v0.1.1

Additional instruction for installingOpenAI Baselines needed for running theadvanced examples is given in thecorresponding readme.

Dynamixels can be controlled by drivers written using either ctypes byRobotis or pyserial, which can be chosen by passing eitherTrue (ctypes) orFalse (pyserial) as an argument to theuse_ctypes_driver parameter of a Dynamixel-based task (e.g., seeexamples/advanced/dxl_reacher.py). We found the ctypes-based driver to provide substantially more timely and precise communication compared to the pyserial-based one.

In order to use the CType-based driver, we need to install gcc and relevant packages for compiling the C libraries:

sudo apt-get install gcc-5 build-essential gcc-multilib g++-multilib

Then run the following script to download and compile the Dynamixel driver C libraries:

sudo bash setup_dxl.sh

For additional setup and troubleshooting information regarding Dynamixels, please seeDXL Docs.

You can check whether SenseAct is installed correctly by running the included unit tests.

cd SenseActpython -m unittest discover -b

Installation problems? Feature requests? General questions?

• read through examples and APIdocumentation
• create github issues on theSenseAct project
• join the mailing listhttps://groups.google.com/forum/#!forum/senseact

This project is developed by theKindred AI Research team.Rupam Mahmood,Dmytro Korenkevych, andBrent Komer originally developed the computational framework and the UR tasks.William Ma developed the Create 2 tasks and contributed substantially by adding new features to SenseAct.Gautham Vasan developed the DXL tasks.Francois Hogan developed the simulated task.

James Bergstra provided support and guidance throughout the development.Adrian Martin,Scott Rostrup, andJonathan Yep developed the pyserial DXL driver for a Kindred project, which was used for the SenseAct DXL Communicator.Daniel Snider,Oliver Limoyo,Dylan Ashley, andCraig Sherstan tested the framework, provided thoughtful suggestions, and confirmed the reproducibility of learning by running experiments on real robots.

For the SenseAct computational framework and the UR-Reacher tasks, please cite Mahmood et al. (2018a). For the DXL and the Create 2 tasks, please cite Mahmood et al. (2018b).

• Mahmood, A. R., Korenkevych, D., Komer,B. J., Bergstra, J. (2018a).Setting up a reinforcement learning task with a real-world robot. InIEEE/RSJ International Conference on Intelligent Robots and Systems.

• Mahmood, A. R., Korenkevych, D., Vasan, G., Ma, W., Bergstra, J. (2018b).Benchmarking reinforcement learning algorithms on real-world robots. InProceedings of the 2nd Annual Conference on Robot Learning.