Repository files navigation

This repository contains code to incrementally build 3D Scene Graphs in real-time and is based on the papers:

• "Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization"
• "Foundations of Spatial Perception for Robotics: Hierarchical Representations and Real-time Systems"

If you find this code relevant for your work, please consider citing one or both of these papers. A bibtex entry is provided below:

@article{hughes2022hydra,title={Hydra: A Real-time Spatial Perception System for {3D} Scene Graph Construction and Optimization},fullauthor={Nathan Hughes, Yun Chang, and Luca Carlone},author={N. Hughes and Y. Chang and L. Carlone},booktitle={Robotics: Science and Systems (RSS)},pdf={http://www.roboticsproceedings.org/rss18/p050.pdf},year={2022},}@article{hughes2024foundations,title={Foundations of Spatial Perception for Robotics: Hierarchical Representations and Real-time Systems},fullauthor={Nathan Hughes and Yun Chang and Siyi Hu and Rajat Talak and Rumaisa Abdulhai and Jared Strader and Luca Carlone},author={N. Hughes and Y. Chang and S. Hu and R. Talak and R. Abdulhai and J. Strader and L. Carlone},journal={The International Journal of Robotics Research},doi={10.1177/02783649241229725},url={https://doi.org/10.1177/02783649241229725},year={2024},}

This work was partially funded by the AIA CRA FA8750-19-2-1000, ARL DCIST CRA W911NF-17-2-0181, and ONR RAIDER N00014-18-1-2828.

Research was sponsored by the United States Air Force Research Laboratory and the United States Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750-19-2-1000. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the United States Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.

Update (06/26/24): We've released the latest version of Hydra.This release also includes the following features:

• Room category classification networks from our most recent paper, availablehere
• Python bindings around Hydra and our interface for running with the habitat simulator (details below)
• Updates to Hydra to useconfig_utilities

Note
We've changed (and simplified) the dependencies required to build Hydra. This includes moving towards the ROS packaged version of GTSAM, dropping Kimera-Semantics and replacing voxblox withspatial_hash.Please make sure to double-check the rosinstall files to make sure you have all new dependencies (and feel free to remove unused old dependencies).

Update (06/26/23): We've released initial changes from the our newest paper.We also plan to release additional code, most notably for training the room classification networks and GNN-based descriptors as described in the above paper.We will link to the new repository once this is done.

Note
As part of the this release, we have moved ROS related code to a new repository locatedhere. This code (and our installation process) still do rely on the ROS ecosystem.

Hydra has been tested on Ubuntu 20.04 and ROS Noetic

You can follow the instructionshere to install ROS if you haven't already.Then, make sure you have some general requirements:

sudo apt install python3-rosdep python3-catkin-tools python3-vcstool

Finally, if you haven't set up rosdep yet:

sudo rosdep initrosdep update

⚠️ Warning
We don't support other platforms. Issues requesting support on other platforms (e.g., Ubuntu 18.04, Windows) will be summarily closed.

Depending on the nature of the issue, it may be helpful to browsethis page about debugging Hydra first.

To get started:

mkdir -p catkin_ws/srccd catkin_wscatkin initcatkin config -DCMAKE_BUILD_TYPE=Releasecd srcgit clone git@github.com:MIT-SPARK/Hydra.git hydravcs import . < hydra/install/hydra.rosinstallrosdep install --from-paths . --ignore-src -r -ycd ..catkin build

Note
Depending on the amount of RAM available on your machine and whether or not you are compiling Kimera-VIO as well, you may run out of memory when compiling withcatkin build directly (which will result in aGCC killed error). If this occurs, you can either specify fewer threads for catkin viacatkin build -j NUM_THREADS or compile certain larger packages directly first by building them specifically.

⚠️ In thevcs import step, GitHub may block too many concurrent requests. If you receivekex_exchange_identification: read: Connection reset by peer errors, try runningvcs import . < hydra/install/hydra.rosinstall --workers 1.

Please help us by creating new issues when you run into problems with these instructions!

To test Hydra out, you can just download a single scene (the office scene without humans is recommended, and can be foundhere.Make sure to decompress the rosbag (rosbag decompress path/to/bagfile) before running!

⚠️ Warning
Also make sure to source the workspace before starting.
This is typicallysource path/to/catkin_ws/devel/setup.bash, though if you use zsh you should use the correct setup file for that.

To start Hydra:

roslaunch hydra_ros uhumans2.launch

Then, start the rosbag in a separate terminal:

rosbag play path/to/rosbag --clock

Seehere for detailed instructions discussing how to run Hydra using ROS.These also detail how to use Hydra withKimera-VIO, including how to build Kimera-VIO alongside Hydra.

Seehere for information

Seehere for information

Note
This package is not public (yet)

Addsemantic_recolor to your workspace via:

roscd && cd ../srcvcs import . < hydra/install/semantic_overlay.rosinstall

Then, follow the instructions to install cuda and other dependencies for thesemantic_recolor package (which can be foundhere).

Finally, build your workspace:

catkin build