Repository files navigation

This ishloc, a modular toolbox for state-of-the-art 6-DoF visual localization. It implementsHierarchical Localization, leveraging image retrieval and feature matching, and is fast, accurate, and scalable. This codebase combines and makes easily accessible years of research on image matching and Structure-from-Motion.

Withhloc, you can:

• Reproduce state-of-the-art results on multiple indoor and outdoor visual localization benchmarks
• Run Structure-from-Motion with SuperPoint+SuperGlue to localize with your own datasets
• Evaluate your own local features or image retrieval for visual localization
• Implement new localization pipelines and debug them easily 🔥

Hierachical Localization uses both image retrieval and feature matching

Build 3D maps with Structure-from-Motion and localize any Internet image right from your browser!You can now runhloc and COLMAP in Google Colab with GPU for free. The notebookdemo.ipynb shows how to run SfM and localization in just a few steps. Try it with your own data and let us know!

hloc requires Python >=3.7 and PyTorch >=1.1. Installing the package locally pulls the other dependencies:

git clone --recursive https://github.com/cvg/Hierarchical-Localization/cd Hierarchical-Localization/python -m pip install -e.

All dependencies are listed inrequirements.txt.Starting withhloc-v1.3, installing COLMAP is not required anymore. This repository includes external local features as git submodules – don't forget to pull submodules withgit submodule update --init --recursive.

We also provide a Docker image:

docker build -t hloc:latest.docker run -it --rm -p 8888:8888 hloc:latest# for GPU support, add `--runtime=nvidia`jupyter notebook --ip 0.0.0.0 --port 8888 --no-browser --allow-root

The toolbox is composed of scripts, which roughly perform the following steps:

1. Extract local features, likeSuperPoint orDISK, for all database and query images
2. Build a reference 3D SfM model
   1. Find covisible database images, with retrieval or a prior SfM model
   2. Match these database pairs withSuperGlue or the fasterLightGlue
   3. Triangulate a new SfM model with COLMAP
3. Find database images relevant to each query, using retrieval
4. Match the query images
5. Run the localization
6. Visualize and debug

The localization can then be evaluated onvisuallocalization.net for the supported datasets. When 3D Lidar scans are available, such as for the indoor dataset InLoc, step 2. can be skipped.

Strcture of the toolbox:

• hloc/*.py : top-level scripts
• hloc/extractors/ : interfaces for feature extractors
• hloc/matchers/ : interfaces for feature matchers
• hloc/pipelines/ : entire pipelines for multiple datasets

hloc can be imported as an external package withimport hloc or called from the command line with:

python -m hloc.name_of_script --arg1 --arg2

We provide step-by-step guides to localize with Aachen, InLoc, and to generate reference poses for your own data using SfM. Just download the datasets and you're reading to go!

Have a look atpipeline_Aachen.ipynb for a step-by-step guide on localizing with Aachen. Play with the visualization, try new local features or matcher, and have fun! Don't like notebooks? You can also run all scripts from the command line.

The notebookpipeline_InLoc.ipynb shows the steps for localizing with InLoc. It's much simpler since a 3D SfM model is not needed.

We show inpipeline_SfM.ipynb how to run 3D reconstruction for an unordered set of images. This generates reference poses, and a nice sparse 3D model suitable for localization with the same pipeline as Aachen.

• Supported local feature extractors:SuperPoint,DISK,D2-Net,SIFT, andR2D2.
• Supported feature matchers:SuperGlue, its faster follow-upLightGlue, and nearest neighbor search with ratio test, distance test, and/or mutual check. hloc also supports dense matching withLoFTR.
• Supported image retrieval:NetVLAD,AP-GeM/DIR,OpenIBL, andMegaLoc.

Using NetVLAD for retrieval, we obtain the following best results:

Check outvisuallocalization.net/benchmark for more details and additional baselines.

We provide inhloc/pipelines/ scripts to run the reconstruction and the localization on the following datasets: Aachen Day-Night (v1.0 and v1.1), InLoc, Extended CMU Seasons, RobotCar Seasons, 4Seasons, Cambridge Landmarks, and 7-Scenes. For example, after downloading the datasetwith the instructions given here, we can run the Aachen Day-Night pipeline with SuperPoint+SuperGlue using the command:

python -m hloc.pipelines.Aachen.pipeline [--outputs ./outputs/aachen]

If you report any of the above results in a publication, or use any of the tools provided here, please consider citing bothHierarchical Localization andSuperGlue papers:

@inproceedings{sarlin2019coarse,  title     = {From Coarse to Fine: Robust Hierarchical Localization at Large Scale},  author    = {Paul-Edouard Sarlin and               Cesar Cadena and               Roland Siegwart and               Marcin Dymczyk},  booktitle = {CVPR},  year      = {2019}}@inproceedings{sarlin2020superglue,  title     = {{SuperGlue}: Learning Feature Matching with Graph Neural Networks},  author    = {Paul-Edouard Sarlin and               Daniel DeTone and               Tomasz Malisiewicz and               Andrew Rabinovich},  booktitle = {CVPR},  year      = {2020},}

opts=dict(camera_model='SIMPLE_RADIAL',camera_params=','.join(map(str, (f,cx,cy,k))))model=reconstruction.main(...,image_options=opts)

python -m hloc.reconstruction [...] --image_options camera_model='"SIMPLE_RADIAL"' camera_params='"256,256,256,0"'

reconstruction.main(...,mapper_options=dict(ba_refine_focal_length=False,ba_refine_extra_params=False))

python -m hloc.reconstruction [...] --mapper_options ba_refine_focal_length=False ba_refine_extra_params=False

External contributions are very much welcome. Please follow thePEP8 style guidelines using a linter like flake8. This is a non-exhaustive list of features that might be valuable additions:

• support for GPS (extraction from EXIF + guided retrieval)
• covisibility clustering for InLoc
• visualization of the raw predictions (features and matches)
• other local features or image retrieval

Created and maintained byPaul-Edouard Sarlin with the help of many contributors.