Repository files navigation

A unified API for quickly and easily trying 37 (and growing!) image matching models.

Jump to:Install |Use |Models |Add a Model/Contributing |Acknowledgements |Cite

Compare matching models across various scenes. For example, we showSIFT-LightGlue andLoFTR matches on pairs:

(1) outdoor, (2) indoor, (3) satellite remote sensing, (4) paintings, (5) a false positive, and (6) spherical.

You can also extract keypoints and associated descriptors.

If you want to to install from source (easiest to edit, usebenchmark.py,demo.ipynb),

git clone --recursive https://github.com/alexstoken/image-matching-modelscd image-matching-modelspip install -e.

Some models require additional optional dependencies which are not included in the default list. To install these, use

pip install .[all]

This will install all dependencies needed to run all models.

We recommend using torch>=2.2, we didn't test most models with older versions.

You can install directly to your package directory with

pip install git+https://github.com/alexstoken/image-matching-models.git

Similar to the above, to get all optional dependencies, use the[all] addendum:

pip install"image-matching-models[all] @ git+https://github.com/alexstoken/image-matching-models.git"

You can use any of the matchers with

frommatchingimportget_matcherfrommatching.vizimportplot_matchesdevice='cuda'# 'cpu'matcher=get_matcher('superpoint-lg',device=device)# Choose any of our ~30+ matchers listed belowimg_size=512# optionalimg0=matcher.load_image('assets/example_pairs/outdoor/montmartre_close.jpg',resize=img_size)img1=matcher.load_image('assets/example_pairs/outdoor/montmartre_far.jpg',resize=img_size)result=matcher(img0,img1)num_inliers,H,inlier_kpts0,inlier_kpts1=result['num_inliers'],result['H'],result['inlier_kpts0'],result['inlier_kpts1']# result.keys() = ['num_inliers', 'H', 'all_kpts0', 'all_kpts1', 'all_desc0', 'all_desc1', 'matched_kpts0', 'matched_kpts1', 'inlier_kpts0', 'inlier_kpts1']plot_matches(img0,img1,result,save_path='plot_matches.png')

You can also run this as a standalone script, which will perform inference on the the examples inside./assets. You may also resolution (im_size) and number of keypoints (n_kpts). This will take a few seconds on a laptop's CPU, and will produce the same images that you see above.

python main_matcher.py --matcher sift-lg --device cpu --out_dir output_sift-lg

wheresift-lg will useSIFT + LightGlue.

The script will generate an image with the matching keypoints for each pair, under./output_sift-lg.

To use on your images you have three options:

1. create a directory with sub-directories, with two images per sub-directory, just like./assets/example_pairs. Then use aspython main_matcher.py --input path/to/dir
2. create a file with pairs of paths, separate by a space, just likeassets/example_pairs_paths.txt. Then use aspython main_matcher.py --input path/to/file.txt
3. import the matcher package into a script/notebook and use from there, as in the example above

To extract keypoints and descriptions (when available) from a single image, use theextract() method.

frommatchingimportget_matcherdevice='cuda'# 'cpu'matcher=get_matcher('xfeat',device=device)# Choose any of our ~30+ matchers listed belowimg_size=512# optionalimg=matcher.load_image('assets/example_pairs/outdoor/montmartre_close.jpg',resize=img_size)result=matcher.extract(img)# result.keys() = ['all_kpts0', 'all_desc0']plot_kpts(img,result)

As with matching, you can also run extraction from the command line

python main_extractor.py --matcher sift-lg --device cpu --out_dir output_sift-lg --n_kpts 2048

Run either variant via:

# ELoFTR backbone (defaults to 832px NPE size)python main_matcher.py --matcher matchanything-eloftr --device cuda --im_size 832 --out_dir outputs_matchanything-eloftr# RoMa backbone (AMP disabled on CPU automatically)python main_matcher.py --matcher matchanything-roma --device cuda --im_size 832 --out_dir outputs_matchanything-roma

Weights download automatically on first MatchAnything use and are cached undermatching/model_weights/matchanything.For submodule setup and troubleshooting, seedocs/matchanything.md.

You can choose any of the following methods (input toget_matcher()):

Dense:roma, tiny-roma, dust3r, mast3r, minima-roma

Semi-dense:loftr, eloftr, se2loftr, xoftr, minima-loftr, aspanformer, matchformer, xfeat-star, xfeat-star-steerers[-perm/-learned], edm, rdd-star

Sparse:[sift, superpoint, disk, aliked, dedode, doghardnet, gim, xfeat]-lg, dedode, steerers, affine-steerers, xfeat-steerers[-perm/learned], dedode-kornia, [sift, orb, doghardnet]-nn, patch2pix, superglue, r2d2, d2net, gim-dkm, xfeat, omniglue, [dedode, xfeat, aliked]-subpx, [sift, superpoint]-sphereglue, minima-splg, liftfeat, rdd-[sparse,lg, aliked], ripe, lisrd

Most matchers can run on CPU and GPU. MPS is not tested. SeeModel Details for runtimes. If a runtime is ❌, it means that model can not run on that device.

Our implementation of Patch2Pix (+ Patch2PixSuperGlue), R2D2, and D2Net are based on theImage Matching Toolbox (IMT). LoFTR and DeDoDe-Lightglue are fromKornia. Other models are based on the offical repos above.

Runtime benchmark is the average of 5 iterations over the 5 pairs of examples in theassets/example_pairs folder at image size 512x512. Benchmark is done usingbenchmark.py on an NVIDIA RTX A4000 GPU. Results rounded to the hundredths place.

*SphereGlue model runtimes are listed in the order:SIFT, SuperPoint

*LightGlue model runtimes are listed in the order:SIFT, SuperPoint, Disk, ALIKED

*Keypt2Subpx model runtimes are listed in the order:superpoint-lg, aliked-lg, xfeat, dedode

*MINIMA model runtimes are listed in the order:superpoint-lg, LoFTR, RoMa (large), RoMa (tiny)

*RDD model runtimes are listed in the order:sparse, star (semi-dense), lg, aliked

SeeCONTRIBUTING.md for details.

Special thanks to the authors of the respective works that are included in this repo (see their papers above). Additional thanks to@GrumpyZhou for developing and maintaining theImage Matching Toolbox, which we have wrapped in this repo, and themaintainers ofKornia.

This repo was created as part of the EarthMatch paper. Please consider citing EarthMatch if this repo is helpful to you!

@InProceedings{Berton_2024_EarthMatch,    author    = {Berton, Gabriele and Goletto, Gabriele and Trivigno, Gabriele and Stoken, Alex and Caputo, Barbara and Masone, Carlo},    title     = {EarthMatch: Iterative Coregistration for Fine-grained Localization of Astronaut Photography},    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},    month     = {June},    year      = {2024},}