Repository files navigation

• 📊 Modern vector index benchmark with embedding datasets
• 🎯 Includes datasets for both in-distribution and out-out-distribution settings
• 🏆 Includes the most comprehensive collection of state-of-the-art vector search algorithms
• 💎 Support for quantized datasets in both 8-bit integer and binary precision
• 🖥️ Support for HPC environments with Slurm and NUMA
• 🚀 Support for GPU algorithms

The current VIBE results can be viewed on our website:

https://vector-index-bench.github.io

The website also features several other tools and visualizations to explore the results.

E. Jääsaari, V. Hyvönen, M. Ceccarello, T. Roos, M. Aumüller.VIBE: Vector Index Benchmark for Embeddings.arXiv preprint arXiv:2505.17810, 2025.

VIBE is maintained byElias Jääsaari,Matteo Ceccarello, andMartin Aumüller.

The evaluation code and some algorithm implementations in VIBE are based on theann-benchmarks project.

VIBE is available under the MIT License (seeLICENSE). Thepyyaml library is also distributed in thevibe folder under the MIT License.

• Linux
• Apptainer (orSingularity)
• Python 3.6 or newer

For example, to install Apptainer on Ubuntu:

sudo add-apt-repository -y ppa:apptainer/ppasudo apt updatesudo apt install -y apptainer

Some algorithms may require that the CPU supports AVX-512 instructions and some algorithms may require an Intel CPU due to a dependency on Intel MKL. The GPU algorithms assume that an NVIDIA GPU is available.

Building all library images can be done using

./install.sh

The script can be used to either build images for all available libraries (./install.sh) or an image for a single library (e.g../install.sh --algorithm faiss).

The benchmarks for a single dataset can be run usingrun.py. For example:

python3 run.py --dataset agnews-mxbai-1024-euclidean

The run.py script does not depend on any external libraries and can therefore be used without a container or a virtual environment.

Common options for run.py:

• --parallelism n: Usen processes for benchmarking.
• --module mod: Run the benchmark only for algorithms in module (library)mod.
• --algorithm algo: Run the benchmark for only algorithmalgo.
• --count k: Run the benchmarks usingk nearest neighbors (default 100).
• --gpu: Run the benchmark in GPU mode.

For all options, see

python3 run.py --help

The benchmark should take less than 24 hours to run for a given dataset using parallelism > 12. We recommend having at least 16 GB of memory per used core.

You should first build theplot.sif image:

singularity build plot.sif plot.def

Before plotting, the current results must first be exported:

./export_results.sh --parallelism 8

The results for a dataset can then plotted with e.g.:

./plot.sh --dataset agnews-mxbai-1024-euclidean

To plot the radar chart above, use:

./plot.sh --plot-type radar

For all available options, see:

./plot.sh --help

The benchmark code downloads precomputed embedding datasets. However, the datasets can also be recreated from scratch, and it is also possible to create new datasets by modifying thedatasets.py file.

Creating the datasets can be done usingcreate_dataset.sh. It first requires thatdataset.sif is built:

singularity build dataset.sif dataset.def

TheVIBE_CACHE environment variable should be set to a cache directory with at least 200 GB of free space when creating image embeddings using the Landmark or ImageNet datasets. Datasets can then be created using the--dataset argument (the--nv argument specifies that an available GPU can be used):

export VIBE_CACHE=$LOCAL_SCRATCH./create_dataset.sh --singularity-args"--bind$LOCAL_SCRATCH:$LOCAL_SCRATCH --nv" --dataset agnews-mxbai-1024-euclidean

Add your algorithm in the foldervibe/algorithms/{METHOD}/ by providing

• Python wrapper in module.py
• Singularity container defination in image.def
• Hyperparameter grid in config.yml