Repository files navigation

Model2Vec is a technique to turn any sentence transformer into a really small static model, reducing model size by a factor up to 50 and making the models up to 500 times faster, with a small drop in performance. Ourbest model is the most performant static embedding model in the world. See our resultshere, or dive in to see how it works.

Install the lightweight base package with:

pip install model2vec

You can start using Model2Vec by loading one of ourflagship models from the HuggingFace hub. These models are pre-trained and ready to use. The following code snippet shows how to load a model and make embeddings, which you can use for any task, such as text classification, retrieval, clustering, or building a RAG system:

frommodel2vecimportStaticModel# Load a model from the HuggingFace hub (in this case the potion-base-8M model)model=StaticModel.from_pretrained("minishlab/potion-base-8M")# Make embeddingsembeddings=model.encode(["It's dangerous to go alone!","It's a secret to everybody."])# Make sequences of token embeddingstoken_embeddings=model.encode_as_sequence(["It's dangerous to go alone!","It's a secret to everybody."])

Instead of using one of our models, you can also distill your own Model2Vec model from a Sentence Transformer model. First, install thedistillation extras with:

pip install model2vec[distill]

Then, you can distill a model in ~30 seconds on a CPU with the following code snippet:

frommodel2vec.distillimportdistill# Distill a Sentence Transformer model, in this case the BAAI/bge-base-en-v1.5 modelm2v_model=distill(model_name="BAAI/bge-base-en-v1.5",pca_dims=256)# Save the modelm2v_model.save_pretrained("m2v_model")

After distillation, you can also fine-tune your own classification models on top of the distilled model, or on a pre-trained model. First, make sure you install thetraining extras with:

pip install model2vec[training]

Then, you can fine-tune a model as follows:

importnumpyasnpfromdatasetsimportload_datasetfrommodel2vec.trainimportStaticModelForClassification# Initialize a classifier from a pre-trained modelclassifier=StaticModelForClassification.from_pretrained(model_name="minishlab/potion-base-32M")# Load a dataset. Note: both single and multi-label classification datasets are supportedds=load_dataset("setfit/subj")# Train the classifier on text (X) and labels (y)classifier.fit(ds["train"]["text"],ds["train"]["label"])# Evaluate the classifierclassification_report=classifier.evaluate(ds["test"]["text"],ds["test"]["label"])

For advanced usage, please refer to ourusage documentation.

• 12/02/2024: We releasedModel2Vec training, allowing you to fine-tune your own classification models on top of Model2Vec models. Find out more in ourtraining documentation andresults.

• 30/01/2024: We released two new models:potion-base-32M andpotion-retrieval-32M.potion-base-32M is our most performant model to date, using a larger vocabulary and higher dimensions.potion-retrieval-32M is a finetune ofpotion-base-32M that is optimized for retrieval tasks, and is the best performing static retrieval model currently available.

• 30/10/2024: We released three new models:potion-base-8M,potion-base-4M, andpotion-base-2M. These models are trained usingTokenlearn. Find out more in ourblog post. NOTE: for users of any of our old English M2V models, we recommend switching to these new models as theyperform better on all tasks.

• State-of-the-Art Performance: Model2Vec models outperform any other static embeddings (such as GLoVe and BPEmb) by a large margin, as can be seen in ourresults.
• Small: Model2Vec reduces the size of a Sentence Transformer model by a factor of up to 50. Ourbest model is just ~30 MB on disk, and our smallest model just ~8 MB (making it the smallest model onMTEB!).
• Lightweight Dependencies: the base package's only major dependency isnumpy.
• Lightning-fast Inference: up to 500 times faster on CPU than the original model.
• Fast, Dataset-free Distillation: distill your own model in 30 seconds on a CPU, without a dataset.
• Fine-tuning: fine-tune your own classification models on top of Model2Vec models.
• Integrated in many popular libraries: Model2Vec is integrated direclty into popular libraries such asSentence Transformers andLangChain. For more information, see ourintegrations documentation.
• Tightly integrated with HuggingFace hub: easily share and load models from the HuggingFace hub, using the familiarfrom_pretrained andpush_to_hub. Our own models can be foundhere.

Model2vec creates a small, fast, and powerful model that outperforms other static embedding models by a large margin on all tasks we could find, while being much faster to create than traditional static embedding models such as GloVe. Like BPEmb, it can create subword embeddings, but with much better performance. Distillation doesn't needany data, just a vocabulary and a model.

The core idea is to forward pass a vocabulary through a sentence transformer model, creating static embeddings for the indiviudal tokens. After this, there are a number of post-processing steps we do that results in our best models. For a more extensive deepdive, please refer to the following resources:

• Our initialModel2Vec blog post. Note that, while this post gives a good overview of the core idea, we've made a number of substantial improvements since then.
• OurTokenlearn blog post. This post describes the Tokenlearn method we used to train ourpotion models.
• Our officialdocumentation. This document provides a high-level overview of how Model2Vec works.

Our official documentation can be foundhere. This includes:

• Usage documentation: provides a technical overview of how to use Model2Vec.
• Integrations documentation: provides examples of how to use Model2Vec in various downstream libraries.
• Model2Vec technical documentation: provides a high-level overview of how Model2Vec works.

We provide a number of models that can be used out of the box. These models are available on theHuggingFace hub and can be loaded using thefrom_pretrained method. The models are listed below.

We have performed extensive experiments to evaluate the performance of Model2Vec models. The results are documented in theresults folder. The results are presented in the following sections:

• MTEB Results
• Training Results
• Ablations

MIT

If you use Model2Vec in your research, please cite the following:

@software{minishlab2024model2vec,authors ={Stephan Tulkens and Thomas van Dongen},title ={Model2Vec: Fast State-of-the-Art Static Embeddings},year ={2024},url ={https://github.com/MinishLab/model2vec}}