The embedding model trained byJina AI.

jina-embeddings-v3: Multilingual Embeddings With Task LoRA

Quick Start

Blog |Azure |AWS SageMaker |API

Intended Usage & Model Info

jina-embeddings-v3 is amultilingual multi-task text embedding model designed for a variety of NLP applications.Based on theJina-XLM-RoBERTa architecture, this model supports Rotary Position Embeddings to handle long input sequences up to8192 tokens.Additionally, it features 5 LoRA adapters to generate task-specific embeddings efficiently.

Key Features:

• Extended Sequence Length: Supports up to 8192 tokens with RoPE.
• Task-Specific Embedding: Customize embeddings through thetask argument with the following options:
  • retrieval.query: Used for query embeddings in asymmetric retrieval tasks
  • retrieval.passage: Used for passage embeddings in asymmetric retrieval tasks
  • separation: Used for embeddings in clustering and re-ranking applications
  • classification: Used for embeddings in classification tasks
  • text-matching: Used for embeddings in tasks that quantify similarity between two texts, such as STS or symmetric retrieval tasks
• Matryoshka Embeddings: Supports flexible embedding sizes (32, 64, 128, 256, 512, 768, 1024), allowing for truncating embeddings to fit your application.

Supported Languages:

While the foundation model supports 100 languages, we've focused our tuning efforts on the following 30 languages:Arabic, Bengali, Chinese, Danish, Dutch, English, Finnish, French, Georgian, German, Greek, Hindi, Indonesian, Italian, Japanese, Korean, Latvian, Norwegian, Polish, Portuguese, Romanian, Russian, Slovak, Spanish, Swedish, Thai, Turkish, Ukrainian, Urdu, andVietnamese.

⚠️ Important Notice:
We fixed a bug in theencode function#60 whereMatryoshka embedding truncation occurredafter normalization, leading to non-normalized truncated embeddings. This issue has been resolved in the latest code revision.

If you have encoded data using the previous version and wish to maintain consistency, please use the specific code revision when loading the model:AutoModel.from_pretrained('jinaai/jina-embeddings-v3', code_revision='da863dd04a4e5dce6814c6625adfba87b83838aa', ...)

Usage

import torchimport torch.nn.functionalas Ffrom transformersimport AutoTokenizer, AutoModeldefmean_pooling(model_output, attention_mask):    token_embeddings = model_output[0]    input_mask_expanded = (        attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()    )return torch.sum(token_embeddings * input_mask_expanded,1) / torch.clamp(        input_mask_expanded.sum(1),min=1e-9    )sentences = ["How is the weather today?","What is the current weather like today?"]tokenizer = AutoTokenizer.from_pretrained("jinaai/jina-embeddings-v3")model = AutoModel.from_pretrained("jinaai/jina-embeddings-v3", trust_remote_code=True)encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors="pt")task ='retrieval.query'task_id = model._adaptation_map[task]adapter_mask = torch.full((len(sentences),), task_id, dtype=torch.int32)with torch.no_grad():    model_output = model(**encoded_input, adapter_mask=adapter_mask)embeddings = mean_pooling(model_output, encoded_input["attention_mask"])embeddings = F.normalize(embeddings, p=2, dim=1)

The easiest way to start usingjina-embeddings-v3 is with theJina Embedding API.

Alternatively, you can usejina-embeddings-v3 directly via Transformers package:

!pip install transformers torch einops!pip install'numpy<2'

If you run it on a GPU that supportFlashAttention-2. By 2024.9.12, it supports Ampere, Ada, or Hopper GPUs (e.g., A100, RTX 3090, RTX 4090, H100),

!pip install flash-attn --no-build-isolation

from transformersimport AutoModel# Initialize the modelmodel = AutoModel.from_pretrained("jinaai/jina-embeddings-v3", trust_remote_code=True)texts = ["Follow the white rabbit.",# English"Sigue al conejo blanco.",# Spanish"Suis le lapin blanc.",# French"跟着白兔走。",# Chinese"اتبع الأرنب الأبيض.",# Arabic"Folge dem weißen Kaninchen.",# German]# When calling the `encode` function, you can choose a `task` based on the use case:# 'retrieval.query', 'retrieval.passage', 'separation', 'classification', 'text-matching'# Alternatively, you can choose not to pass a `task`, and no specific LoRA adapter will be used.embeddings = model.encode(texts, task="text-matching")# Compute similaritiesprint(embeddings[0] @ embeddings[1].T)

By default, the model supports a maximum sequence length of 8192 tokens. However, if you want to truncate your input texts to a shorter length, you can pass themax_length parameter to theencode function:

embeddings = model.encode(["Very long ... document"], max_length=2048)

In case you want to useMatryoshka embeddings and switch to a different dimension, you can adjust it by passing thetruncate_dim parameter to theencode function:

embeddings = model.encode(['Sample text'], truncate_dim=256)

The latest version (3.1.0) ofSentenceTransformers also supportsjina-embeddings-v3:

!pip install -U sentence-transformers

from sentence_transformersimport SentenceTransformermodel = SentenceTransformer("jinaai/jina-embeddings-v3", trust_remote_code=True)task ="retrieval.query"embeddings = model.encode(    ["What is the weather like in Berlin today?"],    task=task,    prompt_name=task,)

You can fine-tunejina-embeddings-v3 usingSentenceTransformerTrainer. To fine-tune for a specific task, you should set the task before passing the model to the ST Trainer, either during initialization:

model = SentenceTransformer("jinaai/jina-embeddings-v3", trust_remote_code=True, model_kwargs={'default_task':'classification'})

Or afterwards:

model = SentenceTransformer("jinaai/jina-embeddings-v3", trust_remote_code=True)model[0].default_task ='classification'

This way you can fine-tune the LoRA adapter for the chosen task.

However, If you want to fine-tune the entire model, make sure the main parameters are set as trainable when loading the model:

model = SentenceTransformer("jinaai/jina-embeddings-v3", trust_remote_code=True, model_kwargs={'lora_main_params_trainable':True})

This will allow fine-tuning the whole model instead of just the LoRA adapters.

import onnxruntimeimport numpyas npfrom transformersimport AutoTokenizer, PretrainedConfig# Mean pool functiondefmean_pooling(model_output: np.ndarray, attention_mask: np.ndarray):    token_embeddings = model_output    input_mask_expanded = np.expand_dims(attention_mask, axis=-1)    input_mask_expanded = np.broadcast_to(input_mask_expanded, token_embeddings.shape)    sum_embeddings = np.sum(token_embeddings * input_mask_expanded, axis=1)    sum_mask = np.clip(np.sum(input_mask_expanded, axis=1), a_min=1e-9, a_max=None)return sum_embeddings / sum_mask# Load tokenizer and model configtokenizer = AutoTokenizer.from_pretrained('jinaai/jina-embeddings-v3')config = PretrainedConfig.from_pretrained('jinaai/jina-embeddings-v3')# Tokenize inputinput_text = tokenizer('sample text', return_tensors='np')# ONNX sessionmodel_path ='jina-embeddings-v3/onnx/model.onnx'session = onnxruntime.InferenceSession(model_path)# Prepare inputs for ONNX modeltask_type ='text-matching'task_id = np.array(config.lora_adaptations.index(task_type), dtype=np.int64)inputs = {'input_ids': input_text['input_ids'],'attention_mask': input_text['attention_mask'],'task_id': task_id}# Run modeloutputs = session.run(None, inputs)[0]# Apply mean pooling and normalization to the model outputsembeddings = mean_pooling(outputs, input_text["attention_mask"])embeddings = embeddings / np.linalg.norm(embeddings,ord=2, axis=1, keepdims=True)

Contact

Join ourDiscord community and chat with other community members about ideas.

License

jina-embeddings-v3 is listed on AWS & Azure. If you need to use it beyond those platforms or on-premises within your company, note that the models is licensed under CC BY-NC 4.0. For commercial usage inquiries, feel free tocontact us.

Citation

If you findjina-embeddings-v3 useful in your research, please cite the following paper:

@misc{sturua2024jinaembeddingsv3multilingualembeddingstask,      title={jina-embeddings-v3: Multilingual Embeddings With Task LoRA},       author={Saba Sturua and Isabelle Mohr and Mohammad Kalim Akram and Michael Günther and Bo Wang and Markus Krimmel and Feng Wang and Georgios Mastrapas and Andreas Koukounas and Andreas Koukounas and Nan Wang and Han Xiao},      year={2024},      eprint={2409.10173},      archivePrefix={arXiv},      primaryClass={cs.CL},      url={https://arxiv.org/abs/2409.10173}, }