README |leaderboard |submission guideline

Quick start

git clone git@github.com:sbintuitions/JMTEBcd JMTEBpoetry installpoetry run pytest tests

poetry run python -m jmteb \  --embedder SentenceBertEmbedder \  --embedder.model_name_or_path"<model_name_or_path>" \  --save_dir"output/<model_name_or_path>"

Note

In order to gurantee the robustness of evaluation, a validation dataset is mandatorily required for hyperparameter tuning.For a dataset that doesn't have a validation set, we set the validation set the same as the test set.

poetry run python -m jmteb \  --evaluators"src/configs/tasks/jsts.jsonnet" \  --embedder SentenceBertEmbedder \  --embedder.model_name_or_path"<model_name_or_path>" \  --save_dir"output/<model_name_or_path>"

Note

Some tasks (e.g., AmazonReviewClassification in classification, JAQKET and Mr.TyDi-ja in retrieval, esci in reranking) are time-consuming and memory-consuming. Heavy retrieval tasks take hours to encode the large corpus, and use much memory for the storage of such vectors. If you want to exclude them, add--eval_exclude "['amazon_review_classification', 'mrtydi', 'jaqket', 'esci']". Similarly, you can also use--eval_include to include only evaluation datasets you want.

Note

If you want to log model predictions to further analyze the performance of your model, you may want to use--log_predictions true to enable all evaluators to log predictions. It is also available to set whether to log in the config of evaluators.

Multi-GPU support

poetry run python -m jmteb \  --evaluators"src/configs/tasks/jsts.jsonnet" \  --embedder DataParallelSentenceBertEmbedder \  --embedder.model_name_or_path"<model_name_or_path>" \  --save_dir"output/<model_name_or_path>"

MODEL_NAME=<model_name_or_path>MODEL_KWARGS="\{\'torch_dtype\':\'torch.bfloat16\'\}"torchrun \    --nproc_per_node=$GPUS_PER_NODE --nnodes=1 \    src/jmteb/__main__.py --embedder TransformersEmbedder \    --embedder.model_name_or_path${MODEL_NAME} \    --embedder.pooling_mode cls \    --embedder.batch_size 4096 \    --embedder.model_kwargs${MODEL_KWARGS} \    --embedder.max_seq_length 512 \    --save_dir"output/${MODEL_NAME}" \    --evaluators src/jmteb/configs/jmteb.jsonnet