Repository files navigation

⚠️重要 2024/10/8 より多様なタスクにより埋め込みモデルを評価したリーダーボードJMTEBが公開されておりますので、こちらを参照することをお勧めします。
⚠️IMPORTANT UPDATE: we recommend checking outJMTEB, a new leaderboard that evaluates embedding models using a more diverse set of tasks.

• JSTS/JSICK: Spearman's rank correlation coefficient
  • Cosine similarity was used to calculate the similarity of sentence pairs.
• MIRACL: top30 recall

• JSTS valid-v1.1

  • https://github.com/yahoojapan/JGLUE
  • 1,457 sentence pairs

• JSICK test

  • https://github.com/verypluming/JSICK
  • 4,927 sentence pairs

• MIRACL dev

  • https://huggingface.co/datasets/miracl/miracl
  • 860 japanese queries
  • From the 6,953,614 japanese data in miracl/miracl-corpus, the sentences to be searched were selected as follows to reduce computation time.
    1. positive passage for each query
    2. 300 hard negatives for each query
    • Hard negative mining was performed using intfloat/multilingual-e5-base
    • Scores for models other than intfloat/multilingual-e5-base are calculated higher only in the following case, but we believe that they are almost unaffected.
      • A negative that is ranked lower than the top 300 by intfloat/multilingual-e5-base is ranked within the top 30 by that model, which pushes the positive into the top 30 or lower.
  • Some queries contain more than 30 potential positive documents in the miracl-corpus. In this case, even a very good model may not be able to rank the ground truth positive documents within the top 30. We estimated such queries to be about 7% to 10% of the total 860 queries. This number was estimated by referring to the tydiqa data for the same query as the corresponding miracl dev query and counting whether the tydiqa answer phrase was in at least 30 of the 300 hard negatives documents.