multilingual-nlp

EMNLP 2023 Papers: Explore cutting-edge research from EMNLP 2023, the premier conference for advancing empirical methods in natural language processing. Stay updated on the latest in machine learning, deep learning, and natural language processing with code included. ⭐ support NLP!

nlp machine-learning natural-language-processing text-mining sentiment-analysis machine-translation word-embeddings transformers named-entity-recognition computational-linguistics gpt language-models emnlp ner bert nlp-applications syntax-and-semantics multilingual-nlp llms emnlp2023

UpdatedMay 18, 2024
Python

cisnlp /Glot500

Star106

Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages -- ACL 2023

multilingual nlp natural-language-processing acl dataset glot xlm multilingual-models xlm-r multilingual-nlp glot500

UpdatedApr 20, 2024
Python

FSoft-AI4Code /TheVault

Star102

[EMNLP 2023] The Vault: A Comprehensive Multilingual Dataset for Advancing Code Understanding and Generation

dataset multilingual-nlp ai4code

UpdatedAug 21, 2024
Jupyter Notebook

shijie-wu /crosslingual-nlp

Star92

This repo supports various cross-lingual transfer learning & multilingual NLP models.

natural-language-processing crosslingual-transfer multilingual-nlp

UpdatedSep 13, 2023
Python

epfl-dlab /llm-latent-language

Star80

Repo accompanying our paper "Do Llamas Work in English? On the Latent Language of Multilingual Transformers".

multilingual-nlp llm mechanistic-interpretability llama2

UpdatedMar 11, 2024
Jupyter Notebook

csebuetnlp /CrossSum

Star53

This repository contains the code, data, and models of the paper titled "CrossSum: Beyond English-Centric Cross-Lingual Summarization for 1,500+ Language Pairs" published in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL’23), July 9-14, 2023.

cross-lingual-summarization cross-lingual-transfer multilingual-nlp

UpdatedMar 26, 2024
Python

techwolf-ai /workrb

Star31

WorkRB: Work Research Benchmark

benchmark text-classification ranking semantic-similarity workbench workerbee multilingual-nlp skill-extraction work-domain workbench-eval skill-normalization job-normalization job-to-skill skill-to-job workrb workrbe

UpdatedDec 16, 2025
Python

posi-olomo /padie-extended

Star28

An open-source Python package that uses AI to predict Nigerian languages, including English, Pidgin, Yoruba, Hausa, and Igbo.

natural-language-processing text-classification transformers language-detection low-resource-languages transformer-architecture huggingface multilingual-nlp nigerian-languages afro-xlmr-base

UpdatedNov 8, 2025
Python

ceferisbarov /TUMLU

Star22

TUMLU: A Unified and Native Language Understanding Benchmark for Turkic Languages

nlp multilingual-nlp llms

UpdatedFeb 25, 2025
Python

kidist-amde /amharic-ir-benchmarks

Star20

Official codebase for the ACL 2025 Findings paper: Optimized Text Embedding Models and Benchmarks for Amharic Passage Retrieval.

information-retrieval bert bm25 passage-retrieval ndcg text-embedding amharic-corpus mrr roberta amharic-nlp huggingface-transformers colbert multilingual-nlp low-resource-nlp dense-retrieval amharic-language retrieval-evaluation academic-benchmark

UpdatedJul 26, 2025
Jupyter Notebook

BatsResearch /LexC-Gen

Star18

Generate synthetic labeled data for extremely low-resource languages using bilingual lexicons.

multilingual sentiment-analysis topic-modeling synthetic-data synthetic-dataset-generation low-resource-languages lexicon-based multilingual-nlp llm

UpdatedOct 3, 2024
Python

BatsResearch /cross-lingual-detox

Star17

Code for "Preference Tuning For Toxicity Mitigation Generalizes Across Languages." Paper accepted at Findings of EMNLP 2024

nlp ai-safety generalization cross-lingual-transfer multilingual-nlp mechanistic-interpretability

UpdatedMar 25, 2025
Jupyter Notebook

swiss-ai /parity-aware-bpe

Star16

Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization [arXiv 2025]

tokenization bpe multilingual-nlp llms multilingual-tokenization

UpdatedDec 10, 2025
Python

cisnlp /MEXA

Star11

🔍 Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignment

multilingual evaluation embeddings evaluation-metrics cross-lingual multilingual-nlp large-language-models decoder-only

UpdatedApr 6, 2025
Python

cambridgeltl /prompt4bli

Star11

On Bilingual Lexicon Induction with Large Language Models (EMNLP 2023). Keywords: Bilingual Lexicon Induction, Word Translation, Large Language Models, LLMs.