Computer Science > Computation and Language
arXiv:2412.04261 (cs)
[Submitted on 5 Dec 2024]
Title:Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontier
Authors:John Dang,Shivalika Singh,Daniel D'souza,Arash Ahmadian,Alejandro Salamanca,Madeline Smith,Aidan Peppin,Sungjin Hong,Manoj Govindassamy,Terrence Zhao,Sandra Kublik,Meor Amer,Viraat Aryabumi,Jon Ander Campos,Yi-Chern Tan,Tom Kocmi,Florian Strub,Nathan Grinsztajn,Yannis Flet-Berliac,Acyr Locatelli,Hangyu Lin,Dwarak Talupuru,Bharat Venkitesh,David Cairuz,Bowen Yang,Tim Chung,Wei-Yin Ko,Sylvie Shang Shi,Amir Shukayev,Sammie Bae,Aleksandra Piktus,Roman Castagné,Felipe Cruz-Salinas,Eddie Kim,Lucas Crawhall-Stein,Adrien Morisot,Sudip Roy,Phil Blunsom,Ivan Zhang,Aidan Gomez,Nick Frosst,Marzieh Fadaee,Beyza Ermis,Ahmet Üstün,Sara Hooker
View a PDF of the paper titled Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontier, by John Dang and 44 other authors
View PDFHTML (experimental)Abstract:We introduce the Aya Expanse model family, a new generation of 8B and 32B parameter multilingual language models, aiming to address the critical challenge of developing highly performant multilingual models that match or surpass the capabilities of monolingual models. By leveraging several years of research at Cohere For AI and Cohere, including advancements in data arbitrage, multilingual preference training, and model merging, Aya Expanse sets a new state-of-the-art in multilingual performance. Our evaluations on the Arena-Hard-Auto dataset, translated into 23 languages, demonstrate that Aya Expanse 8B and 32B outperform leading open-weight models in their respective parameter classes, including Gemma 2, Qwen 2.5, and Llama 3.1, achieving up to a 76.6% win-rate. Notably, Aya Expanse 32B outperforms Llama 3.1 70B, a model with twice as many parameters, achieving a 54.0% win-rate. In this short technical report, we present extended evaluation results for the Aya Expanse model family and release their open-weights, together with a new multilingual evaluation dataset m-ArenaHard.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2412.04261 [cs.CL] |
| (orarXiv:2412.04261v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2412.04261 arXiv-issued DOI via DataCite |
Full-text links:
Access Paper:
- View PDF
- HTML (experimental)
- TeX Source
- Other Formats
View a PDF of the paper titled Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontier, by John Dang and 44 other authors
References & Citations
Bookmark
Bibliographic and Citation Tools
Bibliographic Explorer(What is the Explorer?)
Connected Papers(What is Connected Papers?)
Litmaps(What is Litmaps?)
scite Smart Citations(What are Smart Citations?)
Code, Data and Media Associated with this Article
alphaXiv(What is alphaXiv?)
CatalyzeX Code Finder for Papers(What is CatalyzeX?)
DagsHub(What is DagsHub?)
Gotit.pub(What is GotitPub?)
Hugging Face(What is Huggingface?)
Papers with Code(What is Papers with Code?)
ScienceCast(What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower(What are Influence Flowers?)
CORE Recommender(What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community?Learn more about arXivLabs.