Repository files navigation

The vigogne (French name for vicuña) is a South American camelid native to the Andes Mountains. It is closely related to the llama, alpaca, and guanaco.

Vigogne is a collection of powerful 🇫🇷 French large language models (LLMs) that are open-source and designed for instruction-following and chat purposes.

The main contributions of this project include:

• Open-sourced 🦙 Vigogne models for French instruction-following and chat
• Efficient training code for fine-tuning LLMs such asLLaMA,Llama-2,Falcon, andFLAN-T5
• Generated, translated, and collected French instruction-following and dialogue datasets, along with the used scripts
• Inference code, Gradio demo, and support for deploying within various libraries such as 🤗Transformers,llama.cpp,FastChat, andvLLM
• Support diverse application ecosystems, includingLangChain.

💡The screencast below shows the current 🦙 Vigogne-7B-Chat model running on Apple M1 Pro using 4GB of weights (no sped up).

• Updates
• Installation
• 🦙 Vigogne Models
• Inference and Deployment
• Application
• Data
• Training
• Bias, Risks, and Limitations
• Acknowledgements
• Citation

• [2023/08/18]: IncludedLangChain integration.
• [2023/08/17]: Introduced Vigogne-Chat-V2.0 models (blog).
• [2023/08/16]: Added support for serving usingFastChat andvLLM.
• [2023/08/02]: Implemented generation script forOrca-style data.
• [2023/07/31]: IntegratedFlashAttention support and implemented training example packing.
• [2023/07/20]: Introduced the latest Vigogne models built uponLlama-2.
• [2023/07/05]: Released Vigogne models based onFalcon andMPT, with commercial-friendly licenses.
• [2023/06/05]: IntegratedQLoRA support for improved training efficiency.
• [2023/05/15]: Introduced Vigogne-Chat models with enhanced conversational capabilities.
• [2023/05/11]: ImplementedSelf-Chat data generation script for conversational data.
• [2023/05/11]: Introduced improved Vigogne-Instruct-V2 models, trained on more diverse data.
• [2023/05/11]: Released annotated seed tasks in French and generation script forSelf-Instruct.
• [2023/04/03]: Expanded training scripts to incorporate seq2seq models.
• [2023/03/29]: Included deployment instructions usingllama.cpp.
• [2023/03/26]: Released initial Vigogne-Instruct models trained on translated Stanford Alpaca data.
• [2023/03/26]: Open-sourced Vigogne project with optimized training scripts (LoRA,LLM.int8()).

1. Clone this repository

git clone https://github.com/bofenghuang/vigogne.gitcd vigogne

2. Install the package

# Install DeepSpeed if want to accelerate training with itpip install deepspeed# Install FlashAttention to further speed up training and reduce memory usage (essential for long sequences)pip install packaging ninja# For FlashAttention 1# pip install --no-build-isolation flash-attn<2# For FlashAttention 2# Might takes 3-5 minutes on a 64-core machinepip install --no-build-isolation flash-attnpip install.

The fine-tuned 🦙 Vigogne models come in two types:instruction-following models andchat models. The instruction-following models are optimized to generate concise and helpful responses to user instructions, similar totext-davinci-003. Meanwhile, the chat models are designed for multi-turn dialogues, but they also perform well in instruction-following tasks, similar togpt-3.5-turbo.

More information can be found invigogne/model.

This repository offers multiple options for inference and deployment, including Google Colab notebooks, Gradio demos,FastChat, andvLLM. It also offers guidance on conducting experiments usingllama.cpp on your personal computer.

More information can be found invigogne/inference.

This repository provides integration examples for incorporating Vigogne models into diverse application ecosystems, includingLangChain.

More information can be found invigogne/application.

The Vigogne models were trained on a variety of datasets, including open-source datasets, ChatGPT-distillation datasets (self-instruct, self-chat, and orca-style data), and translated datasets.

More information can be found invigogne/data.

The Vigogne models were mostly instruction fine-tuned from other foundation models.

More information can be found invigogne/training.

Vigogne is still under development, and there are many limitations that have to be addressed. Please note that it is possible that the model generates harmful or biased content, incorrect information or generally unhelpful answers.

Our project builds upon the following open-source projects for further development. We would like to extend our sincerest gratitude to the individuals involved in the research and development of these projects.

• 🤗 Transformers and🤗 PEFT
• LLaMA
• Stanford Alpaca
• Alpaca-LoRA by @tloen
• Baize
• llama.cpp by @ggerganov
• Axolotl by @OpenAccess-AI-Collective

If you find the model, data, and code in our project useful, please consider citing our work as follows:

@misc{vigogne,  author = {Bofeng Huang},  title = {Vigogne: French Instruction-following and Chat Models},  year = {2023},  publisher = {GitHub},  journal = {GitHub repository},  howpublished = {\url{https://github.com/bofenghuang/vigogne}},}