Repository files navigation

• [2025/08/29] 🔥 LLaVA-Critic-R1 We release LLaVA-Critic-R1, a family of generative critic VLM trained through GRPO using pairwise critic data. LLaVA-Critic-R1 not only demonstrates strong critic capability, but also achieves state-of-the-art policy performance at the 7B scale. Refer toLLaVA-Critic-R1 for more training details.

  📄Explore more:

  • LLaVA-Critic-GRPO Dataset: Download the dataset.
  • LLaVA-Critic-R1-7B: LLaVA-Critic-R1 trained based on Qwen-2.5-VL-7B.
  • LLaVA-Critic-R1-7B-Plus-Qwen: LLaVA-Critic-R1+ trained based on ThinkLite-VL-7B.
  • LLaVA-Critic-R1-7B-Plus-Mimo: LLaVA-Critic-R1+ trained based on MiMo-VL-7B-RL-2508.
  • LLaVA-Critic-R1-7B-Plus-LLaMA32v: LLaVA-Critic-R1+ trained based on Llama-3.2-11B-Vision-Instruct.
  • Paper: Detailed information about LLaVA-Critic-R1.

• [2024/10/04] 🔥 LLaVA-Video (formerly LLaVA-NeXT-Video) has undergone a major upgrade! We are excited to releaseLLaVA-Video-178K, a high-quality synthetic dataset for video instruction tuning. This dataset includes:

  • 178,510 caption entries
  • 960,792 open-ended Q&A pairs
  • 196,198 multiple-choice Q&A items

  Along with this, we’re also releasing theLLaVA-Video 7B/72B models, which deliver competitive performance on the latest video benchmarks, includingVideo-MME,LongVideoBench, andDream-1K.

  📄Explore more:

  • LLaVA-Video-178K Dataset: Download the dataset.
  • LLaVA-Video Models: Access model checkpoints.
  • Paper: Detailed information about LLaVA-Video.
  • LLaVA-Video Documentation: Guidance on training, inference and evaluation.

• [2024/09/13] 🔥🚀LLaVA-OneVision-Chat. The new LLaVA-OV-Chat (7B/72B) significantly improves the chat experience of LLaVA-OV. 📄

  

• [2024/08/06] 🔥🚀LLaVA-OneVision (OV)! The new LLaVA-OV models (0.5B/7B/72B) achieve new state-of-the-art performance across single-image, multi-image, and video benchmarks, sometimes rivaling top commercial models on 47 diverse benchmarks. 📄 Explore More:

  • [Paper]: In-depth insights, new emegerging scenarios, ie, strong video understadning through task transfer from images.
  • [LLaVA-OV Doc]: Model inference and evaluation guidance.
  • [Scripts]: Start training models on your single-image/multi-image/video data.

• [2024/07/16] 🔥LLaVA-NeXT-Video has been upgraded. The new 32B model achieves the best open-source performance on several video benchmarks, includingVideo-MME. Please refer tothis page for details, refer tollava_next-video_demo for demo.

• [2024/06/23] 🔥LLaVA-NeXT-Interleave is released. We utilize image-text interleaved format to unify multi-image, video, and 3D tasks in one LLM and achieveSoTA performance on a wide range of benchmarks. Check outpaper,blog, andcheckpoints to see new capabilities and improved performance! We have released 0.5b, 7b, and 7b-dpo models.

  • An all-round LLM for multi-image, video, and 3D with strong performance [demo]
  • Construct interleave training dataM4-Instruct
  • Construct multi-image benchmarkLLaVA-Interleave Bench

• [2024/05/25] 🔥 Wondering "What Else Influences Visual Instruction Tuning Beyond Data?" Our newblog summarizes empirical explorations to ablate the various design choices in improving LMMs except instruct data itself. Meanwhile, open-source the recapioned high-quality data using LLaVA-NeXT-34B on[COCO][LCS][CC3M].

  • Architectures (LMM & Vision Encoder)
  • Visual Representations (Resolution & # Tokens)
  • Training Strategies (High-quality data & Trainable modules)

• [2024/05/10] 🔥LLaVA-NeXT (Stronger) models are released, with support of stronger LMM inlcuding LLama-3 (8B) and Qwen-1.5 (72B/110B) Check out [blog] and [checkpoints] to see improved performance!

• [2024/05/10] 🔥LLaVA-NeXT (Video) is released. The image-only-trained LLaVA-NeXT model is surprisingly strong on video tasks with zero-shot modality transfer. DPO training with AI feedback on videos can yield significant improvement. [Blog], [checkpoints] and [sglang]

• [2024/01/30] 🔥LLaVA-NeXT is out! With additional scaling to LLaVA-1.5, LLaVA-NeXT-34B outperforms Gemini Pro on some benchmarks. It can now process 4x more pixels and perform more tasks/applications than before. Check out theblog post, and explore thedemo! Models are available inModel Zoo. Training/eval data and scripts coming soon.

Usage and License Notices: This project utilizes certain datasets and checkpoints that are subject to their respective original licenses. Users must comply with all terms and conditions of these original licenses, including but not limited to theOpenAI Terms of Use for the dataset and the specific licenses for base language models for checkpoints trained using the dataset (e.g.Llama-1/2 community license for LLaMA-2 and Vicuna-v1.5,Tongyi Qianwen RESEARCH LICENSE AGREEMENT andLlama-3 Research License). This project does not impose any additional constraints beyond those stipulated in the original licenses. Furthermore, users are reminded to ensure that their use of the dataset and checkpoints is in compliance with all applicable laws and regulations.

git clone https://github.com/LLaVA-VL/LLaVA-NeXTcd LLaVA-NeXT

conda create -n llava python=3.10 -yconda activate llavapip install --upgrade pip# Enable PEP 660 support.pip install -e".[train]"

Please checkout the following page for more inference & evaluation details.

• LLaVA-OneVision: for demo inference. The evaluation code is inlmms-eval.

• LLaVA-NeXT-Image: for image demo inference and evaluation of stronger LMMs usinglmms-eval.

• LLaVA-NeXT-Video: for video inference and evaluation scripts. We recommend to useLMMs-video for evaluation.

• LLaVA-NeXT-Interleave: for multi-image demo and evaluation scripts.

We useSGLang to speed up inference and deployment of LLaVA-NeXT. You could make LLaVA-NeXT as a backend API service with SGLang.

Prepare Environment:Following the instruction in thesglang

Checkout the HTTP Post/Get and SRT usage atsglang/examples/runtime/llava_onevision

Launch and Run on (K) Nodes:

• Go to sglang project

  cd PATH_TO/sglang

• First node:

  bash examples/usage/llava_video/srt_example_llava_v.sh K 0 YOUR_VIDEO_PATH YOUR_MODEL_PATH FRAMES_PER_VIDEO(e.g. bash examples/usage/llava_video/srt_example_llava_v.sh K 0 examples/usage/llava_video/videos/Q98Z4OTh8RwmDonc.mp4 lmms-lab/LLaVA-NeXT-Video-7B-DPO 16)

• Second node:

  bash examples/usage/llava_video/srt_example_llava_v.sh K 1 YOUR_VIDEO_PATH YOUR_MODEL_PATH FRAMES_PER_VIDEO

• The K node:

  bash examples/usage/llava_video/srt_example_llava_v.sh K K-1 YOUR_VIDEO_PATH YOUR_MODEL_PATH FRAMES_PER_VIDEO

If you find it useful for your research and applications, please cite related papers/blogs using this BibTeX:

@article{li2024llava,title={LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models},author={Li, Feng and Zhang, Renrui and Zhang, Hao and Zhang, Yuanhan and Li, Bo and Li, Wei and Ma, Zejun and Li, Chunyuan},journal={arXiv preprint arXiv:2407.07895},year={2024}}@misc{li2024llavanext-ablations,title={LLaVA-NeXT: What Else Influences Visual Instruction Tuning Beyond Data?},url={https://llava-vl.github.io/blog/2024-05-25-llava-next-ablations/},author={Li, Bo and Zhang, Hao and Zhang, Kaichen and Guo, Dong and Zhang, Yuanhan and Zhang, Renrui and Li, Feng and Liu, Ziwei and Li, Chunyuan},month={May},year={2024}}@misc{li2024llavanext-strong,title={LLaVA-NeXT: Stronger LLMs Supercharge Multimodal Capabilities in the Wild},url={https://llava-vl.github.io/blog/2024-05-10-llava-next-stronger-llms/},author={Li, Bo and Zhang, Kaichen and Zhang, Hao and Guo, Dong and Zhang, Renrui and Li, Feng and Zhang, Yuanhan and Liu, Ziwei and Li, Chunyuan},month={May},year={2024}}@misc{zhang2024llavanext-video,title={LLaVA-NeXT: A Strong Zero-shot Video Understanding Model},url={https://llava-vl.github.io/blog/2024-04-30-llava-next-video/},author={Zhang, Yuanhan and Li, Bo and Liu, haotian and Lee, Yong jae and Gui, Liangke and Fu, Di and Feng, Jiashi and Liu, Ziwei and Li, Chunyuan},month={April},year={2024}}@misc{liu2024llavanext,title={LLaVA-NeXT: Improved reasoning, OCR, and world knowledge},url={https://llava-vl.github.io/blog/2024-01-30-llava-next/},author={Liu, Haotian and Li, Chunyuan and Li, Yuheng and Li, Bo and Zhang, Yuanhan and Shen, Sheng and Lee, Yong Jae},month={January},year={2024}}@misc{liu2023improvedllava,title={Improved Baselines with Visual Instruction Tuning},author={Liu, Haotian and Li, Chunyuan and Li, Yuheng and Lee, Yong Jae},publisher={arXiv:2310.03744},year={2023},}@misc{liu2023llava,title={Visual Instruction Tuning},author={Liu, Haotian and Li, Chunyuan and Wu, Qingyang and Lee, Yong Jae},publisher={NeurIPS},year={2023},}

• Vicuna: the codebase we built upon, and our base model Vicuna-13B that has the amazing language capabilities!
• The LLaVA-NeXT project is currently maintained by the team along with our contributors (listed alphabetically by the first names):Bo Li,Dong Guo,Feng Li,Hao Zhang,Kaichen Zhang,Renrui Zhang,Yuanhan Zhang, led byChunyuan Li and with the guidance and help fromHaotian Liu.
• Thelmms-eval framework and its core contributors, including Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, and Kairui Hu, for their support on the evaluation side.

• Instruction Tuning with GPT-4
• LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
• Otter: In-Context Multi-Modal Instruction Tuning

For future project ideas, please check out:

• SEEM: Segment Everything Everywhere All at Once
• Grounded-Segment-Anything to detect, segment, and generate anything by marryingGrounding DINO andSegment-Anything.