Computer Science > Computer Vision and Pattern Recognition
arXiv:2403.08857 (cs)
[Submitted on 13 Mar 2024 (v1), last revised 18 Apr 2025 (this version, v3)]
Title:DialogGen: Multi-modal Interactive Dialogue System for Multi-turn Text-to-Image Generation
Authors:Minbin Huang,Yanxin Long,Xinchi Deng,Ruihang Chu,Jiangfeng Xiong,Xiaodan Liang,Hong Cheng,Qinglin Lu,Wei Liu
View a PDF of the paper titled DialogGen: Multi-modal Interactive Dialogue System for Multi-turn Text-to-Image Generation, by Minbin Huang and 8 other authors
View PDFHTML (experimental)Abstract:Text-to-image (T2I) generation models have significantly advanced in recent years. However, effective interaction with these models is challenging for average users due to the need for specialized prompt engineering knowledge and the inability to perform multi-turn image generation, hindering a dynamic and iterative creation process. Recent attempts have tried to equip Multi-modal Large Language Models (MLLMs) with T2I models to bring the user's natural language instructions into reality. Hence, the output modality of MLLMs is extended, and the multi-turn generation quality of T2I models is enhanced thanks to the strong multi-modal comprehension ability of MLLMs. However, many of these works face challenges in identifying correct output modalities and generating coherent images accordingly as the number of output modalities increases and the conversations go deeper. Therefore, we propose DialogGen, an effective pipeline to align off-the-shelf MLLMs and T2I models to build a Multi-modal Interactive Dialogue System (MIDS) for multi-turn Text-to-Image generation. It is composed of drawing prompt alignment, careful training data curation, and error correction. Moreover, as the field of MIDS flourishes, comprehensive benchmarks are urgently needed to evaluate MIDS fairly in terms of output modality correctness and multi-modal output coherence. To address this issue, we introduce the Multi-modal Dialogue Benchmark (DialogBen), a comprehensive bilingual benchmark designed to assess the ability of MLLMs to generate accurate and coherent multi-modal content that supports image editing. It contains two evaluation metrics to measure the model's ability to switch modalities and the coherence of the output images. Our extensive experiments on DialogBen and user study demonstrate the effectiveness of DialogGen compared with other State-of-the-Art models.
| Comments: | Project page:this https URL. Accepted to NAACL2025 |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2403.08857 [cs.CV] |
| (orarXiv:2403.08857v3 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2403.08857 arXiv-issued DOI via DataCite |
Submission history
From: Minbin Huang [view email][v1] Wed, 13 Mar 2024 18:00:01 UTC (12,989 KB)
[v2] Wed, 3 Jul 2024 13:09:53 UTC (15,810 KB)
[v3] Fri, 18 Apr 2025 11:51:22 UTC (15,810 KB)
Full-text links:
Access Paper:
- View PDF
- HTML (experimental)
- TeX Source
- Other Formats
View a PDF of the paper titled DialogGen: Multi-modal Interactive Dialogue System for Multi-turn Text-to-Image Generation, by Minbin Huang and 8 other authors
References & Citations
Bookmark
Bibliographic and Citation Tools
Bibliographic Explorer(What is the Explorer?)
Connected Papers(What is Connected Papers?)
Litmaps(What is Litmaps?)
scite Smart Citations(What are Smart Citations?)
Code, Data and Media Associated with this Article
alphaXiv(What is alphaXiv?)
CatalyzeX Code Finder for Papers(What is CatalyzeX?)
DagsHub(What is DagsHub?)
Gotit.pub(What is GotitPub?)
Hugging Face(What is Huggingface?)
Papers with Code(What is Papers with Code?)
ScienceCast(What is ScienceCast?)
Demos
Recommenders and Search Tools
Influence Flower(What are Influence Flowers?)
CORE Recommender(What is CORE?)
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community?Learn more about arXivLabs.