vision-language-transformer

A collection of original, innovative ideas and algorithms towards Advanced Literate Machinery. This project is maintained by the OCR Team in the Language Technology Lab, Tongyi Lab, Alibaba Group.

ocr computer-vision artificial-intelligence text-recognition document text-detection document-analysis end-to-end-ocr multimodal scene-text-recognition multimodal-deep-learning scene-text-detection vision-language document-understanding scene-text-detection-recognition document-recognition document-intelligence documentai vision-language-transformer vision-language-model

UpdatedApr 9, 2025
C++

henghuiding /ReLA

Star702

[CVPR2023 Highlight] GRES: Generalized Referring Expression Segmentation

multimodal-learning referring-image-segmentation referring-expression-segmentation referring-expression-comprehension vision-language-transformer cvpr2023

UpdatedSep 5, 2023
Python

shenyunhang /APE

Star572

[CVPR 2024] Aligning and Prompting Everything All at Once for Universal Visual Perception

open-world object-detection image-segmentation referring-expression-comprehension vision-language-transformer

UpdatedMay 8, 2024
Python

henghuiding /Vision-Language-Transformer

Star359

[ICCV2021 & TPAMI2023] Vision-Language Transformer and Query Generation for Referring Segmentation

tensorflow keras transformer vision-language referring-segmentation tpami iccv2021 vision-language-transformer

UpdatedJan 7, 2022
Python

haoliuhl /instructrl

Star52

Instruction Following Agents with Multimodal Transforemrs

machine-learning reinforcement-learning instructions transformer flax jax instruction-following vision-language-transformer

UpdatedNov 3, 2022
Python

sMamooler /CLIP_Explainability

Star33

code for studying OpenAI's CLIP explainability

machine-learning computer-vision gradcam-visualization model-explainability openai-clip vision-language-transformer

UpdatedJan 7, 2022
Jupyter Notebook

yiren-jian /BLIText

Star25

[NeurIPS 2023] Bootstrapping Vision-Language Learning with Decoupled Language Pre-training

multimodal-deep-learning vision-language-transformer vision-language-pretraining

UpdatedDec 5, 2023
Python

akusayudodograu /Agentic-RAG-Story-Generation-with-Multimodal-GenAI

Star12

Multimodal Agentic GenAI Workflow – Seamlessly blends retrieval and generation for intelligent storytelling

story-generation multimodal-learning multimodal multimodal-deep-learning vision-language vision-language-transformer generative-ai vision-language-model multimodal-large-language-models generative-ai-model agentic-workflow agentic-rag agentic-ai internvl2

UpdatedJul 3, 2025

unitaryai /VTC

Star11

VTC: Improving Video-Text Retrieval with User Comments

comments video-understanding multimodal-deep-learning video-text-retrieval vision-language-transformer vision-language-pretraining

UpdatedJun 23, 2025
Python

deepmancer /vlm-toolbox

Star10

Vision-Language Models Toolbox: Your all-in-one solution for multimodal research and experimentation

deep-learning transformers pytorch deep-learning-library clip multi-granularity-dataset vision-framework multimodal-learning vision-and-language multimodal-deep-learning multimodal-datasets hierarchical-classification zero-shot-classification vision-language-transformer prompt-tuning soft-prompt-tuning

UpdatedFeb 16, 2025
Jupyter Notebook

ThomasVonWu /Awesome-VLMs-Strawberry

Star10

A collection of VLMs papers, blogs, and projects, with a focus on VLMs in Autonomous Driving and related reasoning techniques.

multimodal-learning vision-language-transformer llm vlms

UpdatedNov 16, 2024

marialymperaiou /knowledge-enhanced-multimodal-learning

Star7

A list of research papers on knowledge-enhanced multimodal learning

knowledge-graph multi-task-learning visual-reasoning visual-dialog visual-question-answering vision-and-language multimodal-deep-learning visual-storytelling multimodal-retrieval visual-grounding visual-commonsense-reasoning vision-and-language-navigation story-visualization image-text-matching vision-language-transformer image-text-retrieval vision-and-language-pre-training conditional-image-generation knowledge-enhanced-multimodal-learning knowledge-enhanced-vision-language

UpdatedDec 8, 2022

fork123aniket /Multi-Round-VLM-powered-Multimodal-Conversational-AI-Navigation-Bot

Star3

Streamlit App Combining Vision, Language, and Audio AI Models

conversational-interface conversational-ai multimodal-learning multimodal multimodal-deep-learning multimodal-data conversational-agent conversational-bot vision-language vision-language-transformer generative-ai vision-language-model vision-language-navigation multimodal-large-language-models vision-language-learning vision-language-models internvl internvl2

UpdatedJan 27, 2025
Python

fork123aniket /Agentic-RAG-Story-Generation-with-Multimodal-GenAI

Star2

Multimodal Agentic GenAI Workflow – Seamlessly blends retrieval and generation for intelligent storytelling

story-generation multimodal-learning multimodal multimodal-deep-learning multimodal-data vision-language vision-language-transformer generative-ai vision-language-model multimodal-large-language-models vision-language-learning generative-ai-model agentic-workflow agentic-rag agentic-ai internvl2

UpdatedJan 29, 2025
Python

PrateekJannu /Vision-GPT

Star1

Coding a Multi-Modal vision model like GPT-4o from scratch, inspired by@hkproj and PaliGemma

open-source machine-learning google artificial-intelligence gemini transformer-architecture transformer-models vision-transformer vision-language-transformer large-language-models llm large-language-model vision-language-model gpt-4o

UpdatedNov 17, 2024
Python

jhakrraman /Deep-Learning-NYU-Spring-2025

Star1

This reporsitory contains all the Homeworks, and Projects from the Deep Learning Course by Prof. Chinmay Hegde, in Spring 2025, at NYU.

natural-language-processing reinforcement-learning computer-vision deep-learning deep-learning-algorithms reinforcement-learning-algorithms new-york-university nyu natural-language-understanding computer-vision-algorithms deep-learning-models deep-learning-projects computer-vision-projects vision-language-transformer llms vision-language-model vision-language-learning vision-language-models

UpdatedMay 29, 2025

aurooj /VLM_SS

Star1

Mini-batch selective sampling for knowledge adaption of VLMs for mammography.

medical-imaging miccai mammogram multimodal-learning vision-and-language multimodal-retrieval vision-language-transformer multimodal-representation-learning miccai2024 medical-vision-language-model minibatch-selective-sampling

UpdatedOct 7, 2024
Jupyter Notebook

Improve this page

Add a description, image, and links to thevision-language-transformer topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with thevision-language-transformer topic, visit your repo's landing page and select "manage topics."

Learn more

Movatterモバイル変換

Navigation Menu

Search code, repositories, users, issues, pull requests...

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

vision-language-transformer

Here are 21 public repositories matching this topic...

salesforce /LAVIS

IDEA-Research /GroundingDINO

salesforce /BLIP

AlibabaResearch /AdvancedLiterateMachinery

henghuiding /ReLA

shenyunhang /APE

henghuiding /Vision-Language-Transformer

haoliuhl /instructrl

sMamooler /CLIP_Explainability

yiren-jian /BLIText

akusayudodograu /Agentic-RAG-Story-Generation-with-Multimodal-GenAI

unitaryai /VTC

deepmancer /vlm-toolbox

ThomasVonWu /Awesome-VLMs-Strawberry

marialymperaiou /knowledge-enhanced-multimodal-learning

fork123aniket /Multi-Round-VLM-powered-Multimodal-Conversational-AI-Navigation-Bot

fork123aniket /Agentic-RAG-Story-Generation-with-Multimodal-GenAI

PrateekJannu /Vision-GPT

jhakrraman /Deep-Learning-NYU-Spring-2025

aurooj /VLM_SS

Improve this page

Add this topic to your repo