Repository files navigation

The must-have resource for anyone who wants to experiment with and build on theOpenAIVision API. This repository serves asa hub for innovative experiments, showcasing a variety of applications ranging fromsimple image classifications to advanced zero-shot learning models. It's a space forboth beginners and experts to explore the capabilities of the Vision API, share theirfindings, and collaborate on pushing the boundaries of visual AI.

Experimenting with the OpenAI API requires an API 🔑. You can get onehere.

• 100 API requests per single API key per day.
• Can't be used for object detection or image segmentation. We can solve this problem by combining GPT-4V with foundational models like GroundingDINO or Segment Anything (SAM). Please take a look at theexample and read ourblog post.

experiment	complementary materials	authors
WebcamGPT - chat with video stream		@SkalskiP
HotDogGPT - simple image classification application		@SkalskiP
zero-shot image classifier with GPT-4V		@capjamesg
zero-shot object detection with GroundingDINO + GPT-4V		@capjamesg
GPT-4V vs. CLIP		@capjamesg
GPT-4V with Set-of-Mark (SoM)		Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, Jianfeng Gao
GPT-4V on Web		@Jiayi-Pan
automated voiceover of NBA game		@SkalskiP
screenshot-to-code		@abi
GPT with Vision Checkup		Roboflow team

• Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vby Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, Jianfeng Gao
• The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)by Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, Lijuan Wang
• GPT-4 System Card by OpenAI

• How CLIP and GPT-4V Compare for Classification
• Experiments with GPT-4V for Object Detection
• Distilling GPT-4 for Classification with an API
• DINO-GPT4-V: Use GPT-4V in a Two-Stage Detection Model
• First Impressions with GPT-4V(ision)

We would love your help in making this repository even better! Whether you want toadd a new experiment or have any suggestions for improvement,feel free to open anissueorpull request.

If you are up to the task and want to add a new experiment, please look at ourcontribution guide. There you can find all the information you need.