Repository files navigation

• The codes of CLIPN with hand-crafted prompts are released (./hand-crafted).
• The codes of CLIPN with learnable prompts are released (./src).
• Thanks to the valuable suggestions from the reviewers of CVPR 2023 and ICCV 2023, our paper has been significantly improved, allowing it to be published at ICCV 2023.
• If you are interested in CLIP-based open vocabulary tasks, please feel free to visit our another work!"CLIP Surgery for Better Explainability with Enhancement in Open-Vocabulary Tasks" (github).

• CLIPN attains SoTA performance in zero-shot OOD detection, all the while inheriting the in-distribution (ID) classification prowess of CLIP.
• CLIPN offers an approach for unsupervised prompt learning using image-text-paired web-dataset.

• Main python libraries of our experimental environment are shown inrequirements.txt. You can install CLIPN following:

git clone https://github.com/xmed-lab/CLIPN.gitcd CLIPNconda create -n CLIPNconda activate CLIPNpip install -r ./requirements.txt

• Pre-training Dataset, CC3M.To download CC3M dataset as webdataset, please followimg2dataset.

When you have downloaded CC3M, please re-write your data root into./src/run.sh.

• OOD detection datasets.
  • ID dataset, ImageNet-1K: The ImageNet-1k dataset (ILSVRC-2012) can be downloadedhere.
  • OOD dataset, iNaturalist, SUN, Places, and Texture. Please follow instruction from these two repositoriesMOS andMCM to download the subsampled datasets where semantically overlapped classes with ImageNet-1k are removed.

When you have downloaded the above datasets, please re-write your data root into./src/tuning_util.py.

• Pre-train CLIPN on CC3M.This step is to empower "no" logic within CLIP via the web-dataset.
  • The model of CLIPN is defined in./src/open_clip/model.py. Here, you can find a group of learnable 'no' token embeddings defined in Line 527.
  • The function of loading parameters of CLIP is defined in./src/open_clip/factory.py.
  • The loss functions are defined in./src/open_clip/loss.py.
  • You can pre-train CLIPN on ViT-B-32 and ViT-B-16 by:

cd ./srcsh run.sh

• Zero-Shot Evaluate CLIPN on ImageNet-1K.
  • Metrics and pipeline are defined in./src/zero_shot_infer.py. Here you can find three baseline methods, and our two inference algorithms: CTW and ATD (see Line 91-96).
  • Dataset details are defined in./src/tuning_util.py.
  • Inference models are defined in./src/classification.py, including converting the text encoders into classifiers.
  • You can download the models provided in the table below or pre-trained by yourself. Then re-write the path of your models in the main function of./src/zero_shot_infer.py. Finally, evaluate CLIPN by:

python3 zero_shot_infer.py

To ensure the reproducibility of the results, we conducted three repeated experiments under each configuration. The following will exhibit the most recent reproduced results achieved before open-sourcing.

• ImageNet-1K

The performance in this table is better than our paper , because that we add an average learnable "no" prompt (seeLine 600-616 in./src/open_clip/model.py).

There are several important factors that could affect the performance:

• Class prompt texts. In the inference period, we need to use prompt texts to get the weights of classifier (see./src/prompt/prompt.txt). You can hand on the design of high-performance inference prompts for our CLIPN.
• The number of learnable "no" tokens. Now I just define the number of learnable "no" tokens as 16. You can vary it to find an optimal value.
• If you have any ideas to enhance CLIPN or attempt to transfer this idea to other topics, feel free to discuss with me and I am happy to share some ideas with you.

If you find our paper helps you, please kindly consider citing our paper in your publications.

@inproceedings{wang2023clipn,  title={CLIPNfor Zero-Shot OOD Detection: Teaching CLIP to Say No},  author={Wang, Hualiang and Li, Yi and Yao, Huifeng and Li, Xiaomeng},  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},  pages={1802--1812},  year={2023}}

We sincerely appreciate these three highly valuable repositoriesopen_clip,MOS andMCM.