Repository files navigation

This repository contains the official PyTorch implementation of our WACV 2025 Oral paper:"Composed Image Retrieval for Training-FREE DOMain Conversion". [arXiv]

We introduce FREEDOM, atraining-free, composed image retrieval (CIR) method for domain conversion based on vision-language models (VLMs). Given an$\textcolor{orange}{image\ query}$ and a$\it{text\ query}$ that names a domain, images are retrieved having the class of the$\textcolor{orange}{image\ query}$ and the domain of the$\it{text\ query}$. A range of applications is targeted, where classes can be defined at category level (a,b) or instance level (c), and domains can be defined as styles (a, c), or context (b). In the above visualization, for each image query, retrieved images are shown for different text queries.

In this paper, we focus on a specific variant of composed image retrieval, namelydomain conversion, where the text query defines the target domain. Unlike conventional cross-domain retrieval, where models are trained to use queries of a source domain and retrieve items from another target domain, we address a more practical, open-domain setting, where the query and database may be from any unseen domain. We target different variants of this task, where the class of the query object is defined at category-level (a, b) or instance-level (c). At the same time, the domain corresponds to descriptions of style (a, c) or context (b). Even though domain conversion is a subset of the tasks handled by existing CIR methods, the variants considered in our work reflect a more comprehensive set of applications than what was encountered in prior art.

Given a$\textcolor{orange}{query\ image}$ and a$\it{query\ text}$ indicating the target domain, proxy images are first retrieved from the query through an image-to-image search over a visual memory. Then, a set of text labels is associated with each proxy image through an image-to-text search over a textual memory. Each of the most frequent text labels is combined with the$\it{query\ text}$ in the text space, and images are retrieved from the database by text-to-image search. The resulting sets of similarities are linearly combined with the frequencies of occurrence as weights. Below:$k=4$ proxy images,$n=3$ text labels per proxy image,$m=2$ most frequent text labels.

Our experiments were conducted usingpython 3.10. To set up a Python environment, run:

python -m venv~/freedomsource~/freedom/bin/activatepip install -r requirements.txt

To set up a Conda environment, run:

conda create --name freedom python=3.10conda activate freedompip install -r requirements.txt

1. Download theImageNet-R dataset and the validation set ofILSVRC2012 and place them in the directorydata/imagenet-r.
2. Download theLTLL dataset and place it in the directorydata/ltll.
3. Download the four domains of mini-DomainNet:Clipart,painting,real, andsketch and place them in the directorydata/minidn.
4. Download theNICO++ dataset, specifically theDG_Benchmark.zip from the dropbox link and place it in the directorydata/nico.The starting data directory structure should look like this:

freedom/    ├── data/    │   ├── imagenet-r/    │   │   ├── imagenet-r.tar    │   │   ├── ILSVRC2012_img_val.tar    │   │   ├── imgnet_real_query.txt    │   │   ├── imgnet_targets.txt    │   │   └── label_names.csv    │   ├── ltll/    |   |   ├── LTLL.zip    |   |   └── full_files.csv    |   ├── minidn/    |   |   ├── clipart.zip    |   |   ├── painting.zip    |   |   ├── real.zip    |   |   ├── sketch.zip    |   |   ├── database_files.csv    |   |   └── query_files.csv    |   └── nico/    |       ├── DG_Benchmark.zip    |       ├── database_files.csv    |       └── query_files.csv

1. To set-upImageNet-R run:

mkdir -p ./data/imagenet_r/imagenet_val&& tar -xf ./data/imagenet_r/ILSVRC2012_img_val.tar -C ./data/imagenet_r/imagenet_valtar -xf ./data/imagenet_r/imagenet-r.tar -C ./data/imagenet_r/python set_dataset.py --dataset imagenet_r

The scriptset_dataset.py will create the folderreal, that includes the 200 classes that are useful forImageNet-R from the validation set ofILSVRC2012, and it will place them in their corresponding class folder. It will also create the filefull_files.csv that is needed for data loading. After that you no longer need theimagenet_val folder.

2. To set-upltll run:

unzip ./data/ltll/LTLL.zip -d ./data/ltllpython set_dataset.py --dataset ltll

The scriptset_dataset.py will handle some space(" ") characters in directory and file names.

3. To set-upMini-DomainNet run:

unzip ./data/minidn/clipart.zip -d ./data/minidn/unzip ./data/minidn/painting.zip -d ./data/minidn/unzip ./data/minidn/real.zip -d ./data/minidn/unzip ./data/minidn/sketch.zip -d ./data/minidn/

4. To set-upNICO++ run:

unzip ./data/nico/DG_Benchmark.zip -d ./data/nicounzip ./data/nico/NICO_DG_Benchmark.zip -d ./data/nicomv ./data/nico/NICO_DG/* ./data/nico/rmdir ./data/nico/NICO_DGpython set_dataset.py --dataset nico

The scriptset_dataset.py will handle some space(" ") characters in directory names.

The necessary files in the final data directory structure are the following:

freedom/    ├── data/    │   ├── imagenet-r/    │   │   ├── imagenet-r/    │   │   ├── real/    │   │   └── full_files.csv    │   ├── ltll/    |   |   ├── New/    |   |   ├── Old/    |   |   └── full_files.csv    |   ├── minidn/    |   |   ├── clipart/    |   |   ├── painting/    |   |   ├── real/    |   |   ├── sketch/    |   |   ├── database_files.csv    |   |   └── query_files.csv    |   └── nico/    |       ├── autumn/    |       ├── dim/    |       ├── grass/    |       ├── outdoor/    |       ├── rock/    |       ├── water/    |       ├── database_files.csv    |       └── query_files.csv

To run any experiment you first need to extract features with the scriptcreate_features.py. Both the features of the corpus and the features of the wanted dataset are needed, but for a given backbone, they need to be extracted only once. You can specify what features to produce with--dataset that can take the inputscorpus,imagenet_r,nico,minidn, andltll. Also, you will need to specify the--backbone, which can either beclip orsiglip. You can specify the GPU ID with--gpu. For example, to run experiments onImageNet-R withCLIP on theGPU 0 you should extract the following features:

python create_features.py --dataset corpus --backbone clip --gpu 0python create_features.py --dataset imagenet_r --backbone clip --gpu 0

The experiments can run through therun_retrieval.py script. You should specify:

1. --dataset fromimagenet_r,nico,minidn, andltll.
2. --backbone fromclip, andsiglip.
3. --method fromfreedom,image,text,sum, andproduct.For example, you can run:

python run_retrieval.py --dataset imagenet_r --backbone clip --method freedom --gpu 0

Specifically forfreedom experiments, you can change the hyperparameters described in the paper by specifying--kappa,--miu, and--ni.

All mAPs in this paper are calculated as inRadenović et. al.

The experiments conducted with therun_retrieval.py script produce the following Domain Conversion mAP (%) results for thedatasets andmethods outlined above. By running the experiments as described, you should expect the following numbers for the CLIP backbone:

NTUA thanksNVIDIA for the support with the donation of GPU hardware.

This repository is released under the MIT license as found in theLICENSE file.

If you find this repository useful, please consider giving a star 🌟 and citation:

@inproceedings{efth2025composed,  title={Composed Image Retrieval for Training-Free Domain Conversion},  author={Efthymiadis, Nikos and Psomas, Bill and Laskar, Zakaria and Karantzalos, Konstantinos and Avrithis, Yannis and Chum, Ondřej and Tolias, Giorgos},  booktitle={IEEE Winter Conference on Applications of Computer Vision},  year={2025},  organization={IEEE}}