Repository files navigation

Official PyTorch implementation

Pretraining is All You Need for Image-to-Image Translation
Tengfei Wang,Ting Zhang,Bo Zhang,Hao Ouyang,Dong Chen,Qifeng Chen,Fang Wen
2022

paper |project website |video |online demo

We present a simple and universal framework that brings the power of the pretraining to variousimage-to-image translation tasks. You may try ouronline demo if interested.

Diverse samples synthesized by our approach.

git clone https://github.com/PITI-Synthesis/PITI.gitcd PITI

conda env create -f environment.yml

Please download our pre-trained models for bothBase model andUpsample model, and put them in./ckpt.

If you fail to access to these links, you may alternatively find our pretrained modelshere.

We put some example images in./test_imgs, and you can quickly try them.

For COCO dataset, download the images and annotations from theCOCO webpage.

For mask-to-image synthesis, we use the semantic maps in RGB format as inputs. To obtain such semantic maps, run./preprocess/preprocess_mask.py (an example of the raw mask and the processed mask is given inpreprocess/example). Note that we do not need instant masks like previous works.

For sketch-to-image synthesis, we use sketch maps extracted by HED as inputs. To obtain such sketch maps, run./preprocess/preprocess_sketch.py.

Run the following script, and it would create an interactive GUI built by gradio. You can upload input masks or sketches and generate images.

pip install gradiopython inference.py

Modifysample.sh according to the follwing instructions, and run:

bash sample.sh

1. Download and preprocess datasets. For COCO dataset, download the images and annotations from theCOCO webpage. Run./preprocess/preprocess_mask.py or./preprocess/preprocess_sketch.py
2. Download pretrained models bypython preprocess/download.py.

Taking mask-to-image synthesis as an example: (sketch-to-image is the same)

Modifymask_finetune_base.sh and run:

bash mask_finetune_base.sh

Modifymask_finetune_upsample.sh and run:

bash mask_finetune_upsample.sh

If you find this work useful for your research, please cite:

@article{wang2022pretraining, title = {Pretraining is All You Need for Image-to-Image Translation},  author = {Wang, Tengfei and Zhang, Ting and Zhang, Bo and Ouyang, Hao and Chen, Dong and Chen, Qifeng and Wen, Fang},  journal={arXiv:2205.12952},  year = {2022},}

Thanks forGLIDE for sharing their code.