Repository files navigation

Google Colab Demo |Introduction |GFM |AM-2k |BG-20k |Results Demo |Train and Test |Inference Code |Statement

🚀 News

[2021-11-12]: The training code, test code and all the pretrained models are released in thiscode-base page.

[2021-10-22]: The paper has been accepted by the International Journal of Computer Vision (IJCV)! 🎉

[2021-09-21]: The datasetsAM-2k andBG-20k can now beopenly accessed from the links below (both at Google Drive and at Baidu Wangpan) ! Please follow the dataset release agreements to access. Due to some privacy issues, the datasetPM-10k will be published after privacy-preserving from the projectPrivacy-Preserving Portrait Matting (ACM MM 21). You can refer to thisrepo for access and updates.

Dataset	Dataset Link (Google Drive)	Dataset Link (Baidu Wangpan 百度网盘)	Dataset Release Agreement
AM-2k	Link	Link (pw: 29r1)	Agreement (MIT License)
BG-20k	Link	Link (pw: dffp)	Agreement (MIT License)

[2020-11-17]: CreateGoogle Colab demo to benefit users who want to have a try online.

[2020-11-03]: Publish theinference code and apretrained model that can be used to test on your own animal images.

[2020-10-27]: Publish a video demo (YouTube |bilibili |Google drive) contains motivation, network, datasets, and test results on an animal video.

For those who do not have GPUs in their environment or only want to have a simple try online, you can try ourGoogle Colab demo to generate the results for your images easily.

This repository contains the code, datasets, models, test results and a video demo for the paperBridging Composite and Real: Towards End-to-end Deep Image Matting. We propose a novel Glance and Focus Matting network (GFM), which employs a shared encoder and two separate decoders to learn both tasks in a collaborative manner for end-to-end image matting. We also establish a novel Animal Matting dataset (AM-2k) to serve for end-to-end matting task. Furthermore, we investigate the domain gap issue between composition images and natural images systematically, propose a carefully designed composite routeRSSN and a large-scale high-resolution background dataset (BG-20k) to serve as better candidates for composition.

We have released the train code, the test code, the datasets, and the pretrained models in thiscode-base page. We have also prepared aGoogle Colab demo andinference code for you to test on our pre-trained models on your own sample images. For the datasetsAM-2k andBG-20k, please follow the sectionsAM-2k andBG-20k to access. Besides, we prepare a video demo (YouTube |bilibili) to illustrate the motivation, the network, the datasets, and the test results on an animal video

The architecture of our proposed end-to-end methodGFM is illustrated below. We adopt three kinds ofRepresentation of Semantic and Transition Area (RoSTa)-TT, -FT, -BT within our method.

We trainedGFM with four backbones,-(r) (ResNet-34),-(d) (DenseNet-121),-(r2b) (ResNet-34 with 2 extra blocks), and-(r') (ResNet-101). The trained model for each backbone can be downloaded via the link listed below.

Our proposedAM-2k contains 2,000 high-resolution natural animal images from 20 categories along with manually labeled alpha mattes. Some examples are shown as below, more can be viewed in the video demo (YouTube |bilibili |Google drive).

AM-2k can be accessed from here (Google Drive |Baidu Wangpan (pw: 29r1)), please make sure that you have readthis agreement before accessing the dataset. Please refer to thereadme.txt in the dataset folder for more details.

Our proposedBG-20k contains 20,000 high-resolution background images excluded salient objects, which can be used to help generate high quality synthetic data. Some examples are shown as below, more can be viewed in the video demo (YouTube |bilibili |Google drive).

BG-20k can be accessed from here (Google Drive |Baidu Wangpan (pw: dffp)), please make sure that you have readthis agreement before accessing the dataset. Please refer to thereadme.txt in the dataset folder for more details.

We test GFM on our AM-2k test dataset and show the results as below. More results on AM-2k test set can be foundhere.

Here we provide the procedure of testing on sample images by our pretrained models:

1. Setup environment following thisinstruction page.

2. Download pretrained models as shown in sectionGFM, unzip to the foldermodels/pretrained/

3. Save your high-resolution sample images in foldersamples/original/.

4. Setup parameters inscripts/test/test_samples.sh and run it

   chmod +x scripts/*

   ./scripts/test/test_samples.sh

5. The results of alpha matte and transparent color image will be saved in foldersamples/result_alpha/. andsamples/result_color/.

We show some sample images from the internet, the predicted alpha mattes, and their transparent results as below. We adoptbackbone='r34_2b',rosta=TT, hybrid testing strategy and the pretrained model (Google Drive |Baidu Wangpan (pw: 34hf)) for this case.

If you are interested in our work, please consider citing the following:

@article{li2022bridging,  title={Bridging composite and real: towards end-to-end deep image matting},  author={Li, Jizhizi and Zhang, Jing and Maybank, Stephen J and Tao, Dacheng},  journal={International Journal of Computer Vision},  volume={130},  number={2},  pages={246--266},  year={2022},  publisher={Springer}}

This project is under the MIT license. For further questions, please contactJizhizi Li atjili8515@uni.sydney.edu.au.

[1]Deep Automatic Natural Image Matting, IJCAI, 2021 |Paper |Github
     Jizhizi Li, Jing Zhang, and Dacheng Tao

[2]Privacy-Preserving Portrait Matting, ACM MM, 2021 |Paper |Github
     Jizhizi Li^∗, Sihan Ma^∗, Jing Zhang, and Dacheng Tao

[3]Referring Image Matting, CVPR, 2023 |Paper |Github
     Jizhizi Li, Jing Zhang, and Dacheng Tao

[4]Rethinking Portrait Matting with Privacy Preserving, IJCV, 2023 |Paper |Github
     Sihan Ma^∗, Jizhizi Li^∗, Jing Zhang, He Zhang, Dacheng Tao

[5]Deep Image Matting: A Comprehensive Survey, ArXiv, 2023 |Paper |Github
     Jizhizi Li, Jing Zhang, and Dacheng Tao