Repository files navigation

Mosaic ofObject-centric Images asScene-centric Images (MosaicOS) for long-tailed object detection and instance segmentation.

Many objects do not appear frequently enough in complex scenes (e.g., certain handbags in living rooms) fortraining an accurate object detector, but are often found frequently by themselves (e.g., in product images).Yet, these object-centric images are not effectively leveraged for improving object detection in scene-centricimages.

We propose Mosaic of Object-centric images as Scene-centric images (MosaicOS), a simple and novel framework that is surprisingly effective at tackling the challenges of long-tailed object detection. Keys to our approachare three-fold: (i) pseudo scene-centric image construction from object-centric images for mitigating domain differences, (ii) high-quality bounding box imputation usingthe object-centric images’ class labels, and (iii) a multistage training procedure. Check our paper for further details:

MosaicOS:A Simple and Effective Use of Object-Centric Images for Long-Tailed Object Detection. In IEEE/CVF International Conference on Computer Vision (ICCV), 2021.

byCheng Zhang*,Tai-Yu Pan*,Yandong Li,Hexiang Hu,Dong Xuan,Soravit Changpinyo,Boqing Gong,Wei-Lun Chao.

The scriptmosaic.py generates mosaic images and annotaions by given an annotation file in COCO format (for more informationhere). The following command will generate 2x2 mosaic images and the annotation file for COCO training dataset inOUTPUT_DIR/images/ andOUTPUT_DIR/annotation.json with 4 processors.--shuffle is to shuffle the order of images to synthesize and--drop-last is to drop the last couple of images if they are not enough fornrow * ncol.--demo 10 plots 10 synthesized images with annotated boxes inOUTPUT_DIR/demo/ for visualization.

 python mosaic.py --coco-file datasets/coco/annotations/instances_train2017.json --img-dir datasets/coco --output-dir output_mosaics --num-proc 4 --nrow 2 --ncol 2 --shuffle --drop-last --demo 10

*Note: In our work, we sythesize mosaics from object-centric images with pseudo bounding box to find-tune the pre-trained detector.

Our impelementation is based onDetectron2.All models are trained onLVIS training set withRepeated FactorSampling (RFS).

• Object detection

• Instance segmentation

• Object detection

• Instance segmentation

Please cite with the following bibtex if you find it useful.

@inproceedings{zhang2021mosaicos,  title={{MosaicOS}: A Simple and Effective Use of Object-Centric Images for Long-Tailed Object Detection},  author={Zhang, Cheng and Pan, Tai-Yu and Li, Yandong and Hu, Hexiang and Xuan, Dong and Changpinyo, Soravit and Gong, Boqing and Chao, Wei-Lun},  booktitle = {ICCV},  year={2021}}

Feel free to email us if you have any questions.

Cheng Zhang (zhang.7804@osu.edu), Tai-Yu Pan (pan.667@osu.edu), Wei-Lun Harry Chao (chao.209@osu.edu)