Repository files navigation

The Devil is in the Points: Weakly Semi-Supervised Instance Segmentation via Point-Guided Mask Representation
Beomyoung Kim^1,2,Joonhyun Jeong^1,2,Dongyoon Han³,Sung Ju Hwang²

¹_{NAVER Cloud, ImageVision}
²_KAIST
³_{NAVER AI Lab}

We introduce a novel learning scheme named weakly semi-supervised instance segmentation (WSSIS) with point labels for budget-efficient and high-performance instance segmentation. Namely, we consider a dataset setting consisting of a few fully-labeled images and a lot of point-labeled images. Motivated by the main challenge of semi-supervised approaches mainly derives from the trade-off between false-negative and false-positive instance proposals, we propose a method for WSSIS that can effectively leverage the budget-friendly point labels as a powerful weak supervision source to resolve the challenge. Furthermore, to deal with the hard case where the amount of fully-labeled data is extremely limited, we propose a MaskRefineNet that refines noise in rough masks. We conduct extensive experiments on COCO and BDD100K datasets, and the proposed method achieves promising results comparable to those of the fully-supervised model, even with 50% of the fully labeled COCO data (38.8% vs. 39.7%). Moreover, when using as little as 5% of fully labeled COCO data, our method shows significantly superior performance over the state-of-the-art semi-supervised learning method (33.7% vs. 24.9%).

2023-05-19 First Commit, We release the official implementation of PointWSSIS.

Our implementation is based onDetectron2 andAdelaiDet.

Please install Detectron2 and AdelaiDet in our repository (there are some modifications,check here).

git clone https://github.com/clovaai/PointWSSIS.gitcd PointWSSIS/detectron2python3 setup.py build develop --usercd ../AdelaiDetpython3 setup.py build develop --user

Please check environment requirements and official guidelinehttps://github.com/aim-uofa/AdelaiDet#installation.

ForCOCO 2017 dataset, we made subsets 1%, 2%, 5%, 10%, 20%, 30%, 50%.

• coco 5p: strong 5% + weak 95%
• coco 10p: strong 10% + weak 90%
• coco 20p: strong 20% + weak 80%
• coco 50p: strong 50% + weak 50%

All subsets are available:[coco2017 dataset] and[BDD100K dataset]

• instances_train2017_{subset}_s.json : strong (fully-labeled) data subset.
• instances_train2017_{subset}_w.json : weak (point-labeled) data subset.

We also provide our pseudo labels for training the student network:

• instances_train2017_{subset}_sw.json : strong and weak (pseudo labels without MaskRefineNet) annotations
• instances_train2017_{subset}_sw_refined.json : strong and weak (pseudo labels with MaskRefineNet) annotations

data_root/    --- coco/        --- train2017/        --- val2017/        --- annotations/            --- instances_val2017.json            --- instances_train2017.json            --- instances_train2017_10p_s.json            --- instances_train2017_10p_w.json            --- instances_train2017_10p_sw.json (optional)            --- instances_train2017_10p_sw_refined.json (optional)

1. Training teacher network using strong labels
2. Training MaskRefineNet using strong labels
3. Generating pseudo labels using weak labels
4. Training student network using strong and pseudo labels

Run all-in-one script.

bash scripts/coco_5p.sh # coco 5% scenariobash scripts/coco_10p.sh # coco 10% scenariobash scripts/coco_20p.sh # coco 20% scenariobash scripts/coco_50p.sh # coco 50% scenario

@inproceedings{kim2023devil,  title={The Devil is in the Points: Weakly Semi-Supervised Instance Segmentation via Point-Guided Mask Representation},  author={Kim, Beomyoung and Jeong, Joonhyun and Han, Dongyoon and Hwang, Sung Ju},  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},  pages={11360--11370},  year={2023}}

PointWSSIS Copyright (c) 2023-present NAVER Cloud Corp.Licensed under the Apache License, Version 2.0 (the "License");you may not use this file except in compliance with the License.You may obtain a copy of the License at    http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.