Hello, I figured I would get the ball rolling and create a first post in this shiny, new discussion section.
Here is the thing, let's say you have a dataset for single-class segmentation with:

• many images with small labels
• many images without any label
• a few images with very large labels

Now let's say you computeIoU on the validation dataset with batch_size = 1
Regular IoU will compute the IoU of each image and average across all imagesSum(I/U)/N
-> in that case, images without any label will either contribute with a score of 1 (no prediction) or 0 (even a single pixel false-positive will lead to 0) i.e.. you get artificially high IoU as long as empty images are correctly predicted

My first approach was to code apartial-IoU: compute I & U for each image and thenSum(I)/Sum(U)
-> in that case empty images do not inflate the score anymore (but they might not contribute enough)
-> but images with very large labels will now steer the score due to being the largest contributors to partial-IoU

So now I'm wondering if there are any SotA techniques to properly weight the metric and avoid large contirbution of extremes images (no label and large label). My research did not let me find anything yet.
Approaches I had in mind

• partial-IoU: weight each image using the inverse of the number of pixels labels in that image
• IoU: increase epsilon to change the penalty shape of images without labels