What Should Be Balanced in a "Balanced" Face Recognition Dataset?
Haiyu Wu (University of Notre Dame),* Kevin Bowyer (University of Notre Dame)
The 34th British Machine Vision Conference
Abstract
The issue of disparities in face recognition accuracy across demographic groups has attracted increasing attention in recent years. Various face image datasets have been proposed as ’fair’ or ’balanced’ to assess the accuracy of face recognition algorithms across demographics. While these datasets often balance the number of identities and images across demographic groups. It is important to note that the number of identities and images in an evaluation dataset are not the driving factors for 1-to-1 face matching accuracy. Moreover, balancing the number of identities and images does not ensure balance in other factors known to impact accuracy, such as head pose, brightness, and image quality. We demonstrate these issues using several recently proposed datasets. To enhance the capacity for less biased evaluations, we propose a bias-aware toolkit that facilitates the creation of cross-demographic evaluation datasets balanced on factors mentioned in this paper.Video
Citation
@inproceedings{Wu_2023_BMVC,author = {Haiyu Wu and Kevin Bowyer},title = {What Should Be Balanced in a "Balanced" Face Recognition Dataset?},booktitle = {34th British Machine Vision Conference 2023, {BMVC} 2023, Aberdeen, UK, November 20-24, 2023},publisher = {BMVA},year = {2023},url = {https://papers.bmvc2023.org/0235.pdf}}Copyright © 2023The British Machine Vision Association and Society for Pattern Recognition
The British Machine Vision Conference is organised byThe British Machine Vision Association and Society for Pattern Recognition. The Association is a Company limited by guarantee, No.2543446, and a non-profit-making body, registered in England and Wales as Charity No.1002307 (Registered Office: Dept. of Computer Science, Durham University, South Road, Durham, DH1 3LE, UK).