Quality Assessment for Crownsourced Object Annotations
As computer vision datasets grow larger the community is increasingly relying on crowdsourced annotations to train and test our algorithms. Due to the heterogeneous and unpredictable capability of online annotators, various strategies have been proposed to "clean" crowdsourced annotations. However, these strategies typically involve getting more annotations, perhaps different types of annotations (e.g. a grading task), rather than computationally assessing the annotation or image content. In this paper we propose and evaluate several strategies for automatically estimating the quality of a spatial object annotation. We show that one can significantly outperform simple baselines, such as that used by LabelMe, by combining multiple image-based annotation assessment strategies.
Posterposter pdf (1 MB)
Database of LabelMe and Ground Truth Annotations
This database contains 1000 image crops -- 200 each from 5 object categories. For each crop we provide an accurate "ground truth" annotation generated according to LabelMe guidelines.