Skip to main navigation menu Skip to main content Skip to site footer

The Benefits of a Model of Annotation


Standard agreement measures for interannotator reliability are neither necessary nor sufficient to ensure a high quality corpus. In a case study of word sense annotation, conventional methods for evaluating labels from trained annotators are contrasted with a probabilistic annotation model applied to crowdsourced data. The annotation model provides far more information, including a certainty measure for each gold standard label; the crowdsourced data was collected at less than half the cost of the conventional approach. 

PDF (Presented at EMNLP 2014) erratum

Author Biography

Rebecca J. Passonneau

Center for Computational Learning Systems, Director