Exploring Contrast Consistency of Open-Domain Question Answering Systems on Minimally Edited Questions
Published
2023-08-15
Zhihan Zhang
,
Wenhao Yu
,
Zheng Ning
,
Mingxuan Ju
,
Meng Jiang
Zhihan Zhang
University of Notre Dame
Wenhao Yu
University of Notre Dame
Zheng Ning
University of Notre Dame
Mingxuan Ju
University of Notre Dame
Meng Jiang
University of Notre Dame
Abstract
Contrast consistency, the ability of a model to make consistently correct predictions in the presence of perturbations, is an essential aspect in NLP. While studied in tasks such as sentiment analysis and reading comprehension, it remains unexplored in open-domain question answering (OpenQA) due to the difficulty of collecting perturbed questions that satisfy factuality requirements. In this work, we collect minimally edited questions as challenging contrast sets to evaluate OpenQA models. Our collection approach combines both human annotation and large language model generation. We find that the widely used dense passage retriever (DPR) performs poorly on our contrast sets, despite fitting the training set well and performing competitively on standard test sets. To address this issue, we introduce a simple and effective query-side contrastive loss with the aid of data augmentation to improve DPR training. Our experiments on the contrast sets demonstrate that DPR's contrast consistency is improved without sacrificing its accuracy on the standard test sets.
Article at MIT Press
Presented at EMNLP 2023