Skip to main navigation menu Skip to main content Skip to site footer

Comparing Apples to Apple: The Effects of Stemmers on Topic Models

Abstract

Rule-based stemmers such as the Porter stemmer are frequently used to preprocess English corpora for topic modeling. In this work, we train and evaluate topic models on a variety of corpora using several different stemming algorithms. We examine several different quantitative measures of the resulting models, including likelihood, coherence, model stability, and entropy. Despite their frequent use in topic modeling, we find that stemmers produce no meaningful improvement in likelihood and coherence and in fact can degrade topic stability.
PDF (presented at EMNLP 2016)

Author Biography

Alexandra Schofield

PhD Student, Department of Computer Science

David Mimno

Assistant Professor, Department of Information Science