Skip to main navigation menu Skip to main content Skip to site footer

Vol. 8 (2020)

TACL approved

A Primer in BERTology: What We Know About How BERT Works

Published 2021-01-06

  • Anna Rogers
    • ,
  • Olga Kovaleva
    • ,
  • Anna Rumshisky

Anna Rogers
University of Massachusetts Lowell

Olga Kovaleva
University of Massachusetts Lowell

Anna Rumshisky
University of Massachusetts Lowell

Abstract

Transformer-based models have pushed state of the art in many areas of NLP, but our understanding of what is behind their success is still limited. This paper is the first survey of over 150 studies of the popular BERT model. We review the current state of knowledge about how BERT works, what kind of information it learns and how it is represented, common modifications to its training objectives and architecture, the overparameterization issue and approaches to compression. We then outline directions for future research.
Article at MIT Press Presented at NAACL 2021

Author Biography

Anna Rogers

Department of Computer Science, post-doctoral associate