Compressing Large-Scale Transformer-Based Models: A Case Study on BERT
Published
2022-01-04
Prakhar Ganesh
,
Yao Chen
,
Xin Lou
,
Mohammad Ali Khan
,
Yin Yang
,
Hassan Sajjad
,
Preslav Nakov
,
Deming Chen
,
Marianne Winslett
Prakhar Ganesh
Advanced Digital Sciences Center, Singapore
Yao Chen
Advanced Digital Sciences Center, Singapore
Xin Lou
Advanced Digital Sciences Center, Singapore
Mohammad Ali Khan
Advanced Digital Sciences Center, Singapore
Yin Yang
Hamad Bin Khalifa University
Hassan Sajjad
Hamad Bin Khalifa University;
Qatar Computing Research Institute
Preslav Nakov
Hamad Bin Khalifa University;
Qatar Computing Research Institute
Deming Chen
University of Illinois at Urbana-Champaign
Marianne Winslett
University of Illinois at Urbana-Champaign
Abstract
Pre-trained Transformer-based models have achieved state-of-the-art performance for various Natural Language Processing (NLP) tasks. However, these models often have billions of parameters, and thus are too resource-hungry and computation-intensive to suit low-capability devices or applications with strict latency requirements. One potential remedy for this is model compression, which has attracted a lot of research attention. Here, we summarize the research in compressing Transformers, focusing on the especially popular BERT model. In particular, we survey the state of the art in compression for BERT, we clarify the current best practices for compressing large-scale Transformer models, and we provide insights into the workings of various methods. Our categorization and analysis also shed light on promising future research directions for achieving lightweight, accurate, and generic NLP models.
Article at MIT Press
Presented at ACL 2022