Skip to main navigation menu Skip to main content Skip to site footer

Vol. 11 (2023)

TACL approved

Efficient Methods for Natural Language Processing: A Survey

Published 2023-07-24

  • Marcos Treviso
    • ,
  • Ji-Ung Lee
    • ,
  • Tianchu Ji
    • ,
  • Betty van Aken
    • ,
  • Qingqing Cao
    • ,
  • Manuel R. Ciosici
    • ,
  • Michael Hassid
    • ,
  • Kenneth Heafield
    • ,
  • Sara Hooker
    • ,
  • Pedro Henrique Martins
    • ,
  • André Filipe Torres Martins
    • ,
  • Peter Milder
    • ,
  • Colin Raffel
    • ,
  • Jessica Zosa Forde
    • ,
  • Emma Strubell
    • ,
  • Edwin Simpson
    • ,
  • Noam Slonim
    • ,
  • Jesse Dodge
    • ,
  • Iryna Gurevych
    • ,
  • Niranjan Balasubramanian
    • ,
  • Leon Derczynski
    • ,
  • Roy Schwartz

Marcos Treviso
Instituto Superior Técnico, University of Lisbon

Ji-Ung Lee
Technical University of Darmstadt

Tianchu Ji
Stony Brook University

Betty van Aken
Berliner Hochschule für Technik

Qingqing Cao
University of Washington

Manuel R. Ciosici
University of Southern California

Michael Hassid
The Hebrew University of Jerusalem

Kenneth Heafield
University of Edinburgh

Sara Hooker
Cohere For AI

Pedro Henrique Martins
Instituto Superior Técnico, University of Lisbon

André Filipe Torres Martins
Instituto Superior Técnico, University of Lisbon

Peter Milder
Stony Brook University

Colin Raffel
University of North Carolina at Chapel Hill

Jessica Zosa Forde
Brown University

Emma Strubell
Carnegie Mellon University

Edwin Simpson
University of Bristol

Noam Slonim
IBM Research

Jesse Dodge
Allen Institute for AI

Iryna Gurevych
Technical University of Darmstadt

Niranjan Balasubramanian
Stony Brook University

Leon Derczynski
IT University of Copenhagen

Roy Schwartz
The Hebrew University of Jerusalem

Abstract

Recent work in natural language processing (NLP) has yielded appealing results from scaling model parameters and training data; however, using only scale to improve performance means that resource consumption also grows. Such resources include data, time, storage, or energy, all of which are naturally limited and unevenly distributed. This motivates research into efficient methods that require fewer resources to achieve similar results. This survey synthesizes and relates current methods and findings in efficient NLP. We aim to provide both guidance for conducting NLP under limited resources, and point towards promising research directions for developing more efficient methods.
Article at MIT Press Presented at ACL 2023