Skip to main navigation menu Skip to main content Skip to site footer

Vol. 10 (2022)

TACL approved

Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets

Published 2022-01-31

  • Julia Kreutzer
    • ,
  • Isaac Caswell
    • ,
  • Lisa Wang
    • ,
  • Ahsan Wahab
    • ,
  • Daan van Esch
    • ,
  • Nasanbayar Ulzii-Orshikh
    • ,
  • Allahsera Auguste Tapo
    • ,
  • Nishant Subramani
    • ,
  • Artem Sokolov
    • ,
  • Claytone Sikasote
    • ,
  • Monang Setyawan
    • ,
  • Supheakmungkol Sarin
    • ,
  • Sokhar Samb
    • ,
  • Benoît Sagot
    • ,
  • Clara Rivera
    • ,
  • Annette Rios
    • ,
  • Isabel Papadimitriou
    • ,
  • Salomey Osei
    • ,
  • Pedro Ortiz Suarez
    • ,
  • Iroro Fred Ọ̀nọ̀mẹ̀ Orife
    • ,
  • Kelechi Ogueji
    • ,
  • Rubungo Andre Niyongabo
    • ,
  • Toan Nguyen
    • ,
  • Mathias Müller
    • ,
  • André Müller
    • ,
  • Shamsuddeen Hassan Muhammad
    • ,
  • Nanda Muhammad
    • ,
  • Ayanda Mnyakeni
    • ,
  • Jamshidbek Mirzakhalov
    • ,
  • Tapiwanashe Matangira
    • ,
  • Colin Leong
    • ,
  • Nze Lawson
    • ,
  • Sneha Kudugunta
    • ,
  • Yacine Jernite
    • ,
  • Mathias Jenny
    • ,
  • Orhan Firat
    • ,
  • Bonaventure Femi Pancrace Dossou
    • ,
  • Sakhile Dlamini
    • ,
  • Nisansa de Silva
    • ,
  • Sakine Çabuk Ballı
    • ,
  • Stella Biderman
    • ,
  • Alessia Battisti
    • ,
  • Ahmed Baruwa
    • ,
  • Ankur Bapna
    • ,
  • Pallavi Baljekar
    • ,
  • Israel Abebe Azime
    • ,
  • Ayodele Awokoya
    • ,
  • Duygu Ataman
    • ,
  • Orevaoghene Ahia
    • ,
  • Oghenefego Ahia
    • ,
  • Sweta Agrawal
    • ,
  • Mofetoluwa Adeyemi

Julia Kreutzer
Google Research; Masakhane NLP

Isaac Caswell
Google Research

Lisa Wang
Google Research

Ahsan Wahab
Turkic Interlingua

Daan van Esch
Google Research

Nasanbayar Ulzii-Orshikh
Haverford College

Allahsera Auguste Tapo
RobotsMali

Nishant Subramani
Allen Institute for AI; Masakhane NLP

Artem Sokolov
Google Research

Claytone Sikasote
University of Zambia; Masakhane NLP

Monang Setyawan
Google

Supheakmungkol Sarin
Google

Sokhar Samb
AIMS-AMMI; Masakhane NLP

Benoît Sagot
Inria

Clara Rivera
Google Research

Annette Rios
University of Zurich

Isabel Papadimitriou
Stanford University

Salomey Osei
Kwame Nkrumah University of Science and Technology; Masakhane NLP

Pedro Ortiz Suarez
Inria; Sorbonne University

Iroro Fred Ọ̀nọ̀mẹ̀ Orife
Niger-Volta Language Technologies Institute; Masakhane NLP

Kelechi Ogueji
University of Waterloo; Masakhane NLP

Rubungo Andre Niyongabo
University of Electronic Science and Technology of China; Masakhane NLP

Toan Nguyen
University of Notre Dame

Mathias Müller
University of Zurich

André Müller
University of Zurich

Shamsuddeen Hassan Muhammad
Bayero University; Masakhane NLP

Nanda Muhammad
Google

Ayanda Mnyakeni
Google

Jamshidbek Mirzakhalov
University of South Florida; Turkic Interlingua

Tapiwanashe Matangira
Google

Colin Leong
Masakhane NLP

Nze Lawson
Google

Sneha Kudugunta
Google Research

Yacine Jernite
Hugging Face; Masakhane NLP

Mathias Jenny
University of Zurich

Orhan Firat
Google Research

Bonaventure Femi Pancrace Dossou
Jacobs University Bremen; Masakhane NLP

Sakhile Dlamini
Google

Nisansa de Silva
University of Moratuwa

Sakine Çabuk Ballı
University of Zurich

Stella Biderman
EleutherAI

Alessia Battisti
University of Zurich

Ahmed Baruwa
Obafemi Awolowo University; Masakhane NLP

Ankur Bapna
Google Research

Pallavi Baljekar
Google Research

Israel Abebe Azime
AIMS-AMMI; Masakhane NLP

Ayodele Awokoya
University of Ibadan; Masakhane NLP

Duygu Ataman
University of Zurich

Orevaoghene Ahia
Masakhane NLP

Oghenefego Ahia
Google

Sweta Agrawal
University of Maryland

Mofetoluwa Adeyemi
Defence Space Administration, Abuja; Masakhane NLP

Abstract

With the success of large-scale pre-training and multilingual modeling in Natural Language Processing (NLP), recent years have seen a proliferation of large, web-mined text datasets covering hundreds of languages. We manually audit the quality of 205 language-specific corpora released with five major public datasets (CCAligned, ParaCrawl, WikiMatrix, OSCAR, mC4). Lower-resource corpora have systematic issues: At least 15 corpora have no usable text, and a significant fraction contains less than 50% sentences of acceptable quality. In addition, many are mislabeled or use nonstandard/ambiguous language codes. We demonstrate that these issues are easy to detect even for non-proficient speakers, and supplement the human audit with automatic analyses. Finally, we recommend techniques to evaluate and improve multilingual corpora and discuss potential risks that come with low-quality data releases.

Remote Article at MIT Press Presented at ACL 2022