Skip to main navigation menu Skip to main content Skip to site footer

An End-to-End Contrastive Self-Supervised Learning Framework for Language Understanding


Self-supervised learning (SSL) methods such as Word2vec, BERT, and GPT have shown great effectiveness in language understanding. Contrastive learning, as a recent SSL approach, has attracted increasing attention in NLP. Contrastive learning learns data representations by predicting whether two augmented data instances are generated from the same original data example. Previous contrastive learning methods perform data augmentation and contrastive learning separately. As a result, the augmented data may not be optimal for   contrastive learning. To address this problem, we propose a four-level optimization framework which performs data augmentation and contrastive learning end-to-end, to enable the augmented data to be tailored to the contrastive learning task. This framework consists of four learning stages, including training machine translation models for sentence augmentation, pretraining a text encoder using contrastive learning, finetuning a text classification model, and updating  weights of translation data by minimizing the validation loss of the classification model, which are performed in a unified way. Experiments on datasets in the GLUE benchmark and on datasets used in (Gururangan et al., 2020) demonstrate the effectiveness of our method.
Article at MIT Press Presented at EMNLP 2023