Gao 2021 - Simple Contrastive Learning of Sentence Embeddings

Paper Link

This paper proposes an unsupervised and supervised approach to fine-tune sentence encoder models to perform semantic textual similarity tasks. The STS tasks refer to a set of tasks taken from SemEval 2012 to 2017 where given two sentences, the model is to output a score between 1-5 and scored based on correlation to human inputted scores.

Task Dataset

Example sentence pair with a score of 4 (taken from SemEval 2016):

• In May 2010, the troops attempted to invade Kabul.
• The US army invaded Kabul on May 7th last year, 2010

Example sentence pair with a score of 3:

• John said he is considered a witness but not a suspect.
• "He is not a suspect anymore." John said.

Unsupervised SimCSE

SimCSE follows the popular framework of contrastive learning using in-batch negatives, where pairs of related sentences are trained to have high similarity whilst having low similarity with other random sentences.

The idea of unsupervised SimCSE is simple: given a collection of sentences $\{x_i{\}}_{i=1}^m$, treat each sentence itself as its own positive pair, and use dropout noise to introduce random perturbations such that the self similarity is not trivially perfect.

Specifically, if we denote $h_i^z$ as the embedding for $x_i$ under dropout mask $z$, then the loss for unsupervised SimCSE for a mini-batch of $N$ sentences is:

$$l_1=-log\frac{e^{sim(h_i^{z_i},h_i^{z_i^{\prime }})/\tau }}{{\sum }_{j=1}^N{e}^{sim(h_j^{z_j},h_j^{z_j^{\prime }})/\tau }}$$

Note that $\tau$ is the temperature hyperparameter. Importantly, the authors found that setting $\tau =1$ with cosine similarity performs very poorly (64.0), compared to dot product (85.9). However, carefully tuning $\tau$ can lead to similar performance ($\tau =0.05$ had a score of 86.2).

We may view this procedure as data augmentation, analogous to how random pixel distortions and rotations are applied to images to improve computer vision models. The paper shows that this simple unsupervised method significantly outperforms other data augmentation methods. Note that the authors used the default 10% dropout for BERT models.

Supervised SimCSE

The supervised version follows a similar framework, although the positive pairs are taken from an external dataset. They chose the Natural Language Inference (NLI) datasets, where each example is a triplet of sentences. The premise is denoted $x_i$, the entailment sentence is denoted $x_i^{+}$ and the contradiction sentence is denoted $x_i^{-}$. The loss is then formulated as:

$$l_2=-log\frac{e^{sim(h_i,h_i^{+})/\tau }}{{\sum }_{j=1}^N{e}^{sim(h_i,h_j^{+})/\tau }+e^{sim(h_i,h_j^{-})/\tau }}$$

Ablation Studies

• The paper finds that including contradiction sentences as hard negatives has a small but significant improvement in performance
• The paper finds that using the [CLS] token or averaging embeddings across the first and last layer does not make much difference

Alignment and Uniformity

Wang and Isola 2020 propose two metrics for measuring the effectiveness of an embedding method on a set of documents:

• Alignment. Given a distribution of positive pairs of documents $p_{pos}$, alignment desires the expected distance between embeddings of each pair to be small:

$$l_{align}={\mathbf{E}}_{(x,x^{+})\sim p_{pos}}||f(x)-f(x^{+})|{|}^2$$

• Uniformity. Given any two documents drawn from the corpus, uniformity metric should be small (i.e. distance between them is large).

$$l_{uniform}=log{\mathbf{E}}_{(x,y)\sim p_{data}}{e}^{-2||f(x)-f(y)|{|}^2}$$

A common problem pointed out in training language models is anisotropy, in which embeddings are pushed into a narrow cone in the vector space, which severely limits their expressiveness. The anisotropy problem is naturally connected to uniformity, which aims at distributing embeddings evenly in the space. The authors argue that contrastive learning as proposed in this paper addresses this problem through some analysis (omitted for now).

Empirically, they show that the alignment metric for SimCSE is comparable to average BERT, but the uniformity measure is significantly lower, leading to much better performance in terms of accuracy.