MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers
Enormous language models lead to breakthroughs and new tasks, such as open-ended text-gen.
Open-ended text-generation task
Open-ended text-gen (OETG) has a problem: how to choose between equally valid alternatives?
Current solution: human evaluation ➡️ expensive & unreliable
Type 1 is the obvious error, but Type 2 also v important.
KL & reverse KL respectively are natural metrics for these
However, they can also be infinite ⚠️
Mauve Method
Solution: calculate KL with target as weighted mixture of Q & P. This gives the divergence curve.
The area under this curve is Mauve.
If MAUVE = 1 → P & Q are identical; 0 → completely dissimilar
Computing Mauve in practice
We require probabilities of the human documents we compare to. This is generally intractable. The solution has two steps:
So the deep embedding gives a continuous probability distribution (in embedding space). However, this is still quite high-dim to evaluate a probability distribution over. The quantisation method reduces this, making it tractable.
Note that interpretation of actual Mauve score is dependent on the embedding model (e.g. RoBERTa, or GPT2), but the comparison between scores (i.e. rank these three models) tend to be similar across embeddings.
Evaluation
MAUVE correlates strongly with human judgements
Also correlates with improvements coming from e.g. model size
That's it! Super simple. Why hasn't anyone thought of this before??
Questions
- What is nucleus sampling?
- How exactly do we get these distributions? How can both GPT2 and RoBERTa be used?