Neural Text Degeneration

• The Curious Case of Neural Text Degeneration
• deep analysis into the properties of the most common decoding methods for open-ended language generation
• surprising distributional differences between human text and machine text
• decoding strategies alone can dramatically effect the quality of machine text, even when generated from exactly the same neural language model
• likelihood maximizing decoding causes repetition and overly generic language usage
• sampling methods without truncation risk sampling from the low-confidence tail of a model’s predicted distribution
• Nucleus Sampling