Stanford XCS224U: NLU I Behavioral Eval of NLU Models, Pt 7: DynaSent and Conclusion I Spring 2023

TL;DR

This screencast provides an overview of the DynaSent dataset, its creation process, and its potential for adversarial training and testing in natural language understanding (NLU).

Transcript

hello everyone welcome back this is the seventh and final screencast in our series on Advanced behavioral evaluation for nlu in the previous screencast I introduced the idea of having adversarial train sets in the mix then I did that in the context of adversarial nli and I talked briefly about the dynabench platform for this screencast we're going ... Read More

Key Insights

• 🏷️ DynaSent is a substantial dataset with labeled sentences for sentiment analysis.
• 💼 Harvesting sentences with contradictory sentiment predictions provides interesting cases for analysis.
• ❓ Distributional training offers a more nuanced perspective on sentiment judgments.
• 🛀 Model 1 shows the importance of incorporating previous rounds of data for training.
• ✋ Human performance on the DynaSent dataset is high, indicating a high level of agreement and consistency.
• 💼 Sentiment analysis in complex cases requires understanding intricate syntactic structures and non-literal language usage.
• ❓ Adversarial training can improve system performance, but careful calibration is necessary.

Questions & Answers

Q: How was the DynaSent dataset created?

The DynaSent dataset is created by harvesting sentences from the Yelp open dataset, favoring sentences where a model's sentiment prediction contradicts the actual sentiment of a one-star or five-star review. These sentences are then human-validated to ensure quality.

Q: What is the difference between majority label training and distributional training?

Majority label training infers the label for an example based on the majority choice of five crowd workers. In distributional training, each example is repeated five times with each of the labels given by the crowd workers, providing a more nuanced perspective on sentiment judgments.

Q: How does Model 0 perform on external sentiment benchmarks?

Model 0 performs reasonably well on external sentiment benchmarks such as SST3, Yelp, and Amazon, although the results are not exceptional. It serves as a means to find interesting cases for analysis but may not achieve high accuracy in sentiment classification.

Q: How does Model 1 perform on the DynaSent dataset?

Model 1, which is trained on external sentiment benchmarks as well as the round one data from DynaSent, achieves around 80% accuracy on the round one data, with a slight drop in performance on external benchmarks. It shows the importance of focusing on round one data for training.

Summary & Key Takeaways

• The DynaSent dataset is a substantial resource with over 120,000 sentences across two rounds, each labeled with five gold labels from crowd workers.

• The dataset is created using a heuristic that favors sentences where a model's prediction contradicts the actual sentiment of a review, leading to potentially interesting cases for further analysis.

• Two approaches for training on the dataset are suggested: majority label training and distributional training, with distributional training providing a more nuanced and robust perspective on sentiment judgments.