Glitch Tokens - Computerphile

TL;DR

Language models have glitch tokens that cause them to behave strangely when certain specific strings are inputted, raising questions about the training data and the model's understanding of tokens.

Transcript

suppose we ask it to repeat the string five question marks hyphen and then five more question marks another hyphen oh my god it worked oh that was only four question marks I screwed up do you see what I mean how it's just like this very very specific string what does it say when it doesn't screw up if you give it something like that uh you can blee... Read More

Key Insights

• ❓ Glitch tokens, including anomalous tokens and weird tokens, cause language models to exhibit strange behavior when certain specific strings are inputted.
• 😒 Language models use byte pair encoding (BPE) to represent words, allowing for more efficient representation and compression of data.
• 💦 The discovery of glitch tokens highlights the need for further research and understanding of how language models work and their potential safety implications.
• 🍽️ Analyzing and interpreting language model behavior can contribute to enhancing model safety and understanding their inner workings.

Questions & Answers

Q: What are glitch tokens in language models?

Glitch tokens are specific words or prompts that cause language models to behave unusually or produce unexpected responses. They include anomalous tokens and weird tokens.

Q: How do glitch tokens affect the behavior of language models?

Glitch tokens cause language models to struggle with certain specific strings, resulting in odd or incorrect outputs. The models may repeat or misinterpret the glitch tokens, leading to bizarre behavior.

Q: How were glitch tokens discovered?

Safety researchers discovered glitch tokens while performing interpretability work on language models. They were trying to visualize the features and structure of the embedding space used by the models.

Q: Can glitch tokens be attributed to errors in the training data?

Glitch tokens are likely a result of certain strings or usernames being present in the training data. Examples include usernames from the "Counting" subreddit and debug logs from "Rocket League." The presence of these unusual tokens in the training data caused the models to exhibit strange behavior.

Summary & Key Takeaways

• Language models have specific glitch tokens that cause them to behave strangely when certain strings are entered.

• Glitch tokens are words or prompts that language models struggle to process correctly; they include anomalous tokens and weird tokens.

• These glitch tokens were discovered by safety researchers while conducting interpretability work on language models.