Stanford XCS224U: NLU I In-context Learning, Part 2: Core Concepts I Spring 2023

TL;DR

In-context learning involves frozen language models that perform tasks based on prompt text, with few-shot and zero-shot variations. Auto-regressive language models predict scores over the entire vocabulary to determine the next token.

Transcript

welcome back everyone this is part two in our series on in context learning I thought I'd cover some Core Concepts for the most part I think these concepts are a review for you all but I thought it would be good to kind of get them into our common ground to help us think about them as we think about what's happening with in-context learning techniq... Read More

Key Insights

• ❓ In-context learning techniques involve using frozen language models without gradient updates.
• ⚾ The distinction between few-shot and zero-shot in-context learning is based on the presence or absence of examples of the intended behavior in the prompts.
• 😫 Auto-regressive language models predict score vectors, not tokens, and the generation process is determined by rules set by humans.
• ✋ The behavior of language models is primarily driven by high probability continuations based on training experience, rather than explicit world knowledge.
• ⚾ Instruction fine-tuning involves training models on human-curated prompts and ranking model outputs based on quality, involving human supervision.

Questions & Answers

Q: What is in-context learning?

In-context learning refers to frozen language models performing tasks by conditioning on prompt text without gradient updates.

Q: How is few-shot in-context learning different from traditional few-shot learning?

Few-shot in-context learning involves using prompts with examples of the intended behavior, while traditional few-shot learning uses gradient updates on a few examples.

Q: What is zero-shot in-context learning?

Zero-shot in-context learning uses prompts without examples of the intended behavior, but can include instructions or formatting. It is uncertain whether this can be verified in practice.

Q: How do auto-regressive language models work?

Auto-regressive language models predict score vectors over the vocabulary at each time step, using context from previous steps to predict the next token.

Summary & Key Takeaways

• In-context learning is when a frozen language model performs a task based only on prompt text, without gradient updates.

• Few-shot in-context learning uses prompts with examples of the intended behavior, without seeing those examples during training.

• Zero-shot in-context learning uses prompts with no examples of the intended behavior, potentially including formatting and instructions.