9.3. Language Models — Dive into Deep Learning 1.0.0-beta0 documentation

d2l.ai

The probability formulae that involve one, two, and three variables are typically referred to as unigram, bigram, and trigram models, respectively. In order to compute the language model, we need to calculate the probability of words and the conditional probability of a word given the previous few w

1 Users

0 Comments

4 Highlights

0 Notes

- The probability formulae that involve one, two, and three variables are typically referred to as unigram, bigram, and trigram models, respectively. In order to compute the language model, we need to calculate the probability of words and the conditional probability of a word given the previous few words.
- A common strategy is to perform some form of Laplace smoothing. The solution is to add a small constant to all counts.
- perplexity
- Suppose that the dataset takes the form of a sequence of � token indices in corpus. We will partition it into subsequences, where each subsequence has � tokens (time steps). To iterate over (almost) all the tokens of the entire dataset for each epoch and obtain all possible length- � subsequences, we can introduce randomness. More concretely, ...

Glasp is a social web highlighter that people can highlight and organize quotes and thoughts from the web, and access other like-minded people’s learning.