How to Build a Bi-Gram Character-Level Language Model

TL;DR

To build a bi-gram character-level language model, start by gathering a dataset of names and counting character pairs to establish probabilities. Use these probabilities to generate new, unique names that resemble names from the dataset. Implement model smoothing techniques to handle cases of zero probability for certain character combinations, and evaluate the model's quality using negative log likelihood.

Transcript

hi everyone hope you're well and next up what i'd like to do is i'd like to build out make more like micrograd before it make more is a repository that i have on my github webpage you can look at it but just like with micrograd i'm going to build it out step by step and i'm going to spell everything out so we're going to build it out slowly and tog... Read More

Key Insights

• 😃 Make More is a bi-gram character-level language model that generates new names based on training data.
• 0️⃣ Model smoothing is used to avoid issues with zero probabilities in the model.
• 🧑‍💻 The negative log likelihood is used to evaluate the quality of the model.

Questions & Answers

Q: How is the Make More model different from Micrograd?

The Make More model is focused on building a character-level language model, while Micrograd is a machine learning library.

Q: How does Make More generate new names?

Make More generates new names by training on a dataset of existing names and learning the patterns of name-like sequences of characters. It then uses this knowledge to generate unique names that sound similar to real names.

Q: What is the purpose of character-level language modeling?

Character-level language modeling aims to predict the next character in a sequence based on the previous characters. It can be used for various applications, such as generating text, speech recognition, and machine translation.

Q: What is model smoothing and why is it used?

Model smoothing is a technique used to prevent zero probabilities in the model. It involves adding a small count or probability to every possible outcome, ensuring that no probability is zero. This helps avoid issues such as infinite loss when evaluating the model.

Q: How is the quality of the model evaluated?

The quality of the model is evaluated using negative log likelihood, which measures how well the model predicts the next character in a sequence. A lower negative log likelihood indicates a better model.

Key Insights:

• Make More is a bi-gram character-level language model that generates new names based on training data.
• Model smoothing is used to avoid issues with zero probabilities in the model.
• The negative log likelihood is used to evaluate the quality of the model.
• Character-level language modeling is a powerful technique for generating text and predicting sequences.

Summary & Key Takeaways

• The content explores the process of building a bi-gram character-level language model using count-based techniques.

• The model is trained on a dataset of names and is used to generate new names that sound name-like but are unique.

• The probability distribution of the model is evaluated using negative log likelihood to assess its quality.

• Model smoothing is applied to address cases where the model assigns zero probability to certain word combinations.