How Does Deep Learning Represent Language?

TL;DR

Natural language must be converted from discrete symbols into vectors before neural networks can process it. One-hot vectors support simple retrieval and classification, but they encode no semantic similarity, while distributional methods build richer representations from the words that occur around each target word and can support language modeling, translation, understanding, and classification.

Transcript

I'll get started so just some housekeeping before I start and this is just a sanity check so I'm told it's a two-hour lecture so we're done by 4:00 is that correct and you're it's probably expecting a break of like 10 minutes or so about halfway through okay so we'll try and make that happen after at some point tonight the slides will be put on wha... Read More

Key Insights

• Language is part of the human cognitive toolkit because people use it to formulate abstractions, communicate ideas, talk internally, and make plans. Solving language is therefore presented as an important objective for developing artificial intelligence with capabilities resembling higher forms of human intelligence.
• Text is a sequence of discrete symbols such as words, characters, or ideograms. Neural networks expect vectors as inputs, so natural language processing systems need a method for mapping these discrete textual units into numerical representations before models can process them.
• A one-hot word representation is a vocabulary-sized vector with one value set to one at the token's assigned index and every other value set to zero. This simple representation is sparse and treats every distinct word as mathematically orthogonal to every other word.
• Classical information retrieval can represent documents by summing the one-hot vectors of their words. A query can be treated as a small document, compared with document vectors using an inner product or cosine distance, and used to return the documents with the highest scores.
• One-hot vectors contain no direct information about semantic similarity or relationships. Words such as cat and kitten refer to related concepts for humans, yet their one-hot representations remain orthogonal, which motivates the construction of richer vector representations that express meaningful similarities.
• Distributional semantics is based on understanding a word through the company it keeps. The surrounding words provide information about the target word's syntactic and semantic roles, allowing representations to reflect usage patterns rather than relying only on separate token identities.
• Count-based distributional methods require a context vocabulary and a word-window size. The context vocabulary contains words expected to be informative, while the window determines how many words to the left and right of each target occurrence contribute to its representation.
• A count-based word vector records how often each selected context word occurs near a target word. Iterating through target occurrences, collecting neighboring words, and placing their counts at fixed context indices produces dense representations based on observed linguistic context.

Questions & Answers

Q: How are words represented for neural networks?

Words can be mapped to vectors by assigning every unique token an index in a vocabulary. In the simplest approach, each token becomes a vocabulary-sized one-hot vector with a one at its assigned position and zeros everywhere else. This converts discrete language symbols into numerical inputs that neural networks can accept, although the resulting vectors are sparse and do not express semantic similarity.

Q: What is a one-hot word vector in natural language processing?

A one-hot word vector is a vector whose length equals the vocabulary size. Each word receives a unique index, and its vector contains a one at that index with zeros in all other positions. The representation is general and can work in simple systems, but distinct words are mathematically orthogonal, so their vectors contain no information about semantic resemblance or relationships.

Q: Why are one-hot vectors limited for representing word meaning?

One-hot vectors identify tokens without encoding how their meanings relate. Every different word occupies a separate orthogonal direction, even when two words refer to similar concepts. Cat and kitten, for example, are understood by humans as related, but their one-hot vectors show no similarity. This limitation motivates richer representations derived from contexts, usage patterns, or learning tasks.

Q: How does classical information retrieval use word vectors?

Classical information retrieval can represent a document by summing the one-hot vectors for every word it contains, producing a vocabulary-sized vector containing word counts. A query is treated as a small document and represented similarly. The system compares the query with document vectors using a similarity measure, such as an inner product or cosine distance, and returns the highest-scoring documents.

Q: What is distributional semantics in natural language processing?

Distributional semantics is the idea that a word can be understood by examining the words that occur around it. Nearby words reveal information about the target word's semantic and syntactic roles. Instead of treating vocabulary items as unrelated identifiers, distributional approaches construct representations from observed usage. The lecture groups these approaches into count-based, predictive, and task-based or end-to-end methods.

Q: How are count-based word vectors constructed?

A count-based method first defines a basis vocabulary of context words and chooses a window size. It then visits occurrences of each target word and counts how often every selected context word appears within the specified number of positions to the left or right. Those counts are stored at fixed indices, producing a vector that summarizes the target word's observed contexts.

Q: Why does context vocabulary selection matter for word vectors?

The context vocabulary determines which neighboring words can contribute dimensions to a count-based representation. It may include all vocabulary items when memory permits, but the lecture describes selecting salient context words, such as a set of frequent words, and possibly excluding common stop words that provide limited information. This choice affects what usage patterns and distinctions the resulting vectors can capture.

Q: What does the window size control in distributional word representations?

The window size specifies how many words on each side of a target occurrence are treated as its context. A method might collect several words to the left and right, then count selected vocabulary items within that region. Because the window defines which neighboring words contribute evidence, its size is a meaningful hyperparameter that influences the representations produced from the corpus.

Summary & Key Takeaways

• Language is a fundamental part of human intelligence because it supports abstraction, communication, planning, and internal thought. Deep learning for natural language processing therefore addresses more than a narrow application. The lecture introduces word representations, language modeling, conditional language modeling, translation, sequence-to-sequence models with attention, language understanding, composition, and classification.

• Text consists of sequences of discrete symbols, including words, characters, and ideograms, while neural networks expect vector inputs. A basic solution assigns each unique token an index and represents it with a one-hot vector whose corresponding position is one and whose remaining positions are zero. This creates sparse, mutually orthogonal word representations.

• Distributional semantics seeks richer representations by examining how words are used and which words appear around them. Count-based methods select a context vocabulary and window size, count nearby context words for every target word, and place those counts into vectors. Predictive and task-based approaches provide additional ways to learn representations from language data.