Deep Learning 6: How Does Deep Learning for NLP Represent Language?

TL;DR

Deep learning for NLP represents language by converting discrete symbols, such as words, characters, or ideograms, into vectors that neural networks can accept. A basic method assigns each unique token an index and creates a vocabulary-sized one-hot vector, while document retrieval can sum these vectors and rank results using an inner product or cosine distance. Read on to understand the method, its uses, and its limitations.

Transcript

I'll get started so just some housekeeping before I start and this is just a sanity check so I'm told it's a two-hour lecture so we're done by 4:00 is that correct and you're it's probably expecting a break of like 10 minutes or so about halfway through okay so we'll try and make that happen after at some point tonight the slides will be put on wha... Read More

Key Insights

• Language is part of the human cognitive toolkit because people use it to formulate abstractions, communicate ideas, talk internally, and make plans. Solving language is therefore presented as an important objective for developing artificial intelligence with capabilities resembling higher forms of human intelligence.
• Text is a sequence of discrete symbols such as words, characters, or ideograms. Neural networks expect vectors as inputs, so natural language processing systems need a method for mapping these discrete textual units into numerical representations before models can process them.
• A one-hot word representation is a vocabulary-sized vector with one value set to one at the token's assigned index and every other value set to zero. This simple representation is sparse and treats every distinct word as mathematically orthogonal to every other word.
• Classical information retrieval can represent documents by summing the one-hot vectors of their words. A query can be treated as a small document, compared with document vectors using an inner product or cosine distance, and used to return the documents with the highest scores.
• One-hot vectors contain no direct information about semantic similarity or relationships. Words such as cat and kitten refer to related concepts for humans, yet their one-hot representations remain orthogonal, which motivates the construction of richer vector representations that express meaningful similarities.
• Distributional semantics is based on understanding a word through the company it keeps. The surrounding words provide information about the target word's syntactic and semantic roles, allowing representations to reflect usage patterns rather than relying only on separate token identities.
• Count-based distributional methods require a context vocabulary and a word-window size. The context vocabulary contains words expected to be informative, while the window determines how many words to the left and right of each target occurrence contribute to its representation.
• A count-based word vector records how often each selected context word occurs near a target word. Iterating through target occurrences, collecting neighboring words, and placing their counts at fixed context indices produces dense representations based on observed linguistic context.
• "text is a sequence of discrete symbols so words or characters or ideograms in other languages" (4:40)
• "especially in your networks which expect vectors as input" (4:53)
• "the document and the query vectors are just going to be super positions of word vectors" (5:44)
• "you simply sum all the vectors of all the words in the webpage" (5:55)
• "you return the documents or the K documents that give you the highest score" (6:14)

Questions & Answers

Q: How does deep learning represent language for NLP?

It represents text as vectors because neural networks expect vectors as input. Since text is a sequence of discrete symbols, words, characters, or ideograms must first be mapped into numerical representations.

Q: What is a one-hot word vector?

A one-hot word vector has the same size as the vocabulary. Each unique token receives an index, with a one at that index and zeros everywhere else.

Q: Why do neural networks need word vectors?

Neural networks expect vectors as input, but natural language text consists of discrete symbols. Word vectors provide a representation that allows text to be fed into these models.

Q: Can one-hot vectors work in practical systems?

Yes, the lecture says this apparently naive representation is fairly successful in very simple systems. Classical information retrieval uses this kind of representation for document retrieval.

Q: How are documents represented in classical information retrieval?

A document is represented by summing the vectors of all its words. The result is one vocabulary-sized vector containing the counts of words that occur in the document.

Q: How is a search query represented for document retrieval?

The query is treated as a small document and represented as a superposition of word vectors. That query vector is then executed against the retrieval system.

Q: How does classical information retrieval rank documents?

It compares query and document vectors with a similarity metric, such as an inner product or cosine distance. The system returns the document or K documents with the highest score.

Q: What topics does Deep Learning 6 cover for NLP?

The lecture covers word representations, language modeling, conditional language modeling, and translation models. It also introduces sequence-to-sequence models with attention, language processing and understanding applications, composition, and classification.

Summary & Key Takeaways

• Language is a fundamental part of human intelligence because it supports abstraction, communication, planning, and internal thought. Deep learning for natural language processing therefore addresses more than a narrow application. The lecture introduces word representations, language modeling, conditional language modeling, translation, sequence-to-sequence models with attention, language understanding, composition, and classification.

• Text consists of sequences of discrete symbols, including words, characters, and ideograms, while neural networks expect vector inputs. A basic solution assigns each unique token an index and represents it with a one-hot vector whose corresponding position is one and whose remaining positions are zero. This creates sparse, mutually orthogonal word representations.

• Distributional semantics seeks richer representations by examining how words are used and which words appear around them. Count-based methods select a context vocabulary and window size, count nearby context words for every target word, and place those counts into vectors. Predictive and task-based approaches provide additional ways to learn representations from language data.