What Are Embeddings and How Do They Work?

TL;DR

An embedding is a vector whose numerical positions encode semantic meaning, allowing text and other concepts to be compared mathematically. In the demonstrated Python workflow, GPT-3 converts an input string into an embedding, NumPy calculates dot products, and the resulting similarity scores can be used to match an input such as a life form against categories like plant, reptile, mammal, and fish.

Transcript

morning everybody david shapiro here for my third video in the zero to python and gpt3 boot camp um what the heck are embeddings i get this question all the time it is by far the biggest hottest topic so this is why i'm doing it as episode three but before we get started i'm going to ask that uh you consider liking and subscribing this video and al... Read More

Key Insights

• A vector is an array of numbers that can also be described mathematically as a one-dimensional matrix or, in Python-oriented language, a list. This numerical structure is the basic form used to introduce embeddings and the later similarity comparisons.
• An embedding is a vector with semantic meaning. Although vectors and embeddings have the same mathematical form, an embedding uses its numerical positions to represent aspects of meaning, allowing concepts expressed as text to be processed and compared numerically.
• A simplified embedding can assign interpretable meanings to individual dimensions. The example uses one coordinate for social power and another for gender, with values between negative 1.0 and 1.0, to illustrate how different people or roles can occupy different semantic positions.
• GPT-3 embeddings distribute semantic meaning across many dimensions. The transcript identifies Ada as having 1,024 dimensions and Davinci as having 12,000 dimensions, making their representations far more detailed and nuanced than the two-dimensional teaching example.
• The Ada embedding of a phrase is a list of 1,024 floating-point numbers. Passing “bald eagle” to the text-similarity Ada engine demonstrates that a familiar phrase can be converted quickly into a large numerical representation for later comparison.
• Vector similarity is calculated in the example with a dot product implemented through NumPy. The demonstrated rule is that a higher dot-product score indicates greater similarity, while substantially different vectors can produce lower or negative scores.
• A basic classification system can compare an input embedding with embeddings for predefined categories. The example accepts the name of a life form and prepares to match it against four labels: plant, reptile, mammal, and fish.
• Category embeddings should be stored instead of regenerated during every classification request. The transcript describes repeatedly requesting embeddings inside the loop as inefficient and proposes saving the category representations in a later or more developed implementation.

Questions & Answers

Q: What is an embedding in machine learning?

An embedding is a vector with semantic meaning. It consists of an array of numbers, just like an ordinary vector, but its numerical positions collectively represent characteristics of a word, sentence, person, object, or other concept. This representation makes it possible to compare meanings mathematically instead of relying only on older structures such as explicit word webs and predefined relationships.

Q: What is the difference between a vector and an embedding?

A vector is any array of numbers and can be described as a one-dimensional matrix or list. An embedding is mathematically the same kind of structure, but its numbers carry semantic meaning. The distinction is therefore not the numerical container itself. It is the use of that container to encode meaningful qualities that can support comparison, search, clustering, or classification.

Q: How can embedding dimensions represent semantic meaning?

Each position in a simplified embedding can be treated as representing a meaningful feature. The transcript demonstrates this with two positions: one for social power and one for gender. Values between negative 1.0 and 1.0 place example people at different coordinates. Real GPT-3 embeddings use many more dimensions, and those positions are described as more abstract than the hand-designed example.

Q: How many dimensions do GPT-3 embeddings have?

The transcript states that the smallest GPT-3 embedding model discussed, Ada, has 1,024 dimensions, while Davinci has 12,000 dimensions. A two-dimensional vector is used only as a teaching example. Increasing the number of dimensions allows the model to break semantic meaning into a much larger and more nuanced numerical representation than the manually defined social-power and gender coordinates.

Q: How do you compare two embeddings in Python?

The demonstrated Python method compares two embeddings by calculating their dot product with NumPy. A small similarity function accepts two vectors and returns that value. According to the explanation, a higher dot product means the vectors are more similar. The example produces a score of 1.5 for two emperor-related vectors and negative 1.0 for a contrasting comparison.

Q: What does a GPT-3 embedding look like?

A GPT-3 embedding appears as a long list of floating-point numbers. In the example, the phrase “bald eagle” is passed to the text-similarity Ada engine, which returns 1,024 values. The numbers fall between negative 1.0 and 1.0 and collectively form the semantic embedding that can later be compared with other text embeddings using a mathematical similarity calculation.

Q: How can embeddings classify a life form?

A classification workflow can first convert an entered life form into an embedding and then compare that vector with embeddings for predefined labels. The demonstrated labels are plant, reptile, mammal, and fish. A matching function receives the input vector and category list, generates vectors for the categories, and uses similarity comparisons to determine which category is the closest semantic match.

Q: Why should category embeddings be stored?

Category embeddings should be stored because repeatedly requesting the same vectors during every classification run is described as inefficient. The introductory implementation generates an embedding for each category inside a loop to keep the example straightforward. A longer-term version would calculate the plant, reptile, mammal, and fish embeddings once, save them, and reuse those numerical representations for later comparisons.

Summary & Key Takeaways

• A vector is introduced as an array of numbers, also described as a one-dimensional matrix or list. An embedding has the same mathematical form but carries semantic meaning. A simplified two-dimensional example assigns one position to social power and another to gender, showing how numerical coordinates can represent different people or concepts.

• GPT-3 embedding models represent meaning across many dimensions. The transcript states that Ada produces embeddings with 1,024 dimensions, while Davinci has 12,000 dimensions. When the Ada similarity engine receives a string such as “bald eagle,” it returns a long list of floating-point values that numerically represents the phrase’s semantic meaning.

• Similarity can be calculated by taking the dot product of two vectors, with a higher result indicating greater similarity in the demonstrated method. The Python example uses NumPy for this calculation, then begins building a classifier that embeds an entered life form and compares it with plant, reptile, mammal, and fish category embeddings.