"The Hunter Economy: How to Fine-Tune Your Embeddings for Better Similarity Search"

"The Hunter Economy: How to Fine-Tune Your Embeddings for Better Similarity Search"

In today's digital age, we are witnessing a shift in the way we consume and discover content. With the rise of the internet and social media, there is an abundance of information and options available to us. This has led to the emergence of what can be called "The Hunter Economy." In this economy, individuals are not just consumers but also curators, tastemakers, and early adopters.

One aspect of The Hunter Economy is curation. With so many people creating music, for example, there is a need for someone to curate and pick out the best songs to be heard. This applies to various industries, from music to technology. Platforms like Product Hunt and Gartner have capitalized on this need and become trusted sources for discovering new products and trends.

Another aspect of The Hunter Economy is positional scarcity. As prosperity and abundance increase, it becomes harder for individuals to preserve and distinguish high status. For example, with the widespread availability of cars, how can one ensure that their car stands out and is seen as the most impressive? Companies like Porsche and Tesla have mastered the art of creating prestige and maintaining high status in a crowded market.

Access is another key factor in The Hunter Economy. With an abundance of people vying for attention and patronage, there is a premium placed on being at the front of the line. This can be seen in various aspects of life, from skipping traffic congestion to gaining access to exclusive events. The ability to skip the line and get to where you need to go quickly is highly valued.

So, what exactly is The Hunter Economy? It is a phenomenon where individuals gain status as hunters and curators. This status comes from their ability to discover and support trends, people, businesses, and ideas early on. It is about being the first to identify and appreciate something before it becomes mainstream. This early discovery and validation carry social and financial capital, making it highly desirable.

Angel investing is one example of how The Hunter Economy operates. Angel investors not only imply wealth but also taste and access. Their ability to curate and identify promising startups gives them social capital, which can be more valuable than financial capital. The joy of being the first to discover and invest in a promising company is a driving force behind the Hunter Economy.

In the future, we can expect a whole class of startups to be built around The Hunter Economy. These startups will incentivize and reward early adopters economically and socially. Imagine a world where everything you subscribe to lists your place in line, allowing you to demonstrate how early you were in discovering something. This concept can be taken even further by betting on abstract terms or ideas, such as making a seed bet on a person or concept becoming more popular over time.

Now, let's shift our focus to another topic that complements The Hunter Economy: fine-tuning embeddings for better similarity search. Embeddings are representations of data in a lower-dimensional space that capture its underlying structure. By leveraging embeddings, we can enhance the labeling process and improve similarity search.

Similarity search involves selecting a record and finding similar records based on the cosine similarity of their embeddings. By fine-tuning our embeddings, we aim to increase the number of records of the same class within a similarity labeling session. This can be highly beneficial in various domains, including the Kern AI refinery, where labeling plays a crucial role.

Large language models (LLM) are powerful tools that can solve a wide range of tasks. Their effectiveness stems from their architecture, training procedure, and access to vast amounts of training data from the internet. However, LLMs may lack domain-specific expertise due to their generalization across multiple domains. Fine-tuning fills this gap by adjusting the language model to better fit the domain of the data.

Before fine-tuning, it is essential to explore existing fine-tuned models in the Hugging Face model database. This saves time and effort by utilizing pre-trained models that are already fine-tuned on similar data. Additionally, fine-tuning requires a task to solve, such as supervised classification or unsupervised masked token prediction. In the case of similarity learning, class labels define the similarity between records.

To illustrate the process of fine-tuning embeddings, we conducted an experiment using Kern refinery. We selected 20,000 records and manually labeled 261 of them, filtering for a confidence score above 0.7. This resulted in 10,854 usable records for the fine-tuning pipeline. We utilized SimilarityGroupSamples, where class information is the only similarity measure available.

The fine-tuning pipeline involved using a pre-trained LLM as an encoder and adding a SkipConnectionHead on top of it. The goal was to learn a mapping from one embedding to another, focusing on increasing the number of records of the same class in the top 1,000 most similar records. The experiment showed that even with as few as 25 labeled records, the benefits of fine-tuning were evident.

Fine-tuned embeddings not only improve similarity search but also have the potential to benefit classifiers trained on the same data. Traditional methods like basic PCA may not effectively separate embeddings in a two-dimensional space, making annotation processes challenging. Ongoing efforts are focused on developing methods to fine-tune embeddings for better class separation in a 2D space.

In conclusion, The Hunter Economy and fine-tuning embeddings for similarity search are two interconnected concepts that highlight the changing dynamics of our digital world. The desire to be early adopters, curators, and tastemakers drives the Hunter Economy, while fine-tuning embeddings enhances the labeling process and improves similarity search.

For those looking to thrive in The Hunter Economy and optimize their similarity search, here are three actionable pieces of advice:

1. Embrace your role as a hunter and curator: Be proactive in discovering and supporting trends, people, businesses, and ideas early on. This will not only give you social capital but also open doors to new opportunities.

2. Stay updated on existing pre-trained models: Before embarking on fine-tuning, explore available pre-trained models that are already fine-tuned on similar data. This can save time and effort while still achieving excellent results.

3. Experiment with different similarity learning techniques: Fine-tuning embeddings require a task to solve, and there are various techniques available. Explore different approaches, such as using similarity scores, pre-formed triplets, or similarity groups, to find the one that best suits your data and objectives.

By embracing the principles of The Hunter Economy and harnessing the power of fine-tuned embeddings, individuals and organizations can navigate the digital landscape with confidence and make impactful discoveries. The future belongs to those who can curate, predict, and adapt to the ever-changing trends and demands of the modern world.