The Anatomy of a Search Engine and AI: Startup vs Incumbent Value

The Anatomy of a Search Engine and AI: Startup vs Incumbent Value

In the world of technology, innovation and competition go hand in hand. Two areas that have seen significant advancements and fierce competition are search engines and artificial intelligence (AI). In this article, we will explore the anatomy of a search engine and the value distribution between startups and incumbents in the AI industry.

The Anatomy of a Search Engine

Search engines have come a long way since their inception. Back in 1994, the World Wide Web Worm (WWWW) had an index of 110,000 web pages and documents. Fast forward to November 1997, and search engines like Altavista claimed to handle roughly 20 million queries per day. The goal of search engines has always been to improve the quality of search results.

One of the main goals in designing search engines like Google was to create an environment where researchers could process large chunks of the web and produce interesting results. The citation graph of the web, which counts citations or backlinks to a page, has been used to determine a page's importance or quality. PageRank, a model of user behavior, takes into account the importance of links from different pages and normalizes it by the total number of links on a page.

Personalization is another important factor in search engine design. By adding a damping factor to a single page or group of pages, search engines can personalize search results and make it difficult to deliberately manipulate rankings. Currently, the predominant business model for search engines is advertising, which doesn't always align with providing quality search results to users.

AI: Startup vs Incumbent Value

When it comes to the value generated by AI, the distribution between startups and incumbents has been uneven. In the first internet wave, most of the value went to startups like Google, Amazon, and Facebook, with some captured by incumbents like Microsoft and Apple. The split was roughly 60:40 or 70:30 in favor of startups.

In the mobile wave, however, most of the value went to incumbents like Apple and Google, while startups like WhatsApp and Uber still managed to capture significant value. The split in this case was around 20:80 in favor of incumbents.

Crypto, on the other hand, has seen almost 100% startup capture of value, with very little participation from existing financial services or infrastructure companies. Bitcoin, Ethereum, and Coinbase are some examples of startups that have captured value in this space.

To beat an incumbent as a startup in the AI industry, you either need to build a product that is dramatically better or focus on a new customer segment or distribution moat that the incumbent cannot serve. A 10X better product is often required to overcome the distribution, capital, and pre-existing product moats of incumbents.

In the past, incumbents may have had an advantage due to their data advantage. However, as companies now use the broader internet as an initial training set and switch to models that work robustly with smaller data sets, this advantage may be diminishing.

What sets this current wave of AI apart is the speed of innovation across multiple areas. The technology seems dramatically stronger, making it easier to create products that are 10X better than what incumbents offer. While GPT-3, a widely known AI model, has not yet sparked a wave of startups building big businesses on it, a 5-10X better model could create a whole new startup ecosystem while augmenting incumbent products.

Unlike previous AI waves, there are now infrastructure-centric companies with broad adoption and rapidly growing usage. OpenAI, Stability.AI, Hugging Face, and Weights and Biases are some examples of such companies. Additionally, there are highly repetitive, highly paid tasks that lack workflow tools, making AI features a valuable addition to broader workflow tools. Summarization and generation of text or images have become possible at a high fidelity level, opening up new opportunities.

However, it is crucial to avoid the "hammer-looking-for-a-nail" problem. It is important to identify actual end user needs and unserved markets that will benefit from the advancements in AI technology. By focusing on the needs of end users, startups can leverage the exciting technology to create real value.

Actionable Advice:

• 1. Focus on building a product that is at least 10X better than what incumbents offer. This will help overcome the distribution, capital, and pre-existing product moats of incumbents.

• 2. Identify and target new customer segments or distribution moats that incumbents cannot serve. This will give startups a unique advantage and increase their chances of success.

• 3. Always prioritize the needs of the end users. By understanding and addressing their needs, startups can leverage AI technology to create valuable solutions.

In conclusion, the anatomy of a search engine and the value distribution in the AI industry provide valuable insights into the world of technology and innovation. By understanding the challenges and opportunities in these areas, startups can navigate the competitive landscape and make a significant impact in their respective industries. Exciting times lie ahead as technology continues to evolve and startups tap into the potential of AI.