The Anatomy of a Search Engine and AI: Startup vs Incumbent Value

The Anatomy of a Search Engine and AI: Startup vs Incumbent Value

In the world of technology, innovation and competition go hand in hand. Two areas that have seen significant advancements and fierce competition are search engines and artificial intelligence (AI). In this article, we will explore the anatomy of a search engine and the value distribution between startups and incumbents in the AI industry.

The Anatomy of a Search Engine

Search engines have come a long way since their inception. Back in 1994, the World Wide Web Worm (WWWW) had an index of 110,000 web pages and documents. Fast forward to November 1997, and search engines like Altavista claimed to handle roughly 20 million queries per day. The goal of search engines has always been to improve the quality of search results.

One of the main goals in designing search engines like Google was to create an environment where researchers could process large chunks of the web and produce interesting results. The citation graph of the web, which counts citations or backlinks to a page, has been used to determine a page's importance or quality. PageRank, a model of user behavior, takes into account the importance of links from different pages and normalizes it by the total number of links on a page.

Personalization is another important factor in search engine design. By adding a damping factor to a single page or group of pages, search engines can personalize search results and make it difficult to deliberately manipulate rankings. Currently, the predominant business model for search engines is advertising, which doesn't always align with providing quality search results to users.

AI: Startup vs Incumbent Value

When it comes to the value generated by AI, the distribution between startups and incumbents has been uneven. In the first internet wave, most of the value went to startups like Google, Amazon, and Facebook, with some captured by incumbents like Microsoft and Apple. The split was roughly 60:40 or 70:30 in favor of startups.

In the mobile wave, however, most of the value went to incumbents like Apple and Google, while startups like WhatsApp and Uber still managed to capture significant value. The split in this case was around 20:80 in favor of incumbents.

Crypto, on the other hand, has seen almost 100% startup capture of value, with very little participation from existing financial services or infrastructure companies. Bitcoin, Ethereum, and Coinbase are some examples of startups that have captured value in this space.