The Anatomy of a Search Engine: Do Things that Don't Scale

The Anatomy of a Search Engine: Do Things that Don't Scale

In the early days of web search engines, the number of indexed web pages was relatively small. For example, the World Wide Web Worm (WWWW) had an index of 110,000 web pages and web accessible documents in 1994. Fast forward to November 1997, and the top search engines claimed to index anywhere from 2 million to 100 million web documents.

The goal of search engine development has always been to improve the quality of search results. Google, in particular, aimed to create an environment where researchers could process large chunks of the web and produce interesting results quickly. However, one resource that had largely gone unused in existing search engines was the citation (link) graph of the web.

Academic citation literature had been applied to the web by counting citations or backlinks to a given page. This provided an approximation of a page's importance or quality. PageRank, a key component of Google's search algorithm, extended this idea by not counting all links equally and normalizing by the number of links on a page. It factored in the probability that a random user, or "random surfer," would visit a page and introduced a damping factor to account for when the user gets bored and requests another random page.

Another important aspect of search engine design is personalization. By adding the damping factor only to a single page or a group of pages, search engines could personalize results and make it difficult for individuals to manipulate rankings deliberately.

While the focus of search engines is on providing quality search results, the predominant business model for commercial search engines is advertising. This misalignment of goals can sometimes lead to a compromise in the quality of search results.

In the world of startups, there is a common misconception about how successful companies got their start. Founders often underestimate the power of compound growth and the impact of manual user recruitment. At the beginning stages, startups are usually fragile, and founders have to take unscalable actions to recruit users manually.

Shyness and laziness are two common reasons why founders resist going out and recruiting users individually. The absolute numbers may seem small at first, but with compound growth, the user base can expand exponentially. Founders need to recognize the power of this growth and be willing to put in the effort to manually recruit users.

Investors and inexperienced founders often misunderstand the fragility of startups. The focus should not be on whether a startup is taking over the world but rather on how big the company could get if the founders do the right things. Startups that solve their own problems and find their peers can start by targeting a subset of the market where they can quickly gain a critical mass of users.

The product itself does not have to be perfect initially, but the experience of being a user should be insanely great. Engaging directly with early users and gathering feedback is crucial for improving the product and providing an exceptional user experience.

Launching a startup does not require a big launch event. In fact, founders who expect a big launch to attract users easily are often misguided. Effort and perseverance are key in the early stages of a startup. Strategies that rely on external factors or shortcuts should be viewed with suspicion.

Three actionable advice for startups:

1. Embrace manual user recruitment: Don't wait for users to come to you. Go out and get them individually. Put in the effort to personally engage with potential users and gather feedback.

2. Focus on an exceptional user experience: Even with an early, incomplete, or buggy product, you can provide an insanely great experience by being attentive to your users' needs. Prioritize the experience over the perfection of the product itself.

3. Recognize the power of compound growth: Don't underestimate the potential for exponential growth. Start small by targeting a subset of the market where you can quickly gain users and gradually expand from there.

In conclusion, search engines have evolved significantly over the years, with a focus on improving the quality of search results. Startups, on the other hand, need to understand the importance of manual user recruitment and providing an exceptional user experience. By embracing these principles and recognizing the potential for compound growth, startups can lay a strong foundation for success.