Will Bryk on Building Exa's Neural Search Engine

TL;DR

Exa (formerly Metaphor Systems) is a search engine built on link prediction, where a Transformer-inspired model is trained on billions of web links to predict which document follows a piece of text. Founder Will Bryk started it in summer 2021 after GPT-3 showed language models could understand queries that Google could not, like 'shirts without stripes.'

Transcript

hey everyone welcome to the Len space podcast this is cesio partner and CTO and residents and deible partners and I'm joyed my co-host swix founder of smalli hey and today we're in the studio with my good friend and former landlord uh will bre roommate how you doing uh will you're now CEO co-founder of EXA AI used to be metaphor systems what's your... Read More

Key Insights

• Exa is a search engine founded by Will Bryk that began in summer 2021 with the pitch 'we are better than Google, Google 2.0,' motivated by GPT-3's ability to understand nuanced language that Google could not.
• Link prediction is Exa's core training method: the model is given text surrounding a link on the web, the link is hidden, and the model predicts which link follows, done billions of times to build a search engine.
• Link prediction is more accurately called document prediction, because the model retrieves the actual document rather than memorizing and reproducing a URL string.
• Exa's architecture is Transformer-inspired but requires a different architecture than Transformers to actually work for predicting links instead of the next token.
• The name Exa means 10 to the 18th, chosen in provocative contrast to Google, which means 10 to the 100th, and staff wear shirts claiming 10 to the 18th is greater than 10 to the 100th.
• Exa was originally named Metaphor Systems and renamed to Exa around January when it announced its Series A, partly because people could spell Exa but not metaphor.
• Exa's vision has always been perfect search over the web and building a better search algorithm, with the belief that downstream use cases could be figured out later.
• A full search engine requires finding URLs to crawl, crawling them, processing documents into an index (embedding models or older keyword inverted indexes), then serving that at high throughput and low latency via a vector database.

Questions & Answers

Q: What is Exa AI and who founded it?

Exa AI is a search engine founded and led by CEO Will Bryk, who has been running it for three years. It was formerly called Metaphor Systems. Bryk has been interested in high-quality information since he was a kid, built a mini search engine in college, and describes Exa as fulfilling his lifelong goal of solving information needs. The company aims to build a perfect search engine over the web.

Q: How does Exa's link prediction technology work?

Link prediction means the model is given a piece of text and predicts the link that follows. Exa finds links on the web, takes the text surrounding each link, hides the link, and asks the model which link came next. For example, if an article says 'check out this cool Aerospace startup' before spacex.com, the model learns to predict that link. Done billions of times, this builds a search engine.

Q: Why did Will Bryk think Exa could beat Google?

When Exa started in summer 2021, GPT-3 had just come out and could understand a full paragraph and the subtlety of language. Google, by contrast, felt like it had not changed in a decade and could barely understand users, failing queries like 'shirts without stripes' by returning striped shirts. The theory was that applying language model insights to search could create an engine that actually understands you.

Q: What does the name Exa mean?

Exa means 10 to the 18th power. The name was chosen in stark, provocative contrast to Google, which means 10 to the 100th power. The company makes shirts that claim 10 to the 18th is greater than 10 to the 100th, which provokes engineers in Silicon Valley to argue it is not true, then to ask what it actually means. Bryk finds the provocation useful and fun.

Q: What are the major components needed to build a search engine?

According to Bryk, you first have to find the URLs to crawl, which is connected to the crawler. Then you crawl those URLs and process them, which could use an embedding model or older methods like a keyword inverted index, gathering documents into a processed index. Finally you serve that index at high throughput and low latency, which is the vector database. Each of these is a large team at Google but only one or two people at Exa.

Q: Why was Exa renamed from Metaphor Systems?

The company was originally called Metaphor Systems and was renamed to Exa around January when it announced its Series A funding. Bryk did not initially like the name Exa but it grew on him; he liked metaphor, but apparently people could spell Exa while they could not spell metaphor. Bryk says the vision has always been the same regardless of the name: building a better search algorithm for perfect search.

Q: How is Exa similar to OpenAI?

Bryk often says Exa is the OpenAI of search. Both are research companies or research startups. Just as OpenAI released ChatGPT as a research preview without a fully clear game plan and shifted from a research company to a product company, Exa started as a research endeavor with clear eyes that success would become a massive business. Exa does fundamental research into making a kind of AGI for search, then builds business products from that.

Q: Is link prediction actually predicting the URL string?

No. A more accurate name is document prediction. The system is not predicting the URL text itself, which would require it to have memorized URLs. Instead the link refers to the actual document, and the model retrieves that document. This link prediction approach was the initial base model Exa trained on, similar to how a language model starts with a self-supervised objective of predicting the next token before adding synthetic data and supervised fine-tuning.

Summary & Key Takeaways

• Will Bryk is CEO and co-founder of Exa AI, which he has run for three years. He has been interested in high-quality information since childhood, built a mini search engine in college, and interned at SpaceX and Zoox. He entered Y Combinator with the audacious pitch that Exa would be Google 2.0.

• The company launched in summer 2021 when GPT-3 had just appeared and could understand a whole paragraph of nuanced language, while Google felt unchanged for a decade and failed simple queries like 'shirts without stripes.' The theory was to apply large language model insights to build a search engine that actually understands the user.

• Exa trains on link prediction: it finds links on the web, hides the linked URL, and predicts which document follows the surrounding text, done billions of times. Building the search engine requires crawling URLs, processing them into an index with embedding models, and serving results at low latency through a vector database.