Tobi Lutke on Shopify and the AI future

TL;DR

The core point is that AI models are changing how software is built and evaluated, and leaders must live in the near future to design for it. The conversation covers AI evaluation harnesses, unit tests for models, and how Shopify adapts to rapid shifts in technology while maintaining growth. It emphasizes practical experimentation over speculation and learning from beta software updates.

Transcript

Hello acquired listeners. Since we did our episode on Shopify six years ago, they have turned into a giant public company. They went public in 2015 for less than $1.5 billion. Very cute by today's uh standards. They're worth nearly $200 billion now and doing about $10 billion a year in revenue. We've gotten to know Toby and the team pretty well sin... Read More

Key Insights

• AI evaluation harnesses become essential tools for comparing model capabilities across many models.
• Unit tests for AI models are akin to traditional software testing and are used to gauge performance on standard prompts.
• Abstractions in software can be powerful but may hide real system capabilities, requiring careful testing at the edges of what models can do.
• Software evolves forever, and hard rewrites are less common; incremental updates and flexible architectures are favored.
• Beta software is a valuable source of learning and bugs, revealing constraints and opportunities for product design.
• Living in other people’s relative futures helps product teams design for upcoming user experiences rather than predicting the distant future.
• Companies can separate core kernel capabilities from higher level glue code to improve adaptability when AI models change.
• Leadership in tech today involves combining deep engineering discipline with a willingness to experiment with new model capabilities.
• More videos from this channel:
  • How does Costco keep prices low and offer value
  • How Jamie Dimon Built JPMorgan Chase to a Banking Giant
  • How does Andrew Sorkin juggle DealBook, Squawk Box and Billions
• More from this channel: Acquired

Questions & Answers

Q: What is the practical way leaders should test AI models according to the interview?

Leaders should build and use a Toby eval like folder of prompts with expected and judged results and run it against every model to compare performance. This approach creates a repeatable test harness that reveals how different models perform on the same criteria and helps decide which models to adopt or discard. It acts like automated, ongoing unit testing for AI systems and supports fast retesting as new models emerge.

Q: How does the guest describe the evolution of software and the role of abstractions?

The guest explains that abstractions can be powerful still they can be lossy and pretend the system is simpler than it actually is. Over time, abstraction layers stack into a tangled cable network. Effective software design requires meaningful abstractions that preserve capability while hiding complexity, yet still allow downward integration when needed to access real capabilities.

Q: Why is living in the relative future important for Shopify’s product design?

Living in the relative future means experiencing beta updates and upcoming UI paradigms to understand how users will interact with products. This approach helps the company design for upcoming experiences rather than relying on outdated assumptions, ensuring product reviews and design decisions align with how customers will actually work with technology in the near term.

Q: What role does AI play in Shopify’s strategy according to the interview?

AI is central to how Shopify evaluates models, enhances decision making, and delivers capabilities through an AI assistant. The discussion highlights using AI to reshape how teams operate, test capabilities, and create more responsive tools for customers, while maintaining core business growth and robustness.

Q: How did the conversation describe the stock market experience for Shopify and what followed?

The interview notes that Shopify experienced a surge in stock price during a period of high revenue expectations, turning into a meme-like surge, and then a crash. The subsequent period focused on real performance and revenue growth, with the company returning to high levels based on actual business success rather than speculative bets.

Q: What does the guest say about software rewrites versus evolution?

The guest argues that software evolves forever and that heavy rewrites are less common than previously thought. Instead, teams should focus on incremental improvements, maintainable architectures, and the ability to replace or update components without rebuilding everything from scratch.

Q: What is the comparison made between mobile platforms and software development practices?

The conversation references the shift from cross platform or HTML5 style approaches to native or closer to native experiences. It discusses how earlier abstractions often failed to capture hardware capabilities, leading to the realization that modern devices require more accurate access to underlying capabilities for optimal performance.

Q: How does the guest view AI testing in relation to human expertise?

The guest emphasizes that testing AI involves evaluating model outputs with a structured, repeatable process, much like unit tests in software. It also suggests combining machine learning expertise with practical product needs, so that evaluation frameworks reflect real-world usage and edge-case behaviors rather than theoretical performance alone.

Summary & Key Takeaways

• Shopify has grown into a large public company and continues to adapt to AI driven changes while maintaining revenue growth.

• The guest describes building and using AI evaluation prompts and test harnesses to measure model capabilities against defined edges.

• The discussion explores software abstractions, the evolution of software engineering, and the importance of living in future user experiences to guide product design.