A/B Testing for Startups with Hiten Shah

TL;DR

Hiten Shah’s central message is to “Always Be Testing”: startups should run A/B tests continually to support growth, while recognizing that testing cannot replace product-market fit. Speaking to a mostly web-focused audience, he explores testing frequency, tools such as Optimizely, and how repeated interface changes can improve a customer experience. Read on for his practical framing of experimentation and iteration.

Transcript

so I've got some questions first and then I've got a whole bunch of stuff my talk that I'm gonna give is called always be testing but first I have questions so and and Ryan's over there great so this gentleman in the back Ryan is gonna be tell speaking after me and he needs this input as well so I'm gonna jump right into it so the first question is... Read More

Key Insights

• 👨‍💼 Structured A/B testing processes are essential for sustained business growth.
• 🦻 Documentation of testing results and learnings aids in making informed decisions.
• ❓ Continuous testing ensures ongoing optimization and improvement.
• 🥺 Experimentation and implementation of tests lead to actionable insights for business strategy.
• 😃 Return on investment from A/B testing is influenced by systematic implementation and analysis.
• 🏆 Utilization of prioritization frameworks enhances test effectiveness and impact.
• 🥺 Consistency in testing practices leads to continual learning and growth.

Questions & Answers

Q: What is Hiten Shah’s main advice about A/B testing for startups?

His talk is called “Always Be Testing,” reflecting his emphasis on making testing an ongoing practice. He tells the audience that A/B testing helps a startup grow, but it cannot create growth without product-market fit.

Q: Can A/B testing replace product-market fit?

No. Hiten Shah explicitly says that although A/B testing helps companies grow, they cannot grow without product-market fit.

Q: How frequently were audience members running A/B tests?

Hiten Shah asks whether attendees run one, two, or three tests per week, then asks about one or two tests per month. He concludes that many understand A/B testing and do some testing, but are not doing enough of it.

Q: Was the talk mainly aimed at web or mobile-app testing?

The audience includes both mobile-app and web participants, but Hiten Shah observes that it is mostly web-focused. He uses those questions to understand the audience before beginning the main talk.

Q: Which A/B testing tool does Hiten Shah mention?

He asks who in the audience uses Optimizely, and several attendees indicate that they do. He also asks whether anyone uses another external tool or has built an in-house A/B testing system.

Q: What types of startups were represented in the audience?

Audience members identify e-commerce, subscription e-commerce, SaaS, and marketplace businesses. Hiten Shah remarks that marketplaces have many problems and therefore many things to test.

Q: How does Hiten Shah connect a gym experience to startup experimentation?

He shows several versions of a gym’s workout dashboard and notes that the company iterated on its design. The sequence moves from earlier boards toward the version he used that day, illustrating repeated product changes.

Q: What customer behavior does Hiten Shah use to illustrate product-market fit?

He says the gym achieved product-market fit with him because it changed his behavior from rarely attending gyms to going 11 times in three months. He and his wife also moved to an unlimited monthly plan costing roughly $100 to $200 per month for both of them.

Summary & Key Takeaways

• Demonstrates the importance of A/B testing for business growth and optimization.

• Highlights the necessity of structured processes and documentation for effective testing.

• Emphasizes the significance of ongoing testing and learning for sustained growth.