How to Plan Evidence-Based Ecommerce A/B Tests

TL;DR

Clear product copy can outperform image-heavy ecommerce pages when shoppers need more information to understand the offer. In the featured experiment, a version with fewer product images and more important copy increased purchases by 61%, while a separate Facebook test found interest-based targeting delivered a $1.38 acquisition cost versus $2.88 for a look-alike audience.

Transcript

good afternoon everyone depending on what time zone you're joining us from we're going to get right away into the face book test that we recently ran and then I've got a question for you now this will be different than the previous broadcasts I want you to look at this particular test with a view in mind that you helped us conceive and plan it in t... Read More

Key Insights

• Interest-based Facebook targeting produced a $1.38 cost per acquisition in the featured experiment, compared with $2.88 for look-alike targeting. The result represented a 52% improvement for the interest-based approach within this particular test.
• Look-alike targeting works by sampling an audience that has purchased before, or is considered ideal, and allowing Facebook to build a broader audience around that sample. Interest-based targeting instead reaches people through available keywords and audience interests.
• Scientific testing replaces guesses with observed customer behavior. Participants predicted whether look-alike targeting, interest targeting, or neither would win, but the experiment supplied the evidence needed to determine which approach produced the lower acquisition cost.
• The Internet functions as a behavioral research environment because digital technologies make customer choices observable. The organization’s research focuses on a single central question, why people say yes, and studies how human nature appears through measurable online decisions.
• Copy length should be determined by what is necessary to communicate the value proposition and essential components of the sale. Neither short copy nor long copy is inherently the enemy. The central problem is a lack of clarity.
• Product images do not automatically outperform explanatory copy on an ecommerce page. In the featured store test, the version with fewer product pictures and more important copy generated a 61% increase in purchases.
• The ecommerce research program is based on more than 20 years of research, over 20,000 sales-path tests, and a catalog containing experimental abstracts. These accumulated experiments provide evidence for developing and evaluating new ecommerce treatments.
• A swipe file can support test ideation by presenting before-and-after versions from previous experiments. The featured ecommerce file compiles 25 experiments and is intended to help marketers improve their own testing plans while examining three common pitfalls.

Questions & Answers

Q: How should ecommerce marketers decide the right copy length?

Ecommerce copy should be long enough to communicate the value proposition and the essential components of the sale clearly. Short copy is not automatically better, and long copy is not automatically a problem. The deciding issue is clarity. When shoppers need more information to understand the offer, additional copy is necessary, and removing it can weaken the product page.

Q: Can more product copy increase ecommerce purchases?

More product copy can increase purchases when it supplies information customers need to understand the offer. In the featured ecommerce experiment, the tested version contained fewer product pictures but more important copy. That treatment produced a 61% increase in purchases. The result supports testing necessary explanatory copy instead of assuming that a more image-focused page will perform better.

Q: What was the result of the Facebook audience test?

The interest-based audience produced a cost per acquisition of $1.38, while the look-alike audience produced a cost per acquisition of $2.88. The presentation describes this as a 52% improvement for the interest-based approach in that experiment. The result answers the phase-one question of which tested audience method generated the lower acquisition cost under those conditions.

Q: How does look-alike audience targeting work?

Look-alike targeting begins with a sample of people who have purchased previously or who represent the marketer’s ideal audience. Facebook then uses that sample to build a broader list of similar people who can receive the message. In the featured comparison, this approach produced a $2.88 cost per acquisition, higher than the interest-based audience’s $1.38 cost.

Q: How does interest-based Facebook targeting work?

Interest-based targeting attracts people through keywords and audience interests available within Facebook’s option set. Some keywords suggested by participants were unavailable, so the experiment included the suggestions that Facebook supported. This approach generated a $1.38 cost per acquisition, compared with $2.88 for the look-alike method, producing the better result in the featured test.

Q: Why should marketers run experiments instead of relying on predictions?

Predictions reveal expectations, but controlled experiments reveal how an audience actually responds. Participants offered different guesses about whether the look-alike audience or interest-based audience would win. The measured acquisition costs then provided a clear answer. The methodology lets marketers talk to audiences through observed behavior and discover what matters without remaining committed to their initial guesses.

Q: What is the purpose of the 25-experiment ecommerce file?

The file compiles 25 ecommerce experiments and presents before-and-after treatments that marketers can examine when developing their own tests. It serves as a swipe file for improving experimentation rather than as a paid offer or required signup. The presentation uses this collection to teach ecommerce optimization and to explore three pitfalls that marketers should avoid.

Q: What research supports the ecommerce testing recommendations?

The recommendations draw from more than 20 years of research, over 20,000 sales-path tests, and a catalog of experimental abstracts. The organization also developed an early experimental program focused on ecommerce and studied an early major ecommerce cart. The presentation uses this accumulated testing history to identify practical experiment ideas and recurring dangers in ecommerce optimization.

Summary & Key Takeaways

• The presentation uses controlled experiments to examine why customers say yes. It begins with a Facebook audience test comparing look-alike targeting against interest-based targeting. Interest targeting produced a $1.38 cost per acquisition, while the look-alike audience cost $2.88, representing a 52% improvement for the interest-based approach in that experiment.

• The ecommerce discussion draws from more than 20 years of research, over 20,000 sales-path tests, and a catalog of experimental abstracts. A downloadable file compiles 25 ecommerce experiments, showing before-and-after treatments that marketers can use to develop ideas for their own tests and avoid three identified testing pitfalls.

• The first ecommerce pitfall is removing necessary copy in favor of product imagery. Copy should be as long as required to communicate the value proposition and essential parts of the sale clearly. In the featured store experiment, the treatment used fewer product pictures but more important copy and generated a 61% increase in purchases.