Testing

Gerald never imagined his 71 year old grandpa would be the one championing a shopping cart and checkout development initiative to both the CIO and the CFO. Gerald’s grandfather, Isaac, doesn’t carry a smartphone and handwrites notes on printed-for-him emails that he returns to his assistant. Isaac is the founder and CEO of a popular apparel catalog merchant that has been thriving for over forty years.

 [Names have been changed to protect our friend’s privacy. The images you’ll see are to illustrate the points]

 When Gerald was hired by Matt, the CMO who is a veteran catalog merchandiser, as the Director of Ecommerce at the company he knew that he would be under scrutiny. Matt demanded that any family members at the company be at least twice as good as anyone else at their job.

Gerald was indeed doing a great job but he was frustrated. His team had spent almost a year optimizing the shopping cart. Due to Matt’s catalog company pedigree placement and copy were seen as THE critical variables. They tested  and retested the placement, color, shapes and sizes of buttons. Gerald fought hard to eliminate a step and two required form fields in the checkout. The shopping cart and checkout were streamlined. Yet they only realized minimal success with conversion improving 17% from 2.89% to 3.38%.

Matt urged Gerald to turn it up a notch and run even more tests. With little confidence that more testing would improve results and a shortage of new actionable testing ideas Gerald could feel himself between a rock on one side and a hard place on the other.  Gerald knew their shopping cart was lacking features that many apparel sites had but knew any changes that involved significant development time and resources so the investment would be difficult or even impossible to push through.  He obviously couldn’t play the grandson card either and he was sure his Luddite grandfather would side with Matt who after all was responsible for all that web stuff.

Yet a few weeks later here he was, watching grandpa convince his C-suite colleagues.  The best part?  Grandpa, the Luddite, was making a passionate case for a technology change he couldn’t have cared less about before.

Understanding Gerald’s dilemma

Neil Patel wrote about “7 A/B Testing Blunders That Even Experts Make”, and explained in Blunder #3, “Expecting big wins from small changes”;

“If small changes are providing huge gains, something else is wrong with your design or copy. The conversion wins that small changes provide typically don’t hold.

The biggest conversion boosts are going to come from drastic changes. So, if you really want to move your conversion rates, don’t focus on small changes. Instead, focus on drastic changes, as they are the ones that boost your revenue.

When you are starting out, you could try small tweaks to your design and copy to see if your conversion rates increase, but eventually you’ll need to focus on the big wins.

What I like doing is to focus on the drastic changes. Once I feel I’ve maximized their potential, I then focus on the small changes.”

What Patel is describing is the inclination that most companies like Gerald’s have. They test variations of individual elements instead of trying to identify variables that might move the needle. Perhaps this happens because of how testing software is designed to work. Yet, 90% of their tests yield little to no results and it is discouraging.

Gerald knew that continuing to do what he was doing would give him the same result . He knew they should be making changes but didn’t know exactly where to start.

Then Gerald met with me at a conference and we spoke a few minutes. He was intrigued by the idea of the book we were working on. I sent him an early draft on the promise that he wouldn’t share it but that he would provide feedback after he went through the Buyer Legends process.

Below is the portion of one of Gerald’s Buyer Legends that start at the Add to Cart phase of the story. The legend describes the current experience, the possible variation tests, and then a variable test. Please take note of how the likelihood of impact is described in the Legend itself.

Testing Legend – Add to Cart –> Checkout –> Confirm Purchase → Confirmation email

The current experience:

“… Pat clicks the Add to Cart button and is taken to the Checkout page. She looks over to the right and sees the Checkout Now button, and clicks on it. Pat notices that that prominently to the right of the form fields, the company addresses her privacy rights next to her billing and shipping information. Her security is addressed right next to the billing information. Pat feels reassured and comfortable filling out those fields, so she does. Finally, she sees her order and the prominent Complete Your Purchase button. Underneath the Complete Your Purchase button, she sees in a contrasting color one last reassurance; a 100% money back, no-questions-asked guarantee. Pat clicks and confidently completes the purchase. Pat notices when she receives her confirmation email..”

This is a reasonably good customer experience.

Here are some potential variation tests that might improve results:

“… Pat clicks the Add to Cart button and is taken to the Checkout page. She looks over to the right and sees the Checkout Now button and clicks on it. Pat notices that prominently [test copy] to the right of the form fields, the company addresses her privacy right next to her billing and shipping information [test copy]. Her security is addressed right next to the billing information [test copy]. Pat feels reassured and comfortable filling out those fields, so she does. Finally, she sees her order and the Complete Your Purchase button [test copy, button size, color etc.]. Underneath the Complete Your Purchase button, she sees in a contrasting color one last reassurance; a 100% money back, no-questions-asked guarantee. [test copy] Pat clicks and confidently completes the purchase. Pat notices when she receives her confirmation email..”

There are likely small but valuable wins in improving copy and perhaps even a button test. However, do any of these changes fundamentally improve the experience?

Here you’ll see a potentially important variable to test instead:

“… Pat clicks the Add to Cart button and is taken to the Checkout page. On the Checkout page, she confirms that it’s the right item (there is a thumbnail image), the right size, and the right quantity.She looks over to the right and sees the Checkout Now button and clicks on it. Pat notices that prominently to the right of the form fields, the company addresses her privacy right next to her billing and shipping information. Her security is addressed right next to the billing information. Pat feels reassured and comfortable filling out those fields, so she does. Finally, she sees her order details, exactly as she saw them in her shopping cart, and the Complete Your Purchase button. Underneath the Complete Your Purchase button, she sees in a contrasting color one last reassurance; a 100% money back, no-questions-asked guarantee. Pat clicks and confidently completes the purchase. Pat is thrilled when all the information,including the product detail with thumbnail image and reassurances show up in her confirmation email exactly as the appeared on the Checkout page.”

Did you notice the hypothesis embedded in Gerald’s last legend?

The hypothesis is that when at the point of greatest cognitive dissonance, placing the order, we should reassure the buyer in every way that they are getting the right thing. Because of the large abandonment rate at this step Gerald was confident that testing it would impact conversions significantly.

Gerald was a bit surprised when Matt told him he shared the Buyer Legends with Isaac.  It seems Matt found Buyer Legends a useful way to communicate with Isaac. His grandfather was committed to testing this and anything on the site that would “help make things clearer and less confusing for our customers”.

Gerald shouldn’t have been surprised.  Isaac had kept the company viable in up and down times driven by his relentless commitment to the customer.

Before reading Gerald’s Buyer Legend Isaac had always considered the online business as simply an evolved function of operations and cost control; much less intimate than his baby the call center. The Buyer Legend helped him empathize with his customers and he was able to begin examining the web site as an opportunity to better deliver on the company’s promise to the customers.

The ecommerce business went up 29.4% over budget this holiday season. Gerald tells us that a significant part of this is due to using the Buyer Legend process.

The takeaway – are you testing too much for too little reward?

If 90% of your tests yield little to no results and you’re discouraged there is hope.

You may not sell apparel but I hope you can see how using the Buyer Legends process helps to provide the customer’s perspective. It delivers an empathetic jolt of context and relevance to your entire team. Use Buyer Legends to identify the most important variables; then you can make incremental improvements by testing variations.

We want to hear about your success with Buyer Legends. If you’d like to learn more, please read Buyer Legends: The Executive Storyteller’s Guide, look into our trainings or if you don’t care to go it alone we’re always here to help you.