Testing

innovation and the blind spot

Learn From Amazon’s Innovations and Walmart’s Blind Spot

innovation and the blind spotAmazon (NASDAQ: AMZN) should be concerned. Walmart’s (NYSE: WMT) resources are enormous. That gives Walmart time to work on their blind spot. Does Walmart leadership understand what makes Amazon so strong? Probably not. If they do, can they adopt it?

Walmart is changing quickly.  We’ve all seen all their ecommerce acquisitions and a lot of new initiatives they’ve launched and tested in the last few months. These included Bossa Nova robot in stores, Mobile, Express Returns, Scan & Go, Check Out With Me, and JetBlack premium membership.

Walmart is still the largest retailer by volume. Walmart has stores within 20 miles of 90% of Americans. Yet, they correctly perceive Amazon, and others as a threat.

This image from their shareholder meeting that was shared by Lauren Thomas from CNBC summarizes their concern:

walmart top 10 US Retailers 1970 now

 

Walmart doesn’t want to end up like Sears. However it might.

For all Walmart’s “digital transformation”, their ecommerce growth slowed down this quarter. Amazon continues to outperform expectations.

Let’s look at two similar initiatives from Amazon and Walmart to understand the differences.

Testing Walmart’s Scan & Go vs. Amazon’s Amazon Go

Cashier-less checkout was a big theme at ShopTalk. Nordstrom, Macy’s and other retailers are focusing resources to reduce this friction point.

Walmart executives presented their Scan & Go initiative at ShopTalk this year. Amazon launched Amazon Go in Seattle and is expanding. In e-commerce, retailers can either help customers increase their motivation or decrease the friction they encounter during a purchase. In retail, checkout has always been a significant source of friction. No one likes to wait.

Walmart’s Scan & Go Test

In January 2018, Walmart announced the expansion of their Scan & Go cashier-less checkout experiment to 100 stores.

Their promotional video looked promising:

This is how it actually worked.

The demo was “smoother” than the actual experience. During the test shoppers could use their phones with a mobile app or a separate device Walmart provided. As you walked through the store you scan the items added to your physical shopping cart. Then shoppers advanced checkout at a special lane near their current self service checkout.

The Scan & Go initiative was cancelled.  Walmart executives said they did learn from it.

What did Walmart learn? It seems like they were asking: If you reduce the friction in checkout will you sell more? No surprise! They likely weren’t impressed with the results of their test.

Walmart essentially moved an already not super popular self checkout scanner onto a mobile device. Self-checkout is an extra job for the customer no matter what part of the store they do it in.  They tested the same concept but moved it to a different location in the store.

Walmart tested a variation of what they were already doing. Tweaking, improving and optimizing variations leads to a local maxima, a dead end.

Let’s examine how Amazon approaches tests to see what we can learn about innovation and testing.

Amazon Go didn’t waste time and resources testing variations. They tested a variable first.

Amazon knows that customers don’t like to wait. It’s something that will never change.

“ I very frequently get the question: ‘What’s going to change in the next 10 years?’ And that is a very interesting question; it’s a very common one. I almost never get the question: ‘What’s not going to change in the next 10 years?’ And I submit to you that that second question is actually the more important of the two — because you can build a business strategy around the things that are stable in time. … [I]n our retail business, we know that customers want low prices, and I know that’s going to be true 10 years from now. They want fast delivery; they want vast selection. It’s impossible to imagine a future 10 years from now where a customer comes up and says, ‘Jeff I love Amazon; I just wish the prices were a little higher,’ [or] ‘I love Amazon; I just wish you’d deliver a little more slowly.’ Impossible. And so the effort we put into those things, spinning those things up, we know the energy we put into it today will still be paying off dividends for our customers 10 years from now. When you have something that you know is true, even over the long term, you can afford to put a lot of energy into it.” ― Jeff Bezos

Amazon innovates with a central purpose – making things better for the customer.

Amazon, and anyone who visited a store with cashier-less checkout knows, that people aren’t excited about taking over the cashier’s job.  You can see that in every grocery store, Walmart, Lowes and Home Depot that have longs cashiered lines and a handful of people struggling at the self checkout. For stores with higher margins sales people can take over the cashier’s job for the convenience of customers.

Amazon, understood that “checkout” is a lousy experience. Checkout is the variable to test.

Amazon didn’t ask how to improve checkout. They asked: would people buy more if they didn’t have to use checkout at all?

Amazon decided to innovate around replacing the checkout experience. Ironically, it was so successful that there were lines of people to try the experience for themselves.

Amazon proved that customers love their no checkout innovation. Only now will Amazon will test many variations to improve on the concept.

Walmart has a serious blind spot

Walmart focused on optimizing store operations with technology.

Amazon focused on innovating on behalf of their customers.

Walmart doesn’t understand that they don’t have an ecommerce channel problem, Walmart’s problem isn’t Amazon. It isn’t even technology.  Walmart’s problem is a blind spot, a lack of interest in who its customers are. Walmart is struggling to transform from being experts in products, inventory management, supply chain, and logistics to becoming experts about their customers.

To succeed in retail today you need to start with the customer, not the product.

Amazon is beating Walmart, and others, because it knows its customers and takes as an article of faith that if they do right by customers they will succeed.

How about you? Every successful business has a blind spot!

Here’s what often happens:

A business has a unique approach, or a special emphasis that separates them from competitors so they commit to it. It makes them successful.

They eventually reach a plateau.

Most businesses double-down on what brought them success.

They do what they know how to do and improve on the margins. They press down harder on the accelerator..

They get stuck in that gear.

At first their innovation gave them momentum.

And then things began to level off.

They believe in first gear. First gear is where they feel comfortable.

Metaphorically, Walmart is accelerating in first gear.

Businesses don’t have automatic transmissions!  All you have to do is look at the top retailers slide at the beginning of this post.

Few companies ever find second gear.

Let us know if we can help you eliminate your blind spot. Let us work with you to innovate your way to success by reducing customers’ friction and increasing customers’ motivations.

Read More

What To Test First #CX #UX #CRO

choices

We use a 3-step process as part of our Buyer Legends process.

  1. Pre-mortem
  2. Eisenberg’s Hierarchy of Optimization
  3. Scoring Priorities

Of course, you cannot start the 3-step process without first creating actionable personas based on qualitative and quantitative data. Buyer Legends employ storytelling to optimize customer experiences.

Why do we focus on customer experience? We wrote in 2001 that conversion rates are a measure of your ability to persuade visitors to take the action you want them to take. They’re a reflection of your effectiveness at satisfying customers. For you to achieve your goals, visitors must first achieve theirs.

Pre-mortem

The reality is that most companies lose more sales every day than they make. If you are converting less than 15% you need to evaluate what is broken in your customer experience.  

Get to the bottom of what is going wrong, and plan to get it right. That is why, hands down, the pre-mortem step is the most impactful step of our Buyer Legends process. In fact, rarely does this exercise fail to produce at least one a-ha moment for our clients.  When you imagine the sale is already dead it frees up all the mental energy that you used to try and get the sale and points it at all the potential pitfalls and problems in your experience.  

Eisenberg’s Hierarchy of Optimization

After you perform your pre-mortem you will likely end up with a long list of potential proof of Murphy’s law, but not everything on your list is equal.  Some thing are worth your effort some are not.  In my work with clients we often use Eisenberg’s Hierarchy of Optimization to separate the more pressing issues from the tinier ones.  

First sort the list of problems into the follow categories:

  • Functional. Does this product/service do what the prospect needs? How easy is it for a prospect to determine this?
  • Accessible. Can she access it? What are the barriers to her ability to realize the need? Is it affordable, reasonable, and findable?
  • Usable. Is it user-friendly? Are there obstacles?
  • Intuitive. Does the sales process/Web site feel intuitive and natural based on her buying preferences? Is she forced to endure unnatural buying modalities to realize her need?
  • Persuasive. Does she want it? Does she truly understand if it fills her need or solves her problem? Is her expectation reasonable? Will she be delighted?

eisenberg-graphic-1

Once they are sorted simply work your way up the pyramid.  Again, remember not every problem is in search of a solution, and you should focus on the problems that are likely to impact the most customers and problems that you can actually fix. Be practical, don’t get caught up in the problems you can’t fix.

Scoring Priorities

Let’s consider another simple system to enable your organization to prioritize more effectively when planning tests. The system is based on prioritizing all your planned efforts by three factors with  a score from 1 to 5, with 5 being the best and 1 being the worse:

  1. Time – How long will it take to execute a project (a change, a test, or full scale roll-out) until its completion? This includes staff hours/days to execute and the number of calendar days until the project’s impact would be recognized. A score of 5 would be given to a project that takes the minimal amount of time to execute and to realize the impact.
  2. Impact – The amount of revenue potential (or reduced costs) from the execution of your project. Will the project impact all of your customers or only certain segments? Will it increase conversion rates by 1 percent or by 20 percent? A score of 5 is for projects that have the greatest lift or cost reduction potential.
  3. Resources – The associated costs (people, tools, space, etc.) needed to execute a project. Keep in mind: No matter how good a project is, it will not succeed if you do not have resources to execute an initiative. A score of 5 is given when resources needed are few and are available for the project.

Next, take each factor and multiply them (don’t add them because these factors are orthogonal) for each project. The best possible score is 125 (5x5x5). Tackle and complete the highest-ranking projects first. Meet weekly with a cross-functional group to evaluate the status of each project. Be prepared to re-prioritize regularly; once a month or at least once a quarter.

Read More

Accountability Means You Win Big or Try Again

accountability1Marie, the VP of Marketing, predicts that she can increase conversions to sales by at least 40 percent. She wants to conduct a series of disruptive experiments that make everyone nervous. Scott, the VP of Sales, is especially on edge. The lead-to-contract rate is already a respectable 7.1 percent for this not-so-new marketing technology B2B SaaS. If Marie is right, it’s a homerun. But if she’s wrong, Scott is predicting a debacle on an epic scale.

You’re in charge. Would you give her the green light?  Please continue reading this post on the Salesforce blog

Read More

How To Test For What Matters

Gerald never imagined his 71 year old grandpa would be the one championing a shopping cart and checkout development initiative to both the CIO and the CFO. Gerald’s grandfather, Isaac, doesn’t carry a smartphone and handwrites notes on printed-for-him emails that he returns to his assistant. Isaac is the founder and CEO of a popular apparel catalog merchant that has been thriving for over forty years.

 [Names have been changed to protect our friend’s privacy. The images you’ll see are to illustrate the points]

 When Gerald was hired by Matt, the CMO who is a veteran catalog merchandiser, as the Director of Ecommerce at the company he knew that he would be under scrutiny. Matt demanded that any family members at the company be at least twice as good as anyone else at their job.

Gerald was indeed doing a great job but he was frustrated. His team had spent almost a year optimizing the shopping cart. Due to Matt’s catalog company pedigree placement and copy were seen as THE critical variables. They tested  and retested the placement, color, shapes and sizes of buttons. Gerald fought hard to eliminate a step and two required form fields in the checkout. The shopping cart and checkout were streamlined. Yet they only realized minimal success with conversion improving 17% from 2.89% to 3.38%.

Matt urged Gerald to turn it up a notch and run even more tests. With little confidence that more testing would improve results and a shortage of new actionable testing ideas Gerald could feel himself between a rock on one side and a hard place on the other.  Gerald knew their shopping cart was lacking features that many apparel sites had but knew any changes that involved significant development time and resources so the investment would be difficult or even impossible to push through.  He obviously couldn’t play the grandson card either and he was sure his Luddite grandfather would side with Matt who after all was responsible for all that web stuff.

Yet a few weeks later here he was, watching grandpa convince his C-suite colleagues.  The best part?  Grandpa, the Luddite, was making a passionate case for a technology change he couldn’t have cared less about before.

Understanding Gerald’s dilemma

Neil Patel wrote about “7 A/B Testing Blunders That Even Experts Make”, and explained in Blunder #3, “Expecting big wins from small changes”;

“If small changes are providing huge gains, something else is wrong with your design or copy. The conversion wins that small changes provide typically don’t hold.

The biggest conversion boosts are going to come from drastic changes. So, if you really want to move your conversion rates, don’t focus on small changes. Instead, focus on drastic changes, as they are the ones that boost your revenue.

When you are starting out, you could try small tweaks to your design and copy to see if your conversion rates increase, but eventually you’ll need to focus on the big wins.

What I like doing is to focus on the drastic changes. Once I feel I’ve maximized their potential, I then focus on the small changes.”

What Patel is describing is the inclination that most companies like Gerald’s have. They test variations of individual elements instead of trying to identify variables that might move the needle. Perhaps this happens because of how testing software is designed to work. Yet, 90% of their tests yield little to no results and it is discouraging.

Gerald knew that continuing to do what he was doing would give him the same result . He knew they should be making changes but didn’t know exactly where to start.

Then Gerald met with me at a conference and we spoke a few minutes. He was intrigued by the idea of the book we were working on. I sent him an early draft on the promise that he wouldn’t share it but that he would provide feedback after he went through the Buyer Legends process.

Below is the portion of one of Gerald’s Buyer Legends that start at the Add to Cart phase of the story. The legend describes the current experience, the possible variation tests, and then a variable test. Please take note of how the likelihood of impact is described in the Legend itself.

Testing Legend – Add to Cart –> Checkout –> Confirm Purchase → Confirmation email

The current experience:

“… Pat clicks the Add to Cart button and is taken to the Checkout page. She looks over to the right and sees the Checkout Now button, and clicks on it. Pat notices that that prominently to the right of the form fields, the company addresses her privacy rights next to her billing and shipping information. Her security is addressed right next to the billing information. Pat feels reassured and comfortable filling out those fields, so she does. Finally, she sees her order and the prominent Complete Your Purchase button. Underneath the Complete Your Purchase button, she sees in a contrasting color one last reassurance; a 100% money back, no-questions-asked guarantee. Pat clicks and confidently completes the purchase. Pat notices when she receives her confirmation email..”

This is a reasonably good customer experience.

Here are some potential variation tests that might improve results:

“… Pat clicks the Add to Cart button and is taken to the Checkout page. She looks over to the right and sees the Checkout Now button and clicks on it. Pat notices that prominently [test copy] to the right of the form fields, the company addresses her privacy right next to her billing and shipping information [test copy]. Her security is addressed right next to the billing information [test copy]. Pat feels reassured and comfortable filling out those fields, so she does. Finally, she sees her order and the Complete Your Purchase button [test copy, button size, color etc.]. Underneath the Complete Your Purchase button, she sees in a contrasting color one last reassurance; a 100% money back, no-questions-asked guarantee. [test copy] Pat clicks and confidently completes the purchase. Pat notices when she receives her confirmation email..”

There are likely small but valuable wins in improving copy and perhaps even a button test. However, do any of these changes fundamentally improve the experience?

Here you’ll see a potentially important variable to test instead:

“… Pat clicks the Add to Cart button and is taken to the Checkout page. On the Checkout page, she confirms that it’s the right item (there is a thumbnail image), the right size, and the right quantity.She looks over to the right and sees the Checkout Now button and clicks on it. Pat notices that prominently to the right of the form fields, the company addresses her privacy right next to her billing and shipping information. Her security is addressed right next to the billing information. Pat feels reassured and comfortable filling out those fields, so she does. Finally, she sees her order details, exactly as she saw them in her shopping cart, and the Complete Your Purchase button. Underneath the Complete Your Purchase button, she sees in a contrasting color one last reassurance; a 100% money back, no-questions-asked guarantee. Pat clicks and confidently completes the purchase. Pat is thrilled when all the information,including the product detail with thumbnail image and reassurances show up in her confirmation email exactly as the appeared on the Checkout page.”

Did you notice the hypothesis embedded in Gerald’s last legend?

The hypothesis is that when at the point of greatest cognitive dissonance, placing the order, we should reassure the buyer in every way that they are getting the right thing. Because of the large abandonment rate at this step Gerald was confident that testing it would impact conversions significantly.

Gerald was a bit surprised when Matt told him he shared the Buyer Legends with Isaac.  It seems Matt found Buyer Legends a useful way to communicate with Isaac. His grandfather was committed to testing this and anything on the site that would “help make things clearer and less confusing for our customers”.

Gerald shouldn’t have been surprised.  Isaac had kept the company viable in up and down times driven by his relentless commitment to the customer.

Before reading Gerald’s Buyer Legend Isaac had always considered the online business as simply an evolved function of operations and cost control; much less intimate than his baby the call center. The Buyer Legend helped him empathize with his customers and he was able to begin examining the web site as an opportunity to better deliver on the company’s promise to the customers.

The ecommerce business went up 29.4% over budget this holiday season. Gerald tells us that a significant part of this is due to using the Buyer Legend process.

The takeaway – are you testing too much for too little reward?

If 90% of your tests yield little to no results and you’re discouraged there is hope.

You may not sell apparel but I hope you can see how using the Buyer Legends process helps to provide the customer’s perspective. It delivers an empathetic jolt of context and relevance to your entire team. Use Buyer Legends to identify the most important variables; then you can make incremental improvements by testing variations.

We want to hear about your success with Buyer Legends. If you’d like to learn more, please read Buyer Legends: The Executive Storyteller’s Guide, look into our trainings or if you don’t care to go it alone we’re always here to help you.

Read More
What we can offer you

Four Pillars Ongoing Support

After our workshops, we work with only a few select clients. Your business must be committed to the Four Pillars (as described in Be Like Amazon) on a long-term basis .

Workshops

We kick-off the workshop with a two-day onsite visit. We help you create the Four Pillar foundation for your organization. The entire process takes between 4-8 weeks and the typical investment is $30,000 – $100,000.

Speak at Your Event

We can speak at your event. Our fees are $20,000 in North America, and that includes travel. International fees are $20,000 plus business class travel, from Austin, and lodging. Contact us to discuss your event