Website testing is on every site owner’s to-do list, as it should be. We all love stats, and testing data can help us improve our sites and reach our business goals faster. It can also help us avoid making big, costly mistakes.
A heavy testing focus has gotten more than one of the web’s big names a long way. In fact, when I spoke with Neil Patel of KISSMetrics recently, he revealed just how deep his trust in stats runs.
“I’m one of those guys who doesn’t have too much of an opinion,” he said when I asked him how he felt about a certain type of call to action. “Test it out. If it works, great. If it doesn’t, move on to the next test.”
“I believe in A/B testing every little thing.”
In a world rife with speculation, opinion seeking, and self-proclaimed web oracles, that kind of open-mindedness is rare and refreshing—especially coming from someone with so much experience. There’s only one problem with stats.
Ashish Desai, Head of Product for 99designs, puts it simply: “A/B testing can tell you how something does. But it doesn’t really tell you why.”
And as he recently found out, the reasons behind your test results can make all the difference to your business.
Case in point: 99designs’ Readymade Logo Store
Ashish recently managed the redesign of 99designs.com. On the previous version of the site, a link to the Readymade Logo Store appeared at the top of the Logo Design landing page. In the redesign, the link was relegated to the footer. And, after the launch of the new site, Ashish saw a noticeable drop-off in Logo Store conversions.
Although search engine marketing drives much of the traffic to the store, the new figures were jarring enough to prompt Ashish to seek a different way to drive logo store traffic from 99designs’ Logo Design landing page.
“We’d found in previous testing that the customer looking for a readymade logo and the one interested in running a design contest are two very different customers,” Ashish says.
“We saw little evidence of an overlap, and assumed the logo store link served to convert very price-sensitive customers who had zero interest in paying more to run a contest.”
But rather than rely on historical data, Ashish decided to test a new logo store link placement first. He didn’t want it to impact design contest conversion, nor undermine the 99designs’ new brand positioning.
But what he found was completely unexpected.
An unexpected result
The new link text provides more context than the old one.
In the test, in which this descriptive link to the Logo Store was placed about three-quarters of the way down on the Logo Design page, design contest conversions increased by more than 20%.
“Part of me still feels slightly skeptical,” Ashish reveals. “Every logical part of me feels like it can’t be right. But I kept running the test, and it kept returning the same results.”
To interpret these results, Ashish had to let go of his expectations. He had to somehow explain the cold, hard statistical evidence in light of his experience with the brand, the category, and the audience—experience that suggested this outcome shouldn’t be possible.
Again, open-mindedness was key.
Ashish suspects that in the past, many people came to the former Logo Design landing page in search of a logo, immediately saw the Logo Store link, and thought that was what they were supposed to go to get a logo.
“The old link just said ‘Logo Store,’ almost implying that if you wanted a logo, you should click on it, ” he says. “We probably missed out on some customers who would have run a contest, but ended up in the Logo Store instead.”
The updated link more clearly presents the Logo Store as an alternative to a design contest. Ashish describes the user’s possible thought process like this:
“I’m sort of familiar with 99designs contests, but what’s this alternative? Hmm, looking at it, it doesn’t seem right for me. I don’t want a template, I don’t want a logo off the shelf, and I don’t need it in 24 hours. I want an original design created just for me. I’m going to run a contest.”
He points out that the price difference between a readymade logo and the entry-level contest offering is quite small and this probably influences people seeking a unique logo to convert to a contest instead.
Ashish also notes that Logo Store conversions have come back up over time, though so far, his testing hasn’t explained why.
Stats + skill = learning
By putting his expectations aside, and considering the test evidence alongside his own experience of the brand and audience, Ashish was able to draw some new conclusions about the way 99designs’ users understand its offerings.
He’s now using those conclusions to formulate new ideas for future experiments in communication.
“Going forward, we’re looking to explore how best to anchor one offering against another,” he says. “What’s the best way to frame our offerings so customers clearly understand why and when each would meet their needs?”
“In the past we’ve been worried about confusing people. The concept of a design contest can be tricky to get across, and we didn’t want to throw too much else at them. But it’s clear that we do need to provide more context in a way that’s not overwhelming so they can make better-informed decisions.”
It’ll be interesting to see how this new approach tests—and what new insights 99designs can glean from the data.
How good are you at putting aside your preconceptions when you’re reviewing weird test results? We’d love to hear about your experiences in the comments.
Photo credit: Marco Bellucci