> And without A/B testing, every product you use would be worse.
The primary goal of A/B testing is to see what's more profitable.
If that happens to result in better UI that's a side effect.
In fact, it could result in less usability (relevant to this conversation, it probably resulted in the frustrating "algorithm-based" timeline at FB/Twitter/etc).
Which is why A/B testing is an important part of the UX toolkit. It's a tool among others, and is one way to validate assumptions. A good UX designer will try to base their designs on data and reasonable hypotheses drawn from the data, but a new design or flow is necessarily based on some amount of assumptions, so it requires validation.
That said, an A/B test does not tell you why something didn't work. You can make further assumptions based on the results and develop new hypotheses, but it never tells you why. Typically you would do some kind of qualitative UX research on a prototype or even static concepts beforehand to identify these kinds of issues before you even expend the effort to do a live A/B test. Far cheaper to do a study with 6-12 people and a prototype than to build out a full, functioning A/B test experience.
It's possible the flow they created was generally better but perhaps it had one fatal flaw. Perhaps that flaw could easily be remedied once identified.
A/B testing is just one small part of a good UX process.
I think the big error in A/B testing is that expectations are quite often very unrealistic. Designers typically have a reasonably good idea about what will work and what will not. Finding 'million dollar buttons' is rare. Of course a couple of percent or even 10's of percents of improvement is nothing to sneeze at. But thinking that by A/B testing forever you're going to make a shrub grow into a tree is imo not realistic. Aside from the detail that a continuously changing user interface is often in itself a barrier to sales.
Ironically, the companies that have benefited most from A/B testing were the ones that were doing a terrible job of it in the first place so then there is lots of low hanging fruit making the consultants look good.
Yet another item often missed: A/B testing success is a direct function of the length of the lever you are pulling. If that lever commands billions of dollars then it is easy to make it pay for itself. But if you're trying to turn $10000 into $11500 then you likely are wasting your time.
"To me, A/B testing means you don't respect your users. You see them as just one factor in your money machine that can be poked and prodded to optimize how much money you can squeeze out of them. "
Your perspective is extremely short-sighted. A/B testing can result in this type of behaviour but that's just poor A/B testing. Good A/B testing focuses on removing distractions from the experience and helping users derive more value from the product. Bad A/B testing tries to make things more discoverable, where discoverability is often just noise and distractions. Good A/B testing ensures that the money machine, as you put it, pays its dues to users by making the product experience delightful.
Some of these fall premises, are actually admitted to be true in the article... wtf?
Also, a lot of the quotes are not so much against a/b test, but people who claim a/b testing is a panacea, and are recommending the wrong tools to the wrong people. (ie tools based around static content to people with entirely dynamically generated content)
basically the biggest problem with a/b testing is that some of it's most noisy proponents have no idea what they're talking about. (this is true of a lot of things)
I understand the frustration, but A/B testing is one of the more objectively tools we have at our disposal.
While there are good arguments against A/B testing UI changes and doing p-hacking, much of the modern web's current UX and UI improvements are in part due to this. How else would we truly know what affects user on a broad scale?
A/B Testing is a way to conduct an experiment. Instrumentation and talking to users is another good way to gain insights, but it not an experiment. They are two different (and often complementary) activities.
Many, many people have successfully used A/B Testing. I've personally used it to great effect several times. I certainly don't make decisions purely based on the statistical results, but I find it to be an extremely useful input to the decision making process. All models are flawed; some are useful.
> If your change is made only to lure people in, not improve their life in any way, mabe you should not even start the A/B testing at all...
Yes. It is totally possible to misuse the tool and run bad tests. That’s why, at least where I work, we don’t run live A/B tests until we’ve done some initial validation that the idea isn’t just terrible, and we have some level of confidence it will work and not harm the user’s experience. For example, we could segment user feedback and customer support issues we received by test variant so we could look for unexpected issues we might have inadvertently caused with the challenger design.
I can’t speak for how A/B testing is used elsewhere, but every A/B test I’ve run has been because we thought it would improve things for the user.
You certainly could run an A/B test to see which banner ad is more deceitful, just like you could use React to build the website that sells the scam medicine that banner ad is advertising.
> Also, to me, the concept of A/B testing certain things may also have an undesired consequence. For example, I order from amazon every day, but today the but button is blue, what does that actually mean? And I go back to the site later and it's yellow again. There are still many people who get confused by seemingly innocuous changes with the way their computer interacts with them.
Proper A/B tests are supposed to be done on a per-unique-user basis. If you access the shop from the same device or user account, a well-done A/B test should consistently show you the same interface.
This would be 100x more useful if it was "Avoiding common pitfalls of A/B testing" -- since the author admits that poor usage of A/B testing makes most of the premises true.
A perfect example of A/B testing and its effects and limitations. The article reports that it is less comfortable for users to use but increases sales. It also reports that this may or may not decrease sales in the long term but in the short-term it works great.
A/B testing is high on hype and promise but low on actual results if you follow it though to actual metrics. I've done various forms of A/B throughout most of my career and found them to be cinsistent with the OP's results.
A much better approach is to install significant instrumentation and actually talk to users about what's wrong with your sign up form.
That, or actually build a product that users want instead of chasing after pointless metrics. I mean, really, you think changing the color of text or a call-out is going to make up for huge deficiencies in your product or make people buy it? The entire premise seems illogical and just doesn't work. The only time I've seen a/b tests truly help was when it accidentally fixed some cross browser issue or moved a button within reach of a user.
Most of the A/B website optimization industry is an elaborate scam, put off on people who don't know any better and are looking for a magic bullet.
A/B tests don't cover everything that entales the user experience design profession. It's about a strategic user-centered process from the beginning of the design. To build a product and then revise without first considering the users throughtfully is a waste of resources and time.
This is complete BS. I run hundreds of a/b tests each quarter and I specifically refuse to run the types of experiments you allude to. My a/b testing is all about helping users achieve the things (the outcomes) that they want to achieve by using our product in the first place. If we can help them do that, with more ease, then we are creating a better experience.
Perhaps you should just agree that, "not all a/b testing is the same".
Despite popular belief, A/B testing doesn't actually work for usability testing. So successful A/B testing usually means bad usability practices.
reply