No, it's not A/B testing for the reasons I try to explain the in the post. A/B testing can't change the choices while the experiment is running, doesn't adjust to customer preferences in real-time, etc.
You can optimise for revenue. Button presses was just a simple example.
Before you get too far, try an A/A test. By this, I mean let the A and B choices be identical. You would certainly expect the outcomes to be equal. Right?
I have seen statistically significant differences in outcomes in A/A testing.
A/B testing has value but being sure to A/A test may temper your expectations and/or point at problems in your setup before you get too far.
A/B Testing is a way to conduct an experiment. Instrumentation and talking to users is another good way to gain insights, but it not an experiment. They are two different (and often complementary) activities.
Many, many people have successfully used A/B Testing. I've personally used it to great effect several times. I certainly don't make decisions purely based on the statistical results, but I find it to be an extremely useful input to the decision making process. All models are flawed; some are useful.
I doubt there are companies exclusively rely on A/B tests to make decisions, but even if there were, someone has to think about hypothesis and design the right experiment to then confirm with data. You can't just have a computer running around and make experiments and confirm them for youº.
--
º: you kind of can, if you have enough fresh data, to search for and derive statistically significant results automatically.
A/B testing is great, but your data is severely limited. How do you know that sticking with the vision doesn't pay off in the long run? Unless you've been running A/B tests for years with doppleganger-Patrick who eschews the cold, mechanical methods of A/B testing, writes blog posts about how important gut-feeling is in web design, and made every page of BCC have yellow text on a green background because dammit he likes John Deere, how could you know that you're better off now? We know a lot less than we think we do, and running some limited tests doesn't change the fact that most of the time we're just guessing.
Getting statistically useful data out of this will be difficult. A/B tests can tell you which of a set if options performs better for a measurable metric, but not why, and without no visibility into 'metrics' you can't easily measure -- of which there are a lot.
Generally, you can only generate data and do experiments based on your present state.
reply