Though I understand the idea, I'm not too familiar with how A/B tests are performed in practice. I would've assumed you'd need a lot of data to get statistically significant results. Is the volume of data that you need to achieve that statistical significance well-understood and do you actually have the traffic to collect that amount of data? (I guess I imagined that it would require huge amounts of data.)
you could look at a/b testing. it might be interesting to learn how to set that up... (not sure how you'd use it to measure this particular case though)
agreed. was thinking the same thing but you beat me to it. anybody who has ever evaluated an A/B test would look at this and immediately question the significance/validity of the result. if only somebody would actually back up their claim with ALL the data for once (i.e. variance and specific sample sizes)
A/B tests are only useful when you can gather statistically significant amounts of data from them. For a lot of small websites, or infrequently used features on larger websites, that is not the case.
reply