I read news.yc and reddit/programming pretty regularly to keep up with what is going on in the biz. Based on that reading, I can probably name a dozen different systems for building high scale applications (distributed storage, message queues, caching layers, search engines, etc), but I can’t name a single AB testing framework other than Google Website Optimizer. That seems like a serious inversion of priorities for most startups. Everyone with a sign up page should use AB testing. Not everyone needs a message queue.
Is this because:
- Nobody needs anything other than Google Website Optimizer?
- Startups don’t actually do AB testing, possibly because they don’t get enough traffic to get meaningful results, or maybe because they don’t have time?
- AB testing (including the statistical analysis to determine if results are valid) is so simple that everyone just bangs out their own?
- As a largely theoretical issue for most startups, scalability is more fun to talk about on the Internet?
- Everyone that is using AB testing is so happy that they are trying to suppress information about it so their competitors don’t start doing it too?
If everyone is secretly using some great framework please shoot me an email and let me know.
If you haven’t thought much about it before, here is a short paper on AB testing from some folks that made Amazon a ton of money.