Comments on: Where Are the AB Testing Frameworks? http://www.tomkleinpeter.com/2009/01/21/where-are-the-ab-testing-frameworks/ Mon, 30 Apr 2012 05:58:46 +0000 hourly 1 http://wordpress.org/?v=3.1 By: Clark http://www.tomkleinpeter.com/2009/01/21/where-are-the-ab-testing-frameworks/comment-page-1/#comment-22466 Clark Thu, 26 Aug 2010 16:46:48 +0000 http://www.spiteful.com/?p=69#comment-22466 Ronen - noticed this text on your main page: "90% probability this result is statistically significant". That is a nonsensical statement. I think you mean to say the p value was 0.1, which is not considered significant in most cases. But there is no probability of significance, ever. (Note: just commenting on the text -- haven't evaluated the product and don't mean to suggest product is bad because the text is bad. :) Ronen – noticed this text on your main page: “90% probability this result is statistically significant”. That is a nonsensical statement. I think you mean to say the p value was 0.1, which is not considered significant in most cases. But there is no probability of significance, ever.

(Note: just commenting on the text — haven’t evaluated the product and don’t mean to suggest product is bad because the text is bad. :)

]]> By: Ronen Botzer http://www.tomkleinpeter.com/2009/01/21/where-are-the-ab-testing-frameworks/comment-page-1/#comment-14251 Ronen Botzer Mon, 07 Dec 2009 20:10:24 +0000 http://www.spiteful.com/?p=69#comment-14251 It's Vanity, and it's here: http://vanity.labnotes.org/ Sorry for the belated comment, but most likely the answer is still pertinent. It’s Vanity, and it’s here: http://vanity.labnotes.org/

Sorry for the belated comment, but most likely the answer is still pertinent.

]]> By: suhel http://www.tomkleinpeter.com/2009/01/21/where-are-the-ab-testing-frameworks/comment-page-1/#comment-4963 suhel Sun, 05 Apr 2009 05:35:43 +0000 http://www.spiteful.com/?p=69#comment-4963 AS you have said GWO is best tool for ab testing , but i have one doubt, how to do Testing on third party domain like i am testing pages of abc.com but my convesrion page is on XYZ.com which is also belong to me. But in GWO it is clearly mentioned that the coversion domain should be on same domain AS you have said GWO is best tool for ab testing , but i have one doubt, how to do Testing on third party domain like i am testing pages of abc.com but my convesrion page is on XYZ.com which is also belong to me. But in GWO it is clearly mentioned that the coversion domain should be on same domain

]]> By: Infrastructure for Modern Web Sites « random($foo) http://www.tomkleinpeter.com/2009/01/21/where-are-the-ab-testing-frameworks/comment-page-1/#comment-4003 Infrastructure for Modern Web Sites « random($foo) Thu, 29 Jan 2009 04:15:02 +0000 http://www.spiteful.com/?p=69#comment-4003 [...] notably Krow’s Scaling, Systems Required list, but also Tom Kleinpeter’s post asking Where Are the AB Testing Frameworks? reminded me that I had made my own list. I was originally going to start working on these, but [...] [...] notably Krow’s Scaling, Systems Required list, but also Tom Kleinpeter’s post asking Where Are the AB Testing Frameworks? reminded me that I had made my own list. I was originally going to start working on these, but [...]

]]> By: Suhail http://www.tomkleinpeter.com/2009/01/21/where-are-the-ab-testing-frameworks/comment-page-1/#comment-3953 Suhail Fri, 23 Jan 2009 05:49:45 +0000 http://www.spiteful.com/?p=69#comment-3953 Hey Tom, check out http://mixpanel.com, we're sort of invented for people like you. I'd be happy to set you up with an account, just shoot me an email. Hey Tom, check out http://mixpanel.com, we’re sort of invented for people like you. I’d be happy to set you up with an account, just shoot me an email.

]]> By: Greg Dingle http://www.tomkleinpeter.com/2009/01/21/where-are-the-ab-testing-frameworks/comment-page-1/#comment-3944 Greg Dingle Thu, 22 Jan 2009 20:04:40 +0000 http://www.spiteful.com/?p=69#comment-3944 I think there is a good reason why systematic experimentation has not become more common in web application development, despite individual successes (amazon.com, etc) and the general ease of collecting data. It is ultimately very important for ease-of-use that the alternatives being tested in a system are stored in the system itself. Otherwise you run into problems of consistency and control. It is the principle of DRY, code reuse, single authoritative source. Some companies are A/B testing by swapping outgoing text at the HTTP server level. This can't be the way of the future! I think there is a good reason why systematic experimentation has not become more common in web application development, despite individual successes (amazon.com, etc) and the general ease of collecting data.

It is ultimately very important for ease-of-use that the alternatives being tested in a system are stored in the system itself. Otherwise you run into problems of consistency and control. It is the principle of DRY, code reuse, single authoritative source.

Some companies are A/B testing by swapping outgoing text at the HTTP server level. This can’t be the way of the future!

]]> By: wac http://www.tomkleinpeter.com/2009/01/21/where-are-the-ab-testing-frameworks/comment-page-1/#comment-3942 wac Thu, 22 Jan 2009 17:35:56 +0000 http://www.spiteful.com/?p=69#comment-3942 You've definitely hit on something here. People seem to vastly underestimate the impact seemingly simple changes like color can make in usage (and revenue). There is room for something more here though. You can derive some great value out of briefly presenting a few random users/sessions with a moderately different UI to see the impact on usage. I suspect the frameworks for those sort of experiments would wind up being pretty specific to the larger toolchain people are tied to though. (The least of which being the need to tie into logging and export the experiments that were live for a given page that was served.) Not just for UIs: There is also good reason at times to do A/B performance testing of code that is suitably non-deterministic that it's easier model with a fraction of live traffic than with some heavyweight testing framework. You’ve definitely hit on something here. People seem to vastly underestimate the impact seemingly simple changes like color can make in usage (and revenue).

There is room for something more here though. You can derive some great value out of briefly presenting a few random users/sessions with a moderately different UI to see the impact on usage. I suspect the frameworks for those sort of experiments would wind up being pretty specific to the larger toolchain people are tied to though. (The least of which being the need to tie into logging and export the experiments that were live for a given page that was served.)

Not just for UIs: There is also good reason at times to do A/B performance testing of code that is suitably non-deterministic that it’s easier model with a fraction of live traffic than with some heavyweight testing framework.

]]> By: Vanessa Pagan http://www.tomkleinpeter.com/2009/01/21/where-are-the-ab-testing-frameworks/comment-page-1/#comment-3940 Vanessa Pagan Thu, 22 Jan 2009 17:26:41 +0000 http://www.spiteful.com/?p=69#comment-3940 I wish more startup businesses paid attention to split testing. It's the best way to find out what works best for your visitors and in turn you business. Thanks for the link to the pdf research article on split testing. I created a mindmap of how to document a split test so that you can keep track of your experiments and results. Just having a process to follow with Website Optimizer is helpful for small companies to manage their testings for better conversions. Here's the mind map: http://phantomcto.com/blog/business-tech/how-to-document-split-test/ I wish more startup businesses paid attention to split testing. It’s the best way to find out what works best for your visitors and in turn you business. Thanks for the link to the pdf research article on split testing. I created a mindmap of how to document a split test so that you can keep track of your experiments and results.

Just having a process to follow with Website Optimizer is helpful for small companies to manage their testings for better conversions. Here’s the mind map: http://phantomcto.com/blog/business-tech/how-to-document-split-test/

]]> By: Tom http://www.tomkleinpeter.com/2009/01/21/where-are-the-ab-testing-frameworks/comment-page-1/#comment-3934 Tom Thu, 22 Jan 2009 04:36:18 +0000 http://www.spiteful.com/?p=69#comment-3934 @Greg -- thanks for the link, that looks really cool. @Philip -- Ah, but when has that ever stopped people from writing a framework :) I think there is a whole management system (to see test results, create new ones, temporarily disable current ones) that could be common. @Greg — thanks for the link, that looks really cool.

@Philip — Ah, but when has that ever stopped people from writing a framework :) I think there is a whole management system (to see test results, create new ones, temporarily disable current ones) that could be common.

]]> By: Ethan Herdrick http://www.tomkleinpeter.com/2009/01/21/where-are-the-ab-testing-frameworks/comment-page-1/#comment-3932 Ethan Herdrick Thu, 22 Jan 2009 04:25:25 +0000 http://www.spiteful.com/?p=69#comment-3932 Greg, thanks. Genetify looks fantastic. Greg, thanks. Genetify looks fantastic.

]]>