Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Results from Candy Japan box design A/B test

candyjapan.com

Post reply on HN
Print view View original

91–98 of 98 posts

Re: Results from Candy Japan box design A/B test

#91

Post by michaelbuddy »

keiferski wrote: post #4

Have you considered that the new box is too "generic" or "corporate?" (For lack of a better word) In other words, the old box, while plain, adds a certain intimacy. It doesn't look like a huge multinational candy company sent it. The new box, though, looks basically like any other piece of Japanese media.

I often think like this, then I'm reminded how infomercials are so obnoxious and yet they still work. So the conundrum of personal taste and a good gut instinct vs what works with a lot of people.

was the old box that blue plain box? It's not intimate to me. Compared to the new one, it looks kind of homely

Re: Results from Candy Japan box design A/B test

#93

Post by gwern »

jupiter90000 wrote: post #60
Medea wrote: post #57

Just out of curiosity, which statistical test should be used in this case?

Some decent answers on Quora: https://www.quora.com/What-statistical-test-should-be-used-f...

Those answers are bizarre. Use a t-test or z-test?! That is both less interpretable* and less efficient than doing a test of difference in proportions (based on a chi-squared) or better yet, doing a generalized linear regression with a binomial link which lets you test multiple interventions and include covariates and other things. These are little harder than using a t-test and work much better.

(Of course, even better would be a Bayesian decision analysis which takes into account the full posterior distribution, costs, and benefits, allowing one to decide whether the evidence is enough to justify the more expensive box or whether one should extend the experiment further to collect more data.)

* quick, if conversions are 0.25%, what does a 0.1SD mean difference between arms of an A/B test mean? 'Uh...' Exactly.

Re: Results from Candy Japan box design A/B test

#94

Post by jupiter90000 »

gwern wrote: post #93
jupiter90000 wrote: post #60

Earlier quoted context omitted.

Some decent answers on Quora: https://www.quora.com/What-statistical-test-should-be-used-f...

Those answers are bizarre. Use a t-test or z-test?! That is both less interpretable* and less efficient than doing a test of difference in proportions (based on a chi-squared) or better yet, doing a generalized linear regression with a binomial link which lets you test multiple interventions and include covariates and other things. These are little harder than using a t-test and work much better. (Of course, even bet…

To be fair, a couple answers there mentioned the Chi-squared approach, and a z-test could also be appropriate for testing a difference in proportions. I think your Bayesian approach would make the most sense for this kind of stuff, to examine the posterior especially. There's other things that could be done than what you mentioned too to assess a difference. I'm not convinced the parametric assumptions made by the standard tests mentioned (including difference in proportions, whether a z-test or Chi-square test) would hold up well for this kind of data, it would take further examination. I think a z-test or Chi-square test are a decent starting point, and could be fine to use if the assumptions made for those tests are met (to defend some of the Quora answers). The issue is often tests are used without folks either knowing how to check or ignoring the underlying assumptions, but that isn't an issue with the test itself I'd argue.

Re: Results from Candy Japan box design A/B test

#95

Post by gwern »

jupiter90000 wrote: post #94
gwern wrote: post #93

Earlier quoted context omitted.

Those answers are bizarre. Use a t-test or z-test?! That is both less interpretable* and less efficient than doing a test of difference in proportions (based on a chi-squared) or better yet, doing a generalized linear regression with a binomial link which lets you test multiple interventions and include covariates and other things. These are little harder than using a t-test and work much better. (Of course, even bet…

To be fair, a couple answers there mentioned the Chi-squared approach, and a z-test could also be appropriate for testing a difference in proportions. I think your Bayesian approach would make the most sense for this kind of stuff, to examine the posterior especially. There's other things that could be done than what you mentioned too to assess a difference. I'm not convinced the parametric assumptions made by the st…

> I'm not convinced the parametric assumptions made by the standard tests mentioned (including difference in proportions, whether a z-test or Chi-square test) would hold up well for this kind of data, it would take further examination.

I'm not sure what you mean. If all you have is binary data, there's not much you can do but a chi-squared test or binomial regression. As I understand it, a chi-squared is considered nonparametric: https://en.wikipedia.org/wiki/Chi-squared_test https://en.wikipedia.org/wiki/Fisher's_exact_test

Re: Results from Candy Japan box design A/B test

#96

Post by jupiter90000 »

gwern wrote: post #95
jupiter90000 wrote: post #94

Earlier quoted context omitted.

To be fair, a couple answers there mentioned the Chi-squared approach, and a z-test could also be appropriate for testing a difference in proportions. I think your Bayesian approach would make the most sense for this kind of stuff, to examine the posterior especially. There's other things that could be done than what you mentioned too to assess a difference. I'm not convinced the parametric assumptions made by the st…

> I'm not convinced the parametric assumptions made by the standard tests mentioned (including difference in proportions, whether a z-test or Chi-square test) would hold up well for this kind of data, it would take further examination. I'm not sure what you mean. If all you have is binary data, there's not much you can do but a chi-squared test or binomial regression. As I understand it, a chi-squared is considered n…

Yes, you're right that chi-squared is non-parametric, my mistake. However, even non-parametric tests can have assumptions that should probably be checked into at least for a given data set. It may be robust to the violations the data collected may exhibit, I'm not sure. This article lists the assumptions for chi-squared test: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3900058/

Re: Results from Candy Japan box design A/B test

#97

Post by mrmondo »

meric wrote: post #75
jtreminio wrote: post #9

Earlier quoted context omitted.

> - The picture itself is kind of creepy, someone walking past my desk once even described it as 'rapey'. This person may have some deeper issues than just the innocent picture on this box.

If we replaced all pictures of females on candy boxes with males I would hazard a guess that someone would still have something to say about it.

Well put.
Post reply on HN