Live data from Hacker News

Winning A/B results were not translating into improved user acquisition

blog.sumall.com

11–20 of 67 posts

Re: Winning A/B results were not translating into improved user acquisition

#11
post #4

Note on SumAll All users who use SumAll should be wary of their service. We tried them out and we then found out that they used our social media accounts to spam our followers and users with their advertising. We contacted them asking for answers and we never heard from them. Our suggestion: Avoid SumAll.

Hey Antr, Jacob from SumAll here. Sorry to hear you had a bad experience with us. The tweets you're talking about that "spam" your accounts were most likely the performance tweets that you are free to toggle on and off. Here's how you can do that: https://support.sumall.com/customer/portal/articles/1378662-... Best, Jacob

This is opt-out? Srsly?

Re: Winning A/B results were not translating into improved user acquisition

#12
Oh great, another misuse of A/B testing

Here's the thing, stop A/Bing every little thing (and/or "just because") and you'll get more significant results.

Do you think the true success of something is due to A/B testing? A/B testing is optimizing, not archtecting.

Re: Winning A/B results were not translating into improved user acquisition

#13
post #4

Note on SumAll All users who use SumAll should be wary of their service. We tried them out and we then found out that they used our social media accounts to spam our followers and users with their advertising. We contacted them asking for answers and we never heard from them. Our suggestion: Avoid SumAll.

Hey Antr, Jacob from SumAll here. Sorry to hear you had a bad experience with us. The tweets you're talking about that "spam" your accounts were most likely the performance tweets that you are free to toggle on and off. Here's how you can do that: https://support.sumall.com/customer/portal/articles/1378662-... Best, Jacob

So SumAll spams your followers by default but you can turn it off if you know how?

Re: Winning A/B results were not translating into improved user acquisition

#15

This article comes off as a bit boastful and somewhat of an advertisement for the company... "What threw a wrench into the works was that SumAll isn’t your typical company. We’re a group of incredibly technical people, with many data analysts and statisticians on staff. We have to be, as our company specializes in aggregating and analyzing business data. Flashy, impressive numbers aren’t enough to convince us that th…

They are incredible as in literally not credible.

Re: Winning A/B results were not translating into improved user acquisition

#17

this is all fine and good, but if you're goal is to see what works best between X new versions of a page and you are rigorous in creating variants, Optimizely is a great tool for figuring out the best converting variant.

Except, apparently, they aren't actually that good at _that_. If an A/A test to not yield 100% chance of 18% uplift, what gives you any degree of certainty that other tests won't have equally skewed results?

Re: Winning A/B results were not translating into improved user acquisition

#18
The red flag here for me was that Optimizely encourages you to stop the test as soon as it "reaches significance." You shouldn't do that. What you should do is precalculate a sample size based on the statistical power you need, which involves determining your tolerance for the probability of making an error and on the minimum effect size you need to detect. Then, you run the test to completion and crunch the numbers afterward. This helps prevent the scenario where your page tests 18% better than itself by minimizing probability that your "results" are just a consequence of a streak of positive results in one branch of the test.

I was also disturbed that the effect size was taken into account in the sample size selection. You need to know this before you do any type of statistical test. Otherwise, you are likely to get "positive" results that just don't mean anything.

OTOH, I wasn't too concerned that the test was a one-tailed test. Honestly, in a website A/B test, all I really am concerned about is whether my new page is better than the old page. A one-tailed test tells you that. It might be interesting to run two-tailed tests just so you can get an idea what not to do, but for this use I think a one-tailed test is fine. It's not like you're testing drugs, where finding any effect, either positive or negative, can be valuable.

I should also note that I only really know enough about statistics to not shoot myself in the foot in a big, obvious way. You should get a real stats person to work on this stuff if your livelihood depends on it.

Re: Winning A/B results were not translating into improved user acquisition

#20
post #4

Note on SumAll All users who use SumAll should be wary of their service. We tried them out and we then found out that they used our social media accounts to spam our followers and users with their advertising. We contacted them asking for answers and we never heard from them. Our suggestion: Avoid SumAll.

Hey Antr, Jacob from SumAll here. Sorry to hear you had a bad experience with us. The tweets you're talking about that "spam" your accounts were most likely the performance tweets that you are free to toggle on and off. Here's how you can do that: https://support.sumall.com/customer/portal/articles/1378662-... Best, Jacob

http://darkpatterns.org/friend-spam/
Post reply on HN