Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Winning A/B results were not translating into improved user acquisition

blog.sumall.com

Post reply on HN
Print view View original

21–30 of 67 posts

Re: Winning A/B results were not translating into improved user acquisition

#21

Post by pluma »

JacobSumAll wrote: post #6
antr wrote: post #4

Note on SumAll All users who use SumAll should be wary of their service. We tried them out and we then found out that they used our social media accounts to spam our followers and users with their advertising. We contacted them asking for answers and we never heard from them. Our suggestion: Avoid SumAll.

Hey Antr, Jacob from SumAll here. Sorry to hear you had a bad experience with us. The tweets you're talking about that "spam" your accounts were most likely the performance tweets that you are free to toggle on and off. Here's how you can do that: https://support.sumall.com/customer/portal/articles/1378662-... Best, Jacob

As the tweets contain both SumAll-related hash tags and Links to SumAll, this is definitely marketing that should be opt-in, not opt-out. Unless the user of your service is explicitly made aware of these automated tweets in clear terms when they sign up, this is a bit shady and dishonest to say the least.

Re: Winning A/B results were not translating into improved user acquisition

#22

Post by throwawayornot »

spacefight wrote: post #11
JacobSumAll wrote: post #6

Earlier quoted context omitted.

Hey Antr, Jacob from SumAll here. Sorry to hear you had a bad experience with us. The tweets you're talking about that "spam" your accounts were most likely the performance tweets that you are free to toggle on and off. Here's how you can do that: https://support.sumall.com/customer/portal/articles/1378662-... Best, Jacob

This is opt-out? Srsly?

maybe it is a revenue stream for them?

Re: Winning A/B results were not translating into improved user acquisition

#24

Post by pocp2 »

pmiller2 wrote: post #18

The red flag here for me was that Optimizely encourages you to stop the test as soon as it "reaches significance." You shouldn't do that. What you should do is precalculate a sample size based on the statistical power you need, which involves determining your tolerance for the probability of making an error and on the minimum effect size you need to detect. Then, you run the test to completion and crunch the numbers…

For a second I thought you were Evan Miller who wrote about the exact same thing: http://www.evanmiller.org/how-not-to-run-an-ab-test.html

Re: Winning A/B results were not translating into improved user acquisition

#26

Post by spacefight »

pluma wrote: post #21
JacobSumAll wrote: post #6

Earlier quoted context omitted.

Hey Antr, Jacob from SumAll here. Sorry to hear you had a bad experience with us. The tweets you're talking about that "spam" your accounts were most likely the performance tweets that you are free to toggle on and off. Here's how you can do that: https://support.sumall.com/customer/portal/articles/1378662-... Best, Jacob

As the tweets contain both SumAll-related hash tags and Links to SumAll, this is definitely marketing that should be opt-in, not opt-out. Unless the user of your service is explicitly made aware of these automated tweets in clear terms when they sign up, this is a bit shady and dishonest to say the least.

Even if it's in the terms - do it opt-in.

Re: Winning A/B results were not translating into improved user acquisition

#27

Post by wmeredith »

throwawayornot wrote: post #22
spacefight wrote: post #11

Earlier quoted context omitted.

This is opt-out? Srsly?

maybe it is a revenue stream for them?

And...? I'm sure it is. It markets their product at the expense of their user's credibility with their social circles. There's no downside! (For Sumall)

Re: Winning A/B results were not translating into improved user acquisition

#28

Post by [deleted] »

pmiller2 wrote: post #18

The red flag here for me was that Optimizely encourages you to stop the test as soon as it "reaches significance." You shouldn't do that. What you should do is precalculate a sample size based on the statistical power you need, which involves determining your tolerance for the probability of making an error and on the minimum effect size you need to detect. Then, you run the test to completion and crunch the numbers…

[deleted]

Re: Winning A/B results were not translating into improved user acquisition

#29

Post by ssharp »

It seems like I see these articles pop up on a regular basis over at Inbound or GrowthHackers.

I think the problem is two-sided: one on the part of the tester and one on the part of the tools. The tools "statistically significant" winners MUST be taken with a grain of salt.

On the user side, you simply cannot trust the tools. To avoid these pitfalls, I'd recommend a few key things. One, know your conversion rates. If you're new to a site and don't know patterns, run A/A tests, run small A/B tests, dig into your analytics. Before you run a serious A/B test, you'd better know historical conversion rates and recent conversion rates. If you know your variances, it's even better, but you could probably heuristically understand your rate fluctuations just by looking at analytics and doing A/A test. Two, run your tests for long after you get a "winning" result. Three, have the traffic. If you don't have enough traffic, your ability to run A/B tests is greatly reduced and you become more prone to making mistakes because you're probably an ambitious person and want to keep making improvements! The nice thing here is that if you don't have enough traffic to run tests, you're probably better off doing other stuff anyway.

On the tools side (and I speak from using VWO, not Optimizely, so things could be different), but VWO tags are on all my pages. VWO knows what my goals are. Even if I'm not running active tests on pages, why can't they collect data anyway and get a better idea of what my typical conversion rates are? That way, that data can be included and considered before they tell me I have a "winner". Maybe this is nitpicky, but I keep seeing people who are actively involved in A/B testing write articles like this, and I have to think the tools could do a better job in not steering intermediate-level users down the wrong path, let alone novice users.

Re: Winning A/B results were not translating into improved user acquisition

#30

Post by wmt »

pmiller2 wrote: post #18

The red flag here for me was that Optimizely encourages you to stop the test as soon as it "reaches significance." You shouldn't do that. What you should do is precalculate a sample size based on the statistical power you need, which involves determining your tolerance for the probability of making an error and on the minimum effect size you need to detect. Then, you run the test to completion and crunch the numbers…

"Honestly, in a website A/B test, all I really am concerned about is whether my new page is better than the old page. A one-tailed test tells you that."

No, it's the other way around. One tailed test is only usable for testing if the new design worse than the old one, because it being better than the old one does not matter as long it's not worse. If you are testing that is the new design better, you definitely need to test both tails or else you may likely switch to a worse design than the old one.

Post reply on HN