20 lines of code that beat A/B testing (2012) • Hacker News Board

Re: 20 lines of code that beat A/B testing (2012)

#161

Post by tripzilch » Fri, Apr 08, 2016, 3:52 PM UTC

ves wrote: post #158

tripzilch wrote: post #153

Earlier quoted context omitted.

Yes they're very good at "seeming" the part. Until you check the cites/references. Please excuse me for lumping Kahneman, Tversky and Taleb into the same bin (stating beforehand, feel free to dismiss or form your own opinions because of this). My justification is they cite eachother all the time, write about the same topics and are quoted as doing their research by "bouncing ideas off each other" (only to later dig u…

You should read silent risk (currently freely available as a draft on talebs website). It's mathematical, with solid proofs and derivations, so it doesn't really suffer from the same problems as The Black Swan. I had basically the same issues with the Black Swan that you did.

Thanks for the tip, I'll check it out! On that note, I also read Fooled by Randomness, an earlier work by Nicholas Taleb. It covers (somewhat) similar topics as Black Swan does, but from a bit more technical perspective. I found it a lot more pleasant and educational read. IIRC, it talks less about the financial world and rare catastrophal risks ("Black Swans"), and more about how humans are wired to reason intuitively quite well about certain types of probability/estimates/combinatorics, and particularly on how we really suck at certain others.

Fooled by Randomness also doesn't needlessly bash people for wearing ties or being French, like Black Swan does :) ... only later did I learn that Taleb was a personal friend of Benoit Mandelbrot (who was French), which put it in a bit more context (as a friendly poke, I'm assuming), but at the time I found it a bit off-putting and weird, as it really had nothing to do with the subject matter at all.

Re: 20 lines of code that beat A/B testing (2012)

#162

Post by scottlocklin » Fri, Apr 08, 2016, 4:10 PM UTC

gburt wrote: post #134

scottlocklin wrote: post #133

Earlier quoted context omitted.

I'm not sure which thing I mentioned you need a reference on. For p-values on machine learning techniques, http://vovk.net/cp/index.html I'll eventually do a blog post on this subject; it's very good math that all ML people should know about, though Vovk, Schaefer and Gammerman write pretty dense articles. For statistical power... "Statistical Power Analysis for the Behavioral Sciences" by Jacob Cohen.

Sorry, I intended to quote your last sentence, applying a confidence interval to a reinforcement learning system, especially with respect to multi-armed bandits / adaptive experiments, but if I have to dig in to some signal processing stuff I am happy to do that. It seems the conformal prediction link has some relation. I will dig, thanks.

Yes, conformal prediction does exactly this.

Re: 20 lines of code that beat A/B testing (2012)

#163

Post by harperlee » Tue, Apr 12, 2016, 2:28 PM UTC

searine wrote: post #150

squeaky-clean wrote: post #116

Earlier quoted context omitted.

I actually think that's a great idea, because then the users and algorithm decide what's good for you, and it will hide the low quality content automatically. You have to do that sort of "curation" anyways anytime you make something. You have to continually decide whether it's worth it to keep working on something, and then decide if it's good enough to release. People tend to be pretty bad judges of this (especially…

"I promise, there is a diamond somewhere in this bathtub of shit". Efficiency is great, but there still needs to be some quality standards.

Most probably the input is not random words, and instead orasis writes pieces aiming to have quality. He just doesn't bleed each and every word and accepts a maximum time per text piece, accepting that some variation in quality is inescapable, and that it is most efficient to allocate less time per piece so he gets more pieces. Note that this "less time" is relative, and probably does not go to zero. It's just orasis has found a level of effort per piece that is effective, and he relies on the algorithm so he does not need to establish a threshold below which he will not publish - as those pieces will be swallowed by the rest of higher quality corpus.

He might have leaned toward a little bit of hyperbole in "just spew out content regardless of its quality" for the purposes of clarity and expressiveness, which most of the readers have correctly parsed, but I don't think you have the right to tag orasis' output as shit based solely on the information of the page. That's a little bit rude, in fact.

Re: 20 lines of code that beat A/B testing (2012)

#164

Post by gingerlime » Tue, Apr 12, 2016, 8:33 PM UTC

norkakn wrote: post #135

gingerlime wrote: post #128

Earlier quoted context omitted.

> but they never really published much justifying it Not that I'm trying to defend Optimizely (I'm not a huge fan, but for other reasons...). I can't vouch for the quality either, but they did publish something about it[0] - that at least looks quite scientific. Happy to read any critique of course. [0] http://pages.optimizely.com/rs/optimizely/images/stats_engin...

Latex is a wonderful way to make a marketing paper look like a scientific one. It doesn't accurately describe the method, but that isn't really its purpose. It's a more technical description of the blog post, meant for people using the product to understand some of the tradeoffs and get more accurate results. They are still having people make very fundamentally flawed assumptions about the data, which results in inco…

I was tempted to make a snarky comment about using LaTeX, but I'm not sure it's entirely fair. It doesn't seem like just a bunch of MarketingSpeak wrapped in LaTeX to be honest.

My issue with Optimizely are mainly how they essentially ditched us as (paying) customers. We were admittedly small-fish, but we were paying and were willing to pay more, but they switched to Enterprise-vs-Free without any middle grounds. Enterprise was way too expensive for us. Free didn't include essential features, so we were just stuck.

I ended up writing an open-source javascript A/B test client[0] (and recently also an AWS-lambda backend[1]), but it still has a way to go...

[0] https://github.com/Alephbet/alephbet

[1] https://github.com/Alephbet/gimel