Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Whom the gods would destroy, they first give real-time analytics (2013)

mcfunley.com

Post reply on HN
Print view View original

51–60 of 66 posts

Re: Whom the gods would destroy, they first give real-time analytics (2013)

#51

Post by Solvency »

jbandela1 wrote: post #44

As I have grown older, I have realized more and more that the most important things can't really be measured directly. Yes you can measure some related things that give you some hints about the thing you care about, but they are fragile. To borrow from Goodhart, if you make the related things a target, they will stop giving you even these hints. This applies not just in software development, but life in general.

Conversion rate and or sales. That's the most important thing. And it can be measured directly.

The problem is analyzing why or why not it's hitting your desired value.

Re: Whom the gods would destroy, they first give real-time analytics (2013)

#52

Post by Solvency »

Ecstatify wrote: post #22
PeterCorless wrote: post #17

This is very 2013. Meanwhile in 2023, a decade later, you literally have systems detecting credit card fraud in milliseconds. [Disclosure: I work for StarTree, which is powered by Apache Pinot. We eat petabytes of data for breakfast.]

What has that to do with product decisions?

Credit cards are products.

Re: Whom the gods would destroy, they first give real-time analytics (2013)

#53

Post by taeric »

tech_ken wrote: post #47
taeric wrote: post #6

Earlier quoted context omitted.

How well does that dodge the problem? I'd imagine a multi armed bandit should stay such that it is always sampling from many fair coins, as it were. I would be delighted to read a study on that.

I can’t say that I did the proof out, but intuitively I would expect the posterior distribution over arm-probabilities would converge to something equal? The other option is spurious convergence to a bad posterior, which could maybe happen with poor sampling techniques, but I can’t imagine it’s more than an edge case

Right, that is what I meant about it should continue to sample from fair coins. I don't know that I've seen experiments to see how long that takes, though.

There is also the question of how long you'd leave multiple treatments out there. Presumably, even if there is no difference in outcomes, there can be benefits to having fewer deployed behaviors.

I'm now also curious if there are non-transitive situations. For example, three treatments together that all act fair if all deployed, but for reasons any two of them deployed alone will show a preference. Ideally, of course, treatments should be done such that this can't happen, but mistakes are often made.

Edit: Fully cede that this is likely chasing edges. The motivation for fewer deployed arms is far more compelling than the edge cases.

Re: Whom the gods would destroy, they first give real-time analytics (2013)

#54

Post by piker »

mcfunley wrote: post #36

Author here. The main thing that inspired this happened a few years before I wrote it down. Etsy had gotten a new CEO, and they spent one of their first few weeks in long hours at my desk, iterating on the homepage design in what could only be described as a radically fast iteration loop. We'd ship a tweak, look at statsd for ten minutes, then change something else. This would have been a bad idea for all of the reas…

Isn’t that last sentence sort of a reason to prefer real-time analytics? If you can make development a fast paced game, no doubt you’ll keep your team more productive and engaged. Granted, it needs to be engineered in a way to ensure that productivity is aimed correctly (“how we decide which things we do”) as you point out in your great article.

Re: Whom the gods would destroy, they first give real-time analytics (2013)

#55

Post by disgruntledphd2 »

abtinf wrote: post #38
bigger_cheese wrote: post #37

Earlier quoted context omitted.

This is something I have been grappling with for a while - I work with data in an industrial plant there is a lot of legacy data systems here and "data latency" has been a real issue - real time dashboards are seen by some subset of management as the holy grail, being able to look at any part of our plant and see what is happening in real time is extremely attractive to certain people. For a longtime there has been a…

I started in manufacturing with a similar view. What turned me around was reading “The Goal” by Goldratt and “Out of the Crisis” by Deming, and seeing just small applications of their principles yield significant results.

Out of the Crisis (and Deming more generally) deserves to be more widely known and quoted.

It's incredible how much of what he says (especially the stuff about how production workers feel) maps directly to the technology industry.

Re: Whom the gods would destroy, they first give real-time analytics (2013)

#56

Post by disgruntledphd2 »

tech_ken wrote: post #48
slotrans wrote: post #24

Earlier quoted context omitted.

I've built a multi-armed bandit system which lived alongside our A/B system. 1. Product didn't have any idea how to interpret its behavior and therefore never made any decisions based on it 2. Experimentation != product design. It's one thing to look at the results of a test, it's another thing to consider patterns of user behavior observed over months or years, which is what Product Analytics is actually for.

How does an typical NHST A/B system resolve this?

You tend to stop while the experiment is running, and then spend time looking at the results once it's done.

The real benefits here are getting a better understanding of what levers drive your product metrics, as you'll inevitably mess up the first n or so experiments (if I could give you only one piece of advice, it would be to use stratified randomisation, but everyone seems to have to make this mistake for themselves).

Re: Whom the gods would destroy, they first give real-time analytics (2013)

#57

Post by teruakohatu »

piker wrote: post #54
mcfunley wrote: post #36

Author here. The main thing that inspired this happened a few years before I wrote it down. Etsy had gotten a new CEO, and they spent one of their first few weeks in long hours at my desk, iterating on the homepage design in what could only be described as a radically fast iteration loop. We'd ship a tweak, look at statsd for ten minutes, then change something else. This would have been a bad idea for all of the reas…

Isn’t that last sentence sort of a reason to prefer real-time analytics? If you can make development a fast paced game, no doubt you’ll keep your team more productive and engaged. Granted, it needs to be engineered in a way to ensure that productivity is aimed correctly (“how we decide which things we do”) as you point out in your great article.

There is a good chance the OP shipped changes that would have positively impacted the bottom line, but after 10 minutes of real time analytics it was replaced with something else because it performed poorly in a single 10 minute period.

You can ship A/B tests quickly and many websites do, but decisions are made after a statistically relevant time period.

Re: Whom the gods would destroy, they first give real-time analytics (2013)

#60

Post by mcfunley »

piker wrote: post #54
mcfunley wrote: post #36

Author here. The main thing that inspired this happened a few years before I wrote it down. Etsy had gotten a new CEO, and they spent one of their first few weeks in long hours at my desk, iterating on the homepage design in what could only be described as a radically fast iteration loop. We'd ship a tweak, look at statsd for ten minutes, then change something else. This would have been a bad idea for all of the reas…

Isn’t that last sentence sort of a reason to prefer real-time analytics? If you can make development a fast paced game, no doubt you’ll keep your team more productive and engaged. Granted, it needs to be engineered in a way to ensure that productivity is aimed correctly (“how we decide which things we do”) as you point out in your great article.

No, it’s orthogonal to the analytics
Post reply on HN