Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Whom the gods would destroy, they first give real-time analytics (2013)

mcfunley.com

Post reply on HN
Print view View original

41–50 of 66 posts

Re: Whom the gods would destroy, they first give real-time analytics (2013)

#41

Post by danparsonson »

> Let’s say you’re testing a text change for a link on your website. Being an impatient person, you decide to do this over the course of an hour. You observe that 20 people in bucket A clicked, but 30 in bucket B clicked. Satisfied, and eager to move on, you choose bucket B.

Horrifying but all too common. A wise man once taught me that humans feel considerable discomfort in the presence of uncertainty, and will tend to jump on the first solution that presents itself. His answer to this was to strive to stay in 'exploration mode' for as long as possible - explore the solution space until you've hit the sides and the back and only then make a decision.

Re: Whom the gods would destroy, they first give real-time analytics (2013)

#42

Post by alex_lav »

sskates wrote: post #35
alex_lav wrote: post #9

Earlier quoted context omitted.

Last I used Amplitude it was insanely expensive. Is that not still the case?

We’re working on it! Free up to 10M events, more to come later this year. What would reasonably costed look like to you? More on my view on real-time analytics here: https://news.ycombinator.com/item?id=15380607

I need to remember that on HN, sometimes the founders of companies I'm talking about will respond :D. I should've used more words.

I don't really know what reasonably costed would've looked like for the time I used Amplitude (2018-2019), but I know that the value my team extracted from it was not commensurate with its cost. Whether that was because of overzealous assumptions on our part, or something else, I don't really know, but I know we signed up, used for a few months and then canceled/downgraded to a cheaper service whose name is escaping me.

I was mostly challenging the "Just use ____" notion of the commenter above, not really that Amplitude is worth the money for correctly-intentioned businesses. Regardless I appreciate the ask.

Re: Whom the gods would destroy, they first give real-time analytics (2013)

#43

Post by 0xferruccio »

tptacek wrote: post #33
0xferruccio wrote: post #31

I think the point of real-time analytics is not to make product decisions but to get a sense of presence from your product and celebrate with your team. As an engineer on many teams shipping features I've found that it's somehow underwhelming to finally launch something after months of work. You launch and the only thing you get to celebrate is some donuts in the office and if something goes wrong a notification from…

If that's all you're going for, you don't need timeliness, comprehensiveness, accuracy, accessibility, performance, and durability.

Agreed 100% you don’t need the same guarantees.

Just saying that as in the conclusion of the article OP says “what do you need this information for?” and my understanding is that the people asking for “real time metrics” aren’t trying to do anything complex but get a pulse of the product

Re: Whom the gods would destroy, they first give real-time analytics (2013)

#44

Post by jbandela1 »

As I have grown older, I have realized more and more that the most important things can't really be measured directly.

Yes you can measure some related things that give you some hints about the thing you care about, but they are fragile. To borrow from Goodhart, if you make the related things a target, they will stop giving you even these hints.

This applies not just in software development, but life in general.

Re: Whom the gods would destroy, they first give real-time analytics (2013)

#45

Post by naijaboiler »

masswerk wrote: post #16

> I can understand why engineers are predisposed to see instantaneous A/B statistics as self-evidently positive This is the crucial misunderstanding: in actuality, you are running a panel. (There is no such thing as an A/B test outside of marketing. Running a meaningful panel requires some information on the population, your samples, the homogeneity of those, etc, just to pick the right test, to begin with. Also, you…

Basically what op did was when you got software developers with little understanding of stats or analytics cosplaying as data scientists

Re: Whom the gods would destroy, they first give real-time analytics (2013)

#46

Post by tech_ken »

whimsicalism wrote: post #15
tech_ken wrote: post #3

If the main objection to constructing a real-time product monitoring system for A/B(C/D/E...) decisions is that optional stopping is bad why not throw away the null-hypothesis sig testing and instead treat the problem as a multi-armed bandit?

Because it is difficult to map that onto real business decisions and requires oftentimes supporting a large space of possible UI combinations because they haven't been fully ruled out yet.

Doesn’t that problem also exist with NHST based A/B testing?

Re: Whom the gods would destroy, they first give real-time analytics (2013)

#47

Post by tech_ken »

taeric wrote: post #6
tech_ken wrote: post #3

If the main objection to constructing a real-time product monitoring system for A/B(C/D/E...) decisions is that optional stopping is bad why not throw away the null-hypothesis sig testing and instead treat the problem as a multi-armed bandit?

How well does that dodge the problem? I'd imagine a multi armed bandit should stay such that it is always sampling from many fair coins, as it were. I would be delighted to read a study on that.

I can’t say that I did the proof out, but intuitively I would expect the posterior distribution over arm-probabilities would converge to something equal? The other option is spurious convergence to a bad posterior, which could maybe happen with poor sampling techniques, but I can’t imagine it’s more than an edge case

Re: Whom the gods would destroy, they first give real-time analytics (2013)

#48

Post by tech_ken »

slotrans wrote: post #24
tech_ken wrote: post #3

If the main objection to constructing a real-time product monitoring system for A/B(C/D/E...) decisions is that optional stopping is bad why not throw away the null-hypothesis sig testing and instead treat the problem as a multi-armed bandit?

I've built a multi-armed bandit system which lived alongside our A/B system. 1. Product didn't have any idea how to interpret its behavior and therefore never made any decisions based on it 2. Experimentation != product design. It's one thing to look at the results of a test, it's another thing to consider patterns of user behavior observed over months or years, which is what Product Analytics is actually for.

How does an typical NHST A/B system resolve this?

Re: Whom the gods would destroy, they first give real-time analytics (2013)

#49

Post by benjaminsky2 »

I’ve always built analytics daily by default, and certain metrics real-time - via a different system - only when a compelling and permanent use case exists (except operational analytics). I mean something like YouTube view counter/likes or delivering order count/total as a b2b2c so your customers can watch their promotions in real-time. Major bonus, you have the super complicated, hard to maintain, real-time system to point to when someone wants everything real-time but invested the minimum possible.

Re: Whom the gods would destroy, they first give real-time analytics (2013)

#50

Post by Solvency »

danparsonson wrote: post #41

> Let’s say you’re testing a text change for a link on your website. Being an impatient person, you decide to do this over the course of an hour. You observe that 20 people in bucket A clicked, but 30 in bucket B clicked. Satisfied, and eager to move on, you choose bucket B. Horrifying but all too common. A wise man once taught me that humans feel considerable discomfort in the presence of uncertainty, and will tend…

This is what you consider horrifying? This seems like an incredibly tame example.
Post reply on HN