(2013)
Exactly. And it betrays the biasea of the era. This author really got it wrong.
Whom the gods would destroy, they first give real-time analytics (2013)
21–30 of 66 posts
Re: Whom the gods would destroy, they first give real-time analytics (2013)
#22This is very 2013. Meanwhile in 2023, a decade later, you literally have systems detecting credit card fraud in milliseconds. [Disclosure: I work for StarTree, which is powered by Apache Pinot. We eat petabytes of data for breakfast.]
What has that to do with product decisions?
Re: Whom the gods would destroy, they first give real-time analytics (2013)
#23Them: We need metrics to know if the users like this new feature we're pushing on them.
Me: or you know, we could maybe see what the users biggest issues are first and try to build stuff to solve those problems.
Re: Whom the gods would destroy, they first give real-time analytics (2013)
#24If the main objection to constructing a real-time product monitoring system for A/B(C/D/E...) decisions is that optional stopping is bad why not throw away the null-hypothesis sig testing and instead treat the problem as a multi-armed bandit?
I've built a multi-armed bandit system which lived alongside our A/B system.
1. Product didn't have any idea how to interpret its behavior and therefore never made any decisions based on it
2. Experimentation != product design. It's one thing to look at the results of a test, it's another thing to consider patterns of user behavior observed over months or years, which is what Product Analytics is actually for.
Re: Whom the gods would destroy, they first give real-time analytics (2013)
#25This is very 2013. Meanwhile in 2023, a decade later, you literally have systems detecting credit card fraud in milliseconds. [Disclosure: I work for StarTree, which is powered by Apache Pinot. We eat petabytes of data for breakfast.]
I wonder if these are the same systems that "detect fraud" and freeze my bank account requiring manual intervention to fix the 2 times a year I send a random family member less than $2,000
Re: Whom the gods would destroy, they first give real-time analytics (2013)
#26> I can understand why engineers are predisposed to see instantaneous A/B statistics as self-evidently positive This is the crucial misunderstanding: in actuality, you are running a panel. (There is no such thing as an A/B test outside of marketing. Running a meaningful panel requires some information on the population, your samples, the homogeneity of those, etc, just to pick the right test, to begin with. Also, you…
Data scientists also do A/B testing on algorithms to see which one has better fit for a use case against real-world, real-time data.
I mean, if done right – there are dedicated sciences for this –, it's a panel with two samples. To me, the notion of an A/B test is all about tossing the scientific basics over board, in order to get a rough estimate, which we will call good enough. However, there are all those statistical methods, meant to even out the bumps in the road that you will encounter, e.g, if you're running your load-balancer performance A/B test at 2 am versus running it 5 pm, or running it on a Sunday versus running it on a Tuesday.
(In this specific case, as a data analyst, you will probably have an intricate understanding of your population, i.e. your data, the structure of the samples you're running against the algorithms, which you have tailored according to this understanding in the first place. However, while we may assume the best, this may still be what's called pre-scientific knowledge in statistical terms.)
Re: Whom the gods would destroy, they first give real-time analytics (2013)
#27Re: Whom the gods would destroy, they first give real-time analytics (2013)
#28I suspect a lot of bad design comes from a myopic focus on The Analytics. Yes, there were a lot of failures in the bad old days due to Big Design Up Front and only iterating once a year after you ship 1.0, but we've gone to the opposite extreme. A lot of organizations seem to just jerk and twitch as the raw numeric results come in, with no central cognition.
Build for the future you want, don't just head for the local minima. Take risks and damn the statistics.
Re: Whom the gods would destroy, they first give real-time analytics (2013)
#29This is very 2013. Meanwhile in 2023, a decade later, you literally have systems detecting credit card fraud in milliseconds. [Disclosure: I work for StarTree, which is powered by Apache Pinot. We eat petabytes of data for breakfast.]
I'm not sure if you actually read the article with this answer. The article is explicitly talking about using real time data to make design decisions, detecting something like credit card fraud is completely different problem space.
Re: Whom the gods would destroy, they first give real-time analytics (2013)
#30Massive fan of this piece. I linked it in another comment yesterday.
The place its truth has been most obvious to me is in analyzing subscription businesses. Your customers pay once a month, or once a year. Nothing, absolutely nothing you do on a minute-by-minute basis is relevant to strategic business decision making. Yet these businesses will invariably want real-time analytics. It serves no purpose! You simply cannot look at your fancy real-time dashboard, and then take action based on it on that same time scale. Meanwhile it costs you 10x what daily data would.