(2013)
Whom the gods would destroy, they first give real-time analytics (2013)
11–20 of 66 posts
Re: Whom the gods would destroy, they first give real-time analytics (2013)
#12While there are probably all sorts of problems with marxism when it comes to economics, in large companies there should be a 'vanguard party' of statisticians who prevent the masses from making false claims of causality from p-hacked tests.
Re: Whom the gods would destroy, they first give real-time analytics (2013)
#13I believe that the comment about CAP theorem violation / treating the problem as a technically unsolved thing isn't true. Eg. See the dataflow paper that sets up more clear tradeoffs for latency and correctness in large scale data processing [1]. I think it makes sense to always hold a high bar for your technology -- if it's technically feasible, and fits within budgets (time and complexity for the team), accepting artificial limitations because they soften social problems feels like a mistake / believing in a false "ignorance is bliss" belief. I think the problem that is presented is more of a problem of popular understanding of statistics and game theory, and not the technical problem.
[1] "The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing" https://research.google/pubs/pub43864/
Re: Whom the gods would destroy, they first give real-time analytics (2013)
#14I really liked this article, and I thought this statement hit the nail on the head: "Confusing how we do things with how we decide which things to do is a fatal mistake." I've worked at companies that practice what I call "thrash management" (constantly jumping from one priority to the next based on whichever fire happens to be burning brightest that day) and it's no fun, to put it mildly.
That said, once you build a system for operational metrics (i.e. what you need to detect anomalies that indicate outages, security concerns, etc.) you're already a huge way there towards having real-time analytics. I still wholeheartedly agree with the author that these real time metrics should only be in the service of operations, not product planning.
Re: Whom the gods would destroy, they first give real-time analytics (2013)
#15If the main objection to constructing a real-time product monitoring system for A/B(C/D/E...) decisions is that optional stopping is bad why not throw away the null-hypothesis sig testing and instead treat the problem as a multi-armed bandit?
Because it is difficult to map that onto real business decisions and requires oftentimes supporting a large space of possible UI combinations because they haven't been fully ruled out yet.
Re: Whom the gods would destroy, they first give real-time analytics (2013)
#16> I can understand why engineers are predisposed to see instantaneous A/B statistics as self-evidently positive
This is the crucial misunderstanding: in actuality, you are running a panel.
(There is no such thing as an A/B test outside of marketing. Running a meaningful panel requires some information on the population, your samples, the homogeneity of those, etc, just to pick the right test, to begin with. Also, you need a controlled setup, which notably includes a predetermined, fixed timeframe for your panel to run. Before this is over, you have no data, at all. You are merely tossing coins…)
Re: Whom the gods would destroy, they first give real-time analytics (2013)
#17This is very 2013. Meanwhile in 2023, a decade later, you literally have systems detecting credit card fraud in milliseconds. [Disclosure: I work for StarTree, which is powered by Apache Pinot. We eat petabytes of data for breakfast.]
Re: Whom the gods would destroy, they first give real-time analytics (2013)
#18> I can understand why engineers are predisposed to see instantaneous A/B statistics as self-evidently positive This is the crucial misunderstanding: in actuality, you are running a panel. (There is no such thing as an A/B test outside of marketing. Running a meaningful panel requires some information on the population, your samples, the homogeneity of those, etc, just to pick the right test, to begin with. Also, you…
Data scientists also do A/B testing on algorithms to see which one has better fit for a use case against real-world, real-time data.
Re: Whom the gods would destroy, they first give real-time analytics (2013)
#19This has aged well.
Re: Whom the gods would destroy, they first give real-time analytics (2013)
#20Related:
Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013) - https://news.ycombinator.com/item?id=15379660 - Oct 2017 (70 comments)
Whom the Gods Would Destroy, They First Give Real-time Analytics - https://news.ycombinator.com/item?id=6515805 - Oct 2013 (1 comment)
Whom the Gods Would Destroy, They First Give Real-time Analytics - https://news.ycombinator.com/item?id=5032588 - Jan 2013 (55 comments)