Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

mcfunley.com

Post reply on HN
Print view View original

51–60 of 72 posts

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#51

Post by moe9 »

jasonkester wrote: post #49

Indeed, "real-time" is one of the most requested features for S3stat, even though there's really not much you'd be able to do with faster data. In our case, Amazon doesn't even deliver their logfiles until 8 hours or so after the fact, so "real-time" reports would just be a pretty moving picture of the past. I toy around with the idea of building it out as a feature anyway, just so that I can charge a premium to cust…

It's what news orgs have been doing for a long time. The number of people addicted to this crap is much more than the number of people who can do something useful with it.

Why that is, has more to do with psychology and evolution of the mind than it has to do with the actual real time data or news.

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#52

Post by disgruntledphd2 »

j7ake wrote: post #38

Are there theoretical arguments to why random sampling is bad and one should instead use the entire dataset ?

Random sampling is the foundation of statistical inference.

It works really well for estimating population statistics from sampling.

It works less well for predicting individual user behaviour (this tends to be where you need almost all the data rather than a sample).

Like, as your sample size decreases, your errors of estimation increase (1/sqrt(n)) so random sampling is normally the best way to approximate data (e.g. for real-time metrics).

You may wish to consider different sampling strategies, however. For some things, a persistent user (or other dimension of interest) makes lots of sense. For tracking service health, you probably want to sample on requests.

You may also wish to over-sample particular groups, if they are important to you (like maybe an F2P game would sample for most users, but store a larger proportion for paying users).

tl;dr random sampling is awesome, and should be used broadly.

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#53

Post by TheReveller »

scott_karana wrote: post #41
sskates wrote: post #18

Firstly, I'm super biased as I'm the CEO of a product analytics company where one of the value props is getting the data in real time. I agree with this post that people will look at single data points out of context and weight the evidence much more strongly than they should be. Analytics should be one of many tools you use that informs your understanding of how customers are using your product. I also agree that I…

Finding products breakages through analytics is very backwards. If a team is deploying code that untested (or with that poor a process), could you really trust them to make informed judgements either? That needs to be fixed earlier than your product. UX improvements or mistakes in a conversion pipeline, sure though. Fair points there. :)

I believe that's naive. There is always a non-zero chance that deploying new code will cause an issue that is not covered by your unit tests or integration tests.

Good testing methodology means adding to your tests when you find such a case, not promising that all possible cases are covered all the time, because that's unreasonable, and the assumption that there's no way it could break on deploy is the kind of hubris that leads to breakages when you deploy.

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#54

Post by mkasu »

PeachPlum wrote: post #50

> Accuracy (how precise the data is). Everything should be accurate. Accurate and precise are different things. You can be precise: We averaged 1 sale per hour today +- 0.01 You can be accurate: We sold 8 things today You can be precise but innaccurate: We averaged 100 sales per hour today +- 0.01 You can be imprecise and accurate: We sold 8 items today +- 10 And you can be both: We sold 8 items today +- 0.01 And you…

I think he is aware. He made a comment about it in the next paragraph.

> hey, did you know that accuracy and precision are different?

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#55

Post by rusk »

sskates wrote: post #18

Firstly, I'm super biased as I'm the CEO of a product analytics company where one of the value props is getting the data in real time. I agree with this post that people will look at single data points out of context and weight the evidence much more strongly than they should be. Analytics should be one of many tools you use that informs your understanding of how customers are using your product. I also agree that I…

There's a ton of positive value to having real time data. Just off the top of my head: 1) If you've instrumented something incorrectly ... 2) Even worse, if you've accidentally messed up ... 3) You can observe significant changes ... 4) It allows you to have more confidence in deploying multiple times ...

Embracing both yours, and the author's perspectives I think I have a compromise. It seems to me the author's gripe is more specifically that his customers want to "do the same large window analytics in realtime" whereas here you highlight particular usecases where realtime analytics are simple.

In a previous life I was producing near-realtime analytics for network activity for populations numbering 10s of millions, all the time, in realtime, except the definition of realtime was stretched to allow a 15 minute latency.

Reducing this 15 minute latency was oft requested e.g to give help-desk operators an instant view of what is happening for a customer or, to automate responses to particular well-known system issues in realtime.

It wouldn't have been impossible to reduce our 15 minute window but given various architectural limitation of the time it would have been expensive and upon analysing these customer requirements it made more sense to simply siphon off the realtime event-data at our monitoring points into a parallel dataflow.

TL;DR you do different things with realtime vs aggregate data. Do you really need the expense of a system that does both equally well?

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#56

Post by hobofan »

titzer wrote: post #48
nerfhammer wrote: post #22

> The turnaround time also imposes a welcome pressure on experimental design. People are more likely to think carefully about how their controls work and how they set up their measurements when there's no promise of immediate feedback. This seems like a cranky rationalization of the lack of a fairly ordinary system. Sure, you shouldn't draw conclusions for potentially small effects on What this means is you can't tel…

> Versus some semi-realtime system: launch, wait 30 minutes, did we break anything? No? I don't think the author is suggesting that you don't have realtime _correctness_ feedback.

The author explicitly mentions operational metrics are a different thing, and I think that would fall under that.

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#57

Post by AnIdiotOnTheNet »

ACow_Adonis wrote: post #8

"But unless the intention is to make decisions with this data, one might wonder what the purpose of such a system could possibly be." Speaking as an analyst, you'd be amazed (or not) at how much analytics is done either for resume padding, because one guy thinks it's "cool", or to enable marketing or executives to chase their own tails. Indeed, with the general phenomenon of bullshit jobs, infinite instantaneous and…

Metrics for the Metrics God.

They don't really mean anything, you can't draw actionable conclusions, and no one will even read the report after the first week, but you gather the metrics anyway to satiate the Metrics God's hunger.

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#58

Post by folbec »

anuragojha wrote: post #40
ACow_Adonis wrote: post #32

Earlier quoted context omitted.

Almost every metric can be abused by increasing the frequency, resolution and dimensions in which it is measured. In terms of some real world examples: Labour force statistics are almost always reported on an instance level by commentators/media, but its trend that's recommended/meaningful. Net promoter scores and breakdowns are consistently done and reported too frequently on too small a base to establish a trend, a…

Interesting. Thanks for sharing. Do you know of any good writing on the web or print that talks more about such abuses in the analytics and corporate world? Interested to learn how to spot when kooky tactics are at play in a tech/corporate setting, specially when its willful.

"Proofiness" : https://www.amazon.com/Proofiness-Youre-Being-Fooled-Numbers...

It is also a common tread in books by Tufte : https://en.wikipedia.org/wiki/Edward_Tufte

I like "Visual Explanations: Images and Quantities, Evidence and Narrative" : https://www.amazon.com/Visual-Explanations-Quantities-Eviden...

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#59

Post by joncampbelldev »

nerfhammer wrote: post #22

> The turnaround time also imposes a welcome pressure on experimental design. People are more likely to think carefully about how their controls work and how they set up their measurements when there's no promise of immediate feedback. This seems like a cranky rationalization of the lack of a fairly ordinary system. Sure, you shouldn't draw conclusions for potentially small effects on What this means is you can't tel…

The article mentions the difference between operational and product analytics, his complaints of real-time data are only for product analytics.

(I hope) No one in the article or this thread is suggesting your uptime or error metrics should be on a 24 hour delay.

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#60

Post by goblin89 »

rhizome wrote: post #27
gxs wrote: post #17

Earlier quoted context omitted.

It completely depends on your domain IMO. If you're running case management for a large corporation with tight SLA's - you by definition need real time analytics. Similarly for if you're tracking outages, downtime, etc. All that said, the organizations I've been in do a good job of distinguishing operational metrics vs high level analytics done for quarterly reviews or metrics pulled from across the enterprise for th…

If you're running case management for a large corporation with tight SLA's - you by definition need real time analytics. Similarly for if you're tracking outages, downtime, etc. If setting up notifications is "real-time analytics" I need to rewrite parts of my resume.

If setting up notifications is "real-time analytics" I need to rewrite parts of my resume.

If your notifications are issued based on aggregated metrics (e.g., the number of incoming cases grew 500% in the last 15 minutes) then you probably could. If your notifications are issued on someone’s action (e.g., user submitted a case) then that’s another level of complexity.

Post reply on HN