Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

The conclusions of the article are probably correct, but it's important for analytics systems to have real time capabilities for debugging and iteration. When you're wiring up some new events or funnels you need to be able to click through and sanity check that you're collecting the data you expect.

Maybe some of the desire for whole systems to be real time comes from this frustration.

Real-time data gives that nice warm confirmation that the change you just deployed is working as expected. Code change, site failover, any change at all. It drives me batty that Google analytics update interval is unpredictable, so I look at real-time numbers.

Certainly it’s foolish to jump to conclusions from a too-small sample. Increasing latency does nothing to solve this. I’d recommend an introductory stats course instead.

Would you mind giving examples of some of these metrics?

Curious too see exactly what they are.

Firstly, I'm super biased as I'm the CEO of a product analytics company where one of the value props is getting the data in real time. I agree with this post that people will look at single data points out of context and weight the evidence much more strongly than they should be. Analytics should be one of many tools you use that informs your understanding of how customers are using your product. I also agree that I see a lot of early stage startups invest way too much in building out real-time analytics stores without thinking about what they value they get out of it. That's not something you should do until you're in the 100+ engineer range.

That said, this is a post-hoc justification of why having real-time analytics is bad. Delaying the data by 24 hours doesn't automatically make significance testing better or force people to incorporate more context when interpreting their data. If that's the issue, fix that problem, don't blame the tools.

There's a ton of positive value to having real time data. Just off the top of my head: 1) If you've instrumented something incorrectly you can see that and fix it right away 2) Even worse, if you've accidentally messed up a feature with a release you can know about it right away. This happened to one of our customers recently and without a real-time analytics system they wouldn't have caught it quickly (and this is a tech startup that's well regarded for their engineering that everyone here would know the name of). 3) You can observe significant changes to your user base right away, eg during a launch or if you're getting a lot of new users from a specific channel. 4) It allows you to have more confidence in deploying multiple times a day. It's a little crazy to me that the deployment of a product is faster than our ability to measure it.

I think the real issue is that we're in the early days of analytics and people don't have a great understanding of how to leverage their tools properly. It's like web search before Google or file sync before Dropbox. People will look at single data points and conclude totally crazy things. They won't have an understanding of basic things like the amount of fluctuation on a week to week basis. The largest analytics provider in the world (Google) gives away their product as an ancillary service to drive you to purchase more ads, not to have a better understanding of how your customers are using your product. I'm hopeful this will change over the next 5 years though (and for us to be a part of that!)

Let me give you another angle on this.

Asking people to become data analysts when they just want to run a business and have a million other things to worry about is exactly the wrong way to think about this.

People shouldn't need to become data analysts to understand the data the software should be able to analyze and give simple advice when it has it.

I have designed my share of analytics software from the classical dashboards to more complex things like a market replay for Nasdaq.

At the end of the day, analytics primarily is useful if it can be used to make decisions with.

If you really want to build a useful analytics software tool you have to build something that analysis and either do things for me automatically (like seeing a lot of people are suddenly coming to your site because of a keyword and then quickly do some google advertising) or it has to be able to tell you something you can use (ex. You should buy extra hotdogs for Sunday as you are about to get a lot of customers because of bad weather after/before the football match.