Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

mcfunley.com

Post reply on HN
Print view View original

21–30 of 72 posts

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#21

Post by sskates »

jakobegger wrote: post #2

When I started my business, I looked at sales number every day. I got my hopes up when I sold 15 copies of my app on a good day, only to feel completely devasted when I sold only 4 copies the next day. After some time I stopped looking at daily sales numbers, and switched to weekly numbers instead. But even that was too often. Now I fetch my numbers roughly every other week, and don‘t worry at all about an individual…

That's definitely one of these things you have to learn as part of running a business. You tend to over-index on things you can measure and under-index on things that you can't. After a while you realize there's a lot more randomness in what happens than you might expect and so you can't weight any particular data point too highly. If you look at patio11's blogging of Bingo Card Creator, you'll notice he has little idea as to why his userbase grows or shrinks any particular year: http://www.kalzumeus.com/2012/12/29/bingo-card-creator-and-o... And he's one of the most sophisticated product analytics experts out there!

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#22

Post by nerfhammer »

> The turnaround time also imposes a welcome pressure on experimental design. People are more likely to think carefully about how their controls work and how they set up their measurements when there's no promise of immediate feedback.

This seems like a cranky rationalization of the lack of a fairly ordinary system.

Sure, you shouldn't draw conclusions for potentially small effects on What this means is you can't tell whether an experiment launch has gone badly wrong. Small effect size experiments are one thing, but you can surely tell if you've badly broken something in short order.

Contrary to encouraging people to be careful, it can make people risk averse for fear of breaking something. And it slows down the process of running experiments a lot. Every time you want to launch something, you probably have to launch it at a very small % of traffic, then you have to wait a full 24-36 hours to know whether you've broken anything, then increase the experiment size. Versus some semi-realtime system: launch, wait 30 minutes, did we break anything? No? OK, let's crank up the group sizes... Without semi-realtime, you have to basically add two full days times 1 + the probability of doing something wrong and requiring relaunch (compounding of course) to the development time of everything you want to try. Plus, if you have the confidence that you haven't broken anything you can much larger experiment sizes so you get significant results much faster.

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#23

Post by sskates »

ThomPete wrote: post #20
sskates wrote: post #18

Firstly, I'm super biased as I'm the CEO of a product analytics company where one of the value props is getting the data in real time. I agree with this post that people will look at single data points out of context and weight the evidence much more strongly than they should be. Analytics should be one of many tools you use that informs your understanding of how customers are using your product. I also agree that I…

Let me give you another angle on this. Asking people to become data analysts when they just want to run a business and have a million other things to worry about is exactly the wrong way to think about this. People shouldn't need to become data analysts to understand the data the software should be able to analyze and give simple advice when it has it. I have designed my share of analytics software from the classical…

Agree with everything you're saying about data. Given how hard an analytics system is to set up properly, it's one of the last pieces you should be adding in to get an understanding of how to build your business.

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#24

Post by konschubert »

inopinatus wrote: post #13
konschubert wrote: post #9

Earlier quoted context omitted.

I am also checking much less frequently now. But it is still zero :) https://www.konstantinschubert.com/2017/09/27/you-have-0-ins...

Have you considered increasing the price? I'm serious. I'm not a mobile app publisher, I don't know anything about app discovery through the Play store, but I have experienced the paradox of increasing sales by increasing prices in other channels. I'm not super up on the psychology at play, but I've read it's because people use price as a proxy for quality/utility in uncertain markets.

I think that the price is pretty high already for app store standards :)

And honestly, if you look at the app, a higher price would feel like a ripoff.

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#26

Post by PaulHoule »

Early stopping is just fine if you are Bayesian. Not fine if you are doing null-hypothesis significance testing.

Multi-armed bandits beat the heck out of A/B testing because they dynamically balance making money now with taking risks that might mean you make more money later.

https://en.wikipedia.org/wiki/Multi-armed_bandit

I'd think they should be more popular among quantitative marketing types than they are.

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#27

Post by rhizome »

gxs wrote: post #17
habosa wrote: post #12

The conclusions of the article are probably correct, but it's important for analytics systems to have real time capabilities for debugging and iteration. When you're wiring up some new events or funnels you need to be able to click through and sanity check that you're collecting the data you expect. Maybe some of the desire for whole systems to be real time comes from this frustration.

It completely depends on your domain IMO. If you're running case management for a large corporation with tight SLA's - you by definition need real time analytics. Similarly for if you're tracking outages, downtime, etc. All that said, the organizations I've been in do a good job of distinguishing operational metrics vs high level analytics done for quarterly reviews or metrics pulled from across the enterprise for th…

If you're running case management for a large corporation with tight SLA's - you by definition need real time analytics.

Similarly for if you're tracking outages, downtime, etc.

If setting up notifications is "real-time analytics" I need to rewrite parts of my resume.

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#28

Post by alexandercrohde »

Generally really good points. However one great use of realtime analytics is as a way to immediately catch production errors (particularly when connected to alarms, e.g. Splunk).

My thought is that realtime amazing systems as discussed can be an exceedingly difficult engineering problem or a trivial business problem (use google or another hosted solution).

Personally, any engineer who would try to reinvent the wheel on this one (rather than leverage any of the many incredibly refined existing technologies) should not serve in a decision-making capacity.

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#29

Post by shostack »

pedasmith wrote: post #19

"No sampling" is just another way of saying "the queries will be so slow that you will never do any ad-hoc queries. It's also short for "I don't understand statistics".

Sampling can be really bad for web analytics. Take Google Analytics for starters. It heavily samples above a certain threshold of hits. At more granular drill downs, certain reports are at best unusable even directionally and at worst are misleading. Small efforts get lost in the noise of the rest of the data and often show up as zero values.

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#30

Post by tacon »

konschubert wrote: post #9
jakobegger wrote: post #2

When I started my business, I looked at sales number every day. I got my hopes up when I sold 15 copies of my app on a good day, only to feel completely devasted when I sold only 4 copies the next day. After some time I stopped looking at daily sales numbers, and switched to weekly numbers instead. But even that was too often. Now I fetch my numbers roughly every other week, and don‘t worry at all about an individual…

I am also checking much less frequently now. But it is still zero :) https://www.konstantinschubert.com/2017/09/27/you-have-0-ins...

Do you have any way to tell if any free users actually entered their AWS keys into your app? There is a much higher level of trust needed for your app over the average mobile app. To actually use it with production data when I'd never heard of you before, I'd probably generate a honey pot set of keys and check if they leaked, and other tests before I could trust it with real keys. Security apps are really tricky to bootstrap.
Post reply on HN