Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

mcfunley.com

Post reply on HN
Print view View original

41–50 of 72 posts

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#41

Post by scott_karana »

sskates wrote: post #18

Firstly, I'm super biased as I'm the CEO of a product analytics company where one of the value props is getting the data in real time. I agree with this post that people will look at single data points out of context and weight the evidence much more strongly than they should be. Analytics should be one of many tools you use that informs your understanding of how customers are using your product. I also agree that I…

Finding products breakages through analytics is very backwards. If a team is deploying code that untested (or with that poor a process), could you really trust them to make informed judgements either? That needs to be fixed earlier than your product.

UX improvements or mistakes in a conversion pipeline, sure though. Fair points there. :)

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#42

Post by albertgoeswoof »

davedx wrote: post #37
nerfhammer wrote: post #22

> The turnaround time also imposes a welcome pressure on experimental design. People are more likely to think carefully about how their controls work and how they set up their measurements when there's no promise of immediate feedback. This seems like a cranky rationalization of the lack of a fairly ordinary system. Sure, you shouldn't draw conclusions for potentially small effects on What this means is you can't tel…

> What this means is you can't tell whether an experiment launch has gone badly wrong. Personally I prefer automated testing to tell me if a feature has gone badly wrong, not conversion numbers. Then I find out before I launch too. Or do you mean that the UX is so badly designed the users cannot use your software anymore? In which case, maybe there are bigger problems than real time analytics

This approach is for bugs that testing can't / won't pick up, you should be doing both.

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#43

Post by hashmal »

konschubert wrote: post #24
inopinatus wrote: post #13

Earlier quoted context omitted.

Have you considered increasing the price? I'm serious. I'm not a mobile app publisher, I don't know anything about app discovery through the Play store, but I have experienced the paradox of increasing sales by increasing prices in other channels. I'm not super up on the psychology at play, but I've read it's because people use price as a proxy for quality/utility in uncertain markets.

I think that the price is pretty high already for app store standards :) And honestly, if you look at the app, a higher price would feel like a ripoff.

If it's a niche market, low prices is rarely a viable approach because you won't have a lot of volume. Also, all the info you have about app stores and revenue are based on the total mass, or on the top-100 lists, and niche apps fall completely outside of it. There are actually a few narrow categories where you can get away with a high price.

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#44

Post by vog »

j7ake wrote: post #38

Are there theoretical arguments to why random sampling is bad and one should instead use the entire dataset ?

Not sure if that answers your question, but:

If you are looking for seldom-occurring patterns, your random sample almost certainly won't contain it.

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#45

Post by zero_shift »

nerfhammer wrote: post #22

> The turnaround time also imposes a welcome pressure on experimental design. People are more likely to think carefully about how their controls work and how they set up their measurements when there's no promise of immediate feedback. This seems like a cranky rationalization of the lack of a fairly ordinary system. Sure, you shouldn't draw conclusions for potentially small effects on What this means is you can't tel…

> But of course if you've done any real world AB testing, much less any statistics training, you should already know that.

Are most product managers and designers running multivariate tests thus trained? My experience has said 'no'.

In fact, if there's one error I see companies making again and again, it's confusing customer signal from user noise.

I do agree with the rest of your comment however.

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#46

Post by zero_shift »

sskates wrote: post #21
jakobegger wrote: post #2

When I started my business, I looked at sales number every day. I got my hopes up when I sold 15 copies of my app on a good day, only to feel completely devasted when I sold only 4 copies the next day. After some time I stopped looking at daily sales numbers, and switched to weekly numbers instead. But even that was too often. Now I fetch my numbers roughly every other week, and don‘t worry at all about an individual…

That's definitely one of these things you have to learn as part of running a business. You tend to over-index on things you can measure and under-index on things that you can't. After a while you realize there's a lot more randomness in what happens than you might expect and so you can't weight any particular data point too highly. If you look at patio11's blogging of Bingo Card Creator, you'll notice he has little i…

> You tend to over-index on things you can measure and under-index on things that you can't.

I am going to steal this. I am going to deploy it in one of this morning's meetings and pass it off as my own.

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#47

Post by mrec »

zero_shift wrote: post #46
sskates wrote: post #21

Earlier quoted context omitted.

That's definitely one of these things you have to learn as part of running a business. You tend to over-index on things you can measure and under-index on things that you can't. After a while you realize there's a lot more randomness in what happens than you might expect and so you can't weight any particular data point too highly. If you look at patio11's blogging of Bingo Card Creator, you'll notice he has little i…

> You tend to over-index on things you can measure and under-index on things that you can't. I am going to steal this. I am going to deploy it in one of this morning's meetings and pass it off as my own.

It's basically the same as the streetlight effect, isn't it?

https://en.wikipedia.org/wiki/Streetlight_effect

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#48

Post by titzer »

nerfhammer wrote: post #22

> The turnaround time also imposes a welcome pressure on experimental design. People are more likely to think carefully about how their controls work and how they set up their measurements when there's no promise of immediate feedback. This seems like a cranky rationalization of the lack of a fairly ordinary system. Sure, you shouldn't draw conclusions for potentially small effects on What this means is you can't tel…

> Versus some semi-realtime system: launch, wait 30 minutes, did we break anything? No?

I don't think the author is suggesting that you don't have realtime _correctness_ feedback.

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#49

Post by jasonkester »

Indeed, "real-time" is one of the most requested features for S3stat, even though there's really not much you'd be able to do with faster data. In our case, Amazon doesn't even deliver their logfiles until 8 hours or so after the fact, so "real-time" reports would just be a pretty moving picture of the past.

I toy around with the idea of building it out as a feature anyway, just so that I can charge a premium to customers who want to turn it on.

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#50

Post by PeachPlum »

> Accuracy (how precise the data is). Everything should be accurate.

Accurate and precise are different things.

You can be precise: We averaged 1 sale per hour today +- 0.01

You can be accurate: We sold 8 things today

You can be precise but innaccurate: We averaged 100 sales per hour today +- 0.01

You can be imprecise and accurate: We sold 8 items today +- 10

And you can be both: We sold 8 items today +- 0.01

And you can be neither: We sold 100 items today +- 10

Post reply on HN