Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

mcfunley.com

Post reply on HN
Print view View original

61–70 of 72 posts

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#61

Post by konschubert »

tacon wrote: post #30
konschubert wrote: post #9

Earlier quoted context omitted.

I am also checking much less frequently now. But it is still zero :) https://www.konstantinschubert.com/2017/09/27/you-have-0-ins...

Do you have any way to tell if any free users actually entered their AWS keys into your app? There is a much higher level of trust needed for your app over the average mobile app. To actually use it with production data when I'd never heard of you before, I'd probably generate a honey pot set of keys and check if they leaked, and other tests before I could trust it with real keys. Security apps are really tricky to b…

The keys are staying on the device, so I have no way to check if people actually entered them. I'm also not running any analytics.

I mean, I just guess that if people weren't using the app, they would uninstall it eventually? I've had a retention rate of 60% which I thought was pretty good and made the think that people actually use the app.

That being said, I completely understand the trust issue you mention. It's even worse for the online version of the same service, iamproxy.com

Do you have a suggestion how I can improve the trust?

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#62

Post by konschubert »

watt wrote: post #39
konschubert wrote: post #24

Earlier quoted context omitted.

I think that the price is pretty high already for app store standards :) And honestly, if you look at the app, a higher price would feel like a ripoff.

Economically, if you deliver value, you deliver value. Does not matter the price, if it's worth doing and generates profit for me.

I know.

But what value would you assign to "being able to check Kibana on AWS from the phone"?

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#63

Post by konschubert »

danieltillett wrote: post #33
konschubert wrote: post #24

Earlier quoted context omitted.

I think that the price is pretty high already for app store standards :) And honestly, if you look at the app, a higher price would feel like a ripoff.

Well it might be worth a try - you at least don’t have to worry about killing your sales :)

Touché... :D

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#64

Post by scott_karana »

TheReveller wrote: post #53
scott_karana wrote: post #41

Earlier quoted context omitted.

Finding products breakages through analytics is very backwards. If a team is deploying code that untested (or with that poor a process), could you really trust them to make informed judgements either? That needs to be fixed earlier than your product. UX improvements or mistakes in a conversion pipeline, sure though. Fair points there. :)

I believe that's naive. There is always a non-zero chance that deploying new code will cause an issue that is not covered by your unit tests or integration tests. Good testing methodology means adding to your tests when you find such a case, not promising that all possible cases are covered all the time, because that's unreasonable, and the assumption that there's no way it could break on deploy is the kind of hubris…

I agree with "defence in depth": analytics can help catch problems...

However, many here are pitching it as a primary selling point. Relying exclusively on it is a design/process failure, in my book.

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#65

Post by nerfhammer »

joncampbelldev wrote: post #59
nerfhammer wrote: post #22

> The turnaround time also imposes a welcome pressure on experimental design. People are more likely to think carefully about how their controls work and how they set up their measurements when there's no promise of immediate feedback. This seems like a cranky rationalization of the lack of a fairly ordinary system. Sure, you shouldn't draw conclusions for potentially small effects on What this means is you can't tel…

The article mentions the difference between operational and product analytics, his complaints of real-time data are only for product analytics. (I hope) No one in the article or this thread is suggesting your uptime or error metrics should be on a 24 hour delay.

You can totally break some product functionality somehow without necessarily triggering a software exception or server crash! You really do need to know the target events per experiment group.

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#66

Post by nerfhammer »

zero_shift wrote: post #45
nerfhammer wrote: post #22

> The turnaround time also imposes a welcome pressure on experimental design. People are more likely to think carefully about how their controls work and how they set up their measurements when there's no promise of immediate feedback. This seems like a cranky rationalization of the lack of a fairly ordinary system. Sure, you shouldn't draw conclusions for potentially small effects on What this means is you can't tel…

> But of course if you've done any real world AB testing, much less any statistics training, you should already know that. Are most product managers and designers running multivariate tests thus trained? My experience has said 'no'. In fact, if there's one error I see companies making again and again, it's confusing customer signal from user noise. I do agree with the rest of your comment however.

If the people running experiments do not know and cannot be told how to do the fundamental thing that they are trying to do, then you have bigger problems.

Which is ultimately what this post points to: the author doesn't trust his team and isn't listening to them and doesn't expect they will listen to him. Regardless of the degree to which the author is correct in his assumptions, the problem is more than just engineering.

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#67

Post by nerfhammer »

hobofan wrote: post #56
titzer wrote: post #48

Earlier quoted context omitted.

> Versus some semi-realtime system: launch, wait 30 minutes, did we break anything? No? I don't think the author is suggesting that you don't have realtime _correctness_ feedback.

The author explicitly mentions operational metrics are a different thing, and I think that would fall under that.

You can break your product without noticeably affecting the things like the http error 500 rate, cpu utilization %, etc. that you would likely see on some ops dashboard.

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#68

Post by gxs »

goblin89 wrote: post #60
rhizome wrote: post #27

Earlier quoted context omitted.

If you're running case management for a large corporation with tight SLA's - you by definition need real time analytics. Similarly for if you're tracking outages, downtime, etc. If setting up notifications is "real-time analytics" I need to rewrite parts of my resume.

If setting up notifications is "real-time analytics" I need to rewrite parts of my resume. If your notifications are issued based on aggregated metrics (e.g., the number of incoming cases grew 500% in the last 15 minutes) then you probably could. If your notifications are issued on someone’s action (e.g., user submitted a case) then that’s another level of complexity.

This is definitely the use case I meant - those notifications are based on analytics across different systems.

Not sure given the context of the conversation why you'd think I meant adding yourself to an email list when a case get created or when a dashboard that's already built reaches some threshold.

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#69

Post by PeachPlum »

mkasu wrote: post #54
PeachPlum wrote: post #50

> Accuracy (how precise the data is). Everything should be accurate. Accurate and precise are different things. You can be precise: We averaged 1 sale per hour today +- 0.01 You can be accurate: We sold 8 things today You can be precise but innaccurate: We averaged 100 sales per hour today +- 0.01 You can be imprecise and accurate: We sold 8 items today +- 10 And you can be both: We sold 8 items today +- 0.01 And you…

I think he is aware. He made a comment about it in the next paragraph. > hey, did you know that accuracy and precision are different?

I did look to find if he had, I missed that little bit. I hopefully provided more insight

Re: Whom the Gods Would Destroy, They First Give Real-Time Analytics (2013)

#70

Post by vpalanc1 »

nerfhammer wrote: post #67
hobofan wrote: post #56

Earlier quoted context omitted.

The author explicitly mentions operational metrics are a different thing, and I think that would fall under that.

You can break your product without noticeably affecting the things like the http error 500 rate, cpu utilization %, etc. that you would likely see on some ops dashboard.

On our ops dashboard we see stuff like number of ID syncs, number of events processed (by type), etc. - I'd argue that is something is truly "broken" you see it.

If you're using funnel analytics to decide that the product is broken - i'd say you probably do something wrong.

Post reply on HN