Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Do we need to store all that telemetry?

mattklein123.dev

Post reply on HN
Print view View original

11–20 of 74 posts

Re: Do we need to store all that telemetry?

#11

Post by jedberg »

Etheryte wrote: post #6
jedberg wrote: post #5

NO! You don't! I couldn't agree with the author more. Keeping historical records of business metrics makes a ton of sense. But history telemetry (CPU, Memory, Network, error logs) makes little sense. If an issue occurs, then turn on telemetry around that issue until you track it down. If an issue occurs once and never again, did it really matter? This obviously does not apply to security, I'm just speaking of operati…

I feel like this swings the pendulum a little too far to the other side. There's very little harm in having telemetry on at all times, but log rotate once a week/month/whatever works for you. If you have telemetry off to begin with, you might not even notice you have an issue while your users do.

You should have a ton of telemetry on business metrics. You would absolutely notice an issue before your users if you have those. For example at Netflix we monitored stream starts per second -- how often you hit play and it worked. That metric was the most important, and the one that triggered most investigations.

If your CPU and memory aren't affecting the business metrics, then it's not super relevant.

Re: Do we need to store all that telemetry?

#12

Post by TylerE »

jedberg wrote: post #5

NO! You don't! I couldn't agree with the author more. Keeping historical records of business metrics makes a ton of sense. But history telemetry (CPU, Memory, Network, error logs) makes little sense. If an issue occurs, then turn on telemetry around that issue until you track it down. If an issue occurs once and never again, did it really matter? This obviously does not apply to security, I'm just speaking of operati…

Strongly disagree. Having stored telemetry has helped me debug so many things.

Forever is probably too much, but keeping a month or so is totally sane.

Re: Do we need to store all that telemetry?

#14

Post by jcgrillo »

Another facet of this is how do we store telemetry data? Fully indexed instantaneously searchable seems to be the "default" these days but who actually needs that?

I keep harping on this, but compressed utf-8 text (or even worse, compressed json) is a horribly wasteful way to do it. See [1]. Putting a small amount of thought into storing telemetry data seems like it could yield incredible savings at scale.

[1] https://lists.w3.org/Archives/Public/www-logging/1996May/000...

Re: Do we need to store all that telemetry?

#15

Post by dimitar »

Etheryte wrote: post #6
jedberg wrote: post #5

NO! You don't! I couldn't agree with the author more. Keeping historical records of business metrics makes a ton of sense. But history telemetry (CPU, Memory, Network, error logs) makes little sense. If an issue occurs, then turn on telemetry around that issue until you track it down. If an issue occurs once and never again, did it really matter? This obviously does not apply to security, I'm just speaking of operati…

I feel like this swings the pendulum a little too far to the other side. There's very little harm in having telemetry on at all times, but log rotate once a week/month/whatever works for you. If you have telemetry off to begin with, you might not even notice you have an issue while your users do.

Very few things are worth keeping after two weeks, I like short retention policies

Re: Do we need to store all that telemetry?

#16

Post by taneq »

blobcode wrote: post #4

> we can also add local storage of telemetry data in an efficient circular buffer. Typically, local storage is cheap and underutilized, allowing for “free” storage of a finite amount of historical data, that wraps automatically. Local storage provides the ability to “time travel” when a particular event is hit. I think that this is a good idea when storage is concern for high-volume logs / production. Persisting the…

This is a good approach and is pretty common in the embedded world. You use a ring buffer to store a relatively short but detailed log, and then if you encounter an error (or whatever other relevant trigger criteria you use) you snapshot the contents of that ring buffer. Then later you can retrieve the snapshots to figure out what happened.

Re: Do we need to store all that telemetry?

#17

Post by tjoff »

Etheryte wrote: post #6
jedberg wrote: post #5

NO! You don't! I couldn't agree with the author more. Keeping historical records of business metrics makes a ton of sense. But history telemetry (CPU, Memory, Network, error logs) makes little sense. If an issue occurs, then turn on telemetry around that issue until you track it down. If an issue occurs once and never again, did it really matter? This obviously does not apply to security, I'm just speaking of operati…

I feel like this swings the pendulum a little too far to the other side. There's very little harm in having telemetry on at all times, but log rotate once a week/month/whatever works for you. If you have telemetry off to begin with, you might not even notice you have an issue while your users do.

Very little harm... If the telemetry is from your users I'd like you to value them more than that.

Also consider the potential risks of handling personal data and leaks.

Re: Do we need to store all that telemetry?

#18

Post by simonw »

jcgrillo wrote: post #14

Another facet of this is how do we store telemetry data? Fully indexed instantaneously searchable seems to be the "default" these days but who actually needs that? I keep harping on this, but compressed utf-8 text (or even worse, compressed json) is a horribly wasteful way to do it. See [1]. Putting a small amount of thought into storing telemetry data seems like it could yield incredible savings at scale. [1] https:…

I thought compressed JSON was pretty efficient. How much would you expect to save over that with a custom binary format?

Re: Do we need to store all that telemetry?

#20

Post by binary132 »

In general I think many programmers have internalized the idea that it’s best to waste as many computing resources as we can possibly afford as long as it’s not the bottleneck. Then, in the future, if and when it becomes the bottleneck, we’ll have plenty of headroom to optimize and look like heroes for saving the millions of dollars we never had to spend in the first place. It’s really insane (at best) or genuinely a type of grift at worst.
Post reply on HN