Do we need to store all that telemetry? • Hacker News Board

Re: Do we need to store all that telemetry?

#41

Post by jcgrillo » Mon, Apr 15, 2024, 5:45 PM UTC

simonw wrote: post #37

jcgrillo wrote: post #28

Earlier quoted context omitted.

The thing about telemetry data is it's extremely repetitive. Take for example a CLF[1] log line: 127.0.0.1 user-identifier frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326 As written this is 99 bytes (792 bits), but how much information is actually in it? We have an IP address which is taking up 9 bytes but only needs at most 4 (fewer in cases like this where two of the bytes are zero if we e…

My hunch is that JSON using a custom compression dictionary with zlib (see zdict argument to https://docs.python.org/3/library/zlib.html#zlib.compressobj ) or zstandard would get you most of the benefit while still letting you interact with existing JSON tools. I've not put the work in to prove that to myself though!

Yeah that would be an interesting experiment too.

This blog post has some interesting ideas as well: https://www.uber.com/blog/reducing-logging-cost-by-two-order...

Re: Do we need to store all that telemetry?

#42

Post by aPoCoMiLogin » Mon, Apr 15, 2024, 5:47 PM UTC

jedberg wrote: post #33

aPoCoMiLogin wrote: post #31

Earlier quoted context omitted.

when storage is full, and you don't know about that, you can't release anything to enable the logs in first place.

You can poll storage periodically though, you don't need to keep a constant metrics stream of where it's at. Also you can set up each machine to alert when it's own storage fills up. Also, as your storage hits 97%+, you'll probably start seeing effects in your business metrics, and then you can look into it.

I think that you are confusing real-time metrics, streamed with very high precision (below 1s) and metrics that are simply polled every N time (most use-cases).

real-time, high precision metrics aren't necessary. when you say that you don't need metrics and then say that you can poll metrics periodically, you are contradicting yourself.

Re: Do we need to store all that telemetry?

#43

Post by Izkata » Mon, Apr 15, 2024, 5:51 PM UTC

Etheryte wrote: post #6

jedberg wrote: post #5

NO! You don't! I couldn't agree with the author more. Keeping historical records of business metrics makes a ton of sense. But history telemetry (CPU, Memory, Network, error logs) makes little sense. If an issue occurs, then turn on telemetry around that issue until you track it down. If an issue occurs once and never again, did it really matter? This obviously does not apply to security, I'm just speaking of operati…

I feel like this swings the pendulum a little too far to the other side. There's very little harm in having telemetry on at all times, but log rotate once a week/month/whatever works for you. If you have telemetry off to begin with, you might not even notice you have an issue while your users do.

As someone who's been on a maintenance team for years, keeping monitoring (cpu, memory, disk, etc) for at least two weeks is critical, and I'd prefer 6 months to easily identify larger trends and prevent issues before they happen.

Re: Do we need to store all that telemetry?

#44

Post by jedberg » Mon, Apr 15, 2024, 5:53 PM UTC

aPoCoMiLogin wrote: post #42

jedberg wrote: post #33

Earlier quoted context omitted.

You can poll storage periodically though, you don't need to keep a constant metrics stream of where it's at. Also you can set up each machine to alert when it's own storage fills up. Also, as your storage hits 97%+, you'll probably start seeing effects in your business metrics, and then you can look into it.

I think that you are confusing real-time metrics, streamed with very high precision (below 1s) and metrics that are simply polled every N time (most use-cases). real-time, high precision metrics aren't necessary. when you say that you don't need metrics and then say that you can poll metrics periodically, you are contradicting yourself.

I'm not contradicting myself. I'm saying you just poll for storage, you don't store the results. My entire thesis is that those metrics aren't worth storing.

Re: Do we need to store all that telemetry?

#45

Post by Izkata » Mon, Apr 15, 2024, 5:57 PM UTC

jedberg wrote: post #23

TylerE wrote: post #12

Earlier quoted context omitted.

Strongly disagree. Having stored telemetry has helped me debug so many things. Forever is probably too much, but keeping a month or so is totally sane.

Why kind of things did you debug with CPU/Memory/Storage telemetry that you couldn't have debugged by only turning those things on after you knew there was a problem?

Identifying patterns where problems coincide with other processes or times, eventually tracking it down to a release done by another team.

It's happened to me a few times.

Re: Do we need to store all that telemetry?

#46

Post by m3047 » Mon, Apr 15, 2024, 6:11 PM UTC

m3047 wrote: post #34

Agree with the article enough that I did something about it which I call "Poor Fred's SIEM". The heart of it is a DNS proxy for Redis ( https://github.com/m3047/rkvdns ). However it's not targeted at environments where everything is in a "bubble" such that there are no ingress / egress costs. (Lookin' at you, Cloud.) Furthermore "control plane" is an important concept, and it's well understood in the industrial contr…

So I went off and set up an actual "live fire" demo because it's that easy:

  curl http://athena.m3047.net/grafitti.html
  dig @athena.m3047.net grafitti\;*.keys.redis.athena.m3047 txt

Re: Do we need to store all that telemetry?

#47

Post by tjoff » Mon, Apr 15, 2024, 6:21 PM UTC

Etheryte wrote: post #39

tjoff wrote: post #17

Earlier quoted context omitted.

Very little harm... If the telemetry is from your users I'd like you to value them more than that. Also consider the potential risks of handling personal data and leaks.

This only holds if you assume telemetry means personal data, but that is a very big if. Meta, Google and other giants generally deal in telemetry that includes personal data, however for most run of the mill software that's not the case. Outside of advertising, I would argue that for most applications you're already pretty close to being clear of personal data as long as you exclude the user's email and other identif…

A lot of telemetry can become personal data. Filenames etc. are the easy parts.

Telemetry needs to be motivated for it to not be considered spyware. You need to really consider what you are logging and why, and then, is it worth the downsides.

It is not something to take lightly, hardly "no harm".

Re: Do we need to store all that telemetry?

#48

Post by jcgrillo » Mon, Apr 15, 2024, 7:14 PM UTC

h1fra wrote: post #40

I understand the point but I also advocate for the opposite, it's not cool for the planet for sure but having all the data points for at least a couple of months is very useful on any large system and +15months for metrics so you can compare with the year before. I can't count the number of times users (or myself) discovered bug after many weeks because something gradually failed over time. Also it saves a lot of tim…

Yeah I've definitely been saved a bunch of times by long retention, and the BI questions that might arise are impossible to predict. So some sort of retention is definitely necessary, IME.

But let's take the case of metrics as an example--do we need full sample granularity for "old" data? Do we need full tag cardinality? Sample granularity reduction could be done with a transform to rollups at a coarser time granularity. That's a 60x reduction going from Hz to 1/min. You might lose a bunch of frequency information this way, but maybe that's ok?

Numbers are really nice in ways that text is not.

Re: Do we need to store all that telemetry?

#49

Post by ChrisCooney » Mon, Apr 15, 2024, 9:06 PM UTC

jcgrillo wrote: post #14

Another facet of this is how do we store telemetry data? Fully indexed instantaneously searchable seems to be the "default" these days but who actually needs that? I keep harping on this, but compressed utf-8 text (or even worse, compressed json) is a horribly wasteful way to do it. See [1]. Putting a small amount of thought into storing telemetry data seems like it could yield incredible savings at scale. [1] https:…

I was gonna make a post but this took the words out of my mouth. I have a whole talk about this exact topic, but the summary is that the paradigm of hot storage and then 2 weeks later, compressed archive, is the most wasteful way we could possibly organize this data. I discuss this at length in the talk below:

https://www.youtube.com/watch?v=XXgBJmqv0ok

Re: Do we need to store all that telemetry?

#50

Post by ChrisCooney » Mon, Apr 15, 2024, 9:09 PM UTC

klabb3 wrote: post #26

Isn’t the issue more that off-the-shelf solutions optimize for features and not cost? For instance, if I sell you an observability product, I want to show off all the cool realtime debugging features and such. And since there’s a cost to having all these features available (retention, indexing, sampling), we end up paying for features we don’t need. In a world of usage-based XaaS, there’s very little incentive to be…

Hey! Just wanna wave a flag for Coralogix (I work there, disclaimer). We've built a ton of cost optimization because we know that the industry is just ridiculous right now. Your assessment is absolutely correct, and there are more than a few multi-billion dollar companies whose bottom line is predicated on their customers wasting money and being inefficient. We're not one of them!

https://coralogix.com/