Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Do we need to store all that telemetry?

mattklein123.dev

Post reply on HN
Print view View original

51–60 of 74 posts

Re: Do we need to store all that telemetry?

#51

Post by jcgrillo »

ChrisCooney wrote: post #49
jcgrillo wrote: post #14

Another facet of this is how do we store telemetry data? Fully indexed instantaneously searchable seems to be the "default" these days but who actually needs that? I keep harping on this, but compressed utf-8 text (or even worse, compressed json) is a horribly wasteful way to do it. See [1]. Putting a small amount of thought into storing telemetry data seems like it could yield incredible savings at scale. [1] https:…

I was gonna make a post but this took the words out of my mouth. I have a whole talk about this exact topic, but the summary is that the paradigm of hot storage and then 2 weeks later, compressed archive, is the most wasteful way we could possibly organize this data. I discuss this at length in the talk below: https://www.youtube.com/watch?v=XXgBJmqv0ok

Nice talk. The first (and best!) logs search solution I experienced in my career was simply a gigantic tree of compressed logs on a hadoop cluster. As someone who spent a bunch of time analyzing logs, the "query interface" being "anything you can sling at the hadoop cluster" was phenomenally awesome. The basic computering tools are programming languages, and eventually you encounter problems where you need a real (Turing-complete) one.

One great side effect of this was service developers weren't afraid to write logs. We logged excessively, and it didn't cost too much. If we'd been indexing everything in ES it would have bankrupted us.

These days with S3 and the cloud, hadoop (or the EMR suite) per se probably isn't the way to go, but I'd sure like to see observability solutions giving me a first-class programming model that I as a user can interact with--not some bespoke "query DSL", and for them to accept that instantaneous indexed retrieval isn't important.

This paper is really interesting: https://www.usenix.org/system/files/osdi21-rodrigues.pdf

Stuff like this gives me hope we can have it both ways. With highly tuned compression and programmatic access the user is empowered and the cost is minimized.

Re: Do we need to store all that telemetry?

#52

Post by dog321 »

jedberg wrote: post #25
rezonant wrote: post #22

Earlier quoted context omitted.

> I can't recall a single time when having more than a day's with of history was ever useful in tracking down an operational issue. User writes into support 3 days after the problem occurred, and support goes back and forth covering level 1 possibilities for an additional 2 days before escalating. It's common for 1 support complaint to represent some larger factor of users who never complain, so it would be useful to…

I think you missed my key point -- I'm talking about operational metrics not business metrics. With business metrics you can get historical context, but I don't see how CPU/Memory/Storage/App logs will help you.

Here is a good piece on gaining value from long term operational metrics. https://danluu.com/metrics-analytics/

Re: Do we need to store all that telemetry?

#53

Post by akira2501 »

hooverd wrote: post #19
thisislife2 wrote: post #8

"Data is the new oil" - if you don't collect your customer data, and treat it as an asset, you are guilty of mismanagement . /s

Does that mean Google is the new Saudi Arabia?

More like Iraq or Daniel Plainsview. Drilling diagonally to tap neighboring fields.

Re: Do we need to store all that telemetry?

#54

Post by dragonwriter »

akira2501 wrote: post #53
hooverd wrote: post #19

Earlier quoted context omitted.

Does that mean Google is the new Saudi Arabia?

More like Iraq or Daniel Plainsview. Drilling diagonally to tap neighboring fields.

That sounds like a popular misunderstanding of the mutual accusations before the 1990 invasion between Iraq and Kuwait of overpumping from the oil field that crosses their border (which did not involve “slant drilling”).

Re: Do we need to store all that telemetry?

#55

Post by zug_zug »

I think most places don't collect enough telemetry in the right formats.

It's also possible they collect too much in the wrong formats.

But the ability to vet a hypothesis (I bet our users are confused about feature X, which we can test by looking at how many times they go to page X, then Y, then X again in 30 second window) in an hour versus 2 sprints is vastly underappreciated/underutilized.

I feel like this article paints with too broad a brush.

Re: Do we need to store all that telemetry?

#56

Post by ryandrake »

thisislife2 wrote: post #8

"Data is the new oil" - if you don't collect your customer data, and treat it as an asset, you are guilty of mismanagement . /s

Data is more like uranium than oil.[1] Valuable for its limited purpose, but dangerous to just collect and hold on to forever.

1: https://www.forbes.com/sites/forbestechcouncil/2022/10/03/th...

Re: Do we need to store all that telemetry?

#57

Post by boundlessdreamz »

aPoCoMiLogin wrote: post #38
jedberg wrote: post #32

Earlier quoted context omitted.

> if you don't have metrics for cpu/memory/storage how do you know when to scale the app When the business metrics start to fail. You don't need constant metics on storage, you can poll it every so often. If your app is constrained by CPU or RAM, then the business metrics will reflect that, and then you can turn on collection of those metrics to identify the problem. > i feel like you have never touched servers/backe…

where you worked doesn't matter to me very much, when what are you saying contradicts what you probably did ("experience in large scale systems"), also it sounds like argument from authority. not having cpu/mem/hdd metrics is just plain bogus and sounds like fantasy world, where everything works like we expect it to work, and there is no bugs at all. ridiculous

You question his competence.

> i feel like you have never touched servers/backend in anything more than simple projects (or at all). with full storage/memory there could be an issue that you won't be able to ssh to the server, so it speaks about your knowledge in this matter.

He was answering that.

If instead of dismissing someone outright and question their competence, you had raised specific concerns, this would have been a more productive conversation

Re: Do we need to store all that telemetry?

#58

Post by rezonant »

aPoCoMiLogin wrote: post #38
jedberg wrote: post #32

Earlier quoted context omitted.

> if you don't have metrics for cpu/memory/storage how do you know when to scale the app When the business metrics start to fail. You don't need constant metics on storage, you can poll it every so often. If your app is constrained by CPU or RAM, then the business metrics will reflect that, and then you can turn on collection of those metrics to identify the problem. > i feel like you have never touched servers/backe…

where you worked doesn't matter to me very much, when what are you saying contradicts what you probably did ("experience in large scale systems"), also it sounds like argument from authority. not having cpu/mem/hdd metrics is just plain bogus and sounds like fantasy world, where everything works like we expect it to work, and there is no bugs at all. ridiculous

> i feel like you have never touched servers/backend in anything more than simple projects (or at all)

I feel like if you are going to go out on a limb and call someone's expertise into question...

> I ran all of ops for reddit for four years and headed up SRE at Netflix

And they provide excellent credentials which you failed to check...

> where you worked doesn't matter to me very much

You can't just weasel out of it by pretending like you didn't start the interaction by calling someone's expertise into question.

Re: Do we need to store all that telemetry?

#59

Post by rezonant »

jedberg wrote: post #25
rezonant wrote: post #22

Earlier quoted context omitted.

> I can't recall a single time when having more than a day's with of history was ever useful in tracking down an operational issue. User writes into support 3 days after the problem occurred, and support goes back and forth covering level 1 possibilities for an additional 2 days before escalating. It's common for 1 support complaint to represent some larger factor of users who never complain, so it would be useful to…

I think you missed my key point -- I'm talking about operational metrics not business metrics. With business metrics you can get historical context, but I don't see how CPU/Memory/Storage/App logs will help you.

Yep, I missed that. I definitely have hit cases where having a larger window of infrastructure metrics has been very useful. Being able to correlate it against other observability factors can help to understand what caused a problem. But I agree that you don't have to keep it forever. I think a few weeks is fine, assuming the scale of the system doesn't mean that a few weeks is an unwieldy amount of data

Re: Do we need to store all that telemetry?

#60

Post by rixed »

simonw wrote: post #18
jcgrillo wrote: post #14

Another facet of this is how do we store telemetry data? Fully indexed instantaneously searchable seems to be the "default" these days but who actually needs that? I keep harping on this, but compressed utf-8 text (or even worse, compressed json) is a horribly wasteful way to do it. See [1]. Putting a small amount of thought into storing telemetry data seems like it could yield incredible savings at scale. [1] https:…

I thought compressed JSON was pretty efficient. How much would you expect to save over that with a custom binary format?

Storing date in compressed json consist of:

- converting every number into its sequence of digits in decimal notation,

- writing those one character at a time,

- also write the string representation of the label of each value repeatedly for every record,

- compress all this with a structure-unaware generic text compression algorithm based on longest match search.

Each time you want to read that data, undo all of the above in reverse order.

You can optimize to some degree, but that's basically it.

I expect that not doing any of this saves the time spent doing it. I also expect data type aware compression to be much more efficient than text compressing the text expansion.

In numbers, I expect 2 to 3 orders of magnitude difference in time and also in space (for non random data).

Post reply on HN