Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How much of my observability data is waste?

usetero.com

Post reply on HN
Print view View original

41–50 of 61 posts

Re: How much of my observability data is waste?

#41

Post by binarylogic »

gmuslera wrote: post #39

Reminded me a note I heard about backups. You don't want backups, it is a waste of time, bandwidth and disk space, by far most if not all of it will end being discarded without being ever used. What you really want is something to restore from if anything breaks. That is the cost that should matter to you. What if you don't have anything meaningful to make a restore from? With observability is not the volume of data,…

I agree with the framing. The goal isn't less data for its own sake. The goal is understanding your systems and being able to debug when things break.

But here's the thing: most teams aren't drowning in data because they're being thorough. They're drowning because no one knows what's valuable and what's not. Health checks firing every second aren't helping anyone debug anything. Debug logs left in production aren't insurance, they're noise.

The question isn't "can you do with less?" It's "do you even know what you have?" Most teams don't. They keep everything just in case, not because they made a deliberate choice, but because they can't answer the question.

Once you can answer it, you can make real tradeoffs. Keep the stuff that matters for debugging. Cut the stuff that doesn't.

Re: How much of my observability data is waste?

#42

Post by xmprt »

stackskipton wrote: post #8

As Ops (DevOps/Sysadmin/SREish) person here, excellent article. However, as always, the problem is more political than technical and those are hardest problems to solve and another service with more cost IMO won't solve it. However, there is plenty of money to be made in attempting to solve it so go get that bag. :) At end of day, it's back to DevOps mentality and it's never caught on at most companies. Devs don't ca…

The first step to solving this is correct cost attribution. And then once you do that, it's easy to go to org leads and tell them that their logs are costing them $X and you can save them 40% by applying these suggestions. They'll be happy to accept your help at that point. But if the costs are all on the Ops team, then why would the product teams care about any cost optimizations which just takes away development time from them.

Re: How much of my observability data is waste?

#43

Post by gmuslera »

binarylogic wrote: post #41
gmuslera wrote: post #39

Reminded me a note I heard about backups. You don't want backups, it is a waste of time, bandwidth and disk space, by far most if not all of it will end being discarded without being ever used. What you really want is something to restore from if anything breaks. That is the cost that should matter to you. What if you don't have anything meaningful to make a restore from? With observability is not the volume of data,…

I agree with the framing. The goal isn't less data for its own sake. The goal is understanding your systems and being able to debug when things break. But here's the thing: most teams aren't drowning in data because they're being thorough. They're drowning because no one knows what's valuable and what's not. Health checks firing every second aren't helping anyone debug anything. Debug logs left in production aren't i…

There is a lot of crap that is and will ever be useless when debugging a problem. But there is a also a lot that you don't know if you will need it, at least, not yet, not when you are defining what information you collect, and may become essential when something in particular (usually unexpected) breaks. And then you won't have the past data you didn't collect.

You can go in a discovering path, can the data you collect explain how and why the system is running now? There are things that are just not relevant when things are normal and when they are not? Understanding the system, and all the moving parts, are a good guide for tuning what you collect, what you should not, and what are the missing pieces. And cycle with that, your understanding and your system will keep changing.

Re: How much of my observability data is waste?

#44

Post by Veserv »

You should not even need a regex; no serious logging system should be emitting formatted strings, JSON, etc. as a storage format. You are immediately incurring on the order of a 5-100x log size and 5-100x log performance overhead with any serialization format that poor. A properly performant logging system should be able to generate on the order of 100 million logs per second per core (assuming relatively small payloads).

At a minimum you should be using message template [1] serialization which is trivial to implement transparently on any logging system/statement with zero code changes to the emitter itself.

Any filtering done on top of that would then just be parsing structured data which is way easier than a regex, though of course that is somewhat beside the point of the article.

[1] https://messagetemplates.org/

Re: How much of my observability data is waste?

#45

Post by HumanOstrich »

nextaccountic wrote: post #38
binarylogic wrote: post #26

Earlier quoted context omitted.

You're right, it's not always binary. That's why we broke it down into categories: https://docs.usetero.com/data-quality/logs/malformed-data You'd be shocked how much obviously-safe waste (redundant attributes, health checks, debug logs left in production) accounts for before you even get to the nuanced stuff. But think about this: if you had a service that was too expensive and you wanted to optimize the data, who w…

I would like to just have a storage engine that can be very aggressive at deduplicating stuff. If some data is redundant, why am I storing it twice?

That's already pretty common, but the goal isn't storing less data for its own sake.

Re: How much of my observability data is waste?

#46

Post by zahlman »

binarylogic wrote: post #20
jldugger wrote: post #16

>Turns out you can compile tens of thousands of patterns and still match at line rate. Well, yea, sort of the magic of the regular expression NFA equality theorem. Any regex can be converted to a state machine. And since you can combine regexes (and NFAs!) procedurally, this is not a surprising result. > I ran it against the first service: ~40% waste. Another: ~60%. Another: ~30%. On average, ~40% waste. I'm surprise…

Yeah, it's funny, I never went down the regex rabbit hole until this, but I was blown away by Hyperscan/Vectorscan. It truly changes the game. Traditional wisdom tells you regex is slow. > I'm surprised it's only 40%. Oh, it's worse. I'm being conservative in the post. That number represents "pure" waste without sampling. You can see how we classify it: https://docs.usetero.com/data-quality/logs/malformed-data . If y…

> Traditional wisdom tells you regex is slow.

Because it's uncomfortably easy to create catastrophic backtracking.

But just logical-ORing many patterns together isn't one of the ways to do that, at least as far as I'm aware.

Re: How much of my observability data is waste?

#47

Post by bluGill »

binarylogic wrote: post #41
gmuslera wrote: post #39

Reminded me a note I heard about backups. You don't want backups, it is a waste of time, bandwidth and disk space, by far most if not all of it will end being discarded without being ever used. What you really want is something to restore from if anything breaks. That is the cost that should matter to you. What if you don't have anything meaningful to make a restore from? With observability is not the volume of data,…

I agree with the framing. The goal isn't less data for its own sake. The goal is understanding your systems and being able to debug when things break. But here's the thing: most teams aren't drowning in data because they're being thorough. They're drowning because no one knows what's valuable and what's not. Health checks firing every second aren't helping anyone debug anything. Debug logs left in production aren't i…

The problem is until I hit a specific bug I don't know what logs might be useful. For every bug I've had to fix 99% of the logs were useless, but I've had to fix many bugs over the years and each one needed a different set of logs. Sometimes I know in the code "this can't happen but I'll log an error just in case" - when I see those in a bug report they are often a clue, but I often need a lot of info bugs that happen normally all the time to figure out how my system got into that state.

"disk getting full" isn't useful unless you understand how/why it got full and that requires logging things that might or might matter to the problem.

Re: How much of my observability data is waste?

#48

Post by nishantmodak »

smithclay wrote: post #15

Kudos to Ben for speaking to one of the elephants in the room in observability: data waste and the impact it has on your bill. All major vendors have a nice dashboard and sometimes alerts to understand usage (broken down by signal type or tags) ... but there's clearly a need for more advanced analysis which Tero seems to be going after. Speaking of the elephant in room in observability: why does storing data on a ven…

Problem has never been the storage. Its running those queries to return in milliseconds - if its for a dashboard, an alert of your new AI agent trying to make sense of it.

Re: How much of my observability data is waste?

#49

Post by pstuart »

binarylogic wrote: post #20
jldugger wrote: post #16

>Turns out you can compile tens of thousands of patterns and still match at line rate. Well, yea, sort of the magic of the regular expression NFA equality theorem. Any regex can be converted to a state machine. And since you can combine regexes (and NFAs!) procedurally, this is not a surprising result. > I ran it against the first service: ~40% waste. Another: ~60%. Another: ~30%. On average, ~40% waste. I'm surprise…

Yeah, it's funny, I never went down the regex rabbit hole until this, but I was blown away by Hyperscan/Vectorscan. It truly changes the game. Traditional wisdom tells you regex is slow. > I'm surprised it's only 40%. Oh, it's worse. I'm being conservative in the post. That number represents "pure" waste without sampling. You can see how we classify it: https://docs.usetero.com/data-quality/logs/malformed-data . If y…

Vectorscan is impressive. It makes a huge difference if you're looping through an eval of dozens (or more) regexps. I have a pending PR to fix it so it'll run as a wasm engine -- this is a good reminder to take that to completion.

Re: How much of my observability data is waste?

#50

Post by stephenlf »

karianna wrote: post #27

Hard agree on the data waste, noise to signal ratio is typically very high and processing, shipping and storing all of that data costs a ton. Previous start-up I worked on (jClarity, exited to Microsoft) mitigated much of this by having a model of only collecting the tiny amount of data that really mattered for a performance bottleneck investigation in a ring buffer and only processing / shipping and storing that dat…

That’s awesome
Post reply on HN