Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How much of my observability data is waste?

usetero.com

Post reply on HN
Print view View original

31–40 of 61 posts

Re: How much of my observability data is waste?

#31

Post by binarylogic »

mr-karan wrote: post #14

Just want to say thanks for creating Vector. We use it heavily at Zerodha and wrote about our setup here: https://zerodha.tech/blog/logging-at-zerodha/ It replaced both Filebeat and Logstash for us with a single binary that actually has sane resource usage (no more JVM nightmares). VRL turned out to be way more powerful than we could imagine - we do all our log parsing, metadata enrichment, and routing to different C…

Thanks for the comment! Yes, I read that post. Great post. Feel free to reach out if you ever need help with Vector or have questions.

Re: How much of my observability data is waste?

#32

Post by binarylogic »

smithclay wrote: post #15

Kudos to Ben for speaking to one of the elephants in the room in observability: data waste and the impact it has on your bill. All major vendors have a nice dashboard and sometimes alerts to understand usage (broken down by signal type or tags) ... but there's clearly a need for more advanced analysis which Tero seems to be going after. Speaking of the elephant in room in observability: why does storing data on a ven…

Thank you! And you're right, it shouldn't cost that much. Financials are public for many of these vendors: 80%+ margins. The cost to value ratio has gotten way out of whack.

But even if storage were free, there's still a signal problem. Junk has a cost beyond the bill: infrastructure works harder, pipelines work harder, network egress adds up. And then there's noise. Engineers are inundated with it, which makes it harder to debug, understand their systems, and iterate on production. And if engineers struggle with noise and data quality, so does AI.

It's all related. Cheap storage is part of the solution, but understanding has to come first.

Re: How much of my observability data is waste?

#33

Post by jldugger »

binarylogic wrote: post #20
jldugger wrote: post #16

>Turns out you can compile tens of thousands of patterns and still match at line rate. Well, yea, sort of the magic of the regular expression NFA equality theorem. Any regex can be converted to a state machine. And since you can combine regexes (and NFAs!) procedurally, this is not a surprising result. > I ran it against the first service: ~40% waste. Another: ~60%. Another: ~30%. On average, ~40% waste. I'm surprise…

Yeah, it's funny, I never went down the regex rabbit hole until this, but I was blown away by Hyperscan/Vectorscan. It truly changes the game. Traditional wisdom tells you regex is slow. > I'm surprised it's only 40%. Oh, it's worse. I'm being conservative in the post. That number represents "pure" waste without sampling. You can see how we classify it: https://docs.usetero.com/data-quality/logs/malformed-data . If y…

> I think you're describing anomaly detection.

Well it's in the same neighborhood. Anomaly detection tends to favor finding unique things that only happened once. I'm interested in the highest volume stuff that only happens on the abnormal state side. But I'm not sure this has a good name.

> Waste isn't about good vs bad, it's about value: does this data help anyone debug anything, ever?

I get your point but: if sorting by the most strongly associated yields root causes (or at least, maximally interesting logs), then sorting in the opposite direction should yield the toxic waste we want to eliminate?

Re: How much of my observability data is waste?

#34

Post by binarylogic »

whazor wrote: post #19

We store the data because we might need to know it. We only discover we didn’t need to know it once we’ve finished knowing it.

Agree to an extent. There are absolutely unknown unknowns. But I think you'd be surprised how much data is obviously waste. Not the grey area, just pure garbage: health checks, debug logs left in production, redundant attributes.

That's why we break waste down into categories: https://docs.usetero.com/data-quality/categories/overview

But we don't stop there. You can go deeper with reasoning to root out the more nuanced waste. It's hard, but it's possible. That's where things get interesting.

Re: How much of my observability data is waste?

#35

Post by pixl97 »

matanyall wrote: post #3

It's so funny, I've never done a cost-benefit analysis of having "good monitoring" and then still not being able to figure out what broke and needing to pull in someone who doesn't need the monitoring at all because they built the thing.

It's probably something along the lines of "Monitoring solves the problems you expect to have".

For example you don't even question when you see latency going up on some service, you can see DB load going up, and you either manually, or script out another instance starting up.

Monitoring all this stuff allows you to call the DBA/app team/etc 20 minutes sooner when you see some component screw off an you have no idea why. Hopefully that person on the app team puts in a new means of showing what the problem was if it ever happens again, then it turns into the first type of problem you never thing about again (or hope was actually fixed in the application).

Re: How much of my observability data is waste?

#36

Post by otterley »

binarylogic wrote: post #25
otterley wrote: post #13

Earlier quoted context omitted.

And how are you solving the problem? The article does not say. > I'm answering the question your observability vendor won't There was no question answered here at all. It's basically a teaser designed to attract attention and stir debate. Respectfully, it's marketing, not problem solving. At least, not yet.

The question is answered in the post: ~40% on average, sometimes higher. That's a real number from real customer data. But I'm an engineer at heart. I wanted this post to shed light on a real problem I've seen over a decade in this space that is causing a lot of pain; not write a product walkthrough. But the solution is very much real. There's deep, hard engineering going on: building semantic understanding of teleme…

I would contend that it is impossible to know a priori what is wasted telemetry and what isn’t, especially over long time horizons. And especially if you treat your logs as the foundational source of truth for answering critical business questions as well as operational ones.

And besides, the value isn’t knowing that the waste rate is 40% (and your methodology isn’t sufficiently disclosed for anyone to evaluate its accuracy). The value in knowing what is or will be wasted. It’s reminiscent of that old marketing complaint: “I know that half my advertising budget is wasted; I just don’t know which half.”

Storage is actually dirt cheap. The real problem, in my view, is not that customers are wasting storage, but that storage is being used inefficiently, that the storage formats aren’t always mechanically sympathetic and cloud-spend-efficient to the ways they data is read and analyzed, and that there’s still this culturally grounded disparate (and artificial) treatment of application and infrastructure logs vs business records.

Re: How much of my observability data is waste?

#37

Post by nextaccountic »

binarylogic wrote: post #20
jldugger wrote: post #16

>Turns out you can compile tens of thousands of patterns and still match at line rate. Well, yea, sort of the magic of the regular expression NFA equality theorem. Any regex can be converted to a state machine. And since you can combine regexes (and NFAs!) procedurally, this is not a surprising result. > I ran it against the first service: ~40% waste. Another: ~60%. Another: ~30%. On average, ~40% waste. I'm surprise…

Yeah, it's funny, I never went down the regex rabbit hole until this, but I was blown away by Hyperscan/Vectorscan. It truly changes the game. Traditional wisdom tells you regex is slow. > I'm surprised it's only 40%. Oh, it's worse. I'm being conservative in the post. That number represents "pure" waste without sampling. You can see how we classify it: https://docs.usetero.com/data-quality/logs/malformed-data . If y…

But if you don't do anomaly detection, how can you possibly know which data is useful for anomaly detection? And thus, which data is valuable to keep

Re: How much of my observability data is waste?

#38

Post by nextaccountic »

binarylogic wrote: post #26
stackskipton wrote: post #24

Earlier quoted context omitted.

I'd be shocked if you can accurately identify waste since you are not ultimately familiar with the product. Sure, I've kicked over what I thought was waste but told it's not or "It is but deal Ops"

You're right, it's not always binary. That's why we broke it down into categories: https://docs.usetero.com/data-quality/logs/malformed-data You'd be shocked how much obviously-safe waste (redundant attributes, health checks, debug logs left in production) accounts for before you even get to the nuanced stuff. But think about this: if you had a service that was too expensive and you wanted to optimize the data, who w…

I would like to just have a storage engine that can be very aggressive at deduplicating stuff. If some data is redundant, why am I storing it twice?

Re: How much of my observability data is waste?

#39

Post by gmuslera »

Reminded me a note I heard about backups. You don't want backups, it is a waste of time, bandwidth and disk space, by far most if not all of it will end being discarded without being ever used. What you really want is something to restore from if anything breaks. That is the cost that should matter to you. What if you don't have anything meaningful to make a restore from?

With observability is not the volume of data, time and bandwidth used on it, is being able to understand your system and properly diagnose and solve problems when they happen. Can you do that with less? For the next problem that you don't know yet? If you can't because of lack of information or information you didn't collect, then spending so much may be was not enough.

Of course that there are more efficient (towards the end result) ways to do it than others. But having the needed information available, even if it is never used, is the real goal here.

Re: How much of my observability data is waste?

#40

Post by otterley »

quadrature wrote: post #17
otterley wrote: post #13

Earlier quoted context omitted.

And how are you solving the problem? The article does not say. > I'm answering the question your observability vendor won't There was no question answered here at all. It's basically a teaser designed to attract attention and stir debate. Respectfully, it's marketing, not problem solving. At least, not yet.

theres more information here https://docs.usetero.com/introduction/how-tero-works the link in the article is broken. They determine what events/fields are not used and then add filters to your observability provider so you dont pay to ingest them.

What’s the differentiation vs., say, Cribl? Telemetry pipeline providers abound.
Post reply on HN