51–60 of 61 posts
Earlier quoted context omitted.
I would like to just have a storage engine that can be very aggressive at deduplicating stuff. If some data is redundant, why am I storing it twice?
That's already pretty common, but the goal isn't storing less data for its own sake.
Isn't it? I was under impression that the problem is the cost storing all this stuff
Hard agree on the data waste, noise to signal ratio is typically very high and processing, shipping and storing all of that data costs a ton. Previous start-up I worked on (jClarity, exited to Microsoft) mitigated much of this by having a model of only collecting the tiny amount of data that really mattered for a performance bottleneck investigation in a ring buffer and only processing / shipping and storing that dat…
Earlier quoted context omitted.
That's already pretty common, but the goal isn't storing less data for its own sake.
> the goal isn't storing less data for its own sake. Isn't it? I was under impression that the problem is the cost storing all this stuff
Earlier quoted context omitted.
> the goal isn't storing less data for its own sake. Isn't it? I was under impression that the problem is the cost storing all this stuff
Nope, you can't just look at cost of storage and try to minimize it. There are a lot of other things that matter.
Earlier quoted context omitted.
Nope, you can't just look at cost of storage and try to minimize it. There are a lot of other things that matter.
What I am asking is, what are the other concerns other than literally the cost ? I have interest in this area and I am seeing everyone saying that observability companies are overcharging their consumers.
I'm not going to quote the article or other threads here to you about why reducing storage just for the sake of cost isn't the answer.
Last I looked (and looked again just now), if we were to take all our structured logs from all services and send them to Datadog with our current retention policy, it would just about double our current IT spend.
Instead, we use Grafana + Loki + ClickHouse and it's been mostly maintenance-free for years. Costs under $100/month.
What am I missing? What's the real value that folks are getting out of these platforms?
Earlier quoted context omitted.
What I am asking is, what are the other concerns other than literally the cost ? I have interest in this area and I am seeing everyone saying that observability companies are overcharging their consumers.
We're currently discussing the cost of _storage_, and you can bet the providers already are deduplicating it. You just don't get those savings - they get increased margins. I'm not going to quote the article or other threads here to you about why reducing storage just for the sake of cost isn't the answer.
At the most absurd, I've seen observability systems replicated across 3 data centers for an application that hadn't been built yet. But don't worry, by the time it was released, they'd have an observability system so good it couldn't fail (narrator: "It did, in fact, fail.")
Sure, nor perfect, but works surprisingly well.
Hard agree on the data waste, noise to signal ratio is typically very high and processing, shipping and storing all of that data costs a ton. Previous start-up I worked on (jClarity, exited to Microsoft) mitigated much of this by having a model of only collecting the tiny amount of data that really mattered for a performance bottleneck investigation in a ring buffer and only processing / shipping and storing that dat…
In broad strokes, I think this is similar to Bitdrift ( https://bitdrift.io ) - though they’re focused on mobile observability.