Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Traceway: MIT-licensed observability stack you can self-host in ~90s

github.com

Post reply on HN
Print view View original

21–30 of 93 posts

Re: Traceway: MIT-licensed observability stack you can self-host in ~90s

#21

Post by dijit »

embedding-shape wrote: post #18
yard2010 wrote: post #15

Earlier quoted context omitted.

I have tried to self host grafana (loki prom and alloy) as o11y stack for prepbook.app. This is hard. I have a bsc in cs not that it says something. I managed to do it eventually, after some research. It was not plug and play in any way. The docs kept saying this solution is not production ready even. I couldn't find the production guide, only the "forget about self hosting and simply pay for us hosting this". After…

FWIW, I have no CS degree and barely attended school at all, and found Grafana + Prometheus + Loki fairly easy to setup, at least compared to what we used to use before those tools were available. Maybe it's because I used NixOS for the setup, but besides learning some new domain-specific things I didn't know since before, I don't recall hitting any particular bumps or roadblocks, I also went the 100% self-hosted rou…

"Getting it running" is the easy part.

"Getting it ready for production" is a different game.

I've fallen on my sword many times by trying to explain that prometheus fails every metric of production ready; in fact Google themselves replaced borgmon (prometheus) for Monarch because the "tiny unreliable time series databases everywhere" was in fact, not the successful and reliable deployment strategy that they had claimed.

But, it is very easy to set up. Just don't go looking for failure modes, because they're everywhere and every single one of them is catastrophic.

Re: Traceway: MIT-licensed observability stack you can self-host in ~90s

#22

Post by denysvitali »

ting0 wrote: post #19
denysvitali wrote: post #13

At KubeCon Europe a very good chunk of booths were observability stacks. Everyone was claiming they're better than the competitors (with some of the just justifying themselves by saying "it's written in Rust). Having dealt with Prometheus (+Thanos) / Grafana / OTEL and other stacks (e.g: custom solution on ClickHouse, Victoria{Metrics,Logs}, Jaeger/Tempo, Loki, ...) and even cloud ones (Google's Monarch rebranded as…

Do you think Prometheus + Grafana is the way to go?

Really depends on the use case. Home lab? Probably.

Production? As soon as you scale you need a proper solution. Prometheus (by itself) doesn't scale - you need Mimir or Thanos (or similar).

Clickhouse (the "clickstack") seems to be the new kid on the block. Looks very promising.

Re: Traceway: MIT-licensed observability stack you can self-host in ~90s

#23

Post by denysvitali »

dijit wrote: post #21
embedding-shape wrote: post #18

Earlier quoted context omitted.

FWIW, I have no CS degree and barely attended school at all, and found Grafana + Prometheus + Loki fairly easy to setup, at least compared to what we used to use before those tools were available. Maybe it's because I used NixOS for the setup, but besides learning some new domain-specific things I didn't know since before, I don't recall hitting any particular bumps or roadblocks, I also went the 100% self-hosted rou…

"Getting it running" is the easy part. "Getting it ready for production " is a different game. I've fallen on my sword many times by trying to explain that prometheus fails every metric of production ready; in fact Google themselves replaced borgmon (prometheus) for Monarch because the "tiny unreliable time series databases everywhere" was in fact, not the successful and reliable deployment strategy that they had cla…

There are ways to scale Prometheus (look at Thanos), but none of the solutions is really bug free.

See this PR for example (https://github.com/prometheus/prometheus/pull/18364) - this used to impact a production deployment I worked on. Prometheus, Thanos and even OpenTelemetry are full of those kind of problems - but at the same time it's the best we have and we should be grateful they're free and open source.

I'd still choose an open source stack (and contribute to it) rather than go for a proprietary solution - we've all seen what happens with DataDog & co.

Please don't take my words lightly, I worked with the rest of my team in a large scale observability platform and scalability should not be underestimated - at the same time DataDog / Splunk prices are simply unjustified. It's ironically cheaper to build a team of engineers that will maintain a sane observability stack instead of feeding the monster(s).

Re: Traceway: MIT-licensed observability stack you can self-host in ~90s

#24

Post by embedding-shape »

dijit wrote: post #21
embedding-shape wrote: post #18

Earlier quoted context omitted.

FWIW, I have no CS degree and barely attended school at all, and found Grafana + Prometheus + Loki fairly easy to setup, at least compared to what we used to use before those tools were available. Maybe it's because I used NixOS for the setup, but besides learning some new domain-specific things I didn't know since before, I don't recall hitting any particular bumps or roadblocks, I also went the 100% self-hosted rou…

"Getting it running" is the easy part. "Getting it ready for production " is a different game. I've fallen on my sword many times by trying to explain that prometheus fails every metric of production ready; in fact Google themselves replaced borgmon (prometheus) for Monarch because the "tiny unreliable time series databases everywhere" was in fact, not the successful and reliable deployment strategy that they had cla…

Well, I am running the stack in production right now, but everyone has a different understanding of what that actually means...

Do you have concrete examples of these catastrophic failures? I've personally havent experienced any myself during these years, but I'm doing very boring and typical stuff, so wouldn't surprise me there was hard edges still.

Re: Traceway: MIT-licensed observability stack you can self-host in ~90s

#25

Post by etiam »

sexylinux wrote: post #12
neya wrote: post #10

Earlier quoted context omitted.

Here's something better than that: https://github.com/plausible/analytics Elixir.

Why is it better? On the internet it is not enough to just say something. You need to deliver some facts and / or a comparison. Please try it.

Do you have any proof of that?

Re: Traceway: MIT-licensed observability stack you can self-host in ~90s

#26

Post by dijit »

embedding-shape wrote: post #24
dijit wrote: post #21

Earlier quoted context omitted.

"Getting it running" is the easy part. "Getting it ready for production " is a different game. I've fallen on my sword many times by trying to explain that prometheus fails every metric of production ready; in fact Google themselves replaced borgmon (prometheus) for Monarch because the "tiny unreliable time series databases everywhere" was in fact, not the successful and reliable deployment strategy that they had cla…

Well, I am running the stack in production right now, but everyone has a different understanding of what that actually means... Do you have concrete examples of these catastrophic failures? I've personally havent experienced any myself during these years, but I'm doing very boring and typical stuff, so wouldn't surprise me there was hard edges still.

There's a difficult distinction here, you're right.

Technically even a single server running LAMP as root but taking frontend traffic meets the definition of in production but I think we all recognise that it's not the right idea.

What I'm referring to is: should the disk start to have issues: what does prometheus do? If the scrapers start to stall due to connection timeouts: what does prometheus do? If you are doing linear interpolation of data and you have massive gaps because you're polling opportunisitically: what does prometheus do.

I'm all about boring technology, but prometheus assumes too much happy path. It assumes that a single node is enough for time series data that is used for alerting.

Which, it is: at very small scale and with best effort reliability.

It's not acceptable as soon as lost data could be critically important in diagnosing major issues in billing systems, or actually billing users, or to infer issues that need to be correlated across multiple systems.

Re: Traceway: MIT-licensed observability stack you can self-host in ~90s

#27

Post by embedding-shape »

dijit wrote: post #26
embedding-shape wrote: post #24

Earlier quoted context omitted.

Well, I am running the stack in production right now, but everyone has a different understanding of what that actually means... Do you have concrete examples of these catastrophic failures? I've personally havent experienced any myself during these years, but I'm doing very boring and typical stuff, so wouldn't surprise me there was hard edges still.

There's a difficult distinction here, you're right. Technically even a single server running LAMP as root but taking frontend traffic meets the definition of in production but I think we all recognise that it's not the right idea. What I'm referring to is: should the disk start to have issues: what does prometheus do? If the scrapers start to stall due to connection timeouts: what does prometheus do? If you are doing…

> should the disk start to have issues

If that happens, is prometheus really the biggest of your worries here? Software breaks left and right when disks disappear from under them, I'm not sure this is neither unexpected or unique to prometheus.

> If the scrapers start to stall due to connection timeouts: what does prometheus do?

I'm having this "issue" all the time, as some of my WiFi connected (less important) cameras are just within the WiFi range, and I'm using prometheus to scrape metrics from them. It seems like the requests times out, then the next time it doesn't, and everything just works? What's the issue you're experiencing with this exactly?

> It's not acceptable as soon as lost data could be critically important in diagnosing major issues in billing systems, or actually billing users, or

Wait what? Billing systems? That stuff would go into your proper database, wouldn't it? Sure, if prometheus/node_exporter fails or whatever, you won't get metrics out of the host, but again, if those things start failing on that host, the host is having bigger issues than "prometheus suck at scale".

I was eagerly awaiting to be educated about potential gaps in my understanding of prometheus, instead it seems like you simply don't happen to like they way they do things? I was under the impression they did something wrong or something was broken, but these things just seems like the typical stuff you have to think about for any service you deploy.

Re: Traceway: MIT-licensed observability stack you can self-host in ~90s

#28

Post by otterley »

denysvitali wrote: post #23
dijit wrote: post #21

Earlier quoted context omitted.

"Getting it running" is the easy part. "Getting it ready for production " is a different game. I've fallen on my sword many times by trying to explain that prometheus fails every metric of production ready; in fact Google themselves replaced borgmon (prometheus) for Monarch because the "tiny unreliable time series databases everywhere" was in fact, not the successful and reliable deployment strategy that they had cla…

There are ways to scale Prometheus (look at Thanos), but none of the solutions is really bug free. See this PR for example ( https://github.com/prometheus/prometheus/pull/18364 ) - this used to impact a production deployment I worked on. Prometheus, Thanos and even OpenTelemetry are full of those kind of problems - but at the same time it's the best we have and we should be grateful they're free and open source. I'd…

> It's ironically cheaper to build a team of engineers that will maintain a sane observability stack instead of feeding the monster(s).

Can you show the math here? This is a very bold claim, and I’m super curious. A shared Google Sheet would work well.

Re: Traceway: MIT-licensed observability stack you can self-host in ~90s

#29

Post by CyberDildonics »

denysvitali wrote: post #13

At KubeCon Europe a very good chunk of booths were observability stacks. Everyone was claiming they're better than the competitors (with some of the just justifying themselves by saying "it's written in Rust). Having dealt with Prometheus (+Thanos) / Grafana / OTEL and other stacks (e.g: custom solution on ClickHouse, Victoria{Metrics,Logs}, Jaeger/Tempo, Loki, ...) and even cloud ones (Google's Monarch rebranded as…

Is "observability stack" the new term for logs and stats?

Re: Traceway: MIT-licensed observability stack you can self-host in ~90s

#30

Post by dijit »

embedding-shape wrote: post #27
dijit wrote: post #26

Earlier quoted context omitted.

There's a difficult distinction here, you're right. Technically even a single server running LAMP as root but taking frontend traffic meets the definition of in production but I think we all recognise that it's not the right idea. What I'm referring to is: should the disk start to have issues: what does prometheus do? If the scrapers start to stall due to connection timeouts: what does prometheus do? If you are doing…

> should the disk start to have issues If that happens, is prometheus really the biggest of your worries here? Software breaks left and right when disks disappear from under them, I'm not sure this is neither unexpected or unique to prometheus. > If the scrapers start to stall due to connection timeouts: what does prometheus do? I'm having this "issue" all the time, as some of my WiFi connected (less important) camer…

Yes, my monitoring system not alerting me when the systems it runs on are failing is the entire problem.

That's not a general "software breaks when disks fail" situation: that's a monitoring system failing at its one job.

Your monitoring system failing silently when your infrastructure is under stress is precisely the failure mode that monitoring exists to prevent.

Zabbix solves this with native HA and self-checks. Prometheus makes it your problem to solve with external tooling, and most people don't, until they need it.

Post reply on HN