Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Traceway: MIT-licensed observability stack you can self-host in ~90s

github.com

Post reply on HN
Print view View original

31–40 of 93 posts

Re: Traceway: MIT-licensed observability stack you can self-host in ~90s

#31

Post by embedding-shape »

dijit wrote: post #30
embedding-shape wrote: post #27

Earlier quoted context omitted.

> should the disk start to have issues If that happens, is prometheus really the biggest of your worries here? Software breaks left and right when disks disappear from under them, I'm not sure this is neither unexpected or unique to prometheus. > If the scrapers start to stall due to connection timeouts: what does prometheus do? I'm having this "issue" all the time, as some of my WiFi connected (less important) camer…

Yes, my monitoring system not alerting me when the systems it runs on are failing is the entire problem. That's not a general "software breaks when disks fail" situation: that's a monitoring system failing at its one job. Your monitoring system failing silently when your infrastructure is under stress is precisely the failure mode that monitoring exists to prevent. Zabbix solves this with native HA and self-checks. P…

Why wouldn't your monitoring system alert you when metrics suddenly disappear? Sounds like you need a better monitoring system, prometheus is not gonna magically solve that problem for you. No wonder you were having issues with prometheus...

Re: Traceway: MIT-licensed observability stack you can self-host in ~90s

#33

Post by dijit »

embedding-shape wrote: post #31
dijit wrote: post #30

Earlier quoted context omitted.

Yes, my monitoring system not alerting me when the systems it runs on are failing is the entire problem. That's not a general "software breaks when disks fail" situation: that's a monitoring system failing at its one job. Your monitoring system failing silently when your infrastructure is under stress is precisely the failure mode that monitoring exists to prevent. Zabbix solves this with native HA and self-checks. P…

Why wouldn't your monitoring system alert you when metrics suddenly disappear? Sounds like you need a better monitoring system, prometheus is not gonna magically solve that problem for you. No wonder you were having issues with prometheus...

I'm not sure what you mean.

Of course the systems that have to alert me to failure have to be designed with mechanisms to alert me to the fact that they themselves are failing.

Zabbix, Nagios, Munin -- practically everything that existed before: understood this.

Prometheus doesn't because it optimised intentionally for being easy to deploy and for there being a hierarchy of prometheus's in a tree-like formation. Which makes sense, but forces a much more distributed and difficult to reason model.

Monitoring systems can't be designed for the happy path. By definition, they only matter when things are going wrong- which is precisely when the happy path isn't available. Prometheus is excellent when everything is fine (scaling aside). That's not when you need your monitoring system to be excellent.

Re: Traceway: MIT-licensed observability stack you can self-host in ~90s

#34

Post by gnyadav »

robertlagrant wrote: post #20
yuppiepuppie wrote: post #17

Earlier quoted context omitted.

I was thinking this might be a result of the Cheap-money (post covid) era ending and everyone scrambling to reduce their Datadog/Cloud costs. Thinking back on 2023/2024, lots of companies were leaking large amounts of capital to those vendors and I imagine lots of people saw an opportunity for creating leaner and cheaper stacks.

This is my instinct too. I've had the pleasure of using DataDog and the pain of negotiating with their salespeople!

[dead]

Re: Traceway: MIT-licensed observability stack you can self-host in ~90s

#37

Post by Ocha »

robertlagrant wrote: post #20
yuppiepuppie wrote: post #17

Earlier quoted context omitted.

I was thinking this might be a result of the Cheap-money (post covid) era ending and everyone scrambling to reduce their Datadog/Cloud costs. Thinking back on 2023/2024, lots of companies were leaking large amounts of capital to those vendors and I imagine lots of people saw an opportunity for creating leaner and cheaper stacks.

This is my instinct too. I've had the pleasure of using DataDog and the pain of negotiating with their salespeople!

Yes. Their sales people don’t even negotiate - they just tell you this is price and done. Dunno why they need sales person if prices are non-negotiable

Re: Traceway: MIT-licensed observability stack you can self-host in ~90s

#38

Post by RGJorge »

The "easy to set up" framing usually skips the hardest part: whether the metric you're alerting on is meaningful. Most stacks pull container memory from cAdvisor's `container_memory_usage_bytes`, which is the same broken `memory_stats.usage` that `docker stats` reports — includes the kernel's reclaimable page cache. For DB containers with hot working sets, that metric stays at 95%+ constantly. Beautiful Grafana dashboards alerting on a structurally wrong number. The fix is computing real anonymous memory (subtract active_file + inactive_file) — most stacks leave that as a custom exporter exercise. Curious how Traceway handles this out of the box.

Re: Traceway: MIT-licensed observability stack you can self-host in ~90s

#39

Post by embedding-shape »

dijit wrote: post #33
embedding-shape wrote: post #31

Earlier quoted context omitted.

Why wouldn't your monitoring system alert you when metrics suddenly disappear? Sounds like you need a better monitoring system, prometheus is not gonna magically solve that problem for you. No wonder you were having issues with prometheus...

I'm not sure what you mean. Of course the systems that have to alert me to failure have to be designed with mechanisms to alert me to the fact that they themselves are failing. Zabbix, Nagios, Munin -- practically everything that existed before: understood this. Prometheus doesn't because it optimised intentionally for being easy to deploy and for there being a hierarchy of prometheus's in a tree-like formation. Whic…

I think we're running really different monitoring setups, I'd never expect my alerting solution to still be able to alert to me if it's down or degraded, nor would I expect my metrics gathering software to alert me if it's down, that's why I have monitoring setup for those things in the first place.

But, I'm sure your setup makes as much sense in your context as mine makes in my context. As long as it works for you, we're all happy :)

Re: Traceway: MIT-licensed observability stack you can self-host in ~90s

#40

Post by NortySpock »

denysvitali wrote: post #13

At KubeCon Europe a very good chunk of booths were observability stacks. Everyone was claiming they're better than the competitors (with some of the just justifying themselves by saying "it's written in Rust). Having dealt with Prometheus (+Thanos) / Grafana / OTEL and other stacks (e.g: custom solution on ClickHouse, Victoria{Metrics,Logs}, Jaeger/Tempo, Loki, ...) and even cloud ones (Google's Monarch rebranded as…

If I can ask a separate question: what scalability problems did you run into with Victoria{Metrics|Logs|Traces}, and at what scale did you hit them?

VictoriaMetrics and Logs have worked fine in my quiet homelab, and VictoriaMetrics appeared to work great for the infrastructure team of an open source online video game I contribute to (say about 10 physical nodes and 20 applications/services ) ... I was going to suggest VictoriaLogs to them next but wanted to ask what roadblocks could come up.

Post reply on HN