Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Traceway: MIT-licensed observability stack you can self-host in ~90s

github.com

Post reply on HN
Print view View original

41–50 of 93 posts

Re: Traceway: MIT-licensed observability stack you can self-host in ~90s

#41

Post by dijit »

embedding-shape wrote: post #39
dijit wrote: post #33

Earlier quoted context omitted.

I'm not sure what you mean. Of course the systems that have to alert me to failure have to be designed with mechanisms to alert me to the fact that they themselves are failing. Zabbix, Nagios, Munin -- practically everything that existed before: understood this. Prometheus doesn't because it optimised intentionally for being easy to deploy and for there being a hierarchy of prometheus's in a tree-like formation. Whic…

I think we're running really different monitoring setups, I'd never expect my alerting solution to still be able to alert to me if it's down or degraded, nor would I expect my metrics gathering software to alert me if it's down, that's why I have monitoring setup for those things in the first place. But, I'm sure your setup makes as much sense in your context as mine makes in my context. As long as it works for you,…

"I have monitoring set up for those things" - but that doesn't solve the ambiguity. When Prometheus misses a scrape, nothing fires. Silence looks identical whether your service is down, the network blipped, or Prometheus itself is struggling. A defensive monitoring system has to treat absence of data as a signal, not just absence of a problem.

Re: Traceway: MIT-licensed observability stack you can self-host in ~90s

#43

Post by sebakubisz »

RGJorge wrote: post #38

The "easy to set up" framing usually skips the hardest part: whether the metric you're alerting on is meaningful. Most stacks pull container memory from cAdvisor's `container_memory_usage_bytes`, which is the same broken `memory_stats.usage` that `docker stats` reports — includes the kernel's reclaimable page cache. For DB containers with hot working sets, that metric stays at 95%+ constantly. Beautiful Grafana dashb…

Curious what LLM model you are.

Re: Traceway: MIT-licensed observability stack you can self-host in ~90s

#44

Post by parliament32 »

denysvitali wrote: post #13

At KubeCon Europe a very good chunk of booths were observability stacks. Everyone was claiming they're better than the competitors (with some of the just justifying themselves by saying "it's written in Rust). Having dealt with Prometheus (+Thanos) / Grafana / OTEL and other stacks (e.g: custom solution on ClickHouse, Victoria{Metrics,Logs}, Jaeger/Tempo, Loki, ...) and even cloud ones (Google's Monarch rebranded as…

FWIW we've also tried all sorts of different things, and honestly the very vanilla (prometheus -> central thanos, fluentbit -> central loki, grafana) ends up on top. The resource consumption is surprisingly minimal (for a sense of scale, we run about 200k eps for metrics and 1k eps for logs). For all these solutions, I find myself asking the same question as you.. what problem are you trying to solve? Is there anything actually different about your product other than less stability than the battle-tested stack?

Re: Traceway: MIT-licensed observability stack you can self-host in ~90s

#45

Post by parliament32 »

denysvitali wrote: post #22
ting0 wrote: post #19

Earlier quoted context omitted.

Do you think Prometheus + Grafana is the way to go?

Really depends on the use case. Home lab? Probably. Production? As soon as you scale you need a proper solution. Prometheus (by itself) doesn't scale - you need Mimir or Thanos (or similar). Clickhouse (the "clickstack") seems to be the new kid on the block. Looks very promising.

Note Clickhouse is quite old (2010ish?) but they've always been a "web server access log analytics" solution. The pivot to "we do observability too" is new, we'll see how that plays out. Not terribly optimistic given how badly a similar pivot went for Elastic, but who knows.

Re: Traceway: MIT-licensed observability stack you can self-host in ~90s

#47

Post by jmalicki »

dijit wrote: post #26
embedding-shape wrote: post #24

Earlier quoted context omitted.

Well, I am running the stack in production right now, but everyone has a different understanding of what that actually means... Do you have concrete examples of these catastrophic failures? I've personally havent experienced any myself during these years, but I'm doing very boring and typical stuff, so wouldn't surprise me there was hard edges still.

There's a difficult distinction here, you're right. Technically even a single server running LAMP as root but taking frontend traffic meets the definition of in production but I think we all recognise that it's not the right idea. What I'm referring to is: should the disk start to have issues: what does prometheus do? If the scrapers start to stall due to connection timeouts: what does prometheus do? If you are doing…

What is the disk? You've already failed by not running distributed. The problem isn't Prometheus, it's "the cloud is too expensive I'll just run on a single VPS"

Re: Traceway: MIT-licensed observability stack you can self-host in ~90s

#48

Post by dijit »

jmalicki wrote: post #47
dijit wrote: post #26

Earlier quoted context omitted.

There's a difficult distinction here, you're right. Technically even a single server running LAMP as root but taking frontend traffic meets the definition of in production but I think we all recognise that it's not the right idea. What I'm referring to is: should the disk start to have issues: what does prometheus do? If the scrapers start to stall due to connection timeouts: what does prometheus do? If you are doing…

What is the disk? You've already failed by not running distributed. The problem isn't Prometheus, it's "the cloud is too expensive I'll just run on a single VPS"

Prometheus does not run a distributed tsdb.
Post reply on HN