Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Traceway: MIT-licensed observability stack you can self-host in ~90s

github.com

Post reply on HN
Print view View original

61–70 of 93 posts

Re: Traceway: MIT-licensed observability stack you can self-host in ~90s

#61

Post by dusanstanojevic »

adenta wrote: post #3
tecoholic wrote: post #2

I was looking into this just yesterday. So the Loki + … comparison is a bit off in the Open Source space. The main ones are Signoz and ClickStack in this space. Both using ClickHouse as the database. Heavy compared to something like Loki, but they are OTEL native and not log monitoring. So not in the same category.

I'm partial to open observe, especially because in Ruby the OTEL stuff isn't great for metrics and logs yet.

When I was starting Traceway I was heavily inspired by skylightio from the Ruby ecosystem. I loved their SLOs/ranking perf issues, but I also wanted the features that Sentry offered in one place.

Re: Traceway: MIT-licensed observability stack you can self-host in ~90s

#62

Post by denysvitali »

parliament32 wrote: post #45
denysvitali wrote: post #22

Earlier quoted context omitted.

Really depends on the use case. Home lab? Probably. Production? As soon as you scale you need a proper solution. Prometheus (by itself) doesn't scale - you need Mimir or Thanos (or similar). Clickhouse (the "clickstack") seems to be the new kid on the block. Looks very promising.

Note Clickhouse is quite old (2010ish?) but they've always been a "web server access log analytics" solution. The pivot to "we do observability too" is new, we'll see how that plays out. Not terribly optimistic given how badly a similar pivot went for Elastic, but who knows.

I mean, the idea of using OTEL with ClickHouse is rather new, and solves the most painful part of metrics: high cardinality. Has its use-cases, but for sure comes with its own problems

Re: Traceway: MIT-licensed observability stack you can self-host in ~90s

#63

Post by denysvitali »

CyberDildonics wrote: post #29
denysvitali wrote: post #13

At KubeCon Europe a very good chunk of booths were observability stacks. Everyone was claiming they're better than the competitors (with some of the just justifying themselves by saying "it's written in Rust). Having dealt with Prometheus (+Thanos) / Grafana / OTEL and other stacks (e.g: custom solution on ClickHouse, Victoria{Metrics,Logs}, Jaeger/Tempo, Loki, ...) and even cloud ones (Google's Monarch rebranded as…

Is "observability stack" the new term for logs and stats?

You have more than that nowadays. Tracing and profiling are part of O11y too

Re: Traceway: MIT-licensed observability stack you can self-host in ~90s

#64

Post by embedding-shape »

SOLAR_FIELDS wrote: post #59
embedding-shape wrote: post #18

Earlier quoted context omitted.

FWIW, I have no CS degree and barely attended school at all, and found Grafana + Prometheus + Loki fairly easy to setup, at least compared to what we used to use before those tools were available. Maybe it's because I used NixOS for the setup, but besides learning some new domain-specific things I didn't know since before, I don't recall hitting any particular bumps or roadblocks, I also went the 100% self-hosted rou…

FWIW, if you come flying in saying you used NixOS to set something up you’re not what we would call a “casual user”

Why not? Hardly unheard of for managing infrastructure. If we were talking about desktop environments, then maybe, and to be fair, I never said I was a casual user, just that I didn't find prometheus particularly difficult to manage in a production environment.

Re: Traceway: MIT-licensed observability stack you can self-host in ~90s

#65

Post by drzaiusx11 »

denysvitali wrote: post #22
ting0 wrote: post #19

Earlier quoted context omitted.

Do you think Prometheus + Grafana is the way to go?

Really depends on the use case. Home lab? Probably. Production? As soon as you scale you need a proper solution. Prometheus (by itself) doesn't scale - you need Mimir or Thanos (or similar). Clickhouse (the "clickstack") seems to be the new kid on the block. Looks very promising.

We're on AWS Managed Prometheus + Grafana in production and it certainly scales just fine, although I'm sure under the hood it's an entirely different beast than FOSS Prometheus, likely only AWS engineers truly know..

Re: Traceway: MIT-licensed observability stack you can self-host in ~90s

#66

Post by dusanstanojevic »

parliament32 wrote: post #45
denysvitali wrote: post #22

Earlier quoted context omitted.

Really depends on the use case. Home lab? Probably. Production? As soon as you scale you need a proper solution. Prometheus (by itself) doesn't scale - you need Mimir or Thanos (or similar). Clickhouse (the "clickstack") seems to be the new kid on the block. Looks very promising.

Note Clickhouse is quite old (2010ish?) but they've always been a "web server access log analytics" solution. The pivot to "we do observability too" is new, we'll see how that plays out. Not terribly optimistic given how badly a similar pivot went for Elastic, but who knows.

Clickhouse is just a database, it has a really neat feature that infrequently accessed data is pushed back to S3 minimizing the costs. It also heavily compresses the data when storing it.

I am the creator of Traceway and it's my all time fav database. Having said that the repositories in Traceway are completely modular, I've implemented the sqlite version so that I can skip docker containers locally and to simplify self hosting for side projects (it runs on like 2$ servers without issues). This is why it's uniquely suitable for telemetry data and why I've used it as the base of Traceway.

They've acquired HyperDX because it was a major Clickhouse user because their whole platform was telemetry on top of Clickhouse. I hope they don't fully pivot into the space as it would be quite awkward, but there are alternatives and I can always redo repositories with a diff storage engine/db.

Re: Traceway: MIT-licensed observability stack you can self-host in ~90s

#67

Post by dusanstanojevic »

denysvitali wrote: post #13

At KubeCon Europe a very good chunk of booths were observability stacks. Everyone was claiming they're better than the competitors (with some of the just justifying themselves by saying "it's written in Rust). Having dealt with Prometheus (+Thanos) / Grafana / OTEL and other stacks (e.g: custom solution on ClickHouse, Victoria{Metrics,Logs}, Jaeger/Tempo, Loki, ...) and even cloud ones (Google's Monarch rebranded as…

Hi, creator of Traceway here. Sorry for the late response, I didn't know this got posted and then my account was rate limiting on comments.

A lot of tools in this space, most pretty good. The goals when I started Traceway were: - simple to host and reason about - cheap to host - comes pre configured for sub 15 dev teams - completely open source, no paid ad-ons

It's not aimed at teams that can afford SREs (yet), the idea was to provide a good tool for smaller teams and startups in the sub 15 dev range.

The base of Traceway is Clickhouse, nothing special there, if you want you can run it with sqlite for self hosting. Sessions are also stored in S3 so the costs are minimal.

It is opinionated, it comes with preconfigured SLOs for flagging issues with endpoints and it will never try to sell you an AI SRE, you can file your exceptions/slo issues with the git integration and run what ever AI you want on it (I was sick of observability tools trying to sell me an AI). The goal is to have a one line setup, for OpenTelemetry, that gets you everything you need in Traceway without anything needing to be additionally configured. It's Datadog/Sentry but combined and fully open sourced.

I'm a huge fan of open source, here is what we've done so far for making existing solutions better:

1 - Session Replays/RUM

Session replays are usually a premium/expensive feature. With Traceway you can self host them and add them to your app in minutes. I am working on making this a standalone feature that ties into the otel sdks for mobile/js so that you can get your spans/logs/metrics/exceptions from any platform connected to your session replays in Traceway. At one point I got nerd snipped into making it work with Flutter, so we are the only solution I know of that has affordable usable session replays for Flutter.

2 - Symfony Otel

Symfony, the php framework, had no library that offered a few line setup and worked out of the box with open telemetry. We wrote one, you can use it with any tool out there.

3 - Symbolicator

We're working on a symbolicator that will be Open Telemetry Collector compatible, so that you can get your stack traces for Js/Flutter/Android/iOS resolved back. From what I can tell no good solution exists for this currently.

I will make a proper HN post at some point with more info on the project, right now I am focusing on building. If you have any ideas or things you'd like to see feel free to comment, join our discord community or open the issue in our git, we're always happy to accept PRs.

Re: Traceway: MIT-licensed observability stack you can self-host in ~90s

#68

Post by dusanstanojevic »

yuppiepuppie wrote: post #17
denysvitali wrote: post #13

At KubeCon Europe a very good chunk of booths were observability stacks. Everyone was claiming they're better than the competitors (with some of the just justifying themselves by saying "it's written in Rust). Having dealt with Prometheus (+Thanos) / Grafana / OTEL and other stacks (e.g: custom solution on ClickHouse, Victoria{Metrics,Logs}, Jaeger/Tempo, Loki, ...) and even cloud ones (Google's Monarch rebranded as…

I was thinking this might be a result of the Cheap-money (post covid) era ending and everyone scrambling to reduce their Datadog/Cloud costs. Thinking back on 2023/2024, lots of companies were leaking large amounts of capital to those vendors and I imagine lots of people saw an opportunity for creating leaner and cheaper stacks.

No need to guess, I'll tell you the exact story of why I made Traceway!

Last Dec I had a customer complaint, took me 2 days to find the issue. I had to pay $800 for Sentry and a bit more for New Relic. The issue was a locking problem that happened only in very very specific cases, erroring in diff places and timing out in others, unfortunately power users were running into it often. I had two systems, no SLO to catch this and they were completely disconnected. Super annoying.

Anyhow, I spent a day looking at those and eventually went, screw this, I'm gonna just make this actually work. So I spent a few hours, hooked it up, no auth or anything nice, pulled the traces and found the issue. Turns out it was locking due to a long transaction existing in a scheduled task, it existed for years.

The big things for me is it automatically flagging issues, prioritizing them and taking into account: errors, response codes, timing. That's why I'm making it, no venture capital, funded by actual revenue from the start (not paying for Sentry or New Relic anymore). It's really a dev focused tool to help smallish teams find and fix issues before customers even have time to complain.

Anyhow, hope that explains it, kinda related to cloud costs, mostly just my personal frustration with existing tools. Also I did NOT want to host a 5 service stack (grafana, otel collector, prometheus, mimir, loki, k8s) for something that can be done in a 60mb go binary that runs on a 3$ server...

Re: Traceway: MIT-licensed observability stack you can self-host in ~90s

#69

Post by dusanstanojevic »

NortySpock wrote: post #40
denysvitali wrote: post #13

At KubeCon Europe a very good chunk of booths were observability stacks. Everyone was claiming they're better than the competitors (with some of the just justifying themselves by saying "it's written in Rust). Having dealt with Prometheus (+Thanos) / Grafana / OTEL and other stacks (e.g: custom solution on ClickHouse, Victoria{Metrics,Logs}, Jaeger/Tempo, Loki, ...) and even cloud ones (Google's Monarch rebranded as…

If I can ask a separate question: what scalability problems did you run into with Victoria{Metrics|Logs|Traces}, and at what scale did you hit them? VictoriaMetrics and Logs have worked fine in my quiet homelab, and VictoriaMetrics appeared to work great for the infrastructure team of an open source online video game I contribute to (say about 10 physical nodes and 20 applications/services ) ... I was going to sugges…

I honestly think you are a bot. When ever I see Victoria mentioned it is always the same, always asking about hitting a scaling problem + promoting it, never responding to any comments. Hope I'm wrong, but it's been one too many. I refuse to use a product that is this dishonest.

Re: Traceway: MIT-licensed observability stack you can self-host in ~90s

#70

Post by dusanstanojevic »

embedding-shape wrote: post #18
yard2010 wrote: post #15

Earlier quoted context omitted.

I have tried to self host grafana (loki prom and alloy) as o11y stack for prepbook.app. This is hard. I have a bsc in cs not that it says something. I managed to do it eventually, after some research. It was not plug and play in any way. The docs kept saying this solution is not production ready even. I couldn't find the production guide, only the "forget about self hosting and simply pay for us hosting this". After…

FWIW, I have no CS degree and barely attended school at all, and found Grafana + Prometheus + Loki fairly easy to setup, at least compared to what we used to use before those tools were available. Maybe it's because I used NixOS for the setup, but besides learning some new domain-specific things I didn't know since before, I don't recall hitting any particular bumps or roadblocks, I also went the 100% self-hosted rou…

Hi, I'm the creator of Traceway.

I have created Traceway because I looked at that stack and decided that I'm not going to add 7 more services to my stack that could all fail that I now have to maintain as well. Here is the list: Grafana, Otel Collector (to forward metrics), Prometheus, Loki, Tempo, Mimir, K8s.

This is not maintainable in production, unless you have a person to manage it. My app had about 500-1000 req/sec, this sounds like a lot but it's extremely light from the observability perspective. Why would I add 7 more points of failure and services to monitor for proper resource allocation for something like this? To add insult to injury I would have to keep building my SLOs, they wouldn't be tracked automatically by default, I would have to keep paying for Sentry because the issue tracking is quite lacking on Grafana. Oh almost forgot, I would also have to get an alerting provider or pay for that (maybe I'm wrong, it was 6 mo ago).

Anyhow, Traceway is a 60mb binary in Go, it works with Clickhouse or Sqlite and the data is stored on S3 when not used. That means you can host it with sqlite on the 2$ server or even free tier and have it working for your side projects, you can host it with managed clickhouse and get auto scalability on the db level.

The goal is to provide full observability and tools to fix issues directly for developers. What we have so far: alerts, notifications, SSO (google & github), integrations, metrics, preconfigured SLOs, distributed tracing, RUM/session recordings (js & flutter).

Almost forgot, you'd need a symbolicator as well, or your fe/mobile exception stack traces will be messed up in Grafana, I don't even know which tool they have for that, but it's always a new service to host and maintain...

Post reply on HN