Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: HyperDX – open-source dev-friendly Datadog alternative

github.com

Post reply on HN
Print view View original

141–150 of 186 posts

Re: Show HN: HyperDX – open-source dev-friendly Datadog alternative

#141

Post by mikeshi42 »

Wulfheart wrote: post #139

So do I understand the landing page correctly: It is possible to run Clickhouse using an Object Storage like S3? What are the performance implications?

You can definitely run Clickhouse directly on S3 [1] - though we don't run _just_ on S3 for performance reasons but instead use a layered disk strategy.

A few of the weaknesses of S3 are:

1. API calls are expensive, while storage in S3 is cheap, writing/reading into it is expensive. Using only S3 for storage will incur lots of API calls as Clickhouse will work on merging objects together (which require downloading the files again from S3 and uploading a merged part) continuously in the background. And searching on recent data on S3 can incur high costs as well, if you're constantly needing to do so (ex. alert rules)

2. Latency and bandwidth of S3 are limited, SSDs are an order of magnitude faster to respond to IO requests, and also on-device SSDs typically have higher bandwidth available. This typically is a bottleneck for reads, but typically not a concern for writes. This can be mitigated by scaling out network-optimized instances, but is just another thing to keep in mind.

3. We've seen some weird behavior on skip indices that can negatively impact performance in S3 specifically, but haven't been able to identify exactly why yet. I don't recall if that's the only weirdness we see happen in S3, but it's one that sticks out right now.

Depending on your scale and latency requirements - writing directly to S3 or a simple layered disk + S3 strategy might work well for your case. Though we've found scaling S3 to work at the latencies/scales our customers typically ask for require a bit of work (as with scaling any infra tool for production workloads).

[1] https://clickhouse.com/docs/en/integrations/s3

Re: Show HN: HyperDX – open-source dev-friendly Datadog alternative

#142

Post by mikeshi42 »

cheema33 wrote: post #140

I am new to this space and was considering a self hosted install of Sentry software. Sentry is also opensource and appears to be similar to datadog and HyperDX in some ways. Do you know Sentry and can you tell us how your product is different? Thanks.

Very familiar with Sentry! I think we have a bit of overlap in that we both do monitoring and help devs debug though here's where I think we differ:

HyperDX:

- Can collect all server logs (to help debug issues even if an exception isn't thrown)

- We can collect server metrics as well (CPU, memory, etc.)

- We accept OpenTelemetry for all your data (logs, metrics, traces) - meaning you only need to instrument once and choose to switch vendors at any time if you'd like without re-instrumenting.

- We can visualize arbitrary data (what's the response time of endpoint X, how many users did action Y, how many times do users hit endpoint X grouped by user id?) - Sentry is a lot more limited in what it can visualize (mainly because it collects more limited amounts of data).

Sentry:

- Great for exception capture, it tries to capture any exception and match them with sourcemap properly so you can get to the right line of code where the issue occurred. We don't have proper sourcemap support yet - so our stack traces point to minified file locations currently.

- Gives you a "inbox" view of all your exceptions so you can see which ones are firing currently, though you can do something similar in HyperDX (error logs, log patterns, etc.) theirs is more opinionated to be email-style inbox, whereas our is more about searching errors.

- Link your exceptions to your project tracker, so you can create Jira, Linear, etc. tickets directly from exceptions in Sentry.

I don't think it's an either/or kind of situation - we have many users that use both because we cover slightly different areas today. In the future we will be working towards accepting exception instrumentation as well, to cover some of our shortfalls when it comes to Sentry v HyperDX (since one common workflow is trying to correlate your Sentry exception to the HyperDX traces and logs).

Hope that gives you an idea! Happy to chat more on our Discord if you'd like as well.

Re: Show HN: HyperDX – open-source dev-friendly Datadog alternative

#143

Post by mikeshi42 »

specialist wrote: post #117
mikeshi42 wrote: post #102

Earlier quoted context omitted.

Yes! It should definitely be thoughtful about what you log and how you expect to use it. My biggest gripe with logs is often people writing them never think about "how would I use this when things are on fire?" and tend to log useless information or fail to tag them in ways that are actually searchable. Tagging the right IDs are a huge thing - customer X is saying their instance is really slow, but if none of your lo…

PS- I just skimmed https://opentelemetry.io , which your readme.md links to. Good stuff. Much industry progress since I was last in the arena. Their site has words about manual and automatic instrumentation. I'd have to dig a bit to see what they mean. -- So. Remembering a bit more... Will try to keep this brief; you're a busy person. > tend to log useless information or fail to tag them in ways that are actually sea…

Yes OpenTelemetry is awesome in what it's done for the industry - it was really early when I was still at Mezmo/LogDNA but it's matured a lot, though I think still has a ways more to go.

For automatic logging - I think you'd enjoy OpenTelemetry's automatic tracing implementation, it helps pull out standard telemetry from things like your Redis requests and correlate them with trace IDs so you can tie everything together from the moment your server starts accepting the HTTP request to the Redis and DB requests and what was sent in each request (without needing to do it manually)

For capturing HTTP req/res - we actually have a few options depending on the language (ex. we do this for Python and Node.js) to enable more advanced network capture (so you can actually get the full req/res information, or whatever subset you're interested in actually storing)! It's actually been asked by a number of teams to make it easier to debug tricky API issues they're running into.

Proper end-to-end logging definitely makes it easier to find the right clue among a sea of logs, hopefully we make it easier to get there!

Re: Show HN: HyperDX – open-source dev-friendly Datadog alternative

#145

Post by mx20 »

mikeshi42 wrote: post #29
addisonj wrote: post #19

Wow, there is a lot here and what here is to a pretty impressive level of polish for how far along this is. The background of someone with a DX background comes through! I will be looking into this a lot more. Here are a few comments, notes, and questions: * I like the focus on DX (especially compared to other OSS solutions) in your messaging here, and I think your hero messaging tells that story, but it isn't reinfo…

Thank you, really appreciate the feedback and encouragement! > It seems like clickhouse is obviously a big piece of the tech here, which is an obvious choice, but from my experience with high data rate ingest, especially logs, you can run into issues at larger scale. Is that something you expect to give options around in open source? Scaling any system can be challenging - our experience so far is that Clickhouse is…

MIT License allows Amazon and other Cloud providers to offer Cloud Solutions as well. That's why most SaaS changed to AGPL or better versions that explicitly disallow Cloud offerings.

Re: Show HN: HyperDX – open-source dev-friendly Datadog alternative

#146

Post by mnahkies »

One thing I appreciate about sentry compared to datadog is the ability to configure hard caps on ingestion to control cost. AFAIK the mechanism is basically that the server starts rate limiting/rejecting requests and the client SDKs are written to handle this and enter a back off state or start sampling events.

I think this could be a nice point of difference to explore that can help people avoid unexpected bills

Re: Show HN: HyperDX – open-source dev-friendly Datadog alternative

#147

Post by datadeft »

mikeshi42 wrote: post #29
addisonj wrote: post #19

Wow, there is a lot here and what here is to a pretty impressive level of polish for how far along this is. The background of someone with a DX background comes through! I will be looking into this a lot more. Here are a few comments, notes, and questions: * I like the focus on DX (especially compared to other OSS solutions) in your messaging here, and I think your hero messaging tells that story, but it isn't reinfo…

Thank you, really appreciate the feedback and encouragement! > It seems like clickhouse is obviously a big piece of the tech here, which is an obvious choice, but from my experience with high data rate ingest, especially logs, you can run into issues at larger scale. Is that something you expect to give options around in open source? Scaling any system can be challenging - our experience so far is that Clickhouse is…

> While we need to make money at the end of the day

Honest question: What makes you think that you are not turning into a Datadog (price wise) once reach a certain scale?

The problem what I see with software companies that the pricing is dominated by investor requirements and when a company reaches a certain milestone change up the licensing model and the pricing with it.

Re: Show HN: HyperDX – open-source dev-friendly Datadog alternative

#149

Post by mikeshi42 »

mnahkies wrote: post #146

One thing I appreciate about sentry compared to datadog is the ability to configure hard caps on ingestion to control cost. AFAIK the mechanism is basically that the server starts rate limiting/rejecting requests and the client SDKs are written to handle this and enter a back off state or start sampling events. I think this could be a nice point of difference to explore that can help people avoid unexpected bills

Agreed on needing better tooling for surprise bills - definitely no stranger to that problem!

For now we're trying to make the base price cheap enough where those kinds of considerations don't need to be top of mind today and a policy that can be forgiving when it occasionally happens, but certainly as we continue to scale and grow, we'll need to put in proper controls to allow users to define what should happen if events are spiking unexpectedly (how to shed events via sampling, what needs to be explicitly perserved for compliance reasons, when to notify, etc.)

I do like Sentry's auto-sampling algorithm which is a really neat way to solve that issue.

Re: Show HN: HyperDX – open-source dev-friendly Datadog alternative

#150

Post by tmd83 »

Sytten wrote: post #118

Anyone has objectives blogs/videos that tested/compared all those new platforms? I feel like I see a new one on HN every month. From my quick research: signoz, openobserve, uptrace, highlight.io, opstrace. I would like to recommend some alternatives to my clients, but I don't have time to test them all and keep up with their progress. I am also worried about long term viability of those platforms. Consolidation is bo…

I would love to read something like that too. I find such tools are fairly hard to evaluate since some of the challenges only comes with scale and you often need a real/realistic scenario to actually figure out if the tool will be useful in a pinch.
Post reply on HN