Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Opstrace (YC S19) – open-source Datadog

news.ycombinator.com

Post reply on HN
Print view View original

101–110 of 120 posts

Re: Launch HN: Opstrace (YC S19) – open-source Datadog

#101

Post by fat-apple »

Denzel wrote: post #96
dang wrote: post #69

Earlier quoted context omitted.

Sorry about that! I didn't see anything wrong with the title but I'm not familiar with Datadog's feature set. If you take the title as aspirational, it may make more sense.

No need to be sorry, you do a great job moderating HN dang! I just felt let down because the promise of the title (which I was excited by) doesn’t match reality in that it’s quite impossible to deliver Datadog’s feature set for metrics, traces, and logs — and tie all three together — on top of prometheus+grafana because the underlying TSDB doesn’t even support the notion of user-customizable indexing. Prometheus inde…

We’re truly sorry that you felt let down! You raise a great point that is worth clarifying though. All logs, metrics and future traces are stored in S3/GCS (and yes metrics are stored in TSDB format). While this does not allow a single query language to ask questions across all these sources in one query (yet), it is absolutely possible to build what Datadog is with a new user interface. To go even further, now that it’s all in one place (S3/GCS) other technologies can be leveraged to create a higher level query language across everything, such as PrestoDB (one would have to build backends for it), or even a new dedicated open source columnar store.

Re: Launch HN: Opstrace (YC S19) – open-source Datadog

#102

Post by englambert »

xyzzy_plugh wrote: post #95
dudeinjapan wrote: post #78

Hi there, at TableCheck (www.tablecheck.com) we recently adopted Lightstep. In a nutshell, running all these various components (Grafana, etc) is a royal pain in the neck. Even if `opstrace create` spawns them easily, the problem is running/maintaining them. We want someone to run these for us as a SaaS/PaaS and we're happy to pay them. Re: your principles: (1) The customer should always own their data --> we agree.…

Lightstep was bananas expensive and had several limitations that lead to us moving away from it. Hopefully it's easier to scrub PII from it these days.

Heh, yes, the PII scrubbing need is very real. This is definitely a contributing factor to our data ownership commitment--just keep your data in your own account.

On the cost topic, last week we published a blog post analyzing the cost of running Opstrace on AWS (https://opstrace.com/blog/pulling-cost-curtain-back). (In fact, feel free to do a local repro to confirm our results.) As mentioned elsewhere here on HN, we are incentivized to provide total transparency in terms of what you spend on your cloud infrastructure. We haven’t compared ourselves to everyone, but feel confident that letting our customers pay S3 directly is the best deal possible.

Re: Launch HN: Opstrace (YC S19) – open-source Datadog

#103

Post by englambert »

jgehrcke wrote: post #80
mh- wrote: post #77

Earlier quoted context omitted.

As a heads up, I think you meant to link https://cortexmetrics.io/

Thanks for the correction! You linked to the right Cortex, not to be confused with https://github.com/TheHive-Project/Cortex , haha. https://github.com/cortexproject/cortex is what we talk about. Naming is hard.

:facepalm: Yes, indeed, I conflated the website and the GitHub org. Mea culpa.

Re: Launch HN: Opstrace (YC S19) – open-source Datadog

#104

Post by cyberpunk »

brodouevencode wrote: post #22

We use [insert very large application performance monitoring tool here] for workloads running in [insert very, very large cloud provider here] and after examining our deployments, concluded that we were spending nearly $13k/mo for data transfer out expenditures because the monitoring agents have crazy aggressive defaults. Seems like running our own (which may be worthwhile) would alleviate anything like that.

Can hurt yourself that way too -- happened to us, but with not a lot of data, and all down to Thanos aggregating/reducing/whatever-ing meeeeeeeeeelions of metrics inside a s3 bucket to the tune of about 7k a month :/

Re: Launch HN: Opstrace (YC S19) – open-source Datadog

#105

Post by dudeinjapan »

xyzzy_plugh wrote: post #95
dudeinjapan wrote: post #78

Hi there, at TableCheck (www.tablecheck.com) we recently adopted Lightstep. In a nutshell, running all these various components (Grafana, etc) is a royal pain in the neck. Even if `opstrace create` spawns them easily, the problem is running/maintaining them. We want someone to run these for us as a SaaS/PaaS and we're happy to pay them. Re: your principles: (1) The customer should always own their data --> we agree.…

Lightstep was bananas expensive and had several limitations that lead to us moving away from it. Hopefully it's easier to scrub PII from it these days.

Lightstep has recently changed their pricing, it's now the most cost effective option out there. You pay $100 per service with basically unlimited data ingestion, it's a fabulous deal. https://lightstep.com/pricing/

As for scrubbing PII, they are now supporting the OpenTelemetry tracing API which does this as standard. For query endpoints you will see something like "QUERY Users where name=? email=?", i.e. masking with "?" chars as you only care about the keys since those what determine your indexing or lack thereof. (This is handled in the OpenTelemetry application library/plugin level.)

As an aside, PII scrubbing should be done that whether or not you own the cluster, because even if you own the cluster you generally don't want your support staff seeing PII esp. as organization grows larger.

Re: Launch HN: Opstrace (YC S19) – open-source Datadog

#106

Post by dudeinjapan »

jgehrcke wrote: post #82
dudeinjapan wrote: post #78

Hi there, at TableCheck (www.tablecheck.com) we recently adopted Lightstep. In a nutshell, running all these various components (Grafana, etc) is a royal pain in the neck. Even if `opstrace create` spawns them easily, the problem is running/maintaining them. We want someone to run these for us as a SaaS/PaaS and we're happy to pay them. Re: your principles: (1) The customer should always own their data --> we agree.…

Jan-Philip from Opstrace here. This is lovely feedback! > is a royal pain in the neck. It's fun to see how different people put the same unpleasant experience into words in this thread. Thanks for adding your personal touch. Every time we hear something like that, we're re-assured that we're on the right track. > Even if `opstrace create` spawns them easily, the problem is running/maintaining them Yes. You're right.…

Will definitely keep an eye on your service. Again, don't underestimate users willingness to pay for a PaaS product you make, even if you also do a dual PaaS/self-hosted option like Gitlab or MongoDB does, for example. We'd definitely prefer the PaaS, and that's where the big $$$ is made these days.

By the way, we are requiring that any vendor we choose in this area support OpenTelemetry as we've already instrumented our apps with it. Lightstep, Datadog, and others already supporting.

Re: Launch HN: Opstrace (YC S19) – open-source Datadog

#107

Post by dudeinjapan »

BringerOfChaos wrote: post #91
dudeinjapan wrote: post #78

Hi there, at TableCheck (www.tablecheck.com) we recently adopted Lightstep. In a nutshell, running all these various components (Grafana, etc) is a royal pain in the neck. Even if `opstrace create` spawns them easily, the problem is running/maintaining them. We want someone to run these for us as a SaaS/PaaS and we're happy to pay them. Re: your principles: (1) The customer should always own their data --> we agree.…

@dudeinjapan, check out https://grafana.com/products/cloud/ The first line on the page says, "Your observability, managed as a service" It might not fit your use case...but it might.

We used self-hosted Grafana until late 2020. We looked at Grafana cloud but ultimate chose a combination of NewRelic One for metric monitoring and Lightstep for request tracing. Those two were the "best-of-breed" for their respective areas with the best pricing for our use case.

Re: Launch HN: Opstrace (YC S19) – open-source Datadog

#108

Post by alexhutcheson »

One pain point with Prometheus is that is has relatively weak support for quantiles, histograms, and sets[1]:

- Histograms require manually specifying the distribution of your data, which is time-consuming, lossy, and can introduce significant error bands around your quantile estimates.

- Quantiles calculated via the Prometheus "summary" feature are specific to a given host, and not aggregatable, which is almost never what you want (you normally want to see e.g. the 95th percentile value of request latency for all servers of a given type, or all servers within a region). Quantiles can be calculated from histograms instead, but that requires a well-specified histogram and can be expensive at query time.

- As far as I know, Prometheus doesn't have any explicit support for unique sets. You can compute this at query time, but persisting and then querying high-cardinality data in this way is expensive.

Understanding the distribution of your data (rather than just averages) is arguably the most important feature you want from a monitoring dashboard, so the weak support for quantiles is very limiting.

Veneur[2] addresses these use-cases for applications that use DogStatsD[3] by using clever data structures for approximate histograms[4] and approximate sets[5], but I believe its integration with Prometheus is limited and currently only one-way - there is a CLI app to poll Prometheus metrics and push them into Veneur[6], but there's no output sink for Veneur to write to Prometheus (or expose metrics for a Prometheus instance to poll), and you aren't able to use the approximate histogram or approximate set datatypes if you go that route, because they can't be expressed as Prometheus metrics.

It would be extremely useful to have something similar for Prometheus, either by integrating with Veneur or implementing those data structures as an extension to Prometheus.

[1] https://prometheus.io/docs/practices/histograms/

[2] https://github.com/stripe/veneur

[3] https://docs.datadoghq.com/developers/dogstatsd/

[4] https://github.com/stripe/veneur#approximate-histograms

[5] https://github.com/stripe/veneur#approximate-sets

[6] https://github.com/stripe/veneur/tree/master/cmd/veneur-prom...

Re: Launch HN: Opstrace (YC S19) – open-source Datadog

#109

Post by erinnh »

spahl wrote: post #42
erinnh wrote: post #40

Earlier quoted context omitted.

Will you have a small(er) plan for homelabs? I like supporting open source projects, and while SSO is pretty useless to me, I always like custom domains.

We are still experimenting with pricing and what can be open and closed. To be completely transparent we chose custom domains because we know companies care a lot. When we have more features on the commercial side we can start to chat about supporting it in the open version. Still early in our journey, happy to discuss anything, like a small plan with just custom domains. Would you pay for that?

Yeah. That’s kind of what I mean. I have no problem paying some money to support you guys and not have to host it on my own. I generally prefer my monitoring etc to not be done by myself anyway cause I take myself offline way too often

Only point was of there was going to be a smaller plan for homelabs or the like that doesn’t have the raw amount of Traffic or features as the enterprise plans do.

Re: Launch HN: Opstrace (YC S19) – open-source Datadog

#110

Post by Denzel »

fat-apple wrote: post #101
Denzel wrote: post #96

Earlier quoted context omitted.

No need to be sorry, you do a great job moderating HN dang! I just felt let down because the promise of the title (which I was excited by) doesn’t match reality in that it’s quite impossible to deliver Datadog’s feature set for metrics, traces, and logs — and tie all three together — on top of prometheus+grafana because the underlying TSDB doesn’t even support the notion of user-customizable indexing. Prometheus inde…

We’re truly sorry that you felt let down! You raise a great point that is worth clarifying though. All logs, metrics and future traces are stored in S3/GCS (and yes metrics are stored in TSDB format). While this does not allow a single query language to ask questions across all these sources in one query (yet), it is absolutely possible to build what Datadog is with a new user interface. To go even further, now that…

> other technologies can be leveraged

That’s my point: your software architecture must evolve to deliver on your promises (open-source Datadog) because it’s impossible to satisfy your promises with the architecture as it exists today.

Nonetheless, I appreciate all the responses. Good luck to OpsTrace!

Post reply on HN