Live data from Hacker News

A cryptocurrency company had a $65M bill, per Datadog’s Q1 earnings call

twitter.com

181–190 of 227 posts

Re: A cryptocurrency company had a $65M bill, per Datadog’s Q1 earnings call

#181

Earlier quoted context omitted.

Grafana, Prometheus, Loki, tempo

Sooo nothing yet :D We attempted to migrate from datadog to prometheus at GitHub and that stack did not cover our use case at all. So much tooling had to be recreated. I took a lot of flak when I pointed out numbers made sense to stay on DataDog and migrate to a Microsoft product instead, but the cost savings spoke for itself

Grafana isn't incentived to simplify self hosting IMO, so this is the best you can do without paying them.

Re: A cryptocurrency company had a $65M bill, per Datadog’s Q1 earnings call

#182

Earlier quoted context omitted.

They did build a self hosted alternative based on grafana, Loki, Prometheus Had a whole team of 10+ engineers working on it for 2 quarters, then scrapped it because it performed terribly The only thing that came of it was negotiation leverage with datadog ("give us X% off or we go self hosted")

What kind of scale of logs are we talking here? The company I work for run a self-hosted Grafana LGTM stack ingesting about 1TB of logs per day, it’s pretty snappy and works well enough, and only costs a few thousand dollars per month in GKE costs for the entire observability stack. How much logging are we talking here?

GitHub has over 21TB of source code. Applications consistently pour through this data and emit logs and events. 1TB of data by breakfast maybe? In reality, we're not pushing logs to datadog, just metrics and event tags. Our level of cardinality, however, requires a lot of horsepower on the backend. Our attempted Prometheus transition was just not cost effective when attempting to view large sets of data over a large-ish period of time. Combined with the heavy lift of integration (we depended heavily on dogstatsd) it just didn't seem efficient to move to Prometheus, support the infrastructure required, all while migrating to microsoft's inhouse product.

Re: A cryptocurrency company had a $65M bill, per Datadog’s Q1 earnings call

#183

Earlier quoted context omitted.

None of this should be even remotely necessary. It’s like being frugal with table salt. “We’ll show you how to make sure you don’t have even one crystal fall off the plate.” My personal pet peeve is Azure Application Insights which uses Log Analytics under the hood… at a rate of $2.75 per ingested GB of logs stored for one month. That’s highway robbery . Let that sink in: They charge $2,800 to store a TB of text that…

"They charge $2,800 to store a TB" !!! good god Totally agree about the compressibility of metrics and toying with the scraping interval. I started out working for an enterprise monitoring vendor that had a proprietary agent that already decided sane intervals to emit metrics, when I learned that Prometheus let users configure that to me...just sounds like an expensive foot gun. My real beef with metrics is at least…

> for app layer insights is the waste.

I’m starting to come to the same conclusion, but the point I’m making is a general one: efficient formats would allow finer grained telemetry to be collected without having to be tuned and carefully monitored.

What’s the point of a monitoring system that itself needs baby sitting?!

Re: A cryptocurrency company had a $65M bill, per Datadog’s Q1 earnings call

#184

Earlier quoted context omitted.

So you’re saying it’s $65 million over three years? The headline seems misleading, like it’s for a single month.

yes but 20 million dollars a year. 20 fucking million getting splunk with 1pb a month ingest isnt that expensive.

Please tell me I am allowed to say "you could just use grep and rsync". I must be allowed!

Re: A cryptocurrency company had a $65M bill, per Datadog’s Q1 earnings call

#185
post #93

Earlier quoted context omitted.

Datadog startup journey from my perspective: 1. use Datadog, because it gets you a bunch of stuff without having to really set it up, like anomaly detection, which is poor man's monitoring & alerting 2. once you start getting product-market fit and the number of instances you run grows you notice your monthly bill going crazier and crazier and you now have something that starts to resemble an ops team -> migrate to a…

I'm not familiar with pricing but depending on alternatives maybe this isn't so bad? > and you now have something that starts to resemble an ops team Like what if Datadog just replaces your ops team completely? What if we start to see AI tools that do cost a lot of money but they can replace a team? Just curious.

Datadog does one part of an ops team, even if it could do all functions of an ops team I would make convincing arguments about your sovereignty of fixing your issues and being resistant to price gouging

Re: A cryptocurrency company had a $65M bill, per Datadog’s Q1 earnings call

#186
post #151

Earlier quoted context omitted.

Their sales team is a nightmare, I had to block them on personal and company email.

I briefly worked there. They literally have no idea what they're doing, like not even a little bit.

I think the vast majority of us don't either

Re: A cryptocurrency company had a $65M bill, per Datadog’s Q1 earnings call

#187

Is anyone using the newer Grafana Cloud Mimir/Tempo/Loki stack that can compare pricing for real usage? I know Datadog probably has a hell of a lot more add-on features, I'm more interested in head to head of comparable produts

It depends a lot on your usage pattern, but we are switching to Granfana cloud from Datadog and are looking at about 1/3 the cost per year. This is using logs, metrics, and traces with OpenTelemetry instrumentation.

The Grafana pricing is more cleanly volume based that is disconnected from number of “hosts” which is where datadog really squeezes you in a kubernetes setup with many pods.

Re: A cryptocurrency company had a $65M bill, per Datadog’s Q1 earnings call

#188

Datadog prices are out of this world. I assumed their front facing prices were for the lone dev, so when I reached out as a company looking to integrate multiple of their products I expected a deal, and got very little compromise. I told them up front they were going to have to cut prices by 90% for us to consider them -- no budge. And they are pretty belligerent salesman, not wanting to leave me alone. After a while…

Datadog costs more to monitor your AWS t3.medium instance than the actual instance. I asked them how them can justify that. They recommended I use "modern infrastructure" which means Docker.

> They recommended I use "modern infrastructure" which means Docker.

It looks to me they recommended to never consider Datadog for anything whatsoever, ever again.

Re: A cryptocurrency company had a $65M bill, per Datadog’s Q1 earnings call

#189
post #121

Earlier quoted context omitted.

Nowadays, incel is synonymous with the Andrew Tate “top g”/alpha sex obsession. Backwards? Definitely, but more symbolic of the arrested development/lack of maturity than the lack of sex. Sleeping with others’ significant others, or actively trying to/bragging about it is a perfect example. Low-class behavior that will get you props from low-class folks.

I've certainly heard that fans of Andrew Tate are incels, or that their world view and advice is built for incels, but I've not heard of many people calling Andrew Tate himself an incel. An incel maker though for sure.

I'm far from an incel (thank you height and sales skills), but I've been around the 'manosphere' online even since early "Ladder Theory" and before pickup artists learned how to internet market.

Reality is some guys are not genetically blessed PLUS they have been lied to when it comes to what women really want. If you take an average real incel, and give him advice from his mother+sister+female friends, and advice from Andrew Tate, I promise you he will get further with Andrew Tates advice.

His mom's advice MIGHT get him a girlfriend that uses him for money, and divorced later in life.. andrew Tate will atleast tell him to get actually attractive through working out and performance at work/business.

Re: A cryptocurrency company had a $65M bill, per Datadog’s Q1 earnings call

#190
post #127

Earlier quoted context omitted.

Datadog costs more to monitor your AWS t3.medium instance than the actual instance. I asked them how them can justify that. They recommended I use "modern infrastructure" which means Docker.

Was Datadog pricing per-host? If so, then I guess running a Kubernetes cluster using the biggest available instances is the most "modern" solution to Datadog-using infrastructure.

Basically, yes.
Post reply on HN