161–170 of 227 posts
I worked at Coinbase until very recently and can confirm this is Coinbase They paid upfront for 3 years of usage, and yes they were burning > $20m/year on datadog
So you’re saying it’s $65 million over three years? The headline seems misleading, like it’s for a single month.
Earnings call said it was an upfront cost
Stupid question: What open source solution can give me an easy Datadog(ish) experience and is simple to implement?
OpenTelemetry is going to be an existential threat to DataDog and other companies that effectively rely on vendor lock-in to exploit customers. Not sure how companies rationalize these types of services at scale when there are so many open source options to run for a fraction of the cost. You could hire 100+ engineers and still save money compared to a 65M bill https://news.ycombinator.com/item?id=34540419
OpenTelemetry (or OTel) is in no way an existential threat to DataDog. Primarily because OTel is simply the substrate/protocol by which data is collected from your apps/systems. DD does _a lot_ more than what OTel provides (RUM, SIEM, synthetics, on-call, dashboarding, anomaly detection, and much much more). If anything it's an existential threat to the more legacy vendors that aren't equipped to provide an OTel inge…
OpenTelemetry is going to be an existential threat to DataDog and other companies that effectively rely on vendor lock-in to exploit customers. Not sure how companies rationalize these types of services at scale when there are so many open source options to run for a fraction of the cost. You could hire 100+ engineers and still save money compared to a 65M bill https://news.ycombinator.com/item?id=34540419
I’ve seen an attempted move from DD to OT and it was a nightmare of undocumented features and little compounded issues. Tracing was non functional. It doesn’t seem mature enough yet.
Damn. Why not self-host something on that scale?!
They did build a self hosted alternative based on grafana, Loki, Prometheus Had a whole team of 10+ engineers working on it for 2 quarters, then scrapped it because it performed terribly The only thing that came of it was negotiation leverage with datadog ("give us X% off or we go self hosted")
How much logging are we talking here?
Earlier quoted context omitted.
I’ve seen an attempted move from DD to OT and it was a nightmare of undocumented features and little compounded issues. Tracing was non functional. It doesn’t seem mature enough yet.
Were you watching me? Because that's what happened. I am a user of New Relic. Not because I'm happy. But because OpenTelemetry doesn't come close to the same features. Fortunately, at least OT is about 10x harder to set up with worse documentation. Wait a minute...
New Relic is really really good too, so it was even more painful.
Earlier quoted context omitted.
So you’re saying it’s $65 million over three years? The headline seems misleading, like it’s for a single month.
yes but 20 million dollars a year. 20 fucking million getting splunk with 1pb a month ingest isnt that expensive.
Having never heard of datadog, Wikipedia’s summary is: > Datadog is an observability service for cloud-scale applications, providing monitoring of servers, databases, tools, and services, through a SaaS-based data analytics platform. So it checks if your servers have crashed or slowed down with a nice dashboard? Any better summaries or descriptions of what it does and how coinbase would have used it?
(Disclaimer: I work at Chronosphere) Our company helps avoid these kinds of observability bills and issues like scaling for fast-growing cloud deployments. Generally speaking, many vendors let you fall into the cardinality trap b/c they have an economic incentive to let you do so. One of our biggest selling points is that we provide an observability control plane that helps drill down into wasted queries, shows how m…
“We’ll show you how to make sure you don’t have even one crystal fall off the plate.”
My personal pet peeve is Azure Application Insights which uses Log Analytics under the hood… at a rate of $2.75 per ingested GB of logs stored for one month. That’s highway robbery.
Let that sink in: They charge $2,800 to store a TB of text that takes a few hundred dollars of overpriced cloud disk and maybe $10 of CPU time for the actual processing. That’s the cost of a serviceable used car or a brand new gaming PC!
But wait! There’s more.
In reality that 1 TB is column compressed down to maybe 100 MB, making it about $30K charged per terabyte stored on disk.
It doesn’t stop there! Thanks to misaligned incentives, the ingested data format is fantastically inefficient JSON that re-sends static values for every metric sample collected. Why would anyone ever bother to optimise their only revenue?!
They won’t.
The reality is that a numeric metric collected once a second (not minute!) is just 21 MB if stored as a simple array. Most metrics are highly compressible, and that would easily pack to 100 KB per metric per month.
A typical Windows server has about 15,000 performance metrics. We could be collecting these once a second and use a grand total of… 1.5 GB per month. That’s every metric for every process, every device, every error counter, everything.
Modern server monitoring is inefficient and overpriced by 5 orders of magnitude. It’s that simple.
That fact that your company can exist at all is a testament to that.
Earlier quoted context omitted.
Were you watching me? Because that's what happened. I am a user of New Relic. Not because I'm happy. But because OpenTelemetry doesn't come close to the same features. Fortunately, at least OT is about 10x harder to set up with worse documentation. Wait a minute...
yup at my last company CTO wanted to switch away from New Relic to save money for Open Telemetry and I can tell you we wasted months of work because OpenTelemetry is dogshit. New Relic is really really good too, so it was even more painful.