Live data from Hacker News

Datadog's $65M/year customer mystery solved

blog.pragmaticengineer.com

41–50 of 66 posts

Re: Datadog's $65M/year customer mystery solved

#41

Earlier quoted context omitted.

Assuming there's nothing else you could do with that elbow grease that would create more value than the SaaS bill costs.

Value is not a hard science. I've seen people shelve tech debt in favor of work on a feature that no one ends up using.

[deleted]

Re: Datadog's $65M/year customer mystery solved

#42
post #10
post #6

I wonder how much that no-expense-spared, money-is-no-object attitude to buying SaaS impacts an engineers ability to make sensible decisions around infra and architecture. Coinbase might have been fine blowing 65 mil but take that approach to a new startup and you could trivially eat up a significant amount of runway with it. I won’t single out Datadog on this because the exact same thing happens with cloud spend, an…

the visible cost of burning runway on a bill is very often far less than the invisible cost of burning engineer time rebuilding undifferentiated heavy lifting rather than working on product/customer needs

I wouldn't really say "very often". Occasionally, perhaps.

Even from a pure zero-sum mathematical perspective, it can make sense to invest even as much as 2 or 3 months of engineer time on cloud cost savings measures. If the engineer is making $200K, that's a $30000 - $50000 investment. When you see the eye-watering cloud bills many startups have, you would realize that, that investment is peanuts in comparison to the potential savings over the next several years.

And then you also have to keep in mind that, these things are usually not actually zero-sum. The engineer could be new, and working on the efficiency project helps them onboard to your stack. It could be the case that customers are complaining (or could start complaining in the future) about how slow your product is, so you actually improve the product by improving the infrastructure. Or it could just be the very common case that there isn't actually a higher-value thing for that engineer to be working on at that time.

Re: Datadog's $65M/year customer mystery solved

#43

Earlier quoted context omitted.

Value is not a hard science. I've seen people shelve tech debt in favor of work on a feature that no one ends up using.

1. Leadership doesn’t want to burn engineer cycles on undifferentiated features. 2. Management doesn’t get recognized for working on undifferentiated features. 3. Engineers working on undifferentiated features aren’t recognized when looking for new jobs. Saving money “makes” sense but getting people to actually prioritize it is hard.

Well, saving money is a differentiator, and one of the best things an engineer can put on their CVs.

Re: Datadog's $65M/year customer mystery solved

#44

> To put it in numbers: the company would make around $9M in revenue it would otherwise lose, Now that $10M/year fee practically pays for itself! am i misunderstanding, or is the author saying it's better to spend $10m than $9m?

you spend that extra million to keep customers satisfied in a competitive industry. they have users trading hundreds of thousands - if there's downtime and they lose money because they weren't able to sell their positions at the right time, they might even try to sue, who knows

Re: Datadog's $65M/year customer mystery solved

#46
post #19
post #6

I wonder how much that no-expense-spared, money-is-no-object attitude to buying SaaS impacts an engineers ability to make sensible decisions around infra and architecture. Coinbase might have been fine blowing 65 mil but take that approach to a new startup and you could trivially eat up a significant amount of runway with it. I won’t single out Datadog on this because the exact same thing happens with cloud spend, an…

>I wonder how much that no-expense-spared, money-is-no-object attitude to buying SaaS impacts an engineers ability to make sensible decisions around infra and architecture I saw this a lot at a previous company. Being able to just "have more Lambdas scale up to handle it" got some very mediocre engineers past challenges they encountered. But it did so at the cost of wasting VAST amounts of money and saddling themselv…

It is also not a very hard skill. You do a back of the envelope calculation and if your proposed architecture is crazy expensive for your reasonable load, then you have to figure out if you are a special snowflake or just doing it wrong.

Re: Datadog's $65M/year customer mystery solved

#47
post #19
post #6

I wonder how much that no-expense-spared, money-is-no-object attitude to buying SaaS impacts an engineers ability to make sensible decisions around infra and architecture. Coinbase might have been fine blowing 65 mil but take that approach to a new startup and you could trivially eat up a significant amount of runway with it. I won’t single out Datadog on this because the exact same thing happens with cloud spend, an…

>I wonder how much that no-expense-spared, money-is-no-object attitude to buying SaaS impacts an engineers ability to make sensible decisions around infra and architecture I saw this a lot at a previous company. Being able to just "have more Lambdas scale up to handle it" got some very mediocre engineers past challenges they encountered. But it did so at the cost of wasting VAST amounts of money and saddling themselv…

What's also frustrating is that a lot of times, costs are hidden from engineering.

I don't know if I would call them mediocre, but without a feedback loop its hard to get engineers to agree whether it's worth time reviewing the code to make it faster compared to just making the db one size larger.

Re: Datadog's $65M/year customer mystery solved

#48
post #10

Earlier quoted context omitted.

the visible cost of burning runway on a bill is very often far less than the invisible cost of burning engineer time rebuilding undifferentiated heavy lifting rather than working on product/customer needs

I wouldn't really say "very often". Occasionally, perhaps. Even from a pure zero-sum mathematical perspective, it can make sense to invest even as much as 2 or 3 months of engineer time on cloud cost savings measures. If the engineer is making $200K, that's a $30000 - $50000 investment. When you see the eye-watering cloud bills many startups have, you would realize that, that investment is peanuts in comparison to th…

> It could be the case that customers are complaining (or could start complaining in the future) about how slow your product is

If Jira has taught me anything, it's that ignoring customers when they complain its too slow makes financial sense.

Re: Datadog's $65M/year customer mystery solved

#49

> Assume that Datadog cuts the number of outages by half, by preventing them with early monitoring. That would mean that without Datadog, we’d look at 24 hours’ worth of downtime, not 12. Let’s also assume that using Datadog results in mitigating outages 50% faster than without - thanks to being able to connect health metrics with logs, debug faster, pinpoint the root cause and mitigate faster. In that case, without…

We are moving from Datadog to Prometheus/Grafana and it's really not all a bed of roses. You'll need monitoring on your monitoring.

Ideally you want two independent monitors for critical things.

Re: Datadog's $65M/year customer mystery solved

#50
post #13
post #10

Earlier quoted context omitted.

the visible cost of burning runway on a bill is very often far less than the invisible cost of burning engineer time rebuilding undifferentiated heavy lifting rather than working on product/customer needs

Most of the complexity in observability is clientside. It is not hard to spin up Grafana and VictoriaMetrics (and now VictoriaLogs) and keep them running. It is not hard to build a Grafana dashboard that correlates data across both metrics and logs sources, and alerting functionality is pretty good now. The "heavy lift" is instrumenting your applications and infrastructure to provide valuable metrics and logs without…

I used to be quite fond of Datadog but after one or two completely surprising bills (thanks to their granular but unintuitive pricing model), I wouldn't recommend it to anybody any more. If I were more cynical I would say the pricing model is designed to be confusing so customers spend more than they need, and this is only made worse by the extreme breadth of the platform now.

These days I'd suggest to just suck it up, spin up a Grafana box, and wire up OpenTelemetry.

Post reply on HN