Most companies do not need Snowflake or Databricks

If the price for staying focused is $X, and X has a positive business impact and need, like Snowflake or Databricks could.

And it doesn't mean you won't need to spend on HR, developers, management and so on to run your own OSS solution.

It is actually great to give Snowflake and/or Databricks your money. It's a really expensive service, with a huge markup. That I understand.

But the alternative doesn't look good at all. A company is better staying focused.

Also, if the company sees that it expends a considerable amount of its budget in Snowflake/Databricks, they'll find solutions, that could be negotiating with them, figuring out how to optimize their use etc.

I've worked and seen many companies optimizing their cost structure and run away from Datadog/Newrelic, Snowflake/Databricks and just fail miserably.

The alternatives made developers simply use it less, and also spend more time to do the same that they did.

They even sometimes call it successful, execs point out the money saved, but they don't see the subjective part: developers being less efficient, wasting their time with BS.

There's a long tail of costs related to inefficient systems, it's wasting developer hours, that you need to manage, hire, train...

Not to forget the burden that it is to fire or make people redundant, companies end up losing so much of their culture and soul doing such things. And this is what you need to typically do if you assume those expenses of running things for yourself.

The ideal company are platforms that pays for many SaaS and stay focused on its core business, if it can make money, great!

There is so much literature on cost optimization in business in general, stuff that have been studied for over a century. If people would read on that, they wouldn't be repeating such nonsense as this article.

Also, the article says that you are OVERPAYING for it if you aren't a Fortune 100. It's much more like the opposite, if you have a small company with a small amount of data, it's very likely that Snowflake or Databricks won't cost you that much.

Based also on the other comments on HN, I can only conclude that this is one of those articles that is so full of mistakes and plainly wrong, that there isn't even a real debate over here, just people saying how wrong it is.

It makes sense for Databricks to do this because they're building a product that needs to work for a high cardinality of datasets. Using a DNN for that is defensible because the input shapes are practically infinite. But for individual orgs, it seems much more likely that simple heuristics-driven access pattern optimizations can be done without throwing ML at the problem (though I'll say the predictive IO concept is a cool one, I've done similar ML work for network traffic QoS).

> I've seen lots of DIY data platforms. They're horrible to work with and I can assure you that the code does not write itself.

This one made me scratch my head a bit, because I have seen many DIY platforms as well over a 20 year career. Most of them have been amazing to work with. The code didn't write itself of course, but maintenance burdens were low and specialization/expertise within the org was high as a result. On top of that, I didn't need to argue with an AE about rising costs on a per-annum basis whenever renewal time comes around.

I point this out not in the service of coloring your observations as wrong or misguided, but to highlight that there's going to be a spectrum of varied lived experiences people have with the build-vs-buy conundrum. I expect we'll never see genuine consensus on this issue as a result of that variance (and maybe we don't have to).

I keep a whole bunch of small-ish datasets in Snowflake because it means I don't have to do all the work to janitor my own postgres install. I go with the db someone else maintains for me that I have, not the one that I wish I had.