Live data from Hacker News

Why is Snowflake so expensive

blog.devgenius.io

201–210 of 217 posts

Re: Why is Snowflake so expensive

#201

I'm the author of the article. Didn't expect it to blow up. Let me clarify a few points: 1. I like Snowflake and I think they brought several innovations to the field: Instant scale out/up, time-travel, unstructured data query support. 2. Snowflake obviously makes innovations and performance improvements, otherwise they would not be the market leader they are. But I'm also suspecting that they make just enough perfor…

Totally agree with Redshift sentiments. It's been lovely seeing BigQuery and Redshift step their game up over the past 1.5yrs, because they really should have been doing certain things for many years prior. Re: Firebolt, I don't consider it to be in the same class as Snowflake whatsoever (even though their advertising seems to indicate otherwise). Snowflake is like a very powerful swiss army knife. Firebolt is good f…

> Firebolt is good for a very specific (dare I say niche?) workload but falls all over itself for the vast majority of data org needs.

It runs SQL queries on structured data. Is that niche?

Re: Why is Snowflake so expensive

#202

Snowflake is not expensive. Snowflake is super cheap, IF you know what it is for and how to use it. Compared to if you had to solve the problem on your own. The best way to describe Snowflake is that it is a brute force method to run complex queries without creating indexes. If you have a more traditional database, you will notice you need to set up indexes to be able to get anything from it in finite time. What if y…

Very clear-minded take on what Snowflake is great and not so great for. Snowflake is great and cheap for what it's meant for. It gets expensive when you try to use it for something it was not designed for.

Re: Why is Snowflake so expensive

#203

Earlier quoted context omitted.

Only competition can enforce this. The article ideally demonstrates the problems with monopolies and vendor lock-in.

Snowflake is nowhere near a monopoly, and plenty of customers have moved from other vendors (Teradata, Netezza, etc) to Snowflake - showing that vendor lock-in is not as strong as it might seem.

If that's the case, then why aren't Snowlake and Google targeting query optimiziation with higher priority to lower end-user costs? There's no incentive in the market for them to do so - once you switch to them, you'll eat up the cost of quirks and learn how to avoid them the hard way.

Re: Why is Snowflake so expensive

#204

Earlier quoted context omitted.

It's a terrible article. The author misunderstands competition and how much it drives products in this area. Snowflake is incentivized to make their product better on every dimension. If Snowflake don't improve, customers will leave in droves - like when they moved to Snowflake . In practice, as has been pointed out in other comments, they do improve their performance (for competitive reasons) and it does cost them m…

It is a terrible article. I’ve been on the engineering side of these big data platforms including snowflake in its early days, Paraccel (redshift’s code ancestor), redshift, and others you probably use but don’t realize are actually hyper scale database engines. The author missed the mark consistently. I chortled when he discussed the redshift WLM which I helped design a very long time ago and it’s absolute garbage.…

I'm a snowflake customer and I've felt/am feeling all of the pain that this article talks about. There might be some handwaving over technical complexity that you don't like given your detailed understanding of how the thing is built, but the article is fundamentally right in its message.

The thing it's most right about is the power imbalance and the innovators dilemma. I've had more than one instance of the case where we've found that query performance/cost is too high, complained about it, and Snowflake have "made a configuration change" (undisclosed) that has brought the cost down.

Re: Why is Snowflake so expensive

#205
post #7

Great article. On the surface, it's about Snowflake. At a deeper level, the article is about the perverse incentives motivating SaaS businesses to do seemingly dumb, inefficient things and avoid seemingly obvious optimizations by default. Many SaaS businesses are perfectly happy to let customers shoot themselves in the foot if it generates more revenue. The BigQuery example (presently, by default, `select * from tabl…

It's a terrible article. The author misunderstands competition and how much it drives products in this area. Snowflake is incentivized to make their product better on every dimension. If Snowflake don't improve, customers will leave in droves - like when they moved to Snowflake . In practice, as has been pointed out in other comments, they do improve their performance (for competitive reasons) and it does cost them m…

> It's a terrible article. The author misunderstands competition and how much it drives products in this area.

Agree, but the author has one thing right. Snowflake is not transparent about product behavior, which makes it hard to reason about costs and performance.

Open source data warehouses like ClickHouse and Druid don't have this problem. If you want to know how something works, you can look at the code. Or listen to talks from the committers. This transparency is an enduring strength of open source projects.

Re: Why is Snowflake so expensive

#206

Earlier quoted context omitted.

I worked at a BiqQuery shop and they have a terrific feature where right next to the “Run query” button there is an estimate of the cost of the query, in bytes. It becomes extremely obvious when a query is a full table scan.

Ha! I wonder if we worked at the same place ... it's in the travel space, because when I worked there someone wrote a plugin that did this, and it was a real eye opener at times!

Nope, not travel! For us it was a built in feature, not a plugin. This was earlier this year also.

Re: Why is Snowflake so expensive

#207

Earlier quoted context omitted.

It is a terrible article. I’ve been on the engineering side of these big data platforms including snowflake in its early days, Paraccel (redshift’s code ancestor), redshift, and others you probably use but don’t realize are actually hyper scale database engines. The author missed the mark consistently. I chortled when he discussed the redshift WLM which I helped design a very long time ago and it’s absolute garbage.…

I'm a snowflake customer and I've felt/am feeling all of the pain that this article talks about. There might be some handwaving over technical complexity that you don't like given your detailed understanding of how the thing is built, but the article is fundamentally right in its message. The thing it's most right about is the power imbalance and the innovators dilemma. I've had more than one instance of the case whe…

Don’t you have the same issue with any query optimized product? If I’m using redshift and hit a bad execution plan that I can’t get around by tweaking the query I’m SOL, and redshift engineers aren’t going to tweak a configuration change to help me.

This is why products like DynamoDB were created - cost based optimizers are imperfect and unpredictable, and once you’ve stepped over some limit or threshold performance wildly changes. The reasons can be your query, or the data has changed, or there’s a noisy neighbor consuming a resource you depend on for your query. If you need highly predictable times you can reason about you won’t get it from any RDB solution.

Given that, what about snowflake feels different? That the details are obscured from you so you don’t understand why things are happening? Is the lack of ability to deeply introspect making you uncomfortable? My experience had been the ability to introspect rarely leads to any change in outcome but instead leads to me identify the query optimizer has done something stupid I can not do anything about, but at least I can point to the specific resource being exhausted by it.

Re: Why is Snowflake so expensive

#208

Earlier quoted context omitted.

Snowflake is nowhere near a monopoly, and plenty of customers have moved from other vendors (Teradata, Netezza, etc) to Snowflake - showing that vendor lock-in is not as strong as it might seem.

If that's the case, then why aren't Snowlake and Google targeting query optimiziation with higher priority to lower end-user costs? There's no incentive in the market for them to do so - once you switch to them, you'll eat up the cost of quirks and learn how to avoid them the hard way.

They are. See other comments about snowflake leaving $97m on the table recently, doing exactly that.

Re: Why is Snowflake so expensive

#209

Earlier quoted context omitted.

It's a terrible article. The author misunderstands competition and how much it drives products in this area. Snowflake is incentivized to make their product better on every dimension. If Snowflake don't improve, customers will leave in droves - like when they moved to Snowflake . In practice, as has been pointed out in other comments, they do improve their performance (for competitive reasons) and it does cost them m…

> It's a terrible article. The author misunderstands competition and how much it drives products in this area. Agree, but the author has one thing right. Snowflake is not transparent about product behavior, which makes it hard to reason about costs and performance. Open source data warehouses like ClickHouse and Druid don't have this problem. If you want to know how something works, you can look at the code. Or liste…

Sure but if you want full transparency you don't use Snowflake. They never sold themselves as that.

I wouldn't buy a Ferrari and complain about lack of trunk space.

Re: Why is Snowflake so expensive

#210
post #195

Earlier quoted context omitted.

Stas: "The issue is that majority of the companies don't experience a 70% revenue growth to catch up with the growth in costs" I think you are misunderstanding something very fundamental here. Snowflake has usage pricing and no one is forcing companies to use Snowflake 70% more every year. In my experience, companies are typically evaluating spend on other platforms and after some testing, moving additional workloads…

"In my experience, companies are typically evaluating spend on other platforms and after some testing, moving additional workloads there to displace cost elsewhere" Fair point, some of that net revenue increase is because of consolidation of workloads, although the majority of the cost is likely still driven by consumers expanding usage beyond what they expected. As I mention in my article, the second part of increas…

I read your link. My immediate reaction:

1) I think Andreessen Horowitz has probably oversimplified the issue based on the Dropbox outlier. It's easy to say you can build your own datacenter to manage stuff but the costs in people are really hard to offset, especially with the security posture and level of 999s that most companies need. Not only that, but throw in disaster recovery, so now you've doubled the costs (two data centers). Etc. Plus hardware ages rapidly--you want to pay for the "floor sweeps" (as Teradata used to call them) every few years?

Beyond the complexity, some of these companies simply could not exist without the Cloud. Take Snowflake. How big of a data center would they need? How many servers? How much disk? How do they know if Dropbox wants to load 1 GB, 1 TB, or 1 PB of data? Answer: they don't. This type of model only works if you can leverage the essentially unlimited scale of the cloud providers. I don't miss the days of loads failing due to being out of disk space and having to scramble around trying to find things to delete.

2) Regarding pricing policy, Snowflake makes it very clear which features are included with which edition: https://docs.snowflake.com/en/user-guide/intro-editions.html

Your link also says Snowflake paid 44% of their revenue in 2021 on Cloud. If that is true, perhaps Snowflake loses money at standard edition, and presumably there is a larger internal cost to supporting some of the higher end features like Private Link that Snowflake needs to recapture. Regardless, as a Snowflake customer, I can determine what features I need and decide if the price they are charging is worth it or if I should look elsewhere. I can say from experience that some of these features and even paper certifications aren't easy and can be very expensive to maintain.

I will tell a story that will age me. Decades ago I used to work for a company that needed a "business continuity" plan. We had to show that we could continue to function if our data center was destroyed by natural disaster. We paid a company in another region that had essentially a copy of all of our hardware, and once a year we'd send our backups there and bring up all of the systems to prove we could. As you might imagine, this service was insanely expensive.

Flash forward to now. Snowflake has a feature called failover/failback with connection redirect. With a few commands, you can replicate your entire database elsewhere, you can incrementally keep the remote target up-to-date, and you can test it as often as you like with connections failing over generally in under 1 minute. If your company needs something like this, how much would that cost to build yourself? Maintain? Test? Clearly there must be customers who did that evaluation and decided that Snowflake's approach is way cheaper. If you disagree, don't use that level of service, or build it yourself. You say SaaS providers are "too expensive" and that even "medium scale" companies can do better themselves, but that isn't my experience.

3) As discussed in the previous comment, no doubt Snowflake can make improvements. However, what I see from my limited view as a (probably much smaller) customer, Snowflake is doing that. In fact, two of those improvements you call out are already in private preview and were discussed at their recent conference. If my company was briefed on these features post Summit, I'd be highly surprised if yours wasn't.

Post reply on HN