Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why is Snowflake so expensive

blog.devgenius.io

Post reply on HN
Print view View original

91–100 of 217 posts

Re: Why is Snowflake so expensive

#91

Post by hangonhn »

brianwawok wrote: post #68

Ran into the same exact thing at CircleCI. Me: My builds are really slow CircleCI: Here are a few very low effort answers Me: git checkout is taking literally 60 seconds, but it takes 3 seconds locally, why? CircleCI: Mumble Mumble. They charge per minute, so why would they care if builds are slow? Was about a year of this getting worse and worse, till I finally cancelled the service last week and built my own server…

100% and not just in tech: when a party’s incentives aren’t aligned with yours, you’ll often find yourself getting little help or even working in opposition to each other. We recently experienced this with filing for a health related insurance claim. My wife wondered why they kept losing stuff, not doing what they promised, or asking for more paper work. I kept explaining to her that while not necessarily malicious, they have very little incentive to improve that department.

Always try to find partners or counter parties who win when you do as well. I know we don’t always have that luxury but sometimes a little headache initially is better than being stuck with someone who works in opposition to you in the long run.

Thanks so much for sharing your story. We are in the process of outsourcing some of our Jenkins functionality and these stories are useful to hear.

Re: Why is Snowflake so expensive

#92

Post by twawaaay »

Snowflake is not expensive. Snowflake is super cheap, IF you know what it is for and how to use it. Compared to if you had to solve the problem on your own.

The best way to describe Snowflake is that it is a brute force method to run complex queries without creating indexes.

If you have a more traditional database, you will notice you need to set up indexes to be able to get anything from it in finite time. What if you don't know the indexes upfront? What if you want your users to be able to ask arbitrary queries and get answers before bedtime?

That's what Snowflake is for. It automates using ENORMOUS amount of hardware to get your query executed fast, very inefficiently.

It is not for free though. That inefficiency will cause a lot of resources used for queries. It is meant for those few queries when your users try to get some insight into your data and you can't predict indexes beforehand. Sometimes this is exactly what you want, like when you let your data people in to figure stuff out. Or when you have very rare functionality that allows the user to build their own queries -- which you should avoid like hell (and there are tricks to make it index pretty well) but can't always avoid.

For everything else, whenever you can predict your indexes, you always want to use more traditional database that can be very efficient on queries properly supported by indexes.

The issue is a lot of people try to use Snowflake as a database or to support frequently executing queries of the same kind. This is bad and it will cost you.

Re: Why is Snowflake so expensive

#93

Post by thexumaker »

sremani wrote: post #87
brianwawok wrote: post #68

Ran into the same exact thing at CircleCI. Me: My builds are really slow CircleCI: Here are a few very low effort answers Me: git checkout is taking literally 60 seconds, but it takes 3 seconds locally, why? CircleCI: Mumble Mumble. They charge per minute, so why would they care if builds are slow? Was about a year of this getting worse and worse, till I finally cancelled the service last week and built my own server…

From the front page of CircleCI. __________________________________________________ Industry-leading speed As soon as you think it, you can deliver it. Your developers’ time is too important to waste. No other CI/CD platform takes performance as seriously as we do. Your pipelines should accelerate your business, not slow you down. __________________________________________________ Rule of thumb: Anyone talking about…

We did the same thing but with self hosted runners with github actions.

https://github.com/philips-labs/terraform-aws-github-runner

phillips-labs has some good resources for scaling this up as well.

Re: Why is Snowflake so expensive

#94

Post by gkoberger »

Fatnino wrote: post #88
brianwawok wrote: post #68

Ran into the same exact thing at CircleCI. Me: My builds are really slow CircleCI: Here are a few very low effort answers Me: git checkout is taking literally 60 seconds, but it takes 3 seconds locally, why? CircleCI: Mumble Mumble. They charge per minute, so why would they care if builds are slow? Was about a year of this getting worse and worse, till I finally cancelled the service last week and built my own server…

Those app rental scooters that are littered around city centers: you pay for distance as well as for time. And that's why they don't go very fast.

No, they're legally limited to 15 MPH for safety. You also don't pay for distance, just time. Not everything is a conspiracy theory.

SF: https://www.williamweisslaw.com/sf-e-scooter-laws/ NYC: https://www1.nyc.gov/html/dot/html/bicyclists/ebikes.shtml#:....

Re: Why is Snowflake so expensive

#95

Post by brianwawok »

icedchai wrote: post #86
brianwawok wrote: post #68

Ran into the same exact thing at CircleCI. Me: My builds are really slow CircleCI: Here are a few very low effort answers Me: git checkout is taking literally 60 seconds, but it takes 3 seconds locally, why? CircleCI: Mumble Mumble. They charge per minute, so why would they care if builds are slow? Was about a year of this getting worse and worse, till I finally cancelled the service last week and built my own server…

At a previous startup, we dumped CircleCI and switched to Jenkins on our own EC2 instance. We had a lot less problems. (This was way back in 2016, I'm sure things have improved now.)

Yup!

I ended up doing TeamCity over Jenkins, but they do the same thing.

Amazing how fast a 32C / 64T EYPC server in my basement can be..

Re: Why is Snowflake so expensive

#96

Post by CrazyStat »

Fatnino wrote: post #88
brianwawok wrote: post #68

Ran into the same exact thing at CircleCI. Me: My builds are really slow CircleCI: Here are a few very low effort answers Me: git checkout is taking literally 60 seconds, but it takes 3 seconds locally, why? CircleCI: Mumble Mumble. They charge per minute, so why would they care if builds are slow? Was about a year of this getting worse and worse, till I finally cancelled the service last week and built my own server…

Those app rental scooters that are littered around city centers: you pay for distance as well as for time. And that's why they don't go very fast.

Also because going fast on those things is fucking dangerous, especially when (like most people riding them) you're not wearing a helmet.

Re: Why is Snowflake so expensive

#97

Post by tluyben2 »

cs702 wrote: post #7

Great article. On the surface, it's about Snowflake. At a deeper level, the article is about the perverse incentives motivating SaaS businesses to do seemingly dumb, inefficient things and avoid seemingly obvious optimizations by default. Many SaaS businesses are perfectly happy to let customers shoot themselves in the foot if it generates more revenue. The BigQuery example (presently, by default, `select * from tabl…

Funny that most people here advocate aws while they have tons and tons of foot shooting tools that cost people 1000s of usd all the time. And we just accept it. Like if you want to kill a complex cluster with one api call or button click, it won’t let you for xyz; that’s not because they cannot, it’s because you will just let it be and that makes money.

Re: Why is Snowflake so expensive

#98

Post by SkyMarshal »

Fatnino wrote: post #88
brianwawok wrote: post #68

Ran into the same exact thing at CircleCI. Me: My builds are really slow CircleCI: Here are a few very low effort answers Me: git checkout is taking literally 60 seconds, but it takes 3 seconds locally, why? CircleCI: Mumble Mumble. They charge per minute, so why would they care if builds are slow? Was about a year of this getting worse and worse, till I finally cancelled the service last week and built my own server…

Those app rental scooters that are littered around city centers: you pay for distance as well as for time. And that's why they don't go very fast.

In that particular case another reason could be that they quite reasonably don't want you going very fast for safety and liability concerns.

Re: Why is Snowflake so expensive

#99

Post by benreesman »

Alright I’ll bite finally. What do these companies do? Neither Snowflake’s front-facing website, nor the Wikipedia article, nor this post tell me why people pay all this money.

I know a bit about the effort involved in chucking around 100 petabyte datasets, and there are numerous niches a SaaS could fill in there, but it’s very murky from the outside.

Re: Why is Snowflake so expensive

#100

Post by josephcsible »

brianwawok wrote: post #68

Ran into the same exact thing at CircleCI. Me: My builds are really slow CircleCI: Here are a few very low effort answers Me: git checkout is taking literally 60 seconds, but it takes 3 seconds locally, why? CircleCI: Mumble Mumble. They charge per minute, so why would they care if builds are slow? Was about a year of this getting worse and worse, till I finally cancelled the service last week and built my own server…

> They charge per minute, so why would they care if builds are slow?

It's worse than just not caring: they have a direct financial incentive to make sure your builds are as slow as you'll tolerate.

Post reply on HN