Why is Snowflake so expensive • Hacker News Board

Re: Why is Snowflake so expensive

#81

Post by twistedpair » Mon, Aug 22, 2022, 4:38 PM UTC

cs702 wrote: post #7

Great article. On the surface, it's about Snowflake. At a deeper level, the article is about the perverse incentives motivating SaaS businesses to do seemingly dumb, inefficient things and avoid seemingly obvious optimizations by default. Many SaaS businesses are perfectly happy to let customers shoot themselves in the foot if it generates more revenue. The BigQuery example (presently, by default, `select * from tabl…

FWIW, BigQuery tables can be configured to require a partition filter clause [0] in the SQL query, so that you cannot shoot yourself in the foot like that. Now if they'd just make an Organization Policy to let you turn it on by default for all new tables.

[0] https://cloud.google.com/bigquery/docs/querying-partitioned-...

Re: Why is Snowflake so expensive

#82

Post by morelisp » Mon, Aug 22, 2022, 4:40 PM UTC

nycdatasci wrote: post #80

KingOfCoders wrote: post #37

Earlier quoted context omitted.

In which way not comparable?

From the article: "JOIN's are also not nearly as performant as in other cloud data warehouses." This seems like a pretty significant limitation.

That's... literally comparing them. The comparison for some use cases might not be favorable for ClickHouse, but they're comparable.

(IMO the slowness of ClickHouse joins has been overstated, especially since its many-column table support is so good you'll probably be fine joining on insert instead.)

Re: Why is Snowflake so expensive

#83

Post by mritchie712 » Mon, Aug 22, 2022, 4:40 PM UTC

SnowHill9902 wrote: post #54

mritchie712 wrote: post #34

I predict[0] we'll see more people choosing Clickhouse over Snowflake in the next 5 years. Clickhouse will get reasonably feature compatible with Snowflake and give people a better escape hatch if they want to self-host their data stack. Clickhouse, Inc is building a cloud product that abstracts away the complexity and there's already companies like Altinity that will spin up a cluster for you in minutes. 0 - https:/…

Clickhouse is incredible software. It only feels a little foreign when coming from Postgres (e.g. some CamelCase terms).

Yeah, the CamelCase throws me too, especially since it's mixed in with snake_case (e.g. date_trunc[0])

0 - https://clickhouse.com/docs/en/sql-reference/functions/date-...

Re: Why is Snowflake so expensive

#84

Post by mritchie712 » Mon, Aug 22, 2022, 4:41 PM UTC

nycdatasci wrote: post #80

KingOfCoders wrote: post #37

Earlier quoted context omitted.

In which way not comparable?

From the article: "JOIN's are also not nearly as performant as in other cloud data warehouses." This seems like a pretty significant limitation.

Yes, this is one major hurdle they need to overcome, but I think they'll (Clickhouse Inc + the community) pull it off. It's a current weakness but by no means unsolvable.

Re: Why is Snowflake so expensive

#85

Post by zX41ZdbW » Mon, Aug 22, 2022, 4:47 PM UTC

mritchie712 wrote: post #83

SnowHill9902 wrote: post #54

Earlier quoted context omitted.

Clickhouse is incredible software. It only feels a little foreign when coming from Postgres (e.g. some CamelCase terms).

Yeah, the CamelCase throws me too, especially since it's mixed in with snake_case (e.g. date_trunc[0]) 0 - https://clickhouse.com/docs/en/sql-reference/functions/date-...

camelCase - native functions

SQL_STYLE_CASE - compatible functions

Re: Why is Snowflake so expensive

#86

Post by icedchai » Mon, Aug 22, 2022, 4:49 PM UTC

brianwawok wrote: post #68

Ran into the same exact thing at CircleCI. Me: My builds are really slow CircleCI: Here are a few very low effort answers Me: git checkout is taking literally 60 seconds, but it takes 3 seconds locally, why? CircleCI: Mumble Mumble. They charge per minute, so why would they care if builds are slow? Was about a year of this getting worse and worse, till I finally cancelled the service last week and built my own server…

At a previous startup, we dumped CircleCI and switched to Jenkins on our own EC2 instance. We had a lot less problems. (This was way back in 2016, I'm sure things have improved now.)

Re: Why is Snowflake so expensive

#87

Post by sremani » Mon, Aug 22, 2022, 4:49 PM UTC

brianwawok wrote: post #68

Ran into the same exact thing at CircleCI. Me: My builds are really slow CircleCI: Here are a few very low effort answers Me: git checkout is taking literally 60 seconds, but it takes 3 seconds locally, why? CircleCI: Mumble Mumble. They charge per minute, so why would they care if builds are slow? Was about a year of this getting worse and worse, till I finally cancelled the service last week and built my own server…

From the front page of CircleCI. __________________________________________________ Industry-leading speed As soon as you think it, you can deliver it. Your developers’ time is too important to waste. No other CI/CD platform takes performance as seriously as we do. Your pipelines should accelerate your business, not slow you down. __________________________________________________

Rule of thumb: Anyone talking about their honesty is not honest.

Re: Why is Snowflake so expensive

#88

Post by Fatnino » Mon, Aug 22, 2022, 4:52 PM UTC

brianwawok wrote: post #68

Ran into the same exact thing at CircleCI. Me: My builds are really slow CircleCI: Here are a few very low effort answers Me: git checkout is taking literally 60 seconds, but it takes 3 seconds locally, why? CircleCI: Mumble Mumble. They charge per minute, so why would they care if builds are slow? Was about a year of this getting worse and worse, till I finally cancelled the service last week and built my own server…

Those app rental scooters that are littered around city centers: you pay for distance as well as for time. And that's why they don't go very fast.

Re: Why is Snowflake so expensive

#89

Post by mikewhy » Mon, Aug 22, 2022, 4:52 PM UTC

brianwawok wrote: post #68

Ran into the same exact thing at CircleCI. Me: My builds are really slow CircleCI: Here are a few very low effort answers Me: git checkout is taking literally 60 seconds, but it takes 3 seconds locally, why? CircleCI: Mumble Mumble. They charge per minute, so why would they care if builds are slow? Was about a year of this getting worse and worse, till I finally cancelled the service last week and built my own server…

I love that CircleCI flaunts it's speed compared to other providers, meanwhile we can clearly see the CircleCI steps take the longest in our builds.

Not to mention the constant failures.

Re: Why is Snowflake so expensive

#90

Post by atwebb » Mon, Aug 22, 2022, 4:53 PM UTC

jwie wrote: post #50

dominotw wrote: post #44

Earlier quoted context omitted.

Because snowflake doesn't build foreign key indexes. Imagine clickstream data where every insert is being checked against an index of customers. This isn't a typical usecase for big data warehouses.

I understand that. But why have constraints that don’t do anything?

Metadata

Tools and scripts can work off of it, design decisions are documented, suggestions can be made, inferences can be made (some dangerous, some not).

Why tag S3 objects if it doesn't enforce a schema? Maybe a bad analogue but I'm going quick right now :).