Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Uses and abuses of cloud data warehouses

materialize.com

Post reply on HN
Print view View original

51–60 of 78 posts

Re: Uses and abuses of cloud data warehouses

#51

Post by riku_iki »

biggestdummy wrote: post #49
slotrans wrote: post #38

Earlier quoted context omitted.

Yeah ClickHouse is definitely the way to go here. Its ability to serve queries with low latency and high concurrency is in an entirely different league from Snowflake, Redshift, BigQuery, etc.

StarRocks handles latency and concurrency as well as Clickhouse but also does joins. Less denormalization, and you can use the same platform for traditional BI/ad-hoc queries.

Clickhouse also does joins.

Somehow StarRocks dudes appear in every relevant post with this false claim.

Re: Uses and abuses of cloud data warehouses

#52

Post by hodgesrm »

This article uses an either or definition that leaves out a big set of use cases that combine operational and analytic usage:

> First, a working definition. An operational tool facilitates the day-to-day operation of your business. Think of it in contrast to analytical tools that facilitate historical analysis of your business to inform longer term resource allocation or strategy.

Security event and incident management (SEIM) is a typical example. You want fast notification on events combined with the ability to sift through history extremely quickly to assesss problems. This is precisely the niche occupied by real-time analytic databases like ClickHouse, Druid, and Pinot.

Re: Uses and abuses of cloud data warehouses

#53

Post by mritchie712 »

riku_iki wrote: post #50
mritchie712 wrote: post #7

I caught myself wondering how Google, Microsoft and Amazon let Snowflake win. You can argue they haven't won, but lets assume they have. Two things: 1. SNOW's market cap is $50B. GOOGL, MSFT, AMZN are all over $1T. Owning Snowflake would be a drop in the bucket for any of them (let alone if they were splitting the revenue). 2. Snowflake runs on AWS, GCP or Azure (customers choice), so a good chunk of their revenue go…

> 1. SNOW's market cap is $50B. GOOGL, MSFT, AMZN are all over $1T. Owning Snowflake would be a drop in the bucket for any of them (let alone if they were splitting the revenue). FAANG can't utilize their market cap to buy SNOW, they would need to pay cash, and 50B is very large amount for any of these companies (its about annual Google net income). Also, snow stock is very inflated now, it is heavily income negative…

My point is that they wouldn't want to buy it (or have focused much on building a competitive product) if it's only worth $50B.

Re: Uses and abuses of cloud data warehouses

#54

Post by debarshri »

15 years ago when I joined workforce business intelligence was all the rage. Data world was pretty much straight forward. You had transactional data in OLTP databases which would be shipped to Operational data stores, then rolled into the data warehouse. Datawarehouses were actual specialised hardware appliances (netezza et al) reporting tools were robust too.

Everytime I moved from one org to another, these concepts of data warehouse somehow got muddled.

Re: Uses and abuses of cloud data warehouses

#55

Post by code_biologist »

benjaminwootton wrote: post #26
weego wrote: post #14

Earlier quoted context omitted.

After a series of calls, examples and explanations with them we never managed to get close to a reasonable projection of what our monthly costs would be like on Snowflake. I understand why companies in this field use abstract notions of 'processing' /'compute' units but it's a no go finance wise. Without some close to real world projections we don't have time to consider implementation to find out for ourselves.

Snowflake is one of the easier tools to measure because it’s a simple function of region, instance size, uptime. If you can simulate some real loads and understand the usage then you do have a shot at forecasting. Of course the number is going to be high, but you have to remember it rolls up compute and requires less manpower. This is also a win for finance if they are comfortable with usage based billing.

Who's finance team likes usage based billing? It makes sense for elastic use cases and is definitely "fair", but there are a lot of issues: Forecasting is hard. "dev team had an oops" situations.

I had frog getting boiled situation at one job that was exactly the process described in the posted article: usage of the cloud data warehouse grew as people trusted the infrastructure and used more fresh data for more and more use cases. They were all good, sane use cases. I repeatedly under-forecast our cost growth until we made large changes and it really frustrated the finance people, rightly so.

Re: Uses and abuses of cloud data warehouses

#56

Post by biggestdummy »

riku_iki wrote: post #51
biggestdummy wrote: post #49

Earlier quoted context omitted.

StarRocks handles latency and concurrency as well as Clickhouse but also does joins. Less denormalization, and you can use the same platform for traditional BI/ad-hoc queries.

Clickhouse also does joins. Somehow StarRocks dudes appear in every relevant post with this false claim.

There's a difference between "supports the syntax for joins" and "does joins efficiently enough that they are useful."

My experience with Clickhouse is that its joins are not performant enough to be useful. So the best practice in most cases is to denormalize. I should have been more specific in my earlier comment.

Re: Uses and abuses of cloud data warehouses

#57

Post by tcoff91 »

ed_elliott_asc wrote: post #45
disgruntledphd2 wrote: post #39

Earlier quoted context omitted.

Snowflake are the Oracle of the cloud.

Oh come on snowflake isn’t cheap but there are none of the license auditing nonsense. (Also aren’t oracle the oracle of the cloud?)

Oracle is so far beyond anyone other major player in crookedness that it's not even funny.

Oh, you happened to run your oracle database in a VM and got audited? They'll try to shake you down to pay for however many oracle licenses for every other box you are running hypervisors on, because they claim that you could have transferred the database to any of those other boxes. So if you have a datacenter with 1000 boxes running VMWare, and you ran Oracle on one of them, they try to shake you down for paying for not buying 1000 licenses. Then they say but if you just buy a bunch of cloud credits, we can make your 1000x license violation go away.

Re: Uses and abuses of cloud data warehouses

#58

Post by riku_iki »

biggestdummy wrote: post #56
riku_iki wrote: post #51

Earlier quoted context omitted.

Clickhouse also does joins. Somehow StarRocks dudes appear in every relevant post with this false claim.

There's a difference between "supports the syntax for joins" and "does joins efficiently enough that they are useful." My experience with Clickhouse is that its joins are not performant enough to be useful. So the best practice in most cases is to denormalize. I should have been more specific in my earlier comment.

ack that anonymous user in internet said he couldn't make CLickhouse joins perform well in his case which he didn't describe

Re: Uses and abuses of cloud data warehouses

#59

Post by jamesblonde »

noazdad wrote: post #21
jamesblonde wrote: post #10

Earlier quoted context omitted.

In theory, fine. Then you look at the walled garden that is Snowpark - only "approved" python libraries are allowed there. It will be a very constrictive set of models you can train, and very constrictive feature engineering in Python. And, wait, aren't Python UDFs super-slow (GIL) - what about Pandas UDFs (wait that's PySpark.....)

Disclaimer: Snowflake employee here. You can add any Python library you want - as long as its dependencies are also 100% Python. Takes about a minute: pip install the package, zip it up, upload it to an internal Snowflake stage, then reference it in the IMPORTS=() directive in your Python. I did this with pydicom just the other day - worked a treat. So yes, not the depth and breadth of the entire Python ecosystem, bu…

Good luck with trying to install any non-trivial python library this way. And with AI moving so fast, do you think people will accept that they can't use the libraries they need, because you haven't approved them yet?!?

Re: Uses and abuses of cloud data warehouses

#60

Post by kwillets »

andrenotgiant wrote: post #2

It seems like Snowflake is going all-in on building features and doing marketing that encourage their customers to build applications, serving operational workloads, etc... on them. Things like in-product analytics, usage-based billing, personalization, etc... Anyone here taking them up on it? I'm genuinely curious how it's going.

It's really bad for in-product analytics. Slow and expensive to keep running on a 24/7 SLA.

Even with Vertica doing that, we're seeing 10x costs just doing back-office DWH. My job now is keeping Vertica running so we can pay our Snowflake bill.

Post reply on HN