Uses and abuses of cloud data warehouses • Hacker News Board

Re: Uses and abuses of cloud data warehouses

#11

Post by lokar » Wed, Aug 16, 2023, 2:24 PM UTC

politelemon wrote: post #4

andrenotgiant wrote: post #2

It seems like Snowflake is going all-in on building features and doing marketing that encourage their customers to build applications, serving operational workloads, etc... on them. Things like in-product analytics, usage-based billing, personalization, etc... Anyone here taking them up on it? I'm genuinely curious how it's going.

I've noticed that too. I think the marketing is definitely working, I'm seeing a few organisations starting to shift more and more workloads onto them, and some are also publishing datasets on their marketplace. One of their most interesting offerings coming up is Snowpark which lets you run a Python function as a UDF, within Snowflake. This way you don't have to transfer data around everywhere, just run it as part o…

Also containers:

https://www.snowflake.com/blog/snowpark-container-services-d...

Re: Uses and abuses of cloud data warehouses

#12

Post by GeneralAntilles » Wed, Aug 16, 2023, 2:35 PM UTC

andrenotgiant wrote: post #2

It seems like Snowflake is going all-in on building features and doing marketing that encourage their customers to build applications, serving operational workloads, etc... on them. Things like in-product analytics, usage-based billing, personalization, etc... Anyone here taking them up on it? I'm genuinely curious how it's going.

Yeah, they're providing a path-of-least-resistance for getting stuff done in your existing data environment.

A common challenge in a lot of organizations is IT as a roadblock to deployment of internal tools coming from data teams. Snowflake is answering this with Streamlit. You get an easy platform for data people to use and deploy on and it can all be done within the business firewall under data governance within Snowflake.

Re: Uses and abuses of cloud data warehouses

#13

Post by albert_e » Wed, Aug 16, 2023, 2:43 PM UTC

Arent a lot of businesses being sold on "real time analytics" these days?

That mixes the uses cases of analytics and operations because everyone is led to believe that things that happened in last 10 minutes must go through the analytics lens and yield actionable insights in real time so their operational systems can react/adapt instantly.

Most business processes probably don't need anywhere near such real time analytics capability but it is very easy to think (or be convinced that) we do. Especially if I am a owner of a given business process (with an IT budget) why wouldn't I want the ability to understand trends in real-time and react to it if not get ahead of them and predict/be prepared. Anything less than that is seen as being shamefully behind on the tech curve.

In this context-- the section in article where it says present data is of virtually zero importance to analytics is no longer true. We need a real solution even if we apply those (presumably complex and costly) solutions to only the most deserving use cases (and not abuse them).

What is the current thinking in this space? I am sure there are technical solutions here but what is the framework to evaluate which use case actually deserves pursuing such a setup.

Curious to hear.

Re: Uses and abuses of cloud data warehouses

#14

Post by weego » Wed, Aug 16, 2023, 2:49 PM UTC

andrenotgiant wrote: post #2

It seems like Snowflake is going all-in on building features and doing marketing that encourage their customers to build applications, serving operational workloads, etc... on them. Things like in-product analytics, usage-based billing, personalization, etc... Anyone here taking them up on it? I'm genuinely curious how it's going.

After a series of calls, examples and explanations with them we never managed to get close to a reasonable projection of what our monthly costs would be like on Snowflake. I understand why companies in this field use abstract notions of 'processing' /'compute' units but it's a no go finance wise.

Without some close to real world projections we don't have time to consider implementation to find out for ourselves.

Re: Uses and abuses of cloud data warehouses

#15

Post by dontupvoteme » Wed, Aug 16, 2023, 2:56 PM UTC

The random bolding of words reeks of adtech.

is the usage of such an old html tag itself now a trigger to send something to /dev/null?

Re: Uses and abuses of cloud data warehouses

#16

Post by higeorge13 » Wed, Aug 16, 2023, 3:01 PM UTC

albert_e wrote: post #13

Arent a lot of businesses being sold on "real time analytics" these days? That mixes the uses cases of analytics and operations because everyone is led to believe that things that happened in last 10 minutes must go through the analytics lens and yield actionable insights in real time so their operational systems can react/adapt instantly. Most business processes probably don't need anywhere near such real time analy…

I work in a real time subscription analytics company (chartmogul.com). We fetch, normalize and aggregate various billing systems data and eventually visualize them into graphs and tables.

I had this discussion with key people and i would say it depends on multiple factors. Small companies really like and require real-time analytics: they want to see how a couple invoices translate into updated saas metrics or why they didn’t get a slack/email notification as soon asit happened. Larger ones will check their data less frequently per day or week, but again it depends on the people and their role. Most of them are happy with getting their data once per day into their mailboxes or warehouses.

But we try to make everyone happy so we aim for real time analytics.

Re: Uses and abuses of cloud data warehouses

#17

Post by Eumenes » Wed, Aug 16, 2023, 3:09 PM UTC

albert_e wrote: post #13

Arent a lot of businesses being sold on "real time analytics" these days? That mixes the uses cases of analytics and operations because everyone is led to believe that things that happened in last 10 minutes must go through the analytics lens and yield actionable insights in real time so their operational systems can react/adapt instantly. Most business processes probably don't need anywhere near such real time analy…

From my experience (mostly startups), real time analytics is generally overkill, esp. from a BI perspective. Unless your business is very focused on real time data and transactional processing, you can generally get away with ETL/batch jobs. Show executives, product, and downstream teams some metrics that update a few times per day saves a ton of money over things like Snowflake/Databricks/Redshift stuff. While cloud services can be pricey, tools like dbt are really useful and can be administered by savvy business people or analyst types. Those candidates are way easier to hire compared to data engineers, sql experts, etc.

Re: Uses and abuses of cloud data warehouses

#18

Post by pm90 » Wed, Aug 16, 2023, 3:12 PM UTC

mritchie712 wrote: post #7

I caught myself wondering how Google, Microsoft and Amazon let Snowflake win. You can argue they haven't won, but lets assume they have. Two things: 1. SNOW's market cap is $50B. GOOGL, MSFT, AMZN are all over $1T. Owning Snowflake would be a drop in the bucket for any of them (let alone if they were splitting the revenue). 2. Snowflake runs on AWS, GCP or Azure (customers choice), so a good chunk of their revenue go…

What I heard is that AWS got there first with Redshift but then didn’t really invest as much as was required by users so Snowflake found an opening and pounced on it.

BigQuery in GCP is a pretty great alternative and I know that GCP invests/promotes it heavily, but they were slightly late to the market.

Re: Uses and abuses of cloud data warehouses

#19

Post by atwebb » Wed, Aug 16, 2023, 3:13 PM UTC

politelemon wrote: post #4

andrenotgiant wrote: post #2

It seems like Snowflake is going all-in on building features and doing marketing that encourage their customers to build applications, serving operational workloads, etc... on them. Things like in-product analytics, usage-based billing, personalization, etc... Anyone here taking them up on it? I'm genuinely curious how it's going.

I've noticed that too. I think the marketing is definitely working, I'm seeing a few organisations starting to shift more and more workloads onto them, and some are also publishing datasets on their marketplace. One of their most interesting offerings coming up is Snowpark which lets you run a Python function as a UDF, within Snowflake. This way you don't have to transfer data around everywhere, just run it as part o…

> run a Python function as a UDF

Is that a differentiator? I'm unfamiliar with Snowpark's actual implementation but know SQL Server introduced Python/R in engine in 2016? something like that.

Re: Uses and abuses of cloud data warehouses

#20

Post by atwebb » Wed, Aug 16, 2023, 3:15 PM UTC

albert_e wrote: post #13

Arent a lot of businesses being sold on "real time analytics" these days? That mixes the uses cases of analytics and operations because everyone is led to believe that things that happened in last 10 minutes must go through the analytics lens and yield actionable insights in real time so their operational systems can react/adapt instantly. Most business processes probably don't need anywhere near such real time analy…

Real-time generally means near-real-time and even then I liken it to availability.

If asked people would say "I need to always be up" until they see the costs associated with it, then being out for a few hours a year tends to be ok.