Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Uses and abuses of cloud data warehouses

materialize.com

Post reply on HN
Print view View original

31–40 of 78 posts

Re: Uses and abuses of cloud data warehouses

#31

Post by ramraj07 »

andrenotgiant wrote: post #2

It seems like Snowflake is going all-in on building features and doing marketing that encourage their customers to build applications, serving operational workloads, etc... on them. Things like in-product analytics, usage-based billing, personalization, etc... Anyone here taking them up on it? I'm genuinely curious how it's going.

Snowflake is capturing a large market share in analytics industries thanks to its “just works” feature. I’m a massive fan.

But in the end, snowflake stores the data in S3 as partitions. If you want to update a single value you have to replace the entire s3 partition. Similarly you need to read a reasonable amount of s3 data to retrieve even a single record. Thus you’re never going to get responses shorter than half a second (at best). As long as you don’t try and game around that limitation it works great.

Materialize up here also follows the same model in the end FWIW.

Re: Uses and abuses of cloud data warehouses

#32

Post by jandrewrogers »

albert_e wrote: post #13

Arent a lot of businesses being sold on "real time analytics" these days? That mixes the uses cases of analytics and operations because everyone is led to believe that things that happened in last 10 minutes must go through the analytics lens and yield actionable insights in real time so their operational systems can react/adapt instantly. Most business processes probably don't need anywhere near such real time analy…

The term "real-time" is much abused in marketing copy. It is often treated like a technical metric but it is actually a business metric: am I making operational decisions with the most recent data available? For many businesses, "most recent data available" can be several days old and little operational efficiency would be gained by investing in reducing that latency.

For some businesses, "real-time" can be properly defined as "within the last week". While there are many businesses where reducing that operational tempo to seconds would have an impact, it is by no means universal.

Re: Uses and abuses of cloud data warehouses

#33

Post by atwong »

There are other databases today that do real time analytics (ClickHouse, Apache Druid, StarRocks along with Apache Pinot). I'd look at the ClickHouse Benchmark to see who are the competitors in that space and their relative performance.

Re: Uses and abuses of cloud data warehouses

#34

Post by mrbungie »

higeorge13 wrote: post #16
albert_e wrote: post #13

Arent a lot of businesses being sold on "real time analytics" these days? That mixes the uses cases of analytics and operations because everyone is led to believe that things that happened in last 10 minutes must go through the analytics lens and yield actionable insights in real time so their operational systems can react/adapt instantly. Most business processes probably don't need anywhere near such real time analy…

I work in a real time subscription analytics company (chartmogul.com). We fetch, normalize and aggregate various billing systems data and eventually visualize them into graphs and tables. I had this discussion with key people and i would say it depends on multiple factors. Small companies really like and require real-time analytics: they want to see how a couple invoices translate into updated saas metrics or why the…

I think GP's point is that is not about the perceived value of real time data/analytics, but rather, its actual value. Decision makers may ask for RT or NRT, but most of the time won't make a decision or action in a timeframe that actually justifies RT/NRT data/analytics.

For most operations RT/NRT data stuff normally is about novelty/vanity rather than a real existing business need.

Re: Uses and abuses of cloud data warehouses

#35

Post by datavirtue »

spullara wrote: post #30

These reasons are why Snowflake is building hybrid tables (under the Unistore umbrella). Those tables keep recent data in an operational store and historical data in their typical data warehouse storage systems. Best of both worlds. Still in private preview but definitely the answer to how you build applications that need both without using multiple databases and syncing. https://www.snowflake.com/guides/htap-hybrid-…

Conveniently leave out the issue of cost. Snowflake is piling on features that encourage more compute. Customers abuse the system and they (Snowflake) respond by helping cement them into continuing the abuse (spending more) by developing features to make bad habits and horrible engineering decisions look like something they should be doing. Typical.

Re: Uses and abuses of cloud data warehouses

#36

Post by mrbungie »

I remeber one time I was working as a Data & Analytics Lead (almost a Chief Data Officer but without the title) in a company were I don't work anymore and I was "challenged" by our parent company CDO about our data tech stack and operations. Just for context, my team at the time was me working as the lead and main Data Engineer plus 3 Data Analysts that I was coaching/teaching to convert into DEngs/DScientists.

At the time we were mostly a batch data shop, based on Apache Airflow + K8S + BigQuery + GCS in Google Cloud Platform, with BigQuery + GCS as the central datalake techs for analytics and processing. We still had RT capabilities due to having also some Flink processes running in the K8S cluster, and also having time-critical (time, not latency) processes running in microbatches of minutes for NRT. It was pretty cheap and sufficiently reliable, with both Airflow and Flink having self-healing capabilities at least at the node/process level (and even cluster/region level should we need it and be willing to increase the costs), while also allowing for some changes down the road like moving out of BQ if the costs scaled up too much.

What they wanted us to implement what according to them was the industry "best practices" circa 2021: a Kafka-based Datalake (KSQL and co.), at least other 4 engines (Trino, Pinot, Postgres and Flink) and an external object storage with most of the stuff running inside Docker containers orchestrated by Ansible in N compute instances manually controlled from a bastion instance. For some reason, they insisted on having a real time datalake based on Kafka. It was an insane mix of cargo cult, FOMO, high operational complexity and low reliability in one package.

I resisted the idea until the last second I was in that place. I reunited with some of my team members for drinks months later after my departure and they told me the new CDO was already convinced that said "RT-based" datalake was the way to go forward. I still shudder every time I remember the architectural diagram and I hope they didn't finally follow that terrible advice.

tl;dr: I will never understand the cargo cult around real time data and analytics but it is a thing that appeals to both decision makers and "data workers". Most businesses and operations (especially those whose main focus is not IT by itself) won't act or decide in hours, but rather in days. Build around your main use case and then make exceptions, not the other way around.

Re: Uses and abuses of cloud data warehouses

#37

Post by slotrans »

albert_e wrote: post #13

Arent a lot of businesses being sold on "real time analytics" these days? That mixes the uses cases of analytics and operations because everyone is led to believe that things that happened in last 10 minutes must go through the analytics lens and yield actionable insights in real time so their operational systems can react/adapt instantly. Most business processes probably don't need anywhere near such real time analy…

Just gonna keep linking this til the heat death of the universe: https://mcfunley.com/whom-the-gods-would-destroy-they-first-...

Real-time analytics are worse than useless. At best they are a distracting resource sink, at worst they directly harm the quality of decision-making.

Re: Uses and abuses of cloud data warehouses

#38

Post by slotrans »

atwong wrote: post #33

There are other databases today that do real time analytics (ClickHouse, Apache Druid, StarRocks along with Apache Pinot). I'd look at the ClickHouse Benchmark to see who are the competitors in that space and their relative performance.

Yeah ClickHouse is definitely the way to go here. Its ability to serve queries with low latency and high concurrency is in an entirely different league from Snowflake, Redshift, BigQuery, etc.

Re: Uses and abuses of cloud data warehouses

#39

Post by disgruntledphd2 »

datavirtue wrote: post #35
spullara wrote: post #30

These reasons are why Snowflake is building hybrid tables (under the Unistore umbrella). Those tables keep recent data in an operational store and historical data in their typical data warehouse storage systems. Best of both worlds. Still in private preview but definitely the answer to how you build applications that need both without using multiple databases and syncing. https://www.snowflake.com/guides/htap-hybrid-…

Conveniently leave out the issue of cost. Snowflake is piling on features that encourage more compute. Customers abuse the system and they (Snowflake) respond by helping cement them into continuing the abuse (spending more) by developing features to make bad habits and horrible engineering decisions look like something they should be doing. Typical.

Snowflake are the Oracle of the cloud.

Re: Uses and abuses of cloud data warehouses

#40

Post by disgruntledphd2 »

Pils wrote: post #25
jamesblonde wrote: post #10

Earlier quoted context omitted.

In theory, fine. Then you look at the walled garden that is Snowpark - only "approved" python libraries are allowed there. It will be a very constrictive set of models you can train, and very constrictive feature engineering in Python. And, wait, aren't Python UDFs super-slow (GIL) - what about Pandas UDFs (wait that's PySpark.....)

Having worked with a team using Snowpark, there are a couple things that bother me about it as a platform. For example, it only supported Python 3.8 until 3.9/10 recently entered preview mode. It feels a bit like a rushed project designed to compete with Databricks/Spark at the bullet point level, but not quite at the same quality level. But that's fine! It has only existed for around a year in public preview, and ap…

Yeah it's definitely not ready for modelling. It's pretty rocking for ETL though, and much easier to test and abstract than regular SQL. Granted it's a PySpark clone but our data is already in Snowflake.
Post reply on HN