Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open table formats are inevitable for analytical datasets

ensembleanalytics.io

Post reply on HN
Print view View original

51–60 of 60 posts

Re: Open table formats are inevitable for analytical datasets

#51

Post by hodgesrm »

datadrivenangel wrote: post #41

The one and a half downsides not mentioned here: 1: Separating storage and compute tends to result in relatively high startup latency (Databricks, here's looking at you!) when you have to provision new compute. For a massive batch job, waiting 10 minutes for a cluster to come up is fine, but a lot of orgs don't realize the cost implications and end up with tiny developer/analyst clusters, or aggressively spin inactiv…

> The one and a half downsides not mentioned here:

Make that 2.5. If you are building new analytic products the most significant issue is that 90% of the information in data lakes is already there and not in an open format like Iceberg. Instead, it's heaps of CSV and Parquet files, maybe in Hive format but maybe just named in some unique way. If your query engine can't read these it's like being all dressed up with no place to go.

Re: Open table formats are inevitable for analytical datasets

#52

Post by lysecret »

datadrivenangel wrote: post #40
lysecret wrote: post #33

Earlier quoted context omitted.

Column orientation is extremely important for query + storage efficiency. Furthermore, you need a distributed query engine (Athena, Bigquery etc.) and they all support parquet.

DuckDB is SQLite but column oriented!

Yes we are using DuckDB right now. Love it. Works extremely well if you can partition down your data to manageable (few gbs) size.

Re: Open table formats are inevitable for analytical datasets

#53

Post by datadrivenangel »

hodgesrm wrote: post #51
datadrivenangel wrote: post #41

The one and a half downsides not mentioned here: 1: Separating storage and compute tends to result in relatively high startup latency (Databricks, here's looking at you!) when you have to provision new compute. For a massive batch job, waiting 10 minutes for a cluster to come up is fine, but a lot of orgs don't realize the cost implications and end up with tiny developer/analyst clusters, or aggressively spin inactiv…

> The one and a half downsides not mentioned here: Make that 2.5. If you are building new analytic products the most significant issue is that 90% of the information in data lakes is already there and not in an open format like Iceberg. Instead, it's heaps of CSV and Parquet files, maybe in Hive format but maybe just named in some unique way. If your query engine can't read these it's like being all dressed up with n…

CSV and Parquet are open file formats.

They're not good structured table formats, but they are open compared to the binary storage you get with Oracle or Snowflake.

Re: Open table formats are inevitable for analytical datasets

#54

Post by 62951413 »

benjaminwootton wrote: post #4

Thanks for the upvotes. Here is a more technical deep dive comparing the three main open formats - https://www.onehouse.ai/blog/apache-hudi-vs-delta-lake-vs-ap...

For a Spark shop Delta is the default choice. If you deploy to AWS then Glue encourages you to go with Iceberg. What makes people use Hudi?

Re: Open table formats are inevitable for analytical datasets

#55

Post by nojito »

datadrivenangel wrote: post #41

The one and a half downsides not mentioned here: 1: Separating storage and compute tends to result in relatively high startup latency (Databricks, here's looking at you!) when you have to provision new compute. For a massive batch job, waiting 10 minutes for a cluster to come up is fine, but a lot of orgs don't realize the cost implications and end up with tiny developer/analyst clusters, or aggressively spin inactiv…

The vast majority of analytics needs do not warrant a cluster.

There are many other competitive query engines that do not suffer from the startup latency.

Re: Open table formats are inevitable for analytical datasets

#56

Post by CyberDildonics »

lalaland1125 wrote: post #19
8n4vidtmkvmk wrote: post #14

I don't see any mention of sqlite. Is a sqlite file not the same thing they're talking about here? Pretty sure it has a spec and hasn't changed formats in many years so if you wanted to read it out in something that isn't sqlite, I imagine it wouldn't be too hard.

The main disadvantage of sqlite compared to these other formats is that sqlite is designed for single machine processing. This can be problematic for huge datasets or complex queries. These other formats easily support using clusters to process your data.

If there are no writers it doesn't matter how many readers there are.

Re: Open table formats are inevitable for analytical datasets

#57

Post by datadrivenangel »

nojito wrote: post #55
datadrivenangel wrote: post #41

The one and a half downsides not mentioned here: 1: Separating storage and compute tends to result in relatively high startup latency (Databricks, here's looking at you!) when you have to provision new compute. For a massive batch job, waiting 10 minutes for a cluster to come up is fine, but a lot of orgs don't realize the cost implications and end up with tiny developer/analyst clusters, or aggressively spin inactiv…

The vast majority of analytics needs do not warrant a cluster. There are many other competitive query engines that do not suffer from the startup latency.

100% agree! Most analytics are done in excel, most analytics should be done in a SQLite or DuckDB database!

Re: Open table formats are inevitable for analytical datasets

#58

Post by datadrivenangel »

sys13 wrote: post #50
datadrivenangel wrote: post #41

The one and a half downsides not mentioned here: 1: Separating storage and compute tends to result in relatively high startup latency (Databricks, here's looking at you!) when you have to provision new compute. For a massive batch job, waiting 10 minutes for a cluster to come up is fine, but a lot of orgs don't realize the cost implications and end up with tiny developer/analyst clusters, or aggressively spin inactiv…

(I do some training for Databricks) 1. Yeah, cluster startup time can be not fun. Here are some solutions: - pools (keeps instances around so you don't have to wait for the cloud to provision them - serverless SQL warehouses (viable if you're doing only SQL) - one job with multiple tasks that share the same job cluster. Delta Live Tables does a similar thing but with streaming autoscaling - streaming: cluster never n…

How do you keep the pool costs manageable?

I see a lot of companies that get sold on Databricks and then are surprised by the cost.

Re: Open table formats are inevitable for analytical datasets

#59

Post by sys13 »

datadrivenangel wrote: post #58
sys13 wrote: post #50

Earlier quoted context omitted.

(I do some training for Databricks) 1. Yeah, cluster startup time can be not fun. Here are some solutions: - pools (keeps instances around so you don't have to wait for the cloud to provision them - serverless SQL warehouses (viable if you're doing only SQL) - one job with multiple tasks that share the same job cluster. Delta Live Tables does a similar thing but with streaming autoscaling - streaming: cluster never n…

How do you keep the pool costs manageable? I see a lot of companies that get sold on Databricks and then are surprised by the cost.

Pool costs become more manageable as you have more clusters sharing the same pool. You can also have it have no incremental cost increase by setting the timeout to be 0, but it makes it less useful. You can have more clusters take advantage of it by using the same instance families. If you purchase reserved instances from Azure/AWS, you might as well make a pool with those as well. You may also want to check out fleet instance types.

Re: Open table formats are inevitable for analytical datasets

#60

Post by CRConrad »

hermitcrab wrote: post #18
benjaminwootton wrote: post #9

Earlier quoted context omitted.

Date lake can be thought of a file system. Imagine 100 CSVs in folders, usually stored on S3. Data Lakehouse involves adding things like the ability to query via SQL, the ability to update/insert/delete, transactions. Where before people needed warehouses for BI and lakes for data science, they can now have only one approach. It’s likely to be a big trend as data moves to this format and arrangement and the DBMS vend…

Data warehouse, data lake, data lakehouse. The data world has some terrible terminology.

Wait til you dive into ETL vs ELT or even better, if you've been doing ETL since long before "ELT" was "a thing", but everyone did ETL actually in an ELT fashion...

It's not only the names, but something like 98% of the tools too, that suck.

Post reply on HN