Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache iceberg the Hadoop of the modern-data-stack?

blog.det.life

Post reply on HN
Print view View original

31–40 of 67 posts

Re: Apache iceberg the Hadoop of the modern-data-stack?

#31

Post by simlevesque »

thom wrote: post #29
simlevesque wrote: post #25

I'm working on an alternative Iceberg client to work better in write heavy use cases. Instead of many smaller files it writes on the same file until it's 1mb in size but it gives it a new name. Then I update the manifest to the new filename and checksum. I keep old files on disk for 60 seconds to allow pending queries. I'm also working on auto compaction, when I have ten 1mb files I compact them, same with ten 10mb f…

Interesting. My personal feeling is that we're slowly headed to a world where we can have our cake and eat it: fast bulk ingestion, fast OLAP, fast OLTP, low latency, all together in the same datastore. I'm hoping we just get to collapse whole complex data platforms into a single consistent store with great developer experience, and never look back.

I think it's possible too and the Iceberg spec allows it but the implementations are not suited for every use case.

Re: Apache iceberg the Hadoop of the modern-data-stack?

#32

Post by mritchie712 »

simlevesque wrote: post #25

I'm working on an alternative Iceberg client to work better in write heavy use cases. Instead of many smaller files it writes on the same file until it's 1mb in size but it gives it a new name. Then I update the manifest to the new filename and checksum. I keep old files on disk for 60 seconds to allow pending queries. I'm also working on auto compaction, when I have ten 1mb files I compact them, same with ten 10mb f…

nice! anywhere we can follow your progress?

Re: Apache iceberg the Hadoop of the modern-data-stack?

#33

Post by simlevesque »

mritchie712 wrote: post #30

This is a bit overblown. Is Iceberg "easy" to set up? No. Can you get set up in a week? Yes. If you really need a datalake, spending a week setting it up is not so bad. We have a guide[0] here that will get you started in under an hour. For smaller (e.g. under 10tb) data where you don't need real-time, DuckDB is becoming a really solid option. Here's on setup[1] we've played around with using Arrow Flight. If you don…

I think Iceberg can work in real time but the current implementations make it impossible.

I have a vision for a way to make it work. I made another comment here. Your blog posts were helpful, I digged a bit in the Duck Takes Flight code in python and rust.

Re: Apache iceberg the Hadoop of the modern-data-stack?

#34

Post by simlevesque »

mritchie712 wrote: post #32
simlevesque wrote: post #25

I'm working on an alternative Iceberg client to work better in write heavy use cases. Instead of many smaller files it writes on the same file until it's 1mb in size but it gives it a new name. Then I update the manifest to the new filename and checksum. I keep old files on disk for 60 seconds to allow pending queries. I'm also working on auto compaction, when I have ten 1mb files I compact them, same with ten 10mb f…

nice! anywhere we can follow your progress?

Not right now sadly I have some work obligations taking my time but I can't wait to share more.

I'm using a basic implementation that's not backed by iceberg, just Parquet files in hive partitions that I can query using DuckDB.

Re: Apache iceberg the Hadoop of the modern-data-stack?

#35

Post by ndm000 »

thom wrote: post #29
simlevesque wrote: post #25

I'm working on an alternative Iceberg client to work better in write heavy use cases. Instead of many smaller files it writes on the same file until it's 1mb in size but it gives it a new name. Then I update the manifest to the new filename and checksum. I keep old files on disk for 60 seconds to allow pending queries. I'm also working on auto compaction, when I have ten 1mb files I compact them, same with ten 10mb f…

Interesting. My personal feeling is that we're slowly headed to a world where we can have our cake and eat it: fast bulk ingestion, fast OLAP, fast OLTP, low latency, all together in the same datastore. I'm hoping we just get to collapse whole complex data platforms into a single consistent store with great developer experience, and never look back.

I’ve felt the same way. It’s so inefficient to have two patterns - OLAP and OLTP - both using SQL interfaces but requiring syncing between systems. There are some physical limits at play though. OLAP will always take less processing and disk usage if the data it needs is all right next to each other (columnar storage) where as OLTP’s need for fast writes usually means row based storage is more efficient. I think the solution would be one system that stores data consistently both ways and knows when to use which method for a given query.

Re: Apache iceberg the Hadoop of the modern-data-stack?

#36

Post by FuriouslyAdrift »

simlevesque wrote: post #25

I'm working on an alternative Iceberg client to work better in write heavy use cases. Instead of many smaller files it writes on the same file until it's 1mb in size but it gives it a new name. Then I update the manifest to the new filename and checksum. I keep old files on disk for 60 seconds to allow pending queries. I'm also working on auto compaction, when I have ten 1mb files I compact them, same with ten 10mb f…

so... sharding?

Re: Apache iceberg the Hadoop of the modern-data-stack?

#37

Post by hendiatris »

indoordin0saur wrote: post #19
jasonjmcghee wrote: post #13

Earlier quoted context omitted.

Have you written about your parquet strategy anywhere? Or have suggested reading related to the tuning you've done? Super interested.

Also very interested in the parquet tuning. I have been building my data lake and most optimization I do is just with efficient partitioning.

I will write something up when the dust settles, I’m still testing things out. It’s a project where the data is fairly standardized but there is about a petabyte to deal with, so I think it makes sense to make investments in efficiency at the lower level rather than through tons of resources at it. That has meant a custom parser for the input data written in Rust, lots of analysis of the statistics of the data, etc. It has been a different approach to data engineering and one that I hope we see more of.

Regarding reading materials, I found this DuckDB post to be especially helpful in realizing how parquet could be better leveraged for efficiency: https://duckdb.org/2024/03/26/42-parquet-a-zip-bomb-for-the-...

Re: Apache iceberg the Hadoop of the modern-data-stack?

#38

Post by chehai »

simlevesque wrote: post #25

I'm working on an alternative Iceberg client to work better in write heavy use cases. Instead of many smaller files it writes on the same file until it's 1mb in size but it gives it a new name. Then I update the manifest to the new filename and checksum. I keep old files on disk for 60 seconds to allow pending queries. I'm also working on auto compaction, when I have ten 1mb files I compact them, same with ten 10mb f…

This approach reminds me of ClickHouse's MergeTree.

Also, https://paimon.apache.org/ seems to be better for streaming use cases.

Re: Apache iceberg the Hadoop of the modern-data-stack?

#39

Post by thom »

ndm000 wrote: post #35
thom wrote: post #29

Earlier quoted context omitted.

Interesting. My personal feeling is that we're slowly headed to a world where we can have our cake and eat it: fast bulk ingestion, fast OLAP, fast OLTP, low latency, all together in the same datastore. I'm hoping we just get to collapse whole complex data platforms into a single consistent store with great developer experience, and never look back.

I’ve felt the same way. It’s so inefficient to have two patterns - OLAP and OLTP - both using SQL interfaces but requiring syncing between systems. There are some physical limits at play though. OLAP will always take less processing and disk usage if the data it needs is all right next to each other (columnar storage) where as OLTP’s need for fast writes usually means row based storage is more efficient. I think the…

In a sense, OLAP is just a series of indexing strategies that takes OLTP data and formats it for particular use cases (sometimes with eventual consistency). Some of these indexing strategies in enterprises today involve building out entire bespoke platforms to extract and transform the data. Incremental view maintenance is a step in the right direction - tools like Materialize give you good performance to keep calculated data up to date, and also break out of the streaming world of only paying attention to recent data. But you need to close the loop and also be able to do massive crunchy queries on top of that. I have no doubt we'll get there, really exciting times.

Re: Apache iceberg the Hadoop of the modern-data-stack?

#40

Post by twoodfin »

foobiekr wrote: post #21
twoodfin wrote: post #16

Earlier quoted context omitted.

Very different business, of course, but Databricks v. Fabric reminds me a lot of Slack v. Teams. Regardless of the relative merits now, I think everyone agrees that a few years ago Slack was clearly superior. Microsoft could have certainly bought Slack instead of pumping probably billions into development, marketing, discounts to destroy them. I think Microsoft could and would consider buying Databricks—$80–100B is a…

Anti-trust is the reason a lot of the kinds of deals you’re talking about don’t happen.

I agree. If the anti-trust regime had been different Microsoft would have bought Databricks years ago. Satya Nadella has surely been tapping his foot watching their valuation grow and grow.

The Trump folks have given mixed messages on the Biden-era FTC; I'd put the odds that with the right tap dancing (sigh) Microsoft could make a blockbuster like this in the B2B space work.

Post reply on HN