Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache iceberg the Hadoop of the modern-data-stack?

blog.det.life

Post reply on HN
Print view View original

61–67 of 67 posts

Re: Apache iceberg the Hadoop of the modern-data-stack?

#61

Post by EdwardDiego »

hendiatris wrote: post #2

This is a huge challenge with Iceberg. I have found that there is substantial bang for your buck in tuning how parquet files are written, particularly in terms of row group size and column-level bloom filters. In addition to that, I make heavy use of the encoding options (dictionary/RLE) while denormalizing data into as few files as possible. This has allowed me to rely on DuckDB for querying terabytes of data at low…

Parquet tuning has always been like that, ever since it first came out in 2013.

I worry with Iceberg that people think it's just a case of "use an Iceberg table in Snowflake" and boom, amazingly fast querying of data in S3!

Re: Apache iceberg the Hadoop of the modern-data-stack?

#62

Post by EdwardDiego »

indoordin0saur wrote: post #19
jasonjmcghee wrote: post #13

Earlier quoted context omitted.

Have you written about your parquet strategy anywhere? Or have suggested reading related to the tuning you've done? Super interested.

Also very interested in the parquet tuning. I have been building my data lake and most optimization I do is just with efficient partitioning.

What query engine are you using?

Tends to be that an optimal file size for Parquet is about 1GiB, once again, the "many small files" problem of Hadoop remains.

Then it's things like, can you organise your data in such a way to take advantage of RLE etc.?

Re: Apache iceberg the Hadoop of the modern-data-stack?

#63

Post by orthoxerox »

I think the complexity of Iceberg is overblown. It's just a table format and it's strictly better than the Hive-style /schema/table/partition_key=partition_value/one_of_many_files.parquet

It has a lot of knobs to fiddle with (more than Delta Lake, which tries very hard to come up with good defaults), but even if you don't touch any of them, you already end up with tables that are as good as Hive's, except now your writers don't break your readers.

This is already a massive boon that lets you escape the rigidity of a timetable schedule for your data pipelines. Anything else you can come up with (switching your table to MOR and rewriting it as a separate step etc) is further improvements.

Re: Apache iceberg the Hadoop of the modern-data-stack?

#64

Post by pid-1 »

tsss wrote: post #53
pid-1 wrote: post #42

Earlier quoted context omitted.

If you're already in AWS, why wouldn't you use AWS Glue Catalog + AWS SDK for pandas + Athena? You can setup a data lake, save data and start doing queries in like 10 minutes with this setup.

Athena is really expensive though and you will often run into a hard limit on the size of your query.

Like most things serverless Athena is cheap as long as you don't use it.

My company has 100s of data pipelines that are executed infrequently.

For this use case Athena is ridiculously cheap and easy to use vs most other solutions.

Re: Apache iceberg the Hadoop of the modern-data-stack?

#65

Post by gregw2 »

thom wrote: post #29
simlevesque wrote: post #25

I'm working on an alternative Iceberg client to work better in write heavy use cases. Instead of many smaller files it writes on the same file until it's 1mb in size but it gives it a new name. Then I update the manifest to the new filename and checksum. I keep old files on disk for 60 seconds to allow pending queries. I'm also working on auto compaction, when I have ten 1mb files I compact them, same with ten 10mb f…

Interesting. My personal feeling is that we're slowly headed to a world where we can have our cake and eat it: fast bulk ingestion, fast OLAP, fast OLTP, low latency, all together in the same datastore. I'm hoping we just get to collapse whole complex data platforms into a single consistent store with great developer experience, and never look back.

I'd love it, but I feel like there is another horizon that Iceberg hasn't tackled to truly get us there.

Iceberg (and Delta Table format) is really OLAP-optimized, being built on a columnar datastore, Parquet. This means it will be slow to do writes compared to a traditional row-based datastore and doesn't really have normal/optimal OLTP indexing.

Fast OLTP + Fast OLAP + low latency is best done via HTAP-type databases which store data in both row and columnar form and give you ability in the SELECT clause to pick your latency tolerance and the query engine will pick the OLTP engine if it knows there are still some OLAP writes queued up that entered the system more than ago but aren't fully on disk yet.

Various vendors do have HTAP, but all with proprietary storage engines and query engines. But Iceberg alone doesn't get you there. I haven't seen discussion of this I don't know if anyone has tried to write both Hudi and Iceberg/Delta in parallel so they could do HTAP; maybe they use pure Hudi instead?

I'd have to re-look at Hudi to see if it's deferred compaction is more like this. XTable doesn't seem to target this issue.

Re: Apache iceberg the Hadoop of the modern-data-stack?

#66

Post by Gasp0de »

0cf8612b2e1e wrote: post #57
Gasp0de wrote: post #8

Earlier quoted context omitted.

The problem is that the initial writing is already so expensive, I guess we'd have to write multiple sensors into the same file instead of having one file per sensor per interval. I'll look into parquet access options, if we could write 10k sensors into one file but still read a single sensor from that file that could work.

Why can you not rewrite the initial file into something partitioned by sensor+time? Would the one time job really be that much more additional cost vs the additional complexity of multiple sensors per file? Do you ever go back and reaggregate older data into bigger, sorted files? That is, maybe you originally partitioned by hour, but stale data is so infrequently accessed, you could roll up into partitions per week/m…

The costly thing is the intial writing already. S3 is our cold storage, we don't often read from it. So compaction would only make reading cheaper, but create a writing cost in the process.

Re: Apache iceberg the Hadoop of the modern-data-stack?

#67

Post by indoordin0saur »

EdwardDiego wrote: post #62
indoordin0saur wrote: post #19

Earlier quoted context omitted.

Also very interested in the parquet tuning. I have been building my data lake and most optimization I do is just with efficient partitioning.

What query engine are you using? Tends to be that an optimal file size for Parquet is about 1GiB, once again, the "many small files" problem of Hadoop remains. Then it's things like, can you organise your data in such a way to take advantage of RLE etc.?

Either Spark or Redshift (serverless)
Post reply on HN