Apache Iceberg
iceberg.apache.org
Apache Iceberg
1–10 of 68 posts
Re: Apache Iceberg
#2Curious to what extent Iceberg enables data composability and what the best complements and alternatives are.
Re: Apache Iceberg
#3I'm somewhat surprised to see it here - Iceberg is around for some time already.
Re: Apache Iceberg
#4Curious to what extent Iceberg enables data composability and what the best complements and alternatives are.
Delta Lake is the main competitor. There's a lot of convergence going on, because everyone wants a common format and it's pretty clear what the desirable features are. Ultimately it becomes just boring infrastructure IMO.
Re: Apache Iceberg
#5Hidden partitioning is the most interesting Iceberg feature, because most of the very large datasets are timeseries fact tables.
I don't remember seeing that in Delta Lake [1], which is probably because the industry standard benchmarks use date as a column (tpc-h) or join date as a dimension table (tpc-ds) and do not use timestamp ranges instead of dates.
Re: Apache Iceberg
#6Curious to what extent Iceberg enables data composability and what the best complements and alternatives are.
It allows you to be query engine agnostic - query the same data via Spark, Snowflake or Trino.
Granted, performance may suffer vs Snowflake internal tables somewhat due to certain performance optimizations not being there.
Re: Apache Iceberg
#7I'm somewhat surprised to see it here - Iceberg is around for some time already.
Yeah, I was confused as well. It was like seeing "postage stamps" on the HN front page.
Re: Apache Iceberg
#8Hidden partitioning is the most interesting Iceberg feature, because most of the very large datasets are timeseries fact tables. I don't remember seeing that in Delta Lake [1], which is probably because the industry standard benchmarks use date as a column (tpc-h) or join date as a dimension table (tpc-ds) and do not use timestamp ranges instead of dates. [1] - https://github.com/delta-io/delta/issues/490
Delta Lake now has Hilbert-curve based clustering which solves a lot of the downsides of hive partitioning
Re: Apache Iceberg
#9Iceberg is a pretty cool guy, he consolidates the Parquet and doesn't afraid of anything.
Re: Apache Iceberg
#10Hidden partitioning is the most interesting Iceberg feature, because most of the very large datasets are timeseries fact tables. I don't remember seeing that in Delta Lake [1], which is probably because the industry standard benchmarks use date as a column (tpc-h) or join date as a dimension table (tpc-ds) and do not use timestamp ranges instead of dates. [1] - https://github.com/delta-io/delta/issues/490
Delta Lake now has Hilbert-curve based clustering which solves a lot of the downsides of hive partitioning
> Hilbert-curve based clustering which solves a lot of the downsides of hive partitioning
Yes, that solved the 2-column high NDV partitioning issue - if you had your ip traffic sorted on destination or source, you need Z-curves, which are a little easier with bit twiddling for fixed types to do the same thing.
Hive would write a large number of small files when partitioned like that or you lose efficiencies when scanning on the non-partitioned column.
This does fix the high NDV issue, but in general Netflix wrote hidden partitioning in specifically to avoid sorting on high NDV columns and to reduce the sort complexity on writes (most daily writes won't need any partitioned inserts at all).
While clustering on timestamp will force a sort even if it is a single day.