And yet there's still no straightforward way to write directly to Iceberg tables from Javascript as far as I know.
Apache Iceberg
11–20 of 68 posts
Re: Apache Iceberg
#12Apache Iceberg is one of the emerging Open Table Formats in addition to Delta Lake and Apache Hudi [1].
[1] Open Table Formats:
Re: Apache Iceberg
#13And yet there's still no straightforward way to write directly to Iceberg tables from Javascript as far as I know.
Writing to catalogs is still pretty new. Databricks has recently been pushing delta-kernel-rs that DuckDb has a connector set up for, and there’s support for writing via Python with the Polars package through delta-rs. For small-time developers this has been pretty helpful for me and influential in picking delta lake over iceberg.
Re: Apache Iceberg
#14Earlier quoted context omitted.
Delta Lake now has Hilbert-curve based clustering which solves a lot of the downsides of hive partitioning
> Hilbert-curve based clustering which solves a lot of the downsides of hive partitioning Yes, that solved the 2-column high NDV partitioning issue - if you had your ip traffic sorted on destination or source, you need Z-curves, which are a little easier with bit twiddling for fixed types to do the same thing. Hive would write a large number of small files when partitioned like that or you lose efficiencies when scan…
What is NDV partitioning?
Re: Apache Iceberg
#15How do you query your iceberg tables? We are looking into moving away from Bigquery and Starrocks [1] looks like a good option.
Re: Apache Iceberg
#16Earlier quoted context omitted.
> Hilbert-curve based clustering which solves a lot of the downsides of hive partitioning Yes, that solved the 2-column high NDV partitioning issue - if you had your ip traffic sorted on destination or source, you need Z-curves, which are a little easier with bit twiddling for fixed types to do the same thing. Hive would write a large number of small files when partitioned like that or you lose efficiencies when scan…
What is NDV partitioning?
NDV = Number of distinct values. Here partitioning on high cardinality columns, essentially.
Re: Apache Iceberg
#17How do you query your iceberg tables? We are looking into moving away from Bigquery and Starrocks [1] looks like a good option. [1] https://www.starrocks.io/
Trino is pretty good (open source presto).
Re: Apache Iceberg
#18ClickHouse has a solid Iceberg integration. It has an Iceberg table function[0] and Iceberg table engine[1] for interacting with Iceberg data stored in s3, gcs, azure, hadoop etc.
[0] https://clickhouse.com/docs/en/sql-reference/table-functions...
[1] https://clickhouse.com/docs/en/engines/table-engines/integra...
Re: Apache Iceberg
#19And yet there's still no straightforward way to write directly to Iceberg tables from Javascript as far as I know.
Writing to catalogs is still pretty new. Databricks has recently been pushing delta-kernel-rs that DuckDb has a connector set up for, and there’s support for writing via Python with the Polars package through delta-rs. For small-time developers this has been pretty helpful for me and influential in picking delta lake over iceberg.
> influential in picking delta lake over iceberg
Can you expand on those reasons a bit?
The dependency on a catalog in Iceberg made it more complicated for simple cases than Delta, where a directory hierarchy was sufficient - if I was understanding the PyIceberg docs correctly.
Re: Apache Iceberg
#20I'm somewhat surprised to see it here - Iceberg is around for some time already.
It’s been on the up in recent years though as it appears to have won the format wars. Every vendor is rallying around it and there were new open source catalogues and support from AWS at the end of 2024.