Iceberg on S3 tables is going to be a hot topic in the next few years.
Apache Iceberg
21–30 of 68 posts
Re: Apache Iceberg
#22I've been looking at Iceberg for a while, but in the end went with Delta Lake because it doesn't have a dependency on a catalog. It also has good support for reading and writing from it without needing Spark.
Does anyone know if Iceberg has plans to support similar use cases?
Re: Apache Iceberg
#23How do you query your iceberg tables? We are looking into moving away from Bigquery and Starrocks [1] looks like a good option. [1] https://www.starrocks.io/
Why away from bigquery? Just wondering if it’s a cost thing.
Re: Apache Iceberg
#24Apache Iceberg is one of the emerging Open Table Formats in addition to Delta Lake and Apache Hudi [1]. [1] Open Table Formats: https://www.starburst.io/data-glossary/open-table-formats/
The table on that page makes it look like all three of these are very similar, with schema evolution and partition evolution being the key differences. Is that really it?
I’d also love to see a good comparison between “regular” Iceberg and AWS’s new S3 Tables.
Re: Apache Iceberg
#25I've been looking at Iceberg for a while, but in the end went with Delta Lake because it doesn't have a dependency on a catalog. It also has good support for reading and writing from it without needing Spark. Does anyone know if Iceberg has plans to support similar use cases?
Iceberg has the hdfs catalog, which also relies only on dirs and files.
That said, a catalog (which Delta also can have) helps a lot to keep things tidy. For example, I can write a dataset with Spark, transform it with dbt and a query engine (such as Trino) and consume the resulting dataset with any client that supports Iceberg. If I use a catalog, all happens without having to register the dataset location in each of these components.
Re: Apache Iceberg
#26ClickHouse has a solid Iceberg integration. It has an Iceberg table function[0] and Iceberg table engine[1] for interacting with Iceberg data stored in s3, gcs, azure, hadoop etc. [0] https://clickhouse.com/docs/en/sql-reference/table-functions... [1] https://clickhouse.com/docs/en/engines/table-engines/integra...
I would say it doesn't but it is actively working on it
Re: Apache Iceberg
#27And yet there's still no straightforward way to write directly to Iceberg tables from Javascript as far as I know.
for some reason it's really cumbersome to access this tech
Re: Apache Iceberg
#28What I like about iceberg is that the partitions of the tables are not tightly coupled to the subfolder structure of the storage layer (at least logically, at the end of the day the partitions are still subfolders with files), but at least the metadata is not tied to that, so you can change the partition of the tables going forward and still query a mix of old and new partitions time ranges.
In the other hand, since one of the use cases they created it at Netflix was to consume directly from real time systems, the management of the file creation when updates to the data is less trivial (the CoW vs MoR problem and how to compact small files) which becomes important on multi-petabytes tables with lots of users and frequent updates. This is something I assume not a lot companies put a lot of attention to (heck, not even at Netflix) and have big performance and cost implications.
Re: Apache Iceberg
#29And yet there's still no straightforward way to write directly to Iceberg tables from Javascript as far as I know.
What’s your use case? Iceberg is meant for analytical workloads
Re: Apache Iceberg
#30If you're looking to give Iceberg a spin, here's how to get it running locally, on AWS[0] and on GCP[1]. The posts use DuckDB as the query engine, but you could swap in Trino (or even chdb / clickhouse).