iceberg is plauged with the problems it tries to solve, like being too tied to spark just to write data
Apache Iceberg
41–50 of 68 posts
Re: Apache Iceberg
#42iceberg is plauged with the problems it tries to solve, like being too tied to spark just to write data
huh what? We use iceberg extensively, never used spark.
Re: Apache Iceberg
#43I am stockholder in snowflake and iceberg's ascendance seems to coincide with snow's downfall.
Is the query engine value add justify snowflake's valuation. Their data marketplace thing didn't seem to have actually worked.
Re: Apache Iceberg
#44Earlier quoted context omitted.
for some reason it's really cumbersome to access this tech
I agree, as a long time Business Intelligence developer I‘m still confused and astounded with all the tooling and bits and pieces seemingly necessary to create analytics/dashboards with open source tools. For years I used a proprietary solution like Qlik Sense for the whole journey from data extraction to a finished dashboard (mostly on-prem). Going from raw data to a finished dashboard is a matter of days (not weeks…
I would argue that stuff like Iceberg is really aimed at Data Platform Engineers, not BI analysts. Companies I've worked with in the past have 10-15 people on a Platform team that work directly with stuff like this, to offer analysts and data scientists a view into the company's data.
Re: Apache Iceberg
#45Earlier quoted context omitted.
It’s been on the up in recent years though as it appears to have won the format wars. Every vendor is rallying around it and there were new open source catalogues and support from AWS at the end of 2024.
yeah, I'll admit I was worried when Databricks acquired Tabular[0] that it would hurt Iceberg's momentum (e.g. databricks would push delta instead), but it seems the opposite has happened. 0 - https://www.definite.app/blog/databricks-tabular-acquisition
I was more worried—and continue to be so—that Databricks will bring the rat’s nest of complexity and pseudo-open source model that characterizes Delta to the future of Iceberg.
Re: Apache Iceberg
#46OneHouse also has a fantastic iceberg implementation (they're the team behind Apache Hudi) and does a ton of great interop work: https://www.onehouse.ai/blog/comprehensive-data-catalog-comp... && https://www.onehouse.ai/blog/open-data-foundations-with-apac...
Re: Apache Iceberg
#47I've been looking at Iceberg for a while, but in the end went with Delta Lake because it doesn't have a dependency on a catalog. It also has good support for reading and writing from it without needing Spark. Does anyone know if Iceberg has plans to support similar use cases?
Why don't you want a catalog? The SQL or REST catalogs are pretty light to set up. I have my eye on lakekeeper[0], but Polaris (from Snowflake) is a good option too. PyIceberg is likely the easiest way to write without Spark. 0 - https://github.com/lakekeeper/lakekeeper
PyIceberg is nice but we had to drop it because it's behind Java API and it's unclear when it will match up, so depending on which features are needed I'd look it up
Re: Apache Iceberg
#48Looks good, but come on… at least try to open your website on a mobile device.
Re: Apache Iceberg
#49Why would one choose this instead of DeltaLake?
Re: Apache Iceberg
#50Earlier quoted context omitted.
Why don't you want a catalog? The SQL or REST catalogs are pretty light to set up. I have my eye on lakekeeper[0], but Polaris (from Snowflake) is a good option too. PyIceberg is likely the easiest way to write without Spark. 0 - https://github.com/lakekeeper/lakekeeper
PyIceberg is nice but we had to drop it because it's behind Java API and it's unclear when it will match up, so depending on which features are needed I'd look it up
what are you using instead?