[dead]
Pg_lake: Postgres with Iceberg and data lake access
101–110 of 126 posts
Re: Pg_lake: Postgres with Iceberg and data lake access
#102Earlier quoted context omitted.
You people can’t be serious, right? Ok so I build my data lake on s3 using all open tech. I’m still paying for S3 for puts and reads and lists. Ok I put it on my own hardware. In my own colo. you’re still paying electricity and other things. Everything is lock in. On top of that you’re beholden to an entire community of people and volunteers to make your tech work. Need a feature? Sponsor it. Or write it and fight to…
After running Snowflake in production for 5+ years I would rather have my data on something like Parquet/Iceberg (which Snowflake fully supports...) than in the table format Snowflake has. It's not that deep
Ok. And this flexibility is only really possible since they did a lot of work to make external and internal tables roughly equivalent in performance.
Re: Pg_lake: Postgres with Iceberg and data lake access
#103Earlier quoted context omitted.
After running Snowflake in production for 5+ years I would rather have my data on something like Parquet/Iceberg (which Snowflake fully supports...) than in the table format Snowflake has. It's not that deep
Ok. And this flexibility is only really possible since they did a lot of work to make external and internal tables roughly equivalent in performance.
Yeah, performance depends.
I think a hybrid approach works best (store on Snowflake native and iceberg/tables where needed), and allows you the benefit of Snowflake without paying the cost for certain workloads (which really adds up).
We're going to see more of this (either open or closed source), since Snowflake has acquired Crunchydata, and the last major bastion is "traditional" database Snowflake.
Re: Pg_lake: Postgres with Iceberg and data lake access
#104Earlier quoted context omitted.
Ok. And this flexibility is only really possible since they did a lot of work to make external and internal tables roughly equivalent in performance.
Yeah, performance depends. I think a hybrid approach works best (store on Snowflake native and iceberg/tables where needed), and allows you the benefit of Snowflake without paying the cost for certain workloads (which really adds up). We're going to see more of this (either open or closed source), since Snowflake has acquired Crunchydata, and the last major bastion is "traditional" database Snowflake.
I had no idea they did. This pg lake announcement dropped that nugget and i was surprised.
Re: Pg_lake: Postgres with Iceberg and data lake access
#105Earlier quoted context omitted.
Ok. And this flexibility is only really possible since they did a lot of work to make external and internal tables roughly equivalent in performance.
Yeah, performance depends. I think a hybrid approach works best (store on Snowflake native and iceberg/tables where needed), and allows you the benefit of Snowflake without paying the cost for certain workloads (which really adds up). We're going to see more of this (either open or closed source), since Snowflake has acquired Crunchydata, and the last major bastion is "traditional" database Snowflake.
Agreed btw.
Re: Pg_lake: Postgres with Iceberg and data lake access
#106Two questions: (1) Are there any plans to make this compatible with the ducklake specification? Meaning: Instead of using Iceberg in the background, you would use ducklake with its SQL tables? My knowledge is very limited but to me, besides leveraging duckdb, another big point of ducklake is that it's using SQL for the catalog stuff instead of a confusing mixture of files, thereby offering a bunch of advantages like…
(1) We've thought about it, no current plans. We'd ideally reimplement DuckLake in Postgres directly such that we can preserve Postgres transaction boundaries, rather than reuse the Ducklake implementation that would run in a separate process. The double-edged sword is that there's a bunch of complexity around things like inlined data and passing the inlined data into DuckDB at query time, though if we can do that then you can get pretty high transaction performance.
(2) In principle, it's a bit easier for pg_duckdb to reuse the existing Ducklake implementation because DuckDB sits in every Postgres process and they can call into each other, but we feel that architecture is less appropriate in terms resource management and stability.
Re: Pg_lake: Postgres with Iceberg and data lake access
#107Earlier quoted context omitted.
After running Snowflake in production for 5+ years I would rather have my data on something like Parquet/Iceberg (which Snowflake fully supports...) than in the table format Snowflake has. It's not that deep
Ok. And this flexibility is only really possible since they did a lot of work to make external and internal tables roughly equivalent in performance.
They didn't do it out of good will. They realized that's where the market was going and if their query engine didn't perform as well as others on top of iceberg, then they'd be another Oracle in the long-term.
Re: Pg_lake: Postgres with Iceberg and data lake access
#108Earlier quoted context omitted.
Locked in? I mean they’re your partner. As long as you’re deriving value from them the partnership is still valuable no?
Yes, don’t be obtuse. “Vendor lock-in” is not some foreign unheard of concept.
Teams of the smartest people on earth make these kind of big vendor decisions, vendor lock-in is top of mind, I tell anyone who will listen to avoid databricks live tables and their sleezy sales reps pushing it over cheaper less locked in solutions
Re: Pg_lake: Postgres with Iceberg and data lake access
#109Why would Snowflake develop and release this? Doesn't this cannibalize their main product?
One thing I admire about Snowflake is a real commitment to self-cannibalization. They were super out front with Iceberg even though it could disrupt them, because that's what customers were asking for and they're willing to bet they'll figure out how to make money in that new world Video of their SVP of Product talking about it here: https://youtu.be/PERZMGLhnF8?si=DjS_OgbNeDpvLA04&t=1195
Supporting Iceberg is eventually having people leaving you because they have better elsewhere, but this is birectionnal, it means you can welcome people from Databricks because you have feature parity.
Re: Pg_lake: Postgres with Iceberg and data lake access
#110Earlier quoted context omitted.
Sounds great until you're locked into Snowflake - so glad iceberg is becoming the standard, anything is great. The trap you end up in is you have to pay snowflake to access your data, iceberg and other technology help with the walled garden. Not just snowflake, any pay on use provider. (Context - have spent 5+ years working with Snowflake, it's great, have built drivers for various languages, etc).
Locked in? I mean they’re your partner. As long as you’re deriving value from them the partnership is still valuable no?
Not all vendors are same. Snowflake charges an arm and leg for compute.
It’s 36x more expensive than equivalent EC2 compute.