Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pg_lake: Postgres with Iceberg and data lake access

github.com

Post reply on HN
Print view View original

91–100 of 126 posts

Re: Pg_lake: Postgres with Iceberg and data lake access

#91

Post by gigatexal »

thejosh wrote: post #81
gigatexal wrote: post #79

Earlier quoted context omitted.

Honestly. Just pay snowflake for the amazing DB and ecosystem it is. And then go build cool stuff unless your value add to customers is infra let them handle all that.

Sounds great until you're locked into Snowflake - so glad iceberg is becoming the standard, anything is great. The trap you end up in is you have to pay snowflake to access your data, iceberg and other technology help with the walled garden. Not just snowflake, any pay on use provider. (Context - have spent 5+ years working with Snowflake, it's great, have built drivers for various languages, etc).

Locked in? I mean they’re your partner. As long as you’re deriving value from them the partnership is still valuable no?

Re: Pg_lake: Postgres with Iceberg and data lake access

#92

Post by kortilla »

gigatexal wrote: post #91
thejosh wrote: post #81

Earlier quoted context omitted.

Sounds great until you're locked into Snowflake - so glad iceberg is becoming the standard, anything is great. The trap you end up in is you have to pay snowflake to access your data, iceberg and other technology help with the walled garden. Not just snowflake, any pay on use provider. (Context - have spent 5+ years working with Snowflake, it's great, have built drivers for various languages, etc).

Locked in? I mean they’re your partner. As long as you’re deriving value from them the partnership is still valuable no?

Yes, don’t be obtuse. “Vendor lock-in” is not some foreign unheard of concept.

Re: Pg_lake: Postgres with Iceberg and data lake access

#93

Post by wodenokoto »

fifilura wrote: post #86

How do you use your data lake? For me it is much more than just storing data, it is just as much for crunching numbers in unpredictable ways. And this is where postgres does not cut it. You need some more CPU and RAM than what you pay for in your postgres instance. I.e. a distributed engine where you don't have to worry about how big your database instance is today.

The point about a datalake is to separate computer and storage. Postgres isn’t a compute layer it’s an access layer.

Your compute asks Postgres “what is the current data for these keys?” Or “what was the current data as of two weeks ago for these keys?” And your compute will then download and aggregate your analytics query directly from the parquet files.

Re: Pg_lake: Postgres with Iceberg and data lake access

#94

Post by thejosh »

gigatexal wrote: post #91
thejosh wrote: post #81

Earlier quoted context omitted.

Sounds great until you're locked into Snowflake - so glad iceberg is becoming the standard, anything is great. The trap you end up in is you have to pay snowflake to access your data, iceberg and other technology help with the walled garden. Not just snowflake, any pay on use provider. (Context - have spent 5+ years working with Snowflake, it's great, have built drivers for various languages, etc).

Locked in? I mean they’re your partner. As long as you’re deriving value from them the partnership is still valuable no?

Everytime you want to query your data, you need to pay the compute cost.

If instead you can write to something like Parquet/Iceberg, you're not paying for access your data.

Snowflake is great at aggregations and other stuff (seriously, huge fan of snowflakes SQL capabilities), but let's say you have a visualisation tool, you're paying for pulling data out .

Instead, writing data to something like S3, you instead can hookup your tools to this.

It's expensive to pull data out of Snowflake otherwise.

Re: Pg_lake: Postgres with Iceberg and data lake access

#95

Post by hamasho »

I like data lakes and their SQL-like query languages. They feel like an advanced version of the "Everything is a file" philosophy.

Under "Everything is a file", you can read or manipulate a wide variety of information via simple, open/read/write() APIs. Linux provides APIs to modify system settings via filesystem. Get the screen brightness setting? `cat /sys/class/backlight/device0/brightness`. Update? `echo 500 > /sys/class/backlight/device0/brightness`. No need for special APIs, just generic file operations and the kernel handles everything.

FUSE (Filesystem in Userspace) provides even more flexibility by allowing user space programs to build their own drivers that handle any data operation via the filesystem. You can mount remote systems (via SSH) and google drive, and copying files is as easy as `cp /mnt/remote/data/origin /mnt/googledrive/data/`. Or using unique FUSE like pgfuse and redisfs, updating redis value by postgres DB data is just `cat /mnt/postgres/users/100/full_name > /mnt/redis/user_100_full_name`.

But filesystems are only good for hierarchical data while a lot of real world data is relational. Many FUSE software tries hard to represent inherently non-hierarchical data in a filesystem. Data lake allows to use SQL, the elegant abstraction for relational data, across different underlying data sources. They can be physically distant and have different structures. A lot of real world applications are just CRUD on relational data. You can accomplish much more much easier if those data are just a big single database.

Re: Pg_lake: Postgres with Iceberg and data lake access

#96

Post by gigatexal »

thejosh wrote: post #94
gigatexal wrote: post #91

Earlier quoted context omitted.

Locked in? I mean they’re your partner. As long as you’re deriving value from them the partnership is still valuable no?

Everytime you want to query your data, you need to pay the compute cost. If instead you can write to something like Parquet/Iceberg, you're not paying for access your data. Snowflake is great at aggregations and other stuff (seriously, huge fan of snowflakes SQL capabilities), but let's say you have a visualisation tool, you're paying for pulling data out . Instead, writing data to something like S3, you instead can…

You people can’t be serious, right?

Ok so I build my data lake on s3 using all open tech. I’m still paying for S3 for puts and reads and lists.

Ok I put it on my own hardware. In my own colo. you’re still paying electricity and other things. Everything is lock in.

On top of that you’re beholden to an entire community of people and volunteers to make your tech work. Need a feature? Sponsor it. Or write it and fight to upstream it. On top of that if you do this at scale at a company what about the highly paid team of engineers you have to have to maintain all this?

With snowflake I alone could provide an entire production ready bi stack to a company. And I can do so and sleep well at night knowing it’s managed and taken care of and if it fails entire teams of people are working to fix it.

Are you going to build your own roads, your own power grid, your own police force?

Again my point remains. The vast majority of times people build on a vendor as a partner and then go on to build useful things.

Apple using cloud vendors for iCloud storage. You think they couldn’t do it themselves? They couldn’t find and pay and support all the tech their own? Of course they could. But they have better things to do than to reinvent the wheel I.e building value on top of dumb compute and that’s iCloud.

Re: Pg_lake: Postgres with Iceberg and data lake access

#97

Post by thejosh »

gigatexal wrote: post #96
thejosh wrote: post #94

Earlier quoted context omitted.

Everytime you want to query your data, you need to pay the compute cost. If instead you can write to something like Parquet/Iceberg, you're not paying for access your data. Snowflake is great at aggregations and other stuff (seriously, huge fan of snowflakes SQL capabilities), but let's say you have a visualisation tool, you're paying for pulling data out . Instead, writing data to something like S3, you instead can…

You people can’t be serious, right? Ok so I build my data lake on s3 using all open tech. I’m still paying for S3 for puts and reads and lists. Ok I put it on my own hardware. In my own colo. you’re still paying electricity and other things. Everything is lock in. On top of that you’re beholden to an entire community of people and volunteers to make your tech work. Need a feature? Sponsor it. Or write it and fight to…

After running Snowflake in production for 5+ years I would rather have my data on something like Parquet/Iceberg (which Snowflake fully supports...) than in the table format Snowflake has.

It's not that deep

Re: Pg_lake: Postgres with Iceberg and data lake access

#99

Post by max_streese »

Two questions:

(1) Are there any plans to make this compatible with the ducklake specification? Meaning: Instead of using Iceberg in the background, you would use ducklake with its SQL tables? My knowledge is very limited but to me, besides leveraging duckdb, another big point of ducklake is that it's using SQL for the catalog stuff instead of a confusing mixture of files, thereby offering a bunch of advantages like not having to care about number of snapshots and better concurrent writes.

(2) Might it be possible that pg_duckdb will achieve the same thing in some time or do things not work like that?

Re: Pg_lake: Postgres with Iceberg and data lake access

#100

Post by hexo »

Oh datalakes. The most ridiculous idea in data processing, right after data frames in python.

We've had this discussion like a week ago about how stupid is to use filesystem for this kind of data storage and here we go again. Actually i had to implement this "idea" in practice. What a nonsense.

Post reply on HN