Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pg_lake: Postgres with Iceberg and data lake access

github.com

Post reply on HN
Print view View original

81–90 of 126 posts

Re: Pg_lake: Postgres with Iceberg and data lake access

#81

Post by thejosh »

gigatexal wrote: post #79
ozgune wrote: post #2

This is huge! When people ask me what’s missing in the Postgres market, I used to tell them “open source Snowflake.” Crunchy’s Postgres extension is by far the most ahead solution in the market. Huge congrats to Snowflake and the Crunchy team on open sourcing this.

Honestly. Just pay snowflake for the amazing DB and ecosystem it is. And then go build cool stuff unless your value add to customers is infra let them handle all that.

Sounds great until you're locked into Snowflake - so glad iceberg is becoming the standard, anything is great.

The trap you end up in is you have to pay snowflake to access your data, iceberg and other technology help with the walled garden.

Not just snowflake, any pay on use provider.

(Context - have spent 5+ years working with Snowflake, it's great, have built drivers for various languages, etc).

Re: Pg_lake: Postgres with Iceberg and data lake access

#82

Post by ggregoire »

NeutralCrane wrote: post #80

I’m not a data engineer but work in an adjacent role. Is there anyone here who could dumb the use case down? Maybe an example of a problem this solves. I am struggling to understand the value proposition here.

> Maybe an example of a problem this solves.

Some service writes a lot of data in parquet files stored on S3 (e.g. logs), and now you want that data to be queryable from your application as if it was in postgres (e.g. near real-time analytics dashboard). pg_lake allows you to load these parquet files into postgres and query the data. You can also join that data with existing tables in postgres.

Re: Pg_lake: Postgres with Iceberg and data lake access

#84

Post by claudeomusic »

lysecret wrote: post #68
claudeomusic wrote: post #63

Can someone dumb this down a bit for a non data-engineer? Hard to fully wrap my head around who this is/isn’t best suited for.

One usecase we have (we built it ourselves) is to periodically offload data from Postgres to lake house partitioned data on GCS. The way I see it this can now be done with a single query. Another one is the other way around to use posters as a query engine or to merge offloaded data with your live data.

What services do people primarily use to accomplish these tasks today? All custom work?

Re: Pg_lake: Postgres with Iceberg and data lake access

#85

Post by ijustlovemath »

mslot wrote: post #45
boshomi wrote: post #36

Why not just use Ducklake?[1] That reduces complexity[2] since only DuckDB and PostgreSQL with pg_duckdb are required. [1] https://ducklake.select/ [2] DuckLake - The SQL-Powered Lakehouse Format for the Rest of Us by Prof. Hannes Mühleisen: https://www.youtube.com/watch?v=YQEUkFWa69o

DuckLake is pretty cool, and we obviously love everything the DuckDB is doing. It's what made pg_lake possible, and what motivated part of our team to step away from Microsoft/Citus. DuckLake can do things that pg_lake cannot do with Iceberg, and DuckDB can do things Postgres absolutely can't (e.g. query data frames). On the other hand, Postgres can do a lot of things that DuckDB cannot do. For instance, it can handl…

Not to mention one of my favorite tools for adding a postgres db to your backend service: PostgREST. Insanely powerful DB introspection and automatic REST endpoint. Pretty good performance too!

Re: Pg_lake: Postgres with Iceberg and data lake access

#86

Post by fifilura »

How do you use your data lake? For me it is much more than just storing data, it is just as much for crunching numbers in unpredictable ways.

And this is where postgres does not cut it.

You need some more CPU and RAM than what you pay for in your postgres instance. I.e. a distributed engine where you don't have to worry about how big your database instance is today.

Re: Pg_lake: Postgres with Iceberg and data lake access

#88

Post by smithclay »

ggregoire wrote: post #82
NeutralCrane wrote: post #80

I’m not a data engineer but work in an adjacent role. Is there anyone here who could dumb the use case down? Maybe an example of a problem this solves. I am struggling to understand the value proposition here.

> Maybe an example of a problem this solves. Some service writes a lot of data in parquet files stored on S3 (e.g. logs), and now you want that data to be queryable from your application as if it was in postgres (e.g. near real-time analytics dashboard). pg_lake allows you to load these parquet files into postgres and query the data. You can also join that data with existing tables in postgres.

Been experimenting with OpenTelemetry->Parquet conversion lately for logs, metrics, and traces. Lots of related projects popping up in this area. It's powerful and cheap.

- https://github.com/smithclay/otlp2parquet (shameless plug, based on Clickhouse's Otel schema) - https://github.com/Mooncake-Labs/moonlink (also has OTLP support) - https://github.com/open-telemetry/otel-arrow (official community project under early dev)

Re: Pg_lake: Postgres with Iceberg and data lake access

#89

Post by swasheck »

boshomi wrote: post #36

Why not just use Ducklake?[1] That reduces complexity[2] since only DuckDB and PostgreSQL with pg_duckdb are required. [1] https://ducklake.select/ [2] DuckLake - The SQL-Powered Lakehouse Format for the Rest of Us by Prof. Hannes Mühleisen: https://www.youtube.com/watch?v=YQEUkFWa69o

i have so desperately wanted to love and use ducklake, but have come across some issues with it in practice (pg catalog). they seem to have to do with maintenance activities and ducklake suddenly throwing http/400 errors on files it created. i’m not sure if it’s due to my write patterns (gather data from sources into a polars dataframe and insert into the ducklake table from the df) into the partitioned tables or something else.

it’s ok in dev/test and for me as the person in the team who’s enamored with duckdb, but it’s made the team experience challenging and so i’ve just kinda reverted to hive partitioned parquet files with a duckdb file that has views created on top of the parquet. attach that file as read only and query away.

i may work up a full example to submit as an issue but up until now too may other things are dominating my time.

Re: Pg_lake: Postgres with Iceberg and data lake access

#90

Post by lysecret »

A usecase I see for this personally I have encountered a lot of “hot cache for some time then offload for historical queries” usecases which I have built by hand multiple times. This should be a great fit. E.g. write to Postgres then periodically offload to lakehouse and even query together (if needed). Very cool!
Post reply on HN