Viewing profile — mslot

Makes sense, we just shipped it https://www.linkedin.com/posts/craigkerstiens_barely-over-2-...

The challenge is converting primary key updates/deletes to row offsets in a columnar table. That requires maintaining an expensive mapping or doing expensive scans, and is not some…

In principle, Postgres has an infinite number of possible types :). pg_lake maps types into their Parquet equivalent and otherwise stores as text representation, there are a few li…

(1) We've thought about it, no current plans. We'd ideally reimplement DuckLake in Postgres directly such that we can preserve Postgres transaction boundaries, rather than reuse th…

It's the same team and same project :). Crunchy Data was acquired by Snowflake.

When we first developed pg_lake at Crunchy Data and defined GTM we considered whether it could be a Snowflake competitor, but we quickly realised that did not make sense. Data plat…

Yes, just COPY table TO 's3://mybucket/data.parquet' Or COPY table TO STDOUT WITH (format 'parquet') if you need it on the client side.

Definitely similar goals, from the Mooncake author: https://news.ycombinator.com/item?id=43298145 I think pg_mooncake is still relatively early stage. There's a degree of maturity …

DuckLake is pretty cool, and we obviously love everything the DuckDB is doing. It's what made pg_lake possible, and what motivated part of our team to step away from Microsoft/Citu…

You can use it as a read layer for for specific metadata JSON URL or a table in a REST catalog. The latter got merged quite recently, not yet in docs.

I gave a talk on that at Data Council, then still discussing the pg_lake extensions as part of Crunchy Data Warehouse. https://youtu.be/HZArjlMB6W4?si=BWEfGjMaeVytW8M1 Also, nicer …

There are Postgres roles for read/write access to the S3 object that DuckDB has access to. Those roles can create tables from specific files or at specific locations, and can then …

You could say With DuckLake, the query frontend and query engine are DuckDB, and Postgres is used as a catalog in the background. With pg_lake, the query frontend and catalog are P…

We first launched it as "Crunchy Bridge for Analytics" in April last year. At the time, it could mostly query/import/export Parquet/CSV/JSON. Our goal was to build a data warehouse…

It's a similar idea, but Crunchy Data Warehouse was built by several founding engineers of Citus, which lets us speedrun through it :) It's a generally available (very solid) produ…

I created pg_incremental because I keep running into the same challenge in PostgreSQL: You have a table of raw event data that you insert into, either individual rows when the even…

Not really. S3 tables provides storage & catalog for Iceberg tables, but is not a query/transaction engine. We'll definitely look for ways to take advantage of it as it matures. I …

Azure is not supported as a backend in pg_parquet right now, but shouldn't be hard to add (contributions welcome!) https://github.com/CrunchyData/pg_parquet It would not be safe to…

Fun fact, I created pg_azure_storage :)

(Marco from Crunchy Data) With PostgreSQL extensions, we find it's most effective to have single-purpose modular extensions. For instance, I created pg_cron a few years ago, and it…

Nice! Seems to be pretty well-crafted.

I think it comes down to every aspect of the DBMS being optimized differently. For instance, UDFs in DuckDB have vectors as input and output, while that would be confusing and unne…