Live data from Hacker News

Supabase Wrappers: A framework for building Postgres foreign data wrappers

supabase.com

41–47 of 47 posts

Re: Supabase Wrappers: A framework for building Postgres foreign data wrappers

#41

I love Supabase and what they're doing! I evaluated them heavily when designing architecture for a healthcare product. I'm not sure about this one though - rust is a great systems language, but it wouldn't be my first choice for bridging the db api gap. I wonder why this wasn't built on top of, or an enhancement to, the existing (excellent) multicorn[1] project. Python seems like a better choice of language for deali…

> I wonder why this wasn't built on top of, or an enhancement to, the existing (excellent) multicorn[1] project Have to agree with you there, multicorn is extremely cool. I'm a big sqlalchemy fan so their default SQLA wrapper was a killer feature to give up (although maybe we could do something similar with launchbadge/sqlx[1]). We investigated using multicorn early this year and had a few hiccups. Activity on the or…

This was an excellent write-up, thank you for taking the time to explain, can't wait to see where it goes!

Re: Supabase Wrappers: A framework for building Postgres foreign data wrappers

#42
post #37

Earlier quoted context omitted.

> speaks a particular API over the network it's a interesting idea, and one of the things that we were toying with in our pg_net extension ( https://github.com/supabase/pg_net ). This is a "generic" async network extension, so you can fetch/put/post. It works well for APIs. I think the generic approach works for some things where the data is less "fixed" - for example, an OpenAI API endpoint. But for "fixed" data (da…

I don't think so... in your example those are functions we just call in SQL right? I'm talking about exposing arbitrary data as a table like an FDW, but without having to compile a whole module. So more like, (this is a bit contrived but bear with me): Let's say I wanted to make a table that lists all my keybase proofs (so sad Keybase is basically dead, but I digress). I'd build a graphql endpoint where you can query…

Yes that makes sense.

> define a metadata endpoint that maybe returns a JSON schema

We could use an OpenAPI spec[0], which many REST services expose as a JSON endpoint. I think that could work: develop a generic "OpenAPI" FDW, which you can connect like this:

    create server example_api
      foreign data wrapper example_api_wrapper
      options (
        open_api_url 'https://api.example.com/'
      );
Then we'd map:

    GET -> SELECT (which would map selected columns into a table)
    POST -> INSERT
    PATCH -> UPDATE
    DELETE -> DELETE

[0] OpenAPI: https://swagger.io/specification/

Re: Supabase Wrappers: A framework for building Postgres foreign data wrappers

#43
post #37

Earlier quoted context omitted.

I don't think so... in your example those are functions we just call in SQL right? I'm talking about exposing arbitrary data as a table like an FDW, but without having to compile a whole module. So more like, (this is a bit contrived but bear with me): Let's say I wanted to make a table that lists all my keybase proofs (so sad Keybase is basically dead, but I digress). I'd build a graphql endpoint where you can query…

Yes that makes sense. > define a metadata endpoint that maybe returns a JSON schema We could use an OpenAPI spec[0], which many REST services expose as a JSON endpoint. I think that could work: develop a generic "OpenAPI" FDW, which you can connect like this: create server example_api foreign data wrapper example_api_wrapper options ( open_api_url 'https://api.example.com/' ); Then we'd map: GET -> SELECT (which woul…

Right!

I’m wondering if this is how multicorn works even now. It seems to be an out of process thing. I wasn’t aware of multicorn before.

But python… wish it was elixir :-)

Re: Supabase Wrappers: A framework for building Postgres foreign data wrappers

#44

I really want to love foreign data wrappers for Postgres and this seems like a big improvement over existing Python library, but the lack of support for them in managed databases services makes them a non-starter for so many use-cases. Because RDS, for example, will only support the foreign data wrapper for reading from another "Postgres", what we really need is a server that supports the Postgres wire protocol (easi…

Never tried this with RDS, but it's entirely possible to use postgres_fdw to interact with a foreign table on another postgres server, where you might have more choice about what extensions you run. I'm doing this right now because I have a postgres installation that it's not yet convenient to upgrade beyond v12, but where I'd really like the benefit of the recently-improved JDBC FDW, which requires at least v13.

I am very interested in this idea. Has anyone used postgres_fwd in RDS to connect to their own postgres instance where they run custom extensions?

Re: Supabase Wrappers: A framework for building Postgres foreign data wrappers

#45
post #34

Earlier quoted context omitted.

Wait, you're telling me that you can create materialized views using foreign tables in Postgres? Is there a way to propagate changes from the foreign data source through the FDW to Postgres? Or would it just be some kind of task polls the foreign data source pulling a delta?

You can create a custom function doing all the delta logic and use pg_cron to schedule it to periodically materialize data using FDW.

Yeah, this was the approach I was imagining before someone mentioned Materialized View (proper).

Once I heard those words mentioned, I began to imagine what sort of interaction with a foreign data source's operations log FDW might have.

Of course, and depending on the foreign data source, you could probably even expose an operations log to Postgres via a FDW by defining it as a foreign table... Effectively opening up the possibility of eventually consistent replication. Lots of data stores' operations logs are accessible data collections/tables anyways.

All of this is exciting stuff!

Re: Supabase Wrappers: A framework for building Postgres foreign data wrappers

#46
post #33

Earlier quoted context omitted.

The blog post, repo and your comment still leave me a little confused about what Supabase's wrappers project is, the intent and its current state. So this "wrappers" project is just a framework that simplifies the development and manages the curation and distribution of Postgres Foreign Data Wrappers? At the end of the day, are these just FDWs that run on top of any Postgres instance? Do I have to run Supabase to tak…

| At the end of the day, are these just FDWs that run on top of any Postgres instance? Yes, the FDWs developed by Wrappers can be used on any Postgres db. | Do I have to run Supabase to take advantage of the FDWs created through this framework? No, you don't need to. Those are just normal FDWs can be used on any Postgres db. | Is there a generic JDBC/SQL/etc FDW for any SQL based database like Snowflake, Oracle, etc?…

Thanks! Very awesome and exciting stuff!

I looked into using a postgres FDW years ago when I was trying to come up with an easy way to ELT data out of Mongo into a DB. Effectively using Postgres as the ETL tool. Understanding where to find FDWs, who maintains them, etc... was overwhelming. I can see the merits of this framework that you're building, and def going to keep an eye on it.

Also always excited to hear about Rust being used.

Re: Supabase Wrappers: A framework for building Postgres foreign data wrappers

#47
I'm eager to start using foreign data wrappers in PG, but there are close to zero benchmarks. Is my database going to break when transactions start piling up because an API I'm using is slow? Will my TPS drop significantly due to longer transactions? In other words, what am I paying for all this good stuff?
Post reply on HN