I love Supabase and what they're doing! I evaluated them heavily when designing architecture for a healthcare product. I'm not sure about this one though - rust is a great systems language, but it wouldn't be my first choice for bridging the db api gap. I wonder why this wasn't built on top of, or an enhancement to, the existing (excellent) multicorn[1] project. Python seems like a better choice of language for deali…
> I wonder why this wasn't built on top of, or an enhancement to, the existing (excellent) multicorn[1] project Have to agree with you there, multicorn is extremely cool. I'm a big sqlalchemy fan so their default SQLA wrapper was a killer feature to give up (although maybe we could do something similar with launchbadge/sqlx[1]). We investigated using multicorn early this year and had a few hiccups. Activity on the or…
Supabase Wrappers: A framework for building Postgres foreign data wrappers
41–47 of 47 posts
Re: Supabase Wrappers: A framework for building Postgres foreign data wrappers
#42Earlier quoted context omitted.
> speaks a particular API over the network it's a interesting idea, and one of the things that we were toying with in our pg_net extension ( https://github.com/supabase/pg_net ). This is a "generic" async network extension, so you can fetch/put/post. It works well for APIs. I think the generic approach works for some things where the data is less "fixed" - for example, an OpenAI API endpoint. But for "fixed" data (da…
I don't think so... in your example those are functions we just call in SQL right? I'm talking about exposing arbitrary data as a table like an FDW, but without having to compile a whole module. So more like, (this is a bit contrived but bear with me): Let's say I wanted to make a table that lists all my keybase proofs (so sad Keybase is basically dead, but I digress). I'd build a graphql endpoint where you can query…
> define a metadata endpoint that maybe returns a JSON schema
We could use an OpenAPI spec[0], which many REST services expose as a JSON endpoint. I think that could work: develop a generic "OpenAPI" FDW, which you can connect like this:
create server example_api
foreign data wrapper example_api_wrapper
options (
open_api_url 'https://api.example.com/'
);
Then we'd map: GET -> SELECT (which would map selected columns into a table)
POST -> INSERT
PATCH -> UPDATE
DELETE -> DELETE
[0] OpenAPI: https://swagger.io/specification/Re: Supabase Wrappers: A framework for building Postgres foreign data wrappers
#43Earlier quoted context omitted.
I don't think so... in your example those are functions we just call in SQL right? I'm talking about exposing arbitrary data as a table like an FDW, but without having to compile a whole module. So more like, (this is a bit contrived but bear with me): Let's say I wanted to make a table that lists all my keybase proofs (so sad Keybase is basically dead, but I digress). I'd build a graphql endpoint where you can query…
Yes that makes sense. > define a metadata endpoint that maybe returns a JSON schema We could use an OpenAPI spec[0], which many REST services expose as a JSON endpoint. I think that could work: develop a generic "OpenAPI" FDW, which you can connect like this: create server example_api foreign data wrapper example_api_wrapper options ( open_api_url 'https://api.example.com/' ); Then we'd map: GET -> SELECT (which woul…
I’m wondering if this is how multicorn works even now. It seems to be an out of process thing. I wasn’t aware of multicorn before.
But python… wish it was elixir :-)
Re: Supabase Wrappers: A framework for building Postgres foreign data wrappers
#44I really want to love foreign data wrappers for Postgres and this seems like a big improvement over existing Python library, but the lack of support for them in managed databases services makes them a non-starter for so many use-cases. Because RDS, for example, will only support the foreign data wrapper for reading from another "Postgres", what we really need is a server that supports the Postgres wire protocol (easi…
Never tried this with RDS, but it's entirely possible to use postgres_fdw to interact with a foreign table on another postgres server, where you might have more choice about what extensions you run. I'm doing this right now because I have a postgres installation that it's not yet convenient to upgrade beyond v12, but where I'd really like the benefit of the recently-improved JDBC FDW, which requires at least v13.
Re: Supabase Wrappers: A framework for building Postgres foreign data wrappers
#45Earlier quoted context omitted.
Wait, you're telling me that you can create materialized views using foreign tables in Postgres? Is there a way to propagate changes from the foreign data source through the FDW to Postgres? Or would it just be some kind of task polls the foreign data source pulling a delta?
You can create a custom function doing all the delta logic and use pg_cron to schedule it to periodically materialize data using FDW.
Once I heard those words mentioned, I began to imagine what sort of interaction with a foreign data source's operations log FDW might have.
Of course, and depending on the foreign data source, you could probably even expose an operations log to Postgres via a FDW by defining it as a foreign table... Effectively opening up the possibility of eventually consistent replication. Lots of data stores' operations logs are accessible data collections/tables anyways.
All of this is exciting stuff!
Re: Supabase Wrappers: A framework for building Postgres foreign data wrappers
#46Earlier quoted context omitted.
The blog post, repo and your comment still leave me a little confused about what Supabase's wrappers project is, the intent and its current state. So this "wrappers" project is just a framework that simplifies the development and manages the curation and distribution of Postgres Foreign Data Wrappers? At the end of the day, are these just FDWs that run on top of any Postgres instance? Do I have to run Supabase to tak…
| At the end of the day, are these just FDWs that run on top of any Postgres instance? Yes, the FDWs developed by Wrappers can be used on any Postgres db. | Do I have to run Supabase to take advantage of the FDWs created through this framework? No, you don't need to. Those are just normal FDWs can be used on any Postgres db. | Is there a generic JDBC/SQL/etc FDW for any SQL based database like Snowflake, Oracle, etc?…
I looked into using a postgres FDW years ago when I was trying to come up with an easy way to ELT data out of Mongo into a DB. Effectively using Postgres as the ETL tool. Understanding where to find FDWs, who maintains them, etc... was overwhelming. I can see the merits of this framework that you're building, and def going to keep an eye on it.
Also always excited to hear about Rust being used.