Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Pg_replicate – Build Postgres replication applications in Rust

github.com

Post reply on HN
Print view View original

11–20 of 29 posts

Re: Show HN: Pg_replicate – Build Postgres replication applications in Rust

#11

Post by hermanradtke »

eknkc wrote: post #7

I was trying out the stdout example. Could not get it to log anything. DuckDB example worked so I went digging into the source. Apparently the stdout sink is using tracing and I did not have a `RUST_LOG` env var set. Might be a good idea to have it documented or have the default level set to info for the stdout example. Maybe this is common Rust knowledge and I just don't know what I'm doing though.

I too consider this a footgun.

Most applications will have something like

   tracing_subscriber::registry()
           .with(
               tracing_subscriber::EnvFilter::try_from_default_env()
                   .unwrap_or_else(|_| "my_app=info".into()),
           )
           .with(tracing_subscriber::fmt::layer())
           .init();
in the `main.rs` which will default the tracing if RUST_LOG env var is not set.

Re: Show HN: Pg_replicate – Build Postgres replication applications in Rust

#12

Post by jeremyjh »

rubenfiszel wrote: post #6

This is super timely. Windmill ( https://windmill.dev ) used to only support webhooks to trigger code and flow jobs. We have just added email support building our own MX server, and wanted to add CDC change. We were gonna do it on Debezium but this will allow us to remove the need for a third-party service and just add this as a crate. Thank you supabase for open-sourcing this.

I'm super excited to see a feature based on this in Windmill.

Re: Show HN: Pg_replicate – Build Postgres replication applications in Rust

#13

Post by imor80 »

chucky_z wrote: post #5
imor80 wrote: post #2

Hey HN, For the past few months, as part of my job at Supabase, I have been working on pg_replicate. pg_replicate lets you very easily build applications which can copy data (full table copies and cdc) from Postgres to any other data system. Around six months back I was figuring out what can be built by tailing Postgres' WAL. pg_replicate grew organically out of that effort. Many similar tools, like Debezium, exist a…

I’m curious in your experience how many clients can run pg_replicate at once? With MySQL I saw the interesting use-case of the black hole storage engine to scale out replication logs but ultimately the only usage I’m aware of was for scaling other mysql read replicas. The idea of scaling an application by tailing logs from a database sounds very interesting to me, and I’m curious if you’ve explored this at all. There…

> I’m curious in your experience how many clients can run pg_replicate at once?

I'd expect more clients to put more pressure on the resource usage on the db. But it's not clear whether the relationship between number of clients and resource usage is linear, quadratic or something else since I haven't done benchmarking yet.

> The idea of scaling an application by tailing logs from a database sounds very interesting to me, and I’m curious if you’ve explored this at all. There’s of course things like Kafka (and then things like Debezium), but it’s hard to beat direct!

It doesn't exist yet, but I was thinking of creating a sink which exposes Postgres WAL via websockets. This way the number of clients might scale much better.

Re: Show HN: Pg_replicate – Build Postgres replication applications in Rust

#14

Post by imor80 »

convolvatron wrote: post #3
imor80 wrote: post #2

Hey HN, For the past few months, as part of my job at Supabase, I have been working on pg_replicate. pg_replicate lets you very easily build applications which can copy data (full table copies and cdc) from Postgres to any other data system. Around six months back I was figuring out what can be built by tailing Postgres' WAL. pg_replicate grew organically out of that effort. Many similar tools, like Debezium, exist a…

I've recently been playing with the logical replication protocol, and it enables all kinds of interesting usages. one really cool thing is that you see the transactional boundaries, so not only can you write a cache, you can do so in a way thats internally consistent. its also inherently much nicer than listen/notify, since you don't have to go back and figure out what data was associated with the event

Yes, logical replication complements very nicely the normal way of interacting with the database via queries. This inverted flow makes those apps possible which were hard/impossible to build with just queries.

Re: Show HN: Pg_replicate – Build Postgres replication applications in Rust

#15

Post by imor80 »

phamilton wrote: post #4

Postgres + Rust is one of the most exciting intersections of tech I've seen in a while. There's external tooling like his project, but postgres extensions in Rust are exciting. Full extensions via pgrx have been cool to see, but plrust + pg_tle is also starting to show up. If you aren't familiar with TLE (Trusted Language Extensions), it is a postgres extension from AWS that created some privileged interfaces for pro…

Agree, Rust and Postgres and a perfect match. It feels so much more productive to write Postgres tooling in Rust. E.g. we already have extensions like pg_graphql[0], pg_jsonschema[1] and wrappers[2] which use pgrx. We don't have plrust on the platform yet though.

Full disclosure: I'm a Rust Engieer at Supabase.

[0] https://github.com/supabase/pg_graphql [1] https://github.com/supabase/pg_jsonschema [2] https://github.com/supabase/wrappers

Re: Show HN: Pg_replicate – Build Postgres replication applications in Rust

#16

Post by imor80 »

hermanradtke wrote: post #11
eknkc wrote: post #7

I was trying out the stdout example. Could not get it to log anything. DuckDB example worked so I went digging into the source. Apparently the stdout sink is using tracing and I did not have a `RUST_LOG` env var set. Might be a good idea to have it documented or have the default level set to info for the stdout example. Maybe this is common Rust knowledge and I just don't know what I'm doing though.

I too consider this a footgun. Most applications will have something like tracing_subscriber::registry() .with( tracing_subscriber::EnvFilter::try_from_default_env() .unwrap_or_else(|_| "my_app=info".into()), ) .with(tracing_subscriber::fmt::layer()) .init(); in the `main.rs` which will default the tracing if RUST_LOG env var is not set.

Thanks for trying out, I'll update the code and/or the README to fix this.

Re: Show HN: Pg_replicate – Build Postgres replication applications in Rust

#17

Post by imor80 »

stlava wrote: post #9

Nice! I'm one of the authors of pg-bifrost which is in the same space. Have you thought about / have solved sharding consumption across multiple slots / multi consumers to increase throughput? This is on my radar but not something I've investigated yet. The issue we've ran into is some team at work decides to re-write an entire table and things get backed up until they stop updating rows.

pg-bifrost looks solid.

> Have you thought about / have solved sharding consumption across multiple slots / multi consumers to increase throughput?

Not yet, there has been not performance yet, as the project is still quite young.

Re: Show HN: Pg_replicate – Build Postgres replication applications in Rust

#18

Post by imor80 »

rubenfiszel wrote: post #6

This is super timely. Windmill ( https://windmill.dev ) used to only support webhooks to trigger code and flow jobs. We have just added email support building our own MX server, and wanted to add CDC change. We were gonna do it on Debezium but this will allow us to remove the need for a third-party service and just add this as a crate. Thank you supabase for open-sourcing this.

Please do share your experience using pg_replicate as feedback once you integrate with windmill.

Re: Show HN: Pg_replicate – Build Postgres replication applications in Rust

#19

Post by imor80 »

alexnewman wrote: post #10

Wow I think i build one of the first pg-rust-cdc replication module for postgresql and now it seems mainstream. It was a part of my https://github.com/posix4e/rpgffi project, which is garbage code I'm now embarrassed about. Rust programming has come a long way and we didn't have the best practices. Now it seems actual grownup engineering teams are living the dream. We truly live in the best of all universes.

These days, for writing extensions/FFI pgrx[0] is the best crate.

[0] https://github.com/pgcentralfoundation/pgrx

Post reply on HN