WAL-RUS: a Rust Rewrite of WAL-G for PostgreSQL Backups
clickhouse.com
WAL-RUS: a Rust Rewrite of WAL-G for PostgreSQL Backups
1–10 of 23 posts
Re: WAL-RUS: a Rust Rewrite of WAL-G for PostgreSQL Backups
#2Do you have any benchmarks with a mix of long open transactions and short ones? I've struggled a lot with WAL-E in the past there, and am curious if that changes here.
Re: WAL-RUS: a Rust Rewrite of WAL-G for PostgreSQL Backups
#3Do you have any benchmarks with a mix of long open transactions and short ones? I've struggled a lot with WAL-E in the past there, and am curious if that changes here.
no. but wal-g & wal-rus both have parallelism over wal-e. however are you more asking about handling build up of wal / vacuum prevention caused by long running transactions? those are up to postgres, archive command only keeps pushing wal so that when postgres is ready to get rid of wal it can. seems like your scenario wouldn't care much what the archiver is since wal should be shipped long before postgres is ready to get rid of wal
Re: WAL-RUS: a Rust Rewrite of WAL-G for PostgreSQL Backups
#4Do you have any benchmarks with a mix of long open transactions and short ones? I've struggled a lot with WAL-E in the past there, and am curious if that changes here.
no. but wal-g & wal-rus both have parallelism over wal-e. however are you more asking about handling build up of wal / vacuum prevention caused by long running transactions? those are up to postgres, archive command only keeps pushing wal so that when postgres is ready to get rid of wal it can. seems like your scenario wouldn't care much what the archiver is since wal should be shipped long before postgres is ready t…
Yeah, I'm probably misremembering some details there. Thanks
Re: WAL-RUS: a Rust Rewrite of WAL-G for PostgreSQL Backups
#5I must say I'm quite pleased to see how well Go version works. It does only use 1.5x the CPU and (predictably) much more RAM/VRAM, but not a crazy amount either (the expected increase is 2x).
Of course you can write a more optimal version in C / C++ / Zig / Rust, but at the same time Go is much easier to write and you don't pay for the convenience with an absurd performance loss like in Python or PHP
Re: WAL-RUS: a Rust Rewrite of WAL-G for PostgreSQL Backups
#6Quick one on the benchmark: was the 2.8GB peak virtual or resident? Go reserves a large virtual arena it mostly never faults in, so RSS tends to be a fraction of the virtual peak, and if Postgres headroom was getting squeezed off the virtual number you were sizing against memory the kernel never actually charges for.
Re: WAL-RUS: a Rust Rewrite of WAL-G for PostgreSQL Backups
#7PostgreSQL WAL-RUS, no relation to PostgreSQL WALRUS https://github.com/supabase/walrus
Re: WAL-RUS: a Rust Rewrite of WAL-G for PostgreSQL Backups
#8Quick one on the benchmark: was the 2.8GB peak virtual or resident? Go reserves a large virtual arena it mostly never faults in, so RSS tends to be a fraction of the virtual peak, and if Postgres headroom was getting squeezed off the virtual number you were sizing against memory the kernel never actually charges for.
Correct. We tune overcommit so postgres reliably returns out of memory. It becomes complicated to accurately tune overcommit for every AWS instance type. We configure GOMEMLIMIT/cgroups but those are about RSS. Outliers come together: instances running queries out of memory on our service tend to also be pushing other resource limits, causing wal-g & prometheus exporters to start having more erratic memory usage at the worst time
This helps on both ends of the cost spectrum. Large 64 core instances are where our heuristics fall off the most as variance increases, & tiny instances with 8GB of memory can use every 100MB of RSS we can get
Re: WAL-RUS: a Rust Rewrite of WAL-G for PostgreSQL Backups
#9I must say I'm quite pleased to see how well Go version works. It does only use 1.5x the CPU and (predictably) much more RAM/VRAM, but not a crazy amount either (the expected increase is 2x). Of course you can write a more optimal version in C / C++ / Zig / Rust, but at the same time Go is much easier to write and you don't pay for the convenience with an absurd performance loss like in Python or PHP
indeed, wal-g actually started as a port of wal-e which was Python: https://www.citusdata.com/blog/2017/08/18/introducing-wal-g-...
wal-g was a much larger improvement over wal-e. we're optimizing the margins here
Re: WAL-RUS: a Rust Rewrite of WAL-G for PostgreSQL Backups
#10I must say I'm quite pleased to see how well Go version works. It does only use 1.5x the CPU and (predictably) much more RAM/VRAM, but not a crazy amount either (the expected increase is 2x). Of course you can write a more optimal version in C / C++ / Zig / Rust, but at the same time Go is much easier to write and you don't pay for the convenience with an absurd performance loss like in Python or PHP
indeed, wal-g actually started as a port of wal-e which was Python: https://www.citusdata.com/blog/2017/08/18/introducing-wal-g-... wal-g was a much larger improvement over wal-e. we're optimizing the margins here
I'd like to use the one getting the most community support. So too soon to wait for Rust vs Go. Although on paper, Rust is better.