Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

WAL-RUS: a Rust Rewrite of WAL-G for PostgreSQL Backups

clickhouse.com

Post reply on HN
Print view View original

21–23 of 23 posts

Re: WAL-RUS: a Rust Rewrite of WAL-G for PostgreSQL Backups

#21

Post by pjmlp »

nasretdinov wrote: post #13
__s wrote: post #8

Earlier quoted context omitted.

Correct. We tune overcommit so postgres reliably returns out of memory. It becomes complicated to accurately tune overcommit for every AWS instance type. We configure GOMEMLIMIT/cgroups but those are about RSS. Outliers come together: instances running queries out of memory on our service tend to also be pushing other resource limits, causing wal-g & prometheus exporters to start having more erratic memory usage at t…

You probably could limit the bloating of Go programs by setting GOMAXPROCS to something like 1 or 2 on smaller machines, but then again you wouldn't get the best performance. So IMO good call here to rewrite it in a language without GC.

Or make better use of the language capabilities to use value types.

Re: WAL-RUS: a Rust Rewrite of WAL-G for PostgreSQL Backups

#22

Post by nasretdinov »

pjmlp wrote: post #21
nasretdinov wrote: post #13

Earlier quoted context omitted.

You probably could limit the bloating of Go programs by setting GOMAXPROCS to something like 1 or 2 on smaller machines, but then again you wouldn't get the best performance. So IMO good call here to rewrite it in a language without GC.

Or make better use of the language capabilities to use value types.

Value types do indeed reduce the GC pressure, however large Virtual Memory usage (in my understanding) is mostly due to large thread count by default (one per core), and some unavoidable GC overhead in general.
Post reply on HN