11–20 of 99 posts
Choose database queue technology https://news.ycombinator.com/item?id=37636841
I recall that in the first release that supported LISTEN/NOTIFY there was a performance issue around it (poor locking IIRC), which today the "bad post" mentioned in here corrects in a errata just after their first paragraph. Since the correction apparently dates from May 8th, I think that a post from July 24th might want to acknowledge that the popular post asserting this feature doesn't (didn't?) scale was not made…
> As an aside, there’s been some online discussion of a Postgres patch (https://github.com/postgres/postgres/commit/282b1cde9dedf456...) related to this issue. This patch (to be released in Postgres 19) does not remove the global lock or fix the bottleneck we observed. Instead, it optimizes the narrower case where there are many notification channels and each listener is waiting only on a specific channel.
"Scale" isn't a binary, it's a continuum. "Scales to 60K/s" can be 5 orders of magnitude more than one system needs and 5 orders of magnitude too small for another. Personally I'd knock the general "premature optimization" off the list of "most common developer errors" and put in its place "using techs with the wrong scaling factors". If you use something too small and you exceed its needs, the failure is obvious, bu…
> 5 orders of magnitude too small for another. Nitpick on an otherwise good post, but I don’t think there are very many 6billion RPS systems out there, and those that do exist are almost certainly using bespoke, purpose-built tools
"Scale" isn't a binary, it's a continuum. "Scales to 60K/s" can be 5 orders of magnitude more than one system needs and 5 orders of magnitude too small for another. Personally I'd knock the general "premature optimization" off the list of "most common developer errors" and put in its place "using techs with the wrong scaling factors". If you use something too small and you exceed its needs, the failure is obvious, bu…
> 5 orders of magnitude too small for another. Nitpick on an otherwise good post, but I don’t think there are very many 6billion RPS systems out there, and those that do exist are almost certainly using bespoke, purpose-built tools
The word "scale" does a lot of load-bearing, maybe it's just not a useful or productive word in practice.
Is the optimization only possible using dbos? is not clear to me if this mean a way to tune normal PG
Is the optimization only possible using dbos? is not clear to me if this mean a way to tune normal PG
The core optimization is to buffer notifications in-memory and send them in a batch instead of sending them as part of every transaction. So that's a general-purpose optimization for Postgres apps using LISTEN/NOTIFY.
1. What I find interesting is that the experiment seems to be using a DB server with 96 cores, 384 GB RAM (https://github.com/dbos-inc/dbos-postgres-benchmark/blob/mai...). This is very critical part of any such experiment, it should have been called out. The database is vertically scalable and that too has its limits
2. Who is making connection, and from where has its own impact on performance and overall latency
3. 60k may seem big number, however in real world the things which bring the systems down are the bursts of traffic, not the regular traffic.
Personally I would never start with such a big server unless I am a big business. Its > 100K cost for one production DB cluster if I include read replicas and cross region redundancy