Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Does Postgres Scale?

dbos.dev

Post reply on HN
Print view View original

31–40 of 105 posts

Re: Does Postgres Scale?

#31

Post by tomwheeler »

cyberpunk wrote: post #24
lelandbatey wrote: post #17

Earlier quoted context omitted.

The reason that DBOS isn't as popular is because it's younger. DBOS launched in the form we know it in 2024. Temporal is much older; Temporal is technically a fork of Cadence and Cadence released originally in 2017, with Temporal forking and releasing back in 2020. When all three are trying to be "the same sort of thing" and that thing is new, it's hard to show up 7-8 years after the trailblazers and say "oh yeah, we…

Temporal is a dumpster fire, they've gotten so much VC funding (recently had D, 300M at a 5bn valuation) with ... nothing to build except ways to trap customers into their SAAS. I give them about a year or two before the wheels fall off, then it's off to Broadcom and friends. But I could be wrong as now they're not in the 'durable execution' space at all, it's 'durable execution for ai' according to their latest conf…

Temporal employee here. I'm very surprised by your comment.

It's true that we recently had a Series D and that VC firms recognize the value of what we do. The Temporal Server software is 100% open source (MIT license: https://github.com/temporalio/temporal/blob/main/LICENSE). It's totally free and you don't even need to fill out a registration form, just download precompiled binaries from GitHub or clone the repo and build it yourself. You can self-host it anywhere you like, no restrictions on scale or commercial usage. We offer SaaS (Temporal Cloud), which customers can choose as an alternative self-hosting, based on their needs. The migration path is bi-directional, so not a trap by any definition.

Regarding AI, Temporal is widely used in that space, but that does not negate the thousands of other companies that use Temporal for other things (e.g., order management systems, customer onboarding, loan origination, money movement, cloud infrastructure management, and so on). In fact, our growth in the AI market came about because companies who were already using Temporal for other use cases realized that it also solved the problems they encountered in their AI projects.

And to your last point, we've made dozens of enhancements to the product (here's a small sample: https://temporal.io/blog/categories/product-news). I'd encourage you to follow the news from next week's Replay conference (https://replay.temporal.io/) because we'll be announcing many more.

Re: Does Postgres Scale?

#32

Post by raddan »

literalAardvark wrote: post #13
_3u10 wrote: post #11

Earlier quoted context omitted.

So the point of distributed compute is to reduce the compute needed? I’ve generally found that distributed compute requires more compute than vertical scaling while getting clobbered by network bandwidth / latency. Theoretically with 2 to 10x compute required and in practice 100 to 500x

The point of distributed computing is to do computing that you can't do on a vertically scaled system or to increase availability. If you're doing it for other reasons it's usually a mistake.

The advice I’ve gotten is that you want to move computation to data that is already distributed. The cost of moving large amounts of data usually dwarfs compute costs (usually, not always), and so the performance win comes from distributing the computation and then (depending on the problem) centralizing aggregate results.

Re: Does Postgres Scale?

#33

Post by switchbak »

cyberpunk wrote: post #24
lelandbatey wrote: post #17

Earlier quoted context omitted.

The reason that DBOS isn't as popular is because it's younger. DBOS launched in the form we know it in 2024. Temporal is much older; Temporal is technically a fork of Cadence and Cadence released originally in 2017, with Temporal forking and releasing back in 2020. When all three are trying to be "the same sort of thing" and that thing is new, it's hard to show up 7-8 years after the trailblazers and say "oh yeah, we…

Temporal is a dumpster fire, they've gotten so much VC funding (recently had D, 300M at a 5bn valuation) with ... nothing to build except ways to trap customers into their SAAS. I give them about a year or two before the wheels fall off, then it's off to Broadcom and friends. But I could be wrong as now they're not in the 'durable execution' space at all, it's 'durable execution for ai' according to their latest conf…

[dead]

Re: Does Postgres Scale?

#34

Post by [deleted] »

jghn wrote: post #6

It scales beyond the needs that most people have in most situations. The constant problem is that "big scale" always means "larger than I've seen", so on any project larger than a person has encountered, they assume they need to pull out the big guns. Also, people worry about things like what happens if they really *do* scale 10 years from now. Neither is a practical concern for nearly anyone who will ever face this…

[deleted]

Re: Does Postgres Scale?

#35

Post by lelandbatey »

cyberpunk wrote: post #24
lelandbatey wrote: post #17

Earlier quoted context omitted.

The reason that DBOS isn't as popular is because it's younger. DBOS launched in the form we know it in 2024. Temporal is much older; Temporal is technically a fork of Cadence and Cadence released originally in 2017, with Temporal forking and releasing back in 2020. When all three are trying to be "the same sort of thing" and that thing is new, it's hard to show up 7-8 years after the trailblazers and say "oh yeah, we…

Temporal is a dumpster fire, they've gotten so much VC funding (recently had D, 300M at a 5bn valuation) with ... nothing to build except ways to trap customers into their SAAS. I give them about a year or two before the wheels fall off, then it's off to Broadcom and friends. But I could be wrong as now they're not in the 'durable execution' space at all, it's 'durable execution for ai' according to their latest conf…

Maybe. But as someone who happily self hosting pretty big Temporal workloads for my day job (I inherited it from early adopters circa 2022), it definitely does not feel like a dumpster fire. It chugs along unglamorously and I enjoy working on it.

Re: Does Postgres Scale?

#36

Post by ahachete »

> we find a Postgres server can handle up to 144K of these writes per second. That’s a lot, equivalent to 12 billion writes per day.

Based on the shown graph, this is misleading at best, essentially false. After 120K writes/s p50 spikes from 10ms to 1s (1 second for a write!!!!). That's two orders of magnitude latency spike, and an unacceptable one for an OLTP workload. It clearly shows the server is completely saturated, which is clearly a non operational regime. Quoting 144K is equivalent to quoting the throughput of a highway at the moment traffic comes to a standstill.

Based on this graph the highest number I'd quote is 120K. And probably you want to keep operating the server within a safe margin below peak, but since this is a benchmark, let's call 120K the peak. Because actually p50 is not even the clear-cut. It should be a higher percentile (say p95) at which latency is within reasonable bounds. But for the shake of not over complicating, it could be taken as a reference.

> We found that the bottleneck was in flushing the Postgres write-ahead log (WAL) to disk

Therefore, you are not measuring Postgres peak performance, but rather Postgres performance under the IO constraints of this particular system. Certainly, 120K IOPS is the maximum that this particular instance can have. But it doesn't show if Postgres could do better under a more performant IO disk. Actually, a good test would have been to try the next instance (db.m7i.48xlarge) with 240K IOPS and see if performance doubles (within the same envelop of p50 latency) or not. And afterwards to test on an instance with local NVMe (you won't find this in RDS).

> From [1]: > Postgres insert throughput > uv run python benchmarks/postgres_insert.py --rps 1000 --duration 300

300 seconds test duration?? This is not operational. You are not accounting for checkpoints, background writer, and especially autovacuum. Given that workflow pattern includes UPDATEs, you must validate bloat generation (or, equivalently, bloat removal) by a) observing much longer periods of time (e.g. 1h) and b) making sure the autovacuum configuration (and/or individual table vacuum configuration if required) makes bloat contained in a stable way. Otherwise, shown performance numbers will degrade over time, making them not realistic.

Only after proper autovacuum tuning and under the effects of bgwriter (also tuned!) and checkpoints (all required, especially for write intensive workloads), over much larger periods of time, the benchmark could be considered meaningful.

> We next measure the scalability of Postgres-backed queues.

I'd recommend benchmarking the recently announced PgQue project, that operates bloat-free (one of the largest operational hurdles in queue-like Postgres workloads). See [2] for a previous discussion in HN.

[1]: https://github.com/dbos-inc/dbos-postgres-benchmark [2]: https://news.ycombinator.com/item?id=47817349

Re: Does Postgres Scale?

#37

Post by petergeoghegan »

keithnz wrote: post #27
subhobroto wrote: post #16

Earlier quoted context omitted.

These are good suggestions but I'm apprehensive they might come back and say they have 64 GB (or less) of RAM or they are using PostgreSQL RDS on AWS or something. I asked them for specifics.

I don't think it really matters in terms of their question though, given MySql on the same specs doesn't have the problem and postgres does. Quite clearly it has something to do with indexes and what is the wall postgres is running into that causes the drop off on quite low amounts of rows. If the answer is just get more RAM, it kind of implies postgres is not really that scalable. Especially if the drop off is propo…

Why are you using hash indexes? They're much less widely used than standard B-Tree indexes. The bucket split code likely isn't very scalable [1].

I suggest testing the same workload with your existing hash indexes replaced with equivalent B-Trees.

[1] https://github.com/postgres/postgres/blob/master/src/backend...

Re: Does Postgres Scale?

#39

Post by farsa »

subhobroto wrote: post #2

DBOS is amazing when it comes to Durable Workflows. There are others in the space - the most popular one being Temporal but I argue, Temporal is also the most complicated one. I often say Temporal is like Kubernetes while DBOS is like `docker compose`. (and for those taking me literally, you can use DBOS in Kubernetes!) I don't realize why DBOS is not nearly as popular as Temporal but it has made a world of differenc…

DBOS looks simple (good), but from the docs below, executor elasticity appears to be locked behind license purchase. So it truly is like docker compose, good and bad parts?

https://docs.dbos.dev/production/workflow-recovery#recovery-...

>When self-hosting in a distributed setting without Conductor, it is important to manage workflow recovery so that when an executor crashes, restarts, or is shut down, its workflows are recovered. You should assign each executor running a DBOS application an executor ID through DBOS configuration. Each workflow is tagged with the ID of the executor that started it. When an application with an executor ID restarts, it only recovers pending workflows assigned to that executor ID.

https://docs.dbos.dev/production/hosting-conductor

> Self-hosted Conductor is released under a proprietary license. Self-hosting Conductor for commercial or production use requires a paid license key.

Re: Does Postgres Scale?

#40

Post by mannyv »

Postgres can scale, just like any database can scale.

The issue is there's a lot of lore and esoterica required to get it to scale.

Every time there's an issue with Postgres there's someone that's all "just do xyzzy, it's super obvious that this undocumented setting fixes it."

Just look at the slowdown/big pages problem.

If it works for you, use it.

Post reply on HN