Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Turning PostgreSQL into a queue serving 10k jobs per second (2013)

gist.github.com

Post reply on HN
Print view View original

111–120 of 146 posts

Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)

#111

Post by dimino »

outworlder wrote: post #107
dimino wrote: post #45

Earlier quoted context omitted.

I find this argument to be pretty weak, and seems to be a stand-in for, "I don't want to use a new technology because it'd require me to learn something, so I'm going to shoehorn something I feel more comfortable with, even though it's not the best tool for the job." Edit: I'm rate limited so to elaborate a bit. Queueing technologies are not "new interesting bit of technology", they're tailored solutions to solve a s…

> I cannot stress this enough; fear of learning is anathema to software, and trying to hide it behind a veneer of caution is not only disingenuous but potentially malicious as well, maximizing exclusively for the benefit of the individual against the interests of the group. This is wrong. You are ultimately getting paid to solve the company's problems. If those problems actually require deployment of a new technology…

Sorry, but what you call a "toy", I call a highly reliable technology that's used in production widely across the majority of Fortune 500 companies. No one here is suggesting anything remotely like what you're strawmanning, but it's clear the straw man is the only argument you can take down.

The reason you are completely wrong and a toxic member of your team is that your logic can be used to justify all kinds of terrible, short term wins that sacrifice any kind of long term/sustainable design. You'd rather make a bad technology play so your boss gets off your back, than think through your decision and make a long term investment into a sustainable, maintainable, improvable, and agile technology that's actually suited to solve the real problem you have.

OF COURSE you're there to accomplish the tasks your business needs to succeed. That's a given. What's not a given is that you need to slap together the shittiest piece of software that'll do that job right then and there.

Your attitude is what keeps companies from solving problems, and forces them to pay dollar after dollar to fix the same issue over and over again. You're selfish, you're greedy, and you cost your company insane amounts of money.

You should not be employed in this industry if what you wrote here is how you think, period.

Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)

#112

Post by DLA »

dimino wrote: post #96
tomtheelder wrote: post #84

Earlier quoted context omitted.

I believe you are missing the point a bit here. Postgres is not the optimal tool for queueing. No one is arguing that position. However, it's also true that for a very large number of use cases, it is entirely sufficient. If you already have tooling and knowledge for managing Postgres, and Postgres is sufficient for your queueing needs, then why would you incur the cost of introducing a new service? Most situations d…

I'm not missing the point, I'm discounting the point, because the point that adding technology is additional complexity, and complexity is bad is a nonsense argument that actually stems from a fear of new things. This has nothing to do with getting shit done, and everything to do with people who don't want to learn new things because they're afraid of not being smart enough to make the new thing work.

Every different technology added to an infrastructure adds potential failure points, raises the DevOps burden, and adds surface area subject to bugs and security attacks.

If a PG-based solution meets the needs then this reduces complexity in the large. And this has nothing to do with learning or not learning.

Sorry but your argument just does not hold up.

Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)

#113

Post by dimino »

DLA wrote: post #112
dimino wrote: post #96

Earlier quoted context omitted.

I'm not missing the point, I'm discounting the point, because the point that adding technology is additional complexity, and complexity is bad is a nonsense argument that actually stems from a fear of new things. This has nothing to do with getting shit done, and everything to do with people who don't want to learn new things because they're afraid of not being smart enough to make the new thing work.

Every different technology added to an infrastructure adds potential failure points, raises the DevOps burden, and adds surface area subject to bugs and security attacks. If a PG-based solution meets the needs then this reduces complexity in the large. And this has nothing to do with learning or not learning. Sorry but your argument just does not hold up.

It only adds the burden to DevOps who can't figure out how to monitor/maintain a system, which is not rocket science, and if you're a DevOps person who doesn't know how to handle such a mainstream technology as, say, ZeroMQ or RabbitMQ or even Redis, then maybe you shouldn't be in the position you're in.

The only argument that doesn't hold up is "use one tool for everything". This ideology is for lazy people who want to anchor themselves in a period of time, and refuse to learn new things.

Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)

#114

Post by DLA »

dimino wrote: post #113
DLA wrote: post #112

Earlier quoted context omitted.

Every different technology added to an infrastructure adds potential failure points, raises the DevOps burden, and adds surface area subject to bugs and security attacks. If a PG-based solution meets the needs then this reduces complexity in the large. And this has nothing to do with learning or not learning. Sorry but your argument just does not hold up.

It only adds the burden to DevOps who can't figure out how to monitor/maintain a system, which is not rocket science, and if you're a DevOps person who doesn't know how to handle such a mainstream technology as, say, ZeroMQ or RabbitMQ or even Redis, then maybe you shouldn't be in the position you're in. The only argument that doesn't hold up is "use one tool for everything". This ideology is for lazy people who want…

Nobody is saying use one tool for everything. In fact the argument is why not use PG until you grow out of it (most people won’t).

Agree on devops. Of course.

There’s still the security surface area argument and the overall complexity argument—complexity of the system of systems is NOT reduced by adding technologies.

By way of analogy, using military aircraft, take the F35 vs. the A-10. The F35 is insanely more complex than the nearly indestructible A10 in large part due to the massive number of subsystems, software, sensors, etc. Sure the F35 way more capable (this is NOT a head to head argument about the military capabilities or said aircraft) than the A10, but it also has orders of magnitude more failure modes. More parts. More complexity. More opportunities for failure. This is engineering fact.

Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)

#115

Post by olefoo »

bloody-crow wrote: post #14

One of the interesting unforeseen downsides of RDBMS-based queues is explored here: https://brandur.org/postgres-queues TDLR the lock time grows exponentially depending on the number of dead tuples in the table, which naturally grows as you use long running transactions.

If you are experiencing this problem look at your autovacuum settings; postgres ( versions > 9.0 ) can take per table settings for autovacuum.

Percona has a good article on this https://www.percona.com/blog/2018/08/10/tuning-autovacuum-in...

Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)

#116

Post by bgentry »

grumpy8 wrote: post #110
bgentry wrote: post #27

Earlier quoted context omitted.

I elaborated a bit on this elsewhere ( https://news.ycombinator.com/item?id=21537414 ) but Que’s design has changed significantly and no longer holds open a connection or transaction for the duration of working a job. It holds one database connection per worker process. Each worker process handles all job locking and assignment to the individual worker threads in that process, each of which only use connections that…

> I believe this model is almost always the right starting point for a web application, barring some unusual job requirements or massive initial scale. Would you run this job queue on the same postgres database as the rest of the application or rather use a different one specific for workers?

One of the main benefits of this model is the ability to transactionally enqueue any jobs that are being triggered at the same time as other changes your app is making. Also the ability to transactionally apply changes to your database at the same time as marking the job completed.

You only get this benefit if you’re doing everything in a single Postgres database. By the time you outgrow that setup, you may as well move to a more dedicated job queue or something built on Redis, because you’ve already lost the benefits of having a single system.

I suppose you could maintain the queue in a separate Postgres database, but without sharing ACID with the app I’m not sure What you’d gain vs another system.

Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)

#117

Post by Thaxll »

tomtheelder wrote: post #84
dimino wrote: post #75

Earlier quoted context omitted.

Sorry, but this isn't one of those cases. A RDBMS is objectively inferior for queueing compared to dedicated message broker technologies. This is blatantly obvious if you've ever used both for queueing. You're applying generic heuristics against a problem where folks have specific domain knowledge that contradicts those heuristics. I cannot stress this enough; you are flat wrong if you think Postgres is appropriate t…

I believe you are missing the point a bit here. Postgres is not the optimal tool for queueing. No one is arguing that position. However, it's also true that for a very large number of use cases, it is entirely sufficient. If you already have tooling and knowledge for managing Postgres, and Postgres is sufficient for your queueing needs, then why would you incur the cost of introducing a new service? Most situations d…

Except that PG and other RDMS don't offer "queue" commands in driver because the queue is built on top of other basic commands, when you use a system designed for that you can leverage the API offered by the driver directly.

So if my language doesn't have a library like Que ( no thanks using Ruby ) I can't do anything with PG whereas using Redis or any other solution you can use "subscribe" / "publish" ect ...

Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)

#118

Post by aldoushuxley001 »

dimino wrote: post #111
outworlder wrote: post #107

Earlier quoted context omitted.

> I cannot stress this enough; fear of learning is anathema to software, and trying to hide it behind a veneer of caution is not only disingenuous but potentially malicious as well, maximizing exclusively for the benefit of the individual against the interests of the group. This is wrong. You are ultimately getting paid to solve the company's problems. If those problems actually require deployment of a new technology…

Sorry, but what you call a "toy", I call a highly reliable technology that's used in production widely across the majority of Fortune 500 companies. No one here is suggesting anything remotely like what you're strawmanning, but it's clear the straw man is the only argument you can take down. The reason you are completely wrong and a toxic member of your team is that your logic can be used to justify all kinds of terr…

You gotta relax your tone eh. Not sure why you’re so personally/emotionally invested in this topic but your aggressive tone makes your myriad comments grating.

Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)

#120

Post by dimino »

DLA wrote: post #114
dimino wrote: post #113

Earlier quoted context omitted.

It only adds the burden to DevOps who can't figure out how to monitor/maintain a system, which is not rocket science, and if you're a DevOps person who doesn't know how to handle such a mainstream technology as, say, ZeroMQ or RabbitMQ or even Redis, then maybe you shouldn't be in the position you're in. The only argument that doesn't hold up is "use one tool for everything". This ideology is for lazy people who want…

Nobody is saying use one tool for everything. In fact the argument is why not use PG until you grow out of it (most people won’t). Agree on devops. Of course. There’s still the security surface area argument and the overall complexity argument—complexity of the system of systems is NOT reduced by adding technologies. By way of analogy, using military aircraft, take the F35 vs. the A-10. The F35 is insanely more compl…

People are literally saying to default to using PG for "everything". That's the attitude I'm arguing against.

What's frustrating about this conversation is that I'm literally, right now, supporting two different queueing systems based on PG and Redis, so I get on a very real level, the tradeoffs. I know in great detail the problems that come up, but HN is not conducive to talking at that level of detail. At this point every comment I make is flagged and downvoted, so why would I pay time into a system that has clearly decided my opinion isn't relevant?

Post reply on HN