Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Turning PostgreSQL into a queue serving 10k jobs per second (2013)

gist.github.com

Post reply on HN
Print view View original

121–130 of 146 posts

Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)

#121

Post by dimino »

aldoushuxley001 wrote: post #118
dimino wrote: post #111

Earlier quoted context omitted.

Sorry, but what you call a "toy", I call a highly reliable technology that's used in production widely across the majority of Fortune 500 companies. No one here is suggesting anything remotely like what you're strawmanning, but it's clear the straw man is the only argument you can take down. The reason you are completely wrong and a toxic member of your team is that your logic can be used to justify all kinds of terr…

You gotta relax your tone eh. Not sure why you’re so personally/emotionally invested in this topic but your aggressive tone makes your myriad comments grating.

This isn't an aggressive tone, except to people who have their own emotional investment into the topic.

I put out what is given.

Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)

#122

Post by maxpert »

May be I am being ignorant/arrogant, or have seen similar toy systems crash and burn in prod environments. And if you go through https://www.2ndquadrant.com/en/blog/what-is-select-skip-lock... it clearly states:

> A queue implemented in the RDBMS will never match the performance of a fast dedicated queueing system, even one that makes the same atomicity and durability guarantees as PostgreSQL. Using SKIP LOCKED is better than existing in-database approaches, but you’ll still go faster using a dedicated and highly optimised external queueing engine.

People only realize these warnings when they have built a toy system that is rolled out to a production/product that takes off. By that time you will realize "ohhh I don't need all the ACID guarantees for every job in my system", and I want to run my workers as lambdas/elastic scaling workers (you need more connections). That is where companies will be spending efforts from their best engineers to move away from Postgres as queue. Which comes down to question; why do it in first place? What makes it cheaper (other than local dev) to deploy a system that is bound to fail in future? Don't get me wrong I love Postgres, I just don't believe that it's the right tool for doing this job.

Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)

#123

Post by AlfeG »

dimino wrote: post #113
DLA wrote: post #112

Earlier quoted context omitted.

Every different technology added to an infrastructure adds potential failure points, raises the DevOps burden, and adds surface area subject to bugs and security attacks. If a PG-based solution meets the needs then this reduces complexity in the large. And this has nothing to do with learning or not learning. Sorry but your argument just does not hold up.

It only adds the burden to DevOps who can't figure out how to monitor/maintain a system, which is not rocket science, and if you're a DevOps person who doesn't know how to handle such a mainstream technology as, say, ZeroMQ or RabbitMQ or even Redis, then maybe you shouldn't be in the position you're in. The only argument that doesn't hold up is "use one tool for everything". This ideology is for lazy people who want…

But what about "box" products? We can't force clients have dedicated DevOps just to support our product. Less is better in this case. To have same code in Cloud and offline installs is even better.

Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)

#124

Post by ukd1 »

wjd2030 wrote: post #105
bufferoverflow wrote: post #35

Now compare to OpenAMQ, ZeroMQ, RabbitMQ, NSQ, Kafka. I have seen benchmarks reaching millions of messages per second.

Exactly. It is so easy to achieve using a dedicated queuing system. You could just as easily achieve much higher throughput by passing a jobId as a message to Rabbit and have the worker pop the required data out of the db. Thats assuming you dont want to just pass in a serialized object. All this talk of it not being a durable system is just wrong. Rabbit has strong durability guarantees across a cluster with queue m…

Rabbit and most others do; but only once it reaches the queue itself. The problem that using a fb to queue stuff gets around is guaranteeing it earlier; your changes to database and the queued messages are either all committed, or not - a single transaction. This is hard to impossible with queues outside a dB. However obviously it’s a trade off; it’s slower, it’s more work and connections to your dB. But, you don’t have to code around messages either never sent it from a later rolled back transaction.

Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)

#125

Post by ukd1 »

dimino wrote: post #120
DLA wrote: post #114

Earlier quoted context omitted.

Nobody is saying use one tool for everything. In fact the argument is why not use PG until you grow out of it (most people won’t). Agree on devops. Of course. There’s still the security surface area argument and the overall complexity argument—complexity of the system of systems is NOT reduced by adding technologies. By way of analogy, using military aircraft, take the F35 vs. the A-10. The F35 is insanely more compl…

People are literally saying to default to using PG for "everything". That's the attitude I'm arguing against. What's frustrating about this conversation is that I'm literally, right now, supporting two different queueing systems based on PG and Redis, so I get on a very real level, the tradeoffs. I know in great detail the problems that come up, but HN is not conducive to talking at that level of detail. At this poin…

I mean i think it’s mostly the blunt, binary right/wrong nature of your posts getting you this reaction. For most folks they’ve moved past that and almost all of us have non perfect world systems where we’ve made trade offs. It’s just not my (and I guess a lit of folks) reality to be so black/white on this.

Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)

#126

Post by ukd1 »

ukd1 wrote: post #98
dimino wrote: post #76

Earlier quoted context omitted.

It is very much a bad technology sourcing strategy, and you will kill your business if you attempt it. Period. It comes from a place of ignorance, and you're promoting ignorance. Learn why technologies exist and make an informed decision about tradeoffs, instead of being lazy and incompetent by blindly choosing technology based on what's a very locally maxima for you personally. It's selfish and damaging. Edit since…

Wow, you're super jaded on this, do you sell queuing software for a living? Also wrong: how will this kill a business, at all?

Downvote away; but really “kill” your business!!?

Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)

#127

Post by dimino »

ukd1 wrote: post #125
dimino wrote: post #120

Earlier quoted context omitted.

People are literally saying to default to using PG for "everything". That's the attitude I'm arguing against. What's frustrating about this conversation is that I'm literally, right now, supporting two different queueing systems based on PG and Redis, so I get on a very real level, the tradeoffs. I know in great detail the problems that come up, but HN is not conducive to talking at that level of detail. At this poin…

I mean i think it’s mostly the blunt, binary right/wrong nature of your posts getting you this reaction. For most folks they’ve moved past that and almost all of us have non perfect world systems where we’ve made trade offs. It’s just not my (and I guess a lit of folks) reality to be so black/white on this.

Of course there are trade offs, of course we live in imperfect situations where we have to deal with shit that isn't ideal, but sitting around pretending like everything is alright is just lying to ourselves.

You're stuck in PG because your DevOps team is a bunch of incompetent dolts, so yeah, make the best out of the bad situation and go ahead and use something like Que.

Just don't develop Stockholm Syndrome while doing it.

People in this thread are protecting their own egos by expressing how great PG is as a message broker, but it's harmful to the industry to let that kind of attitude perpetuate.

Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)

#128

Post by dang »

dimino wrote: post #111
outworlder wrote: post #107

Earlier quoted context omitted.

> I cannot stress this enough; fear of learning is anathema to software, and trying to hide it behind a veneer of caution is not only disingenuous but potentially malicious as well, maximizing exclusively for the benefit of the individual against the interests of the group. This is wrong. You are ultimately getting paid to solve the company's problems. If those problems actually require deployment of a new technology…

Sorry, but what you call a "toy", I call a highly reliable technology that's used in production widely across the majority of Fortune 500 companies. No one here is suggesting anything remotely like what you're strawmanning, but it's clear the straw man is the only argument you can take down. The reason you are completely wrong and a toxic member of your team is that your logic can be used to justify all kinds of terr…

Would you please stop posting flamewar comments and crossing into personal attack? This breaks the site guidelines and is not ok here.

https://news.ycombinator.com/newsguidelines.html

Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)

#129

Post by dimino »

dang wrote: post #128
dimino wrote: post #111

Earlier quoted context omitted.

Sorry, but what you call a "toy", I call a highly reliable technology that's used in production widely across the majority of Fortune 500 companies. No one here is suggesting anything remotely like what you're strawmanning, but it's clear the straw man is the only argument you can take down. The reason you are completely wrong and a toxic member of your team is that your logic can be used to justify all kinds of terr…

Would you please stop posting flamewar comments and crossing into personal attack? This breaks the site guidelines and is not ok here. https://news.ycombinator.com/newsguidelines.html

I'm not doing anything like that, but please, pretend that I am, seems like your go-to move here anyway.

Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)

#130

Post by StavrosK »

aldoushuxley001 wrote: post #37
bgentry wrote: post #15

The author of this post, Chris Hanks, created the Que queueing library for Ruby: https://github.com/que-rb/que It’s changed significantly since this post as the 1.x betas use a very different structure which should actually be more efficient, use fewer Postgres connections, cause less lock contention, and cause less table bloat. Not sure if the benchmarks have been run recently or not but I’m definitely curious how t…

Wish I could use that in a Django app. Doesn't seem to be a viable python queueing library that allows using postgresql

Yeah, it would be nice if Dramatiq supported it. django-postgres-queue sounds promising, though.
Post reply on HN