Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Turning PostgreSQL into a queue serving 10k jobs per second (2013)

gist.github.com

Post reply on HN
Print view View original

141–146 of 146 posts

Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)

#141

Post by ubu7737 »

bufferoverflow wrote: post #52
madhadron wrote: post #48

Earlier quoted context omitted.

The article is discussing job queues, not messaging. The MQ systems plus Kafka that you are referring to are message transport systems.

What's the actual difference between a job queue and a message queue filled with job IDs?

When I used PG for my job queue, I was already using Kafka instead, to handle immediate job entries. The reason I needed PG was for jobs scheduled at a future time.

I had no difficulty with long-running jobs because servicing jobs out of PG was simply a matter of pushing them onto the Kafka queue for immediate uptake there.

Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)

#142

Post by dimino »

zentiggr wrote: post #139
dimino wrote: post #129

Earlier quoted context omitted.

I'm not doing anything like that, but please, pretend that I am, seems like your go-to move here anyway.

> You should not be employed in this industry if what you wrote here is how you think, period. That seems pretty personal. Not that I'm hopping on dang's bandwagon, but you can't post that sort of attack and say you're not doing anything like that. I've wound up seeing a lot of your recent posts and I can understand where a lot of your responders are coming from... you are very strongly opinionated and have no proble…

It's not a personal attack, it's my opinion. I've worked with people who think that way, and I'd prefer if they didn't exist in the industry. How would you prefer I shape that comment so it's not crossing some magical line?

I am aware, generally, how things are. My complaining here is not confusion, it's frustration. I know I can change my behavior to elicit a better response, but I'm frustrated that it's relevant. People should have thicker skin, and when people dish it out (as what was going on here), I should be allowed to give it back. Dang's not commenting on anyone else's posts here, though there were many other rule violators.

It's an unfair application of the rules.

Edit: This post is 0 minutes old, how does it have a downvote already? This is the shit I'm talking about...

Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)

#143

Post by shrimpx »

pas wrote: post #136
shrimpx wrote: post #134

Earlier quoted context omitted.

A global job timeout might be unreasonable with high variance in workload. Eg some jobs taking 0.5 seconds and others 30 seconds. You might set a global timeout of say 60s but it sucks to wait 59.5s to reap that short job whose worker crashed. A better system is to make workers update a timestamp on an interval and you reap any jobs that haven't been updated in N seconds.

It's a trade off between updates per sec and latency. Maybe simply using a timeout per job type is a better way. (That of course trades off simplicity.)

I agree. Frequency of updates also becomes more of an issue as you add workers. Say you have 1000 workers each updating every 2 seconds. That's ~500 timestamp update statements per second which is not trivial in terms of added load on the DB.

Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)

#145

Post by Noumenon72 »

qaq wrote: post #104
voldacar wrote: post #81

I'm not very knowledgable about db internals so sorry if this comes off as ignorant, but in an era where cpus execute billions of instructions per second per core , is 10000 jobs per second supposed to be impressive? Is this kind of problem bottlenecked by memory?

by IOPS you fsync transaction to Write Ahead Log (WAL) on commit.

Is this supposed to explain why 10000 per second is impressive? I don't follow.

Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)

#146

Post by qaq »

Noumenon72 wrote: post #145
qaq wrote: post #104

Earlier quoted context omitted.

by IOPS you fsync transaction to Write Ahead Log (WAL) on commit.

Is this supposed to explain why 10000 per second is impressive? I don't follow.

This is supposed to explain that it's orthogonal to CPU performance.
Post reply on HN