Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Choose Postgres queue technology

adriano.fyi

Post reply on HN
Print view View original

201–210 of 369 posts

Re: Choose Postgres queue technology

#201

Post by mrkeen »

liminal wrote: post #99

My main issue with pretty much all queue approaches is that they don't work across platforms. They are built for one technology stack, be it Python/NodeJS/etc. This is fine if you've only got one stack, but in a microservices world it doesn't work where jobs can span multiple systems. You might be able to find some abandoned library that supports that queue tech on the other platforms you need, but now you've basical…

> My main issue with pretty much all queue approaches is that they don't work across platforms.

What technology stack are you working in which doesn't support postgres?

> but in a microservices world it doesn't work where jobs can span multiple systems

The point of the queue system is to be able to span said microservices. You can have an OCaml service picking up from one queue, processing, then writing into another queue. That queue could then be processed by a TypeScript service.

Re: Choose Postgres queue technology

#202

Post by pjmlp »

latchkey wrote: post #169
donatj wrote: post #155

Earlier quoted context omitted.

They ask for work after they finish the previous job (or jobs, they can ask for more than one). Each worker is a single process built just for one task. If there's no work for them there's a small timeout and they ask for more. Simple loop. It's all part of a library we built for building workers. For better or worse, it's all done over http. You are right, though, it is one XFS volume per queue instance. We just run…

I like how GCP cloud tasks reverses the model. Instead of workers pinging the server asking for work, have the queue ping the worker and the worker is effectively a http endpoint. So you send a message to the server, it queues it and then pings a worker with the message. https://cloud.google.com/tasks/docs/dual-overview

Just like using JMS, MSMQ or similar queues, I fail to see what is so great about it.

Re: Choose Postgres queue technology

#204

Post by winrid »

donatj wrote: post #145

We have our own queue, because it was easy, fun and has been exceedingly reliable above all else. Far moreso than other things we had tried. Cough Gearman cough SQS cough One endpoint accepts work to a named queue, writes it to a file in an XFS directory. Another locks a mutex, moves the file to an in progress directory and unlocks the mutex before passing the content to the reader. A third and final endpoint deletes…

What if a task fails/crashes?

Re: Choose Postgres queue technology

#205

Post by RichieAHB »

deredede wrote: post #191
refibrillator wrote: post #87

Earlier quoted context omitted.

PSA: This is a read-modify-write pattern, thus it is not safe under concurrency unless a transaction isolation level of SERIALIZABLE is specified, or some locking mechanism is used (select for update etc).

The part about checking the number of affected rows hints at using `UPDATE ... WHERE ...` which should act as an atomic CAS regardless of isolation level. Edit: To clarify, I mean `SELECT id WHERE used = 0` followed by `UPDATE ... SET used = 1 WHERE id = ... AND used = 0`

This works fine as long as you’re happy to do the same task multiple times. I.e. the task is idempotent and cheap.

Re: Choose Postgres queue technology

#206

Post by vore »

nsonha wrote: post #119
vore wrote: post #103

Earlier quoted context omitted.

You have no ordering guarantees, so how can order be important? If 4 work items are scheduled on 4 independent workers, you have no guarantee which will start first or finish first.

The order matters in the sense that the 5th jobs should not be atempted before those 4.

Then I think what you actually care about is scheduling fairness, and a strict ordering of execution of job 5 after job 4 is unimportant.

Re: Choose Postgres queue technology

#207

Post by fasteo »

Over the years I have used beanstalkd for all things queue. Working flawlessly every single time.

This is an output for our oldest instance (legacy system running Ubuntu 12)

  Trying 127.0.0.1...
  Connected to localhost.
  Escape character is '^]'.
  stats
  OK 952
  ---
  current-jobs-urgent: 0
  current-jobs-ready: 765
  current-jobs-reserved: 2
  current-jobs-delayed: 946
  current-jobs-buried: 0
  cmd-put: 1188640739
  cmd-peek: 2718986
  cmd-peek-ready: 5052
  cmd-peek-delayed: 797
  cmd-peek-buried: 797
  cmd-reserve: 0
  cmd-reserve-with-timeout: 3245006799
  cmd-delete: 1188639093
  cmd-release: 43276760
  cmd-use: 1137988211
  cmd-watch: 166122
  cmd-ignore: 166077
  cmd-bury: 0
  cmd-kick: 1
  cmd-touch: 0
  cmd-stats: 1260
  cmd-stats-job: 43134828
  cmd-stats-tube: 53942209
  cmd-list-tubes: 3251625
  cmd-list-tube-used: 0
  cmd-list-tubes-watched: 0
  cmd-pause-tube: 988
  job-timeouts: 14084
  total-jobs: 1188640739
  max-job-size: 1048576
  current-tubes: 44
  current-connections: 63
  current-producers: 20
  current-workers: 47
  current-waiting: 41
  total-connections: 14996583
  pid: 3959
  version: 1.4.6
  rusage-utime: 170303.331293
  rusage-stime: 399435.543161
  uptime: 321658179
  binlog-oldest-index: 90539
  binlog-current-index: 90983
  binlog-max-size: 10485760

Re: Choose Postgres queue technology

#208

Post by hknmtt »

donatj wrote: post #145

We have our own queue, because it was easy, fun and has been exceedingly reliable above all else. Far moreso than other things we had tried. Cough Gearman cough SQS cough One endpoint accepts work to a named queue, writes it to a file in an XFS directory. Another locks a mutex, moves the file to an in progress directory and unlocks the mutex before passing the content to the reader. A third and final endpoint deletes…

Why files though, and why move them into different directories? You said billions a day. With files, the physical drive must be taking a beating. Not to mention potential issues with directory file limitations(based on OS and file system). Why not use some kvdb?

Re: Choose Postgres queue technology

#209

Post by Karrot_Kream »

mlyle wrote: post #195
Karrot_Kream wrote: post #183

Earlier quoted context omitted.

In my experience, a queue system is the worst thing to find out isn't scaling properly because once you find out your queue system can't architecturally scale, there's no easy fix to avoid data loss. You talk about "several thousand background jobs" but generally, queues are measured in terms of Little's Law [1] for which you need to be talking about rates; according to Little's Law namely average task enqueue rate p…

> and are confident you'll be working at tens of tasks per second forever. It's more like a few thousand per second, and enqueues win, not dequeues like you say... on very small hardware without tuning. If you're at tens of tasks per second, you have a whole lot of breathing room: don't build for 100x current requirements. https://chbussler.medium.com/implementing-queues-in-postgres... > eventually your dequeue queri…

> https://chbussler.medium.com/implementing-queues-in-postgres...

This link is simply raw enqueue/dequeue performance. Factor in workers that perform work or execute remote calls and the numbers change. Also, I find when your jobs have high variance in times, performance degrades significantly.

> This doesn't really make sense to me. To me, the main problem seems to be that you end up with having a lot of snapshots around.

The dequeuer needs to know which tasks to "claim", so this requires some form of locking. Eventually this becomes a bottleneck.

> don't build for 100x current requirements

What happens if you get 100x traffic? Popularity spikes can do it, so can attacks. Is the answer to just accept data loss in those situations? Queue systems are super simple to use. I'm counting "NOTIFY/LISTEN" on Postgres as a queue, because it is a queue from the bottom up.

Re: Choose Postgres queue technology

#210

Post by boruto »

aduffy wrote: post #2

For several projects I’ve opted for the even dumber approach, that works out of the box with every ORM/Query DSL framework in every language: using a normal table with SELECT FOR UPDATE SKIP LOCKED https://www.pgcasts.com/episodes/the-skip-locked-feature-in-... It’s not “web scale” but it easily extends to several thousand background jobs in my experience

Skip locked is useful till you have to maintain order for a group of messages with some "group_id", so that set of related messages are sent one after the other.

Then you probably have to write complicated queries or use partitions in some sort.

Or Just stick to one thread polling the messages.

Post reply on HN