Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

On SQS

tbray.org

Post reply on HN
Print view View original

21–30 of 229 posts

Re: On SQS

#21

Post by varelaz »

philliphaydon wrote: post #19
varelaz wrote: post #14

One important drawback of SQS is that it's eventually consistent, you can read the same message twice from different workers. Nevertheless we keep using it with additional checks when it's critical, it's still the cheapest solution by maintenance.

> you can read the same message twice from different workers If you have a short visibility timeout that it appears back on the queue before you delete it... Sure.

No. By default SQS uses `at-least-once delivery` system. It's distributed and when you connect to API one SQS node that you working with could have different state comparing to another, so you may fetch the same message twice by 2 separate workers even with big visibility timeout.

Re: On SQS

#22

Post by lclarkmichalek »

varelaz wrote: post #14

One important drawback of SQS is that it's eventually consistent, you can read the same message twice from different workers. Nevertheless we keep using it with additional checks when it's critical, it's still the cheapest solution by maintenance.

Making the processing of a message idempotent is the ideal way to handle that limitation

Re: On SQS

#23

Post by redact207 »

Author of https://node-ts.github.io/bus/ here. SQS is definitely one of my most favourite message queues. The ability to have a HA managed solution without having to worry about persistence, scaling or connections is huge.

Most of the complaints are native to message based systems in general. At least once message receives, out of order receives, pretty standard faire that can be handled by applying well established patterns.

My only request would be to please increase the limits of message visibility timeouts! Often I want to delay send a message for receipt in 30 days. SQS forces me to cook some weird delete and resend recipe, or make this a responsibility of a data store. It's be really nice to do away with batch/Cron jobs and deal more with delayed queue events.

Re: On SQS

#24

Post by plasma »

Anyone run a multi-tenant SaaS and handle fairness with jobs “fairly”?

Occasionally we use to have all workers tied up on a single customers long running tasks, we mitigated by using a throttler we wrote that can defer a job if too many resources are in use by the customer, but it’s not ideal.

I’d love a priority based, customer throttled (eg max concurrent tasks) queue.

We can prioritize by low/medium/high using separate queues, and could make a set of queues per customer; but that is starting to explode how many queues we have and feels unmanageable.

Re: On SQS

#25

Post by plasma »

redact207 wrote: post #23

Author of https://node-ts.github.io/bus/ here. SQS is definitely one of my most favourite message queues. The ability to have a HA managed solution without having to worry about persistence, scaling or connections is huge. Most of the complaints are native to message based systems in general. At least once message receives, out of order receives, pretty standard faire that can be handled by applying well established…

RE: visibility timeout beyond 30 days, you may be more after a “saga” that has state and is long running (hours/days/months/years).

You can imagine building a saga system on top of a queue system.

Re: On SQS

#26

Post by varelaz »

plasma wrote: post #24

Anyone run a multi-tenant SaaS and handle fairness with jobs “fairly”? Occasionally we use to have all workers tied up on a single customers long running tasks, we mitigated by using a throttler we wrote that can defer a job if too many resources are in use by the customer, but it’s not ideal. I’d love a priority based, customer throttled (eg max concurrent tasks) queue. We can prioritize by low/medium/high using sep…

We implemented this with additional DB checks. For example: we put only one job to queue at a time per customer, the rest are in database until one that in progress is not processed.

Actually most of the prioritization could be implemented through additional DB. With SQS in most cases you need persistent reflection of job to keep its status, process times, results. You can put to queue only few items that are highest priority to guarantee that workers are busy next 10-30 minutes.

Re: On SQS

#27

Post by ilogik »

cyberferret wrote: post #18

We've used SQS with great results (and reliability) for many years now, but I am interested to hear the author talking about 'replaying queues' to replicate faults. I never realised you could do this with SQS. Or can you? I thought once a queue item was processed and deleted, that was it, it was gone forever, but perhaps you can see historical queue data somewhere? (without having to store it yourself)

maybe once you've processed a message, you resend it to a different queue, just in case?

Re: On SQS

#28

Post by redact207 »

plasma wrote: post #25
redact207 wrote: post #23

Author of https://node-ts.github.io/bus/ here. SQS is definitely one of my most favourite message queues. The ability to have a HA managed solution without having to worry about persistence, scaling or connections is huge. Most of the complaints are native to message based systems in general. At least once message receives, out of order receives, pretty standard faire that can be handled by applying well established…

RE: visibility timeout beyond 30 days, you may be more after a “saga” that has state and is long running (hours/days/months/years). You can imagine building a saga system on top of a queue system.

You're absolutely right, in fact I have a whole package that is just that https://node-ts.github.io/bus/packages/bus-workflow/.

The problem is this. Let's say that I want to trigger a step in a "free trial" saga that sends an email to the customer 10 days after they sign up nudging them to get a paid account. If I can delay send this message for 10 days then it's easy.

However because SQS has a much shorter visibility timeout, I have to find a much more roundabout way of triggering that action.

Re: On SQS

#29

Post by plasma »

redact207 wrote: post #28
plasma wrote: post #25

Earlier quoted context omitted.

RE: visibility timeout beyond 30 days, you may be more after a “saga” that has state and is long running (hours/days/months/years). You can imagine building a saga system on top of a queue system.

You're absolutely right, in fact I have a whole package that is just that https://node-ts.github.io/bus/packages/bus-workflow/ . The problem is this. Let's say that I want to trigger a step in a "free trial" saga that sends an email to the customer 10 days after they sign up nudging them to get a paid account. If I can delay send this message for 10 days then it's easy. However because SQS has a much shorter visibili…

Yeah, that makes total sense. For some of our saga's (we don't use SQS -- we use a custom redis queue), we have the saga potentially wake up and immediately sleep again ("Nothing to do right now, defer again in a few days").

But yes, a quirk.

Re: On SQS

#30

Post by plasma »

varelaz wrote: post #26
plasma wrote: post #24

Anyone run a multi-tenant SaaS and handle fairness with jobs “fairly”? Occasionally we use to have all workers tied up on a single customers long running tasks, we mitigated by using a throttler we wrote that can defer a job if too many resources are in use by the customer, but it’s not ideal. I’d love a priority based, customer throttled (eg max concurrent tasks) queue. We can prioritize by low/medium/high using sep…

We implemented this with additional DB checks. For example: we put only one job to queue at a time per customer, the rest are in database until one that in progress is not processed. Actually most of the prioritization could be implemented through additional DB. With SQS in most cases you need persistent reflection of job to keep its status, process times, results. You can put to queue only few items that are highest…

Thanks for the suggestion, ill reflect on that; we were also considering an "overflow" queue that would receive jobs if there was a recent high insert rate of a customer's jobs to the main queue, but that didn't solve the "cost" problem of a job potentially being large.

Was hoping to avoid having "side car" infrastructure for this, but I don't think I can escape it :)

Post reply on HN