Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

On SQS

tbray.org

Post reply on HN
Print view View original

31–40 of 229 posts

Re: On SQS

#31

Post by plasma »

etaioinshrdlu wrote: post #13

I really wish SQS had reliably lower latency, like Redis, and also supported priority levels. (Also like redis, now, with sorted sets and the https://redis.io/commands/bzpopmax command.) Has anyone measured the performance of Redis on large sorted sets, say millions of items? Hoping that it's still in single digit milliseconds at that size... And can sustain say 1000QPS...

We use Redis as a job queue and its great; the only limitation is being sometimes concerned about job queue size due to memory limits of the Redis server itself.

Re: On SQS

#32

Post by emmelaich »

archgoon wrote: post #7
reallydude wrote: post #3

Nowhere is cost mentioned. Using S3 as an ad-hoc queue is a cheaper solution, which should throw some red flags. You can easily do what SQS does for so much cheaper (this includes horizontal scaling and failure planning), that I'm consistently surprised anyone uses it. Either you are running at a volume where you need high throughput and it's pricey, or you're at such a low throughput you could use any MQ (even redis…

> Using S3 as an ad-hoc queue is a cheaper solution, which should throw some red flags. Interesting. Can you expand on this? How do you ensure that only one worker takes a message from s3? Or do you only use this setup when you have only one worker?

FWIW there is a queue based on maildir which has implementations in Perl, Python, C and Java and probably more.

The Perl implementation was the original AFAIK.

http://search.cpan.org/dist/Directory-Queue/

Re: On SQS

#33

Post by gondo »

redact207 wrote: post #23

Author of https://node-ts.github.io/bus/ here. SQS is definitely one of my most favourite message queues. The ability to have a HA managed solution without having to worry about persistence, scaling or connections is huge. Most of the complaints are native to message based systems in general. At least once message receives, out of order receives, pretty standard faire that can be handled by applying well established…

isn't it dangerous to relay on persistence of MQ for such a long time? shouldn't your system have a way to re-create the queue at any time? if so, you should be able to call such command to fill in the queue within the maximum timeout time allowed by MQ provider.

Re: On SQS

#34

Post by cies »

lclarkmichalek wrote: post #22
varelaz wrote: post #14

One important drawback of SQS is that it's eventually consistent, you can read the same message twice from different workers. Nevertheless we keep using it with additional checks when it's critical, it's still the cheapest solution by maintenance.

Making the processing of a message idempotent is the ideal way to handle that limitation

That's not always possible. Thus a big no-no for SQS if that's what you need. Concluding: SQS cannot replace RabbitMQ in all usecases.

Re: On SQS

#35

Post by kayoone »

> Those messages will age out and vanish after a little while (14 days is currently the max); but before they go, they’re stored carefully and are very unlikely to go missing

can somebody expand on this? I know about the 14 days limitation but this makes it sound like you can store messages for a long time and still recover them somehow?

Re: On SQS

#36

Post by bpicolo »

gondo wrote: post #33
redact207 wrote: post #23

Author of https://node-ts.github.io/bus/ here. SQS is definitely one of my most favourite message queues. The ability to have a HA managed solution without having to worry about persistence, scaling or connections is huge. Most of the complaints are native to message based systems in general. At least once message receives, out of order receives, pretty standard faire that can be handled by applying well established…

isn't it dangerous to relay on persistence of MQ for such a long time? shouldn't your system have a way to re-create the queue at any time? if so, you should be able to call such command to fill in the queue within the maximum timeout time allowed by MQ provider.

> isn't it dangerous to relay on persistence of MQ for such a long time

Most of them are designed and intended as a persistent data store. Data loss for a queue system is typically something you do not want

Re: On SQS

#37

Post by actuator »

kayoone wrote: post #35

> Those messages will age out and vanish after a little while (14 days is currently the max); but before they go, they’re stored carefully and are very unlikely to go missing can somebody expand on this? I know about the 14 days limitation but this makes it sound like you can store messages for a long time and still recover them somehow?

I think what he meant was, Once you successfully put a message to SQS, it is kept till the 'message retention period' unless you explicitly call delete on it. Right now you can configure the period to be upto 14 days. AFAIK, there is no way to recover messages older than their retention period.

Re: On SQS

#38

Post by tybit »

cies wrote: post #34
lclarkmichalek wrote: post #22

Earlier quoted context omitted.

Making the processing of a message idempotent is the ideal way to handle that limitation

That's not always possible. Thus a big no-no for SQS if that's what you need. Concluding: SQS cannot replace RabbitMQ in all usecases.

If you’re message consumer isn’t idempotent then no MQ can help you. Exactly once delivery is impossible other than with at least once delivery and an idempotent consumer.

https://bravenewgeek.com/tag/amazon-sqs/

Re: On SQS

#39

Post by actuator »

sessy wrote: post #2

WE are heavy users of AWS. SQS is the only service where we have had zero downtime. The only downside we have about SQS is you can pull out only 10 messages at a time (without batching). You can have parallel readers but they result in some duplicates. There is SQS FIFO but it is throttled.

It went down on us once for extended period in 2015. It was chaotic as you don't expect it to fail. If memory serves me right, even S3 suffered that day.

Re: On SQS

#40

Post by actuator »

etaioinshrdlu wrote: post #13

I really wish SQS had reliably lower latency, like Redis, and also supported priority levels. (Also like redis, now, with sorted sets and the https://redis.io/commands/bzpopmax command.) Has anyone measured the performance of Redis on large sorted sets, say millions of items? Hoping that it's still in single digit milliseconds at that size... And can sustain say 1000QPS...

I have worked with sorted sets with millions of items. The latency really depends on what you execute. Commands that involve only few elements are fine. Like the ZPOPMAX you mentioned should be way under a ms if you pop say 10 items, and you should be able to get way more than 1k QPS. The thing with sorted sets like most data structures in Redis is to just read the time complexity well in their documentation. For an operation like ZPOPMAX it is linearly proportional to the number of items you pop, so if you pop too many it will take time.
Post reply on HN