Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

On SQS

tbray.org

Post reply on HN
Print view View original

121–130 of 229 posts

Re: On SQS

#121

Post by timbray »

dantillberg wrote: post #94

I've worked with SQS at volumes in thousands of messages per second with varied (non-tiny) payload sizes. SQS is a very simple service, which makes it fairly reliable, though part of the reason for the reliability is that the API's guarantees are weak. And it can be economical, but I've had to build a lot of non-trivial logic in order to interact with SQS robustly, performantly, and efficiently, especially around usi…

Um, I’ve been at AWS since late 2014, and AFAIK the only extended SQS hiccup correlated with the DynamoDB issue in 2016. SQS isn’t perfect but I’m pretty sure “does routinely have temporary failures that generally last for a few ours at a time” is just wrong.

Re: On SQS

#122

Post by noelherrick »

timbray wrote: post #121
dantillberg wrote: post #94

I've worked with SQS at volumes in thousands of messages per second with varied (non-tiny) payload sizes. SQS is a very simple service, which makes it fairly reliable, though part of the reason for the reliability is that the API's guarantees are weak. And it can be economical, but I've had to build a lot of non-trivial logic in order to interact with SQS robustly, performantly, and efficiently, especially around usi…

Um, I’ve been at AWS since late 2014, and AFAIK the only extended SQS hiccup correlated with the DynamoDB issue in 2016. SQS isn’t perfect but I’m pretty sure “does routinely have temporary failures that generally last for a few ours at a time” is just wrong.

I believe GP was talking about particular messages failing, not a total system outage. In my use of AWS, the status page almost never reports an outage even though that AWS service is down for me-as in the most I've ever seen is some hand wavey message that there's elevated error rates. So you could be right, SQS hasn't failed entirely, but that probably means there's a good number of failed requests that are below the margin where AWS would consider it down.

Re: On SQS

#123

Post by gizzlon »

eropple wrote: post #113
gizzlon wrote: post #108

Earlier quoted context omitted.

Guess it depends on the definition of "queue". Potentials: - https://nsq.io/ - https://nats.io/

I am curious what definition of "queue" fits NSQ. It's a distributed messaging platform.

The normal one. It puts stuff in a queue.

"nsqd is the daemon that receives, queues, and delivers messages to clients."

Re: On SQS

#124

Post by appwiz »

oceanbreeze83 wrote: post #51

is SNS + SQS a reasonable solution for realtime irc style topic chatrooms?

AWS AppSync (https://aws.amazon.com/appsync/) is a better fit for the chat room use case because of server pushed events over a persistent connection (WebSockets). Launch the Chat sample in the console to try it out.

Re: On SQS

#125

Post by KirinDave »

varelaz wrote: post #14

One important drawback of SQS is that it's eventually consistent, you can read the same message twice from different workers. Nevertheless we keep using it with additional checks when it's critical, it's still the cheapest solution by maintenance.

Amazon's hand an answer to that with FIFO queues for the better part of years, but even without a queue based system you probably still need some notion of idempotence in the actions your system takes.

Systems "without queues" have just eschewed one queue for N queues, where N is the number of clients buffering actions for retries at any given moment.

Re: On SQS

#126

Post by archgoon »

emmelaich wrote: post #32
archgoon wrote: post #7

Earlier quoted context omitted.

> Using S3 as an ad-hoc queue is a cheaper solution, which should throw some red flags. Interesting. Can you expand on this? How do you ensure that only one worker takes a message from s3? Or do you only use this setup when you have only one worker?

FWIW there is a queue based on maildir which has implementations in Perl, Python, C and Java and probably more. The Perl implementation was the original AFAIK. http://search.cpan.org/dist/Directory-Queue/

Interesting; looks like DirectoryQueue uses directories, rather than file locks (man 2 flock), to lock the queue messages. This might actually work, since mkdir returns an error if you attempt to create a directory that already exists. The implementation seems to be handling most of the obvious failure cases, or at least tries to.

https://metacpan.org/release/Directory-Queue/source/lib/Dire...

So how does one lock a message in s3? Does s3 have a "createIfDoesNotExistOrError"? I'm still having difficulty understanding how the proposed system avoids race conditions.

Re: On SQS

#127

Post by staticassertion »

I think this is a decent response - they really nail what @rbranson misses, that the failures he mentions are actually features we're after.

An example,

> Convert something to an async operation and your system will always return a success response. But there's no guarantee that the request will actually ever be processed successfully.

Great! I don't want service A to be coupled to service B's ability to work. I want A to send off a message and leave it to B to succeed or fail. This separation of state (service A and B can't even talk to each other directly) is part of what makes queues so powerful - it's also the foundation of the actor model, which is known for its powerful resiliency and scalability properties.

The author's suggestion of using synchronous communication with backpressure and sync failures is my last ditch approach. I have to set up circuit breakers just to make something like this anything less than a total disaster with full system failure due to a single service outage.

Like the author, the "good use cases for queues" is very nearly 100% for me. I believe you should reach for queues first, and it's worth remodeling a system to be queue based if you can help it.

Sometimes modeling as synchronous control is easiest, but I'm happy that I can avoid that in almost every case.

Re: On SQS

#128

Post by dantillberg »

noelherrick wrote: post #122
timbray wrote: post #121

Earlier quoted context omitted.

Um, I’ve been at AWS since late 2014, and AFAIK the only extended SQS hiccup correlated with the DynamoDB issue in 2016. SQS isn’t perfect but I’m pretty sure “does routinely have temporary failures that generally last for a few ours at a time” is just wrong.

I believe GP was talking about particular messages failing, not a total system outage. In my use of AWS, the status page almost never reports an outage even though that AWS service is down for me-as in the most I've ever seen is some hand wavey message that there's elevated error rates. So you could be right, SQS hasn't failed entirely, but that probably means there's a good number of failed requests that are below t…

Yes, this is correct, thank you. I updated my comment to indicate that I meant partial failure, though the failure conditions persist from 20 minutes to a few hours. Those partial failures have happened once every two months or so in my experience.

Technically, it's not even really a failure of SQS because the guarantees SQS makes are so weak that those partial failures are really "operating normally."

Re: On SQS

#129

Post by dmix »

staticassertion wrote: post #127

I think this is a decent response - they really nail what @rbranson misses, that the failures he mentions are actually features we're after. An example, > Convert something to an async operation and your system will always return a success response. But there's no guarantee that the request will actually ever be processed successfully. Great! I don't want service A to be coupled to service B's ability to work. I want…

> Convert something to an async operation and your system will always return a success response.

It's funny reading this after using Erlang/Elixir over the last few years. The default is always async with the assumption it will fail - as async processes failing is a core part of the OTP application architecture.

It's not something to be feared but a key part of how your application data-flow works.

Re: On SQS

#130

Post by polskibus »

gizzlon wrote: post #108
polskibus wrote: post #88

Does anyone know a good, low overhead out-of-process message queue, that's lightweight enough that it can be useful for communicating between processes on the same machine, but if necessary it can scale beyond it? In case of a single-machine product that comprises of several services, a message queue can sometimes be useful for pull model, but adding RabbitMQ to the stack makes installation and ops much more complex…

Guess it depends on the definition of "queue". Potentials: - https://nsq.io/ - https://nats.io/

Do they offer at-least-once or better delivery guarantees?
Post reply on HN