Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

On SQS

tbray.org

Post reply on HN
Print view View original

151–160 of 229 posts

Re: On SQS

#151

Post by coredog64 »

emmelaich wrote: post #32
archgoon wrote: post #7

Earlier quoted context omitted.

> Using S3 as an ad-hoc queue is a cheaper solution, which should throw some red flags. Interesting. Can you expand on this? How do you ensure that only one worker takes a message from s3? Or do you only use this setup when you have only one worker?

FWIW there is a queue based on maildir which has implementations in Perl, Python, C and Java and probably more. The Perl implementation was the original AFAIK. http://search.cpan.org/dist/Directory-Queue/

Back in 2000, I worked with a guy that built an entire message queue product using the SMTP protocol with an implementation that was in turn built on top of lex and yacc.

Re: On SQS

#152

Post by dev_dull »

staticassertion wrote: post #127

I think this is a decent response - they really nail what @rbranson misses, that the failures he mentions are actually features we're after. An example, > Convert something to an async operation and your system will always return a success response. But there's no guarantee that the request will actually ever be processed successfully. Great! I don't want service A to be coupled to service B's ability to work. I want…

> The author's suggestion of using synchronous communication with backpressure and sync failures is my last ditch approach

Also back pressure isn’t difficult to implement. Simply read the estimated size of the queue every N minutes and pass sending until it goes down to a more manageable level. Obvious downside is that it’s client side.

Re: On SQS

#153

Post by avip »

dantillberg wrote: post #94

I've worked with SQS at volumes in thousands of messages per second with varied (non-tiny) payload sizes. SQS is a very simple service, which makes it fairly reliable, though part of the reason for the reliability is that the API's guarantees are weak. And it can be economical, but I've had to build a lot of non-trivial logic in order to interact with SQS robustly, performantly, and efficiently, especially around usi…

>you need to put that data somewhere else and pass a pointer to it in the actual SQS message

I was under the impression that's the industry standard - you drop the payload in some redis-like storage and pass keys in messages.

Re: On SQS

#154

Post by mateuszf »

Jemaclus wrote: post #139

We love SQS, but one of the problems we're running into lately is the 256kb per message limitation. We do tens of millions of messages per day, with a small percentage of those reaching the 256kb limit. We're approaching the point where most of our messages will hit that limit. What are our options for keeping SQS but somehow sending large payloads? Only thing I can think of is throwing them into another datastore an…

We have a library that puts the payload in s3 bucket under random key, the bucket has expiration policy of few days. Then we generate http link to the object and send an sqs message with this url in metadata. The reader library gets data from s3, it doesn't even have to remove it. It will disappear automatically later.

We do it "by ourselves", not using the provided lib, because that way it works both for SQS and SNS. The provided lib only supports SQS.

Also our messages aren't typically very big, so we do this only if the payload size demands it.

Re: On SQS

#155

Post by awinder »

haolez wrote: post #131
dmix wrote: post #129

Earlier quoted context omitted.

> Convert something to an async operation and your system will always return a success response. It's funny reading this after using Erlang/Elixir over the last few years. The default is always async with the assumption it will fail - as async processes failing is a core part of the OTP application architecture. It's not something to be feared but a key part of how your application data-flow works.

I've been planning on giving Erlang/Elixir a try, but we are very reliant on serverless and managed cloud services (like SQS) and I get the impression that managing a cluster of worker nodes for Erlang/Elixir would be too much work for us, since we would have to manage the servers, security patches, plan its scaling, etc. Maybe I'm wrong and it's not so much work in the end. Hoping for some feedback.

I’d love to know how this isn’t true as well, but I was in an environment where cross-az network costs were something we were continuously mitigating against. Using stuff like sqs let us build cross-az availability with 0-metered network costs, serverless can come into play because it’s network connections usually come through 0-cost aws services as well. It seems to me like from a cost basis, getting into something with clustered erlang would kill you in many of these cloud environments (or at least you would be on the hook for engineering workarounds to keep traffic within an az w/ failover to other azs)

Re: On SQS

#156

Post by avip »

yilugurlu wrote: post #82

Has anyone ever measured the latency of the sending message to SQS? I was using with ELB in t2.medium instances, and my API (handle => send message to queue => return {status: true}) response times were around 150 - 300 ms and replaced SQS with RabbitMQ, and it went down to around 75-100 ms. Does anyone think that sending message to SQS is slow? Edit: With this update, I was able to process almost 3 x requests with t…

Sounds more like AZ issue than SQS/rabbit?

Re: On SQS

#157

Post by stingraycharles »

avip wrote: post #153
dantillberg wrote: post #94

I've worked with SQS at volumes in thousands of messages per second with varied (non-tiny) payload sizes. SQS is a very simple service, which makes it fairly reliable, though part of the reason for the reliability is that the API's guarantees are weak. And it can be economical, but I've had to build a lot of non-trivial logic in order to interact with SQS robustly, performantly, and efficiently, especially around usi…

>you need to put that data somewhere else and pass a pointer to it in the actual SQS message I was under the impression that's the industry standard - you drop the payload in some redis-like storage and pass keys in messages.

On AWS you would typically use S3 but yes this is definitely the accepted standard: nobody puts big blobs in message queues because they solve a different problem and are (typically) not designed to handle that.

Re: On SQS

#158

Post by xyzzy_plugh »

KirinDave wrote: post #148
gmueckl wrote: post #137

Earlier quoted context omitted.

What possibilities are there for a write to fail that the database server cannot react to? I am under the impression that a write error like this would be reported as a failure to run the statement and the application is responsible for handling that.

The box turning off, the disk interface failing, and/or the link to the database failing in mid instruction. Same as SQS.

All of those would result in a write failure from the application's perspective, which is fine, and must be accounted for regardless (e.g. retry, two phase commit, log an error, whatever).

Re: On SQS

#159

Post by avip »

stingraycharles wrote: post #157
avip wrote: post #153

Earlier quoted context omitted.

>you need to put that data somewhere else and pass a pointer to it in the actual SQS message I was under the impression that's the industry standard - you drop the payload in some redis-like storage and pass keys in messages.

On AWS you would typically use S3 but yes this is definitely the accepted standard: nobody puts big blobs in message queues because they solve a different problem and are (typically) not designed to handle that.

You will definitely not use S3 in a pipeline if latency is an issue.

Re: On SQS

#160

Post by snisarenko »

zug_zug wrote: post #83
andrewstuart wrote: post #15

I use Postgres SKIP LOCKED as a queue. I used to use SQS but Postgres gives me everything I want. I can also do priority queueing and sorting. I gave up on SQS when it couldn't be accessed from a VPC. AWS might have fixed that now. All the other queueing mechanisms I investigated were dramatically more complex and heavyweight than Postgres SKIP LOCKED.

I LOVE this idea. I usually hear other Sr. engineers denigrate it as "hacky," but I think they aren't really looking at the big picture. 1. By combining services, 1 less service to manage in your stack (e.g. do your demo/local/qa envs all connect to Sqs?) 2. Postgres preserves your data if it goes down 3. You already have the tools on each machine and everybody knows the querying language to examine the stack 4. All…

There is another advantage.

If you are using the queue as a log of events (i.e. user actions), you get an atomic guarantee that the db data is updated and the event describing this update has been recorded.

Post reply on HN