On SQS

Post by floatboth » Mon, May 27, 2019, 9:50 PM UTC

LunaSea wrote: post #47

andrewstuart wrote: post #15

I use Postgres SKIP LOCKED as a queue. I used to use SQS but Postgres gives me everything I want. I can also do priority queueing and sorting. I gave up on SQS when it couldn't be accessed from a VPC. AWS might have fixed that now. All the other queueing mechanisms I investigated were dramatically more complex and heavyweight than Postgres SKIP LOCKED.

Do you have an example of such a queue somewhere? I think I have a rough idea about how it works because I implemented something similar about four years ago in PostgreSQL but kept getting locking issues I couldn't get out of: - https://stackoverflow.com/questions/33467813/concurrent-craw... - https://stackoverflow.com/questions/29807033/postgresql-conc... Also, what kind of queue size / concurrency on the polling si…

Post by Jemaclus » Mon, May 27, 2019, 10:54 PM UTC

mateuszf wrote: post #154

Jemaclus wrote: post #139

We love SQS, but one of the problems we're running into lately is the 256kb per message limitation. We do tens of millions of messages per day, with a small percentage of those reaching the 256kb limit. We're approaching the point where most of our messages will hit that limit. What are our options for keeping SQS but somehow sending large payloads? Only thing I can think of is throwing them into another datastore an…

We have a library that puts the payload in s3 bucket under random key, the bucket has expiration policy of few days. Then we generate http link to the object and send an sqs message with this url in metadata. The reader library gets data from s3, it doesn't even have to remove it. It will disappear automatically later. We do it "by ourselves", not using the provided lib, because that way it works both for SQS and SNS…

Post by Jemaclus » Mon, May 27, 2019, 10:56 PM UTC

manish_gill wrote: post #162

Jemaclus wrote: post #139

We love SQS, but one of the problems we're running into lately is the 256kb per message limitation. We do tens of millions of messages per day, with a small percentage of those reaching the 256kb limit. We're approaching the point where most of our messages will hit that limit. What are our options for keeping SQS but somehow sending large payloads? Only thing I can think of is throwing them into another datastore an…

I've successfully been using Kafka in production with a tiny percentage of packets ~1Mb. We utilize gzip compression with chunking on some of those messages (system is legacy, no fancy compressed message format in use). IIRC, only had to modify settings at the broker level and it works perfectly fine.

Post by dmix » Mon, May 27, 2019, 11:09 PM UTC

awinder wrote: post #155

haolez wrote: post #131

Earlier quoted context omitted.

I've been planning on giving Erlang/Elixir a try, but we are very reliant on serverless and managed cloud services (like SQS) and I get the impression that managing a cluster of worker nodes for Erlang/Elixir would be too much work for us, since we would have to manage the servers, security patches, plan its scaling, etc. Maybe I'm wrong and it's not so much work in the end. Hoping for some feedback.

I’d love to know how this isn’t true as well, but I was in an environment where cross-az network costs were something we were continuously mitigating against. Using stuff like sqs let us build cross-az availability with 0-metered network costs, serverless can come into play because it’s network connections usually come through 0-cost aws services as well. It seems to me like from a cost basis, getting into something…

Post by cryptonector » Mon, May 27, 2019, 11:20 PM UTC

sorokod wrote: post #73

napsterbr wrote: post #72

Earlier quoted context omitted.

Not GP but I think this wouldn't be a problem. The consumer dequeues with SELECT FOR UPDATE within a transaction. If it crashes the database would rollback the transaction, and then another consumer would be able to select the work unit. As for acking, I see two common methods: using an additional boolean column, something like is_processed. Consumers skip truthy ones. Or, after the work is done, simply delete the en…

My question assumed a scenario where a consumer dequeues a batch, commits the deqieued change, and then crashes while processing the batch. Offcourse one could delay the commit until all processing is completed but then reasoning about the queue throughput becomes tricky.