One downside of SQS is that it doesn't support fan-out, for eg. S3->SQS->multiple consumers. The recommendation instead seems to be to first push to SNS, and then hookup SQS/other consumers to it. Kinesis/Kafka would appear to be better suited for this (since they support fan-out like SNS and are pull-based like SQS), but aren't as well supported as SNS/SQS (you can't push S3 events directly to Kinesis for eg.) Can someone from AWS comment on why that is? Also, related: when can we expect GA for Kafka (MSK)?
On SQS
41–50 of 229 posts
Re: On SQS
#42> Those messages will age out and vanish after a little while (14 days is currently the max); but before they go, they’re stored carefully and are very unlikely to go missing can somebody expand on this? I know about the 14 days limitation but this makes it sound like you can store messages for a long time and still recover them somehow?
They're saying you've very unlikely to experience data loss after a message has been saved by SQS. Presumably they have multiple layers of redundancy at varying levels in order to provide this guarantee.
Re: On SQS
#43We've used SQS with great results (and reliability) for many years now, but I am interested to hear the author talking about 'replaying queues' to replicate faults. I never realised you could do this with SQS. Or can you? I thought once a queue item was processed and deleted, that was it, it was gone forever, but perhaps you can see historical queue data somewhere? (without having to store it yourself)
Re: On SQS
#44Anyone run a multi-tenant SaaS and handle fairness with jobs “fairly”? Occasionally we use to have all workers tied up on a single customers long running tasks, we mitigated by using a throttler we wrote that can defer a job if too many resources are in use by the customer, but it’s not ideal. I’d love a priority based, customer throttled (eg max concurrent tasks) queue. We can prioritize by low/medium/high using sep…
Using a database lets you make much better decisions on this space
Tbh purpose-built queues are taken way too eagerly by programmers who later end up needing the flexibility offered by a more general data store.
Re: On SQS
#45I recreated an SQS style service on AWS for sole purpose of avoiding being locked into Amazon. It used their autoscale system and was about the dumbest simplest api you can imagine, just depositing messages into a geographically replicated db table for later processing. We also used it as sort of a backup system, where the messages were never truly deleted and everything could be reprocessed as needed since the front end db that held processed records was an absolute amateur disaster just waiting for some malicious sql injection wipes. It was solid as a rock and extremely low maintenance. I think things have become relatively standardized since then (aka lots of duplicate SQS compatible api/services), so unless there was a serious requirement to stay off SQS I'm not sure I'd do it again.
Re: On SQS
#46I really wish SQS had reliably lower latency, like Redis, and also supported priority levels. (Also like redis, now, with sorted sets and the https://redis.io/commands/bzpopmax command.) Has anyone measured the performance of Redis on large sorted sets, say millions of items? Hoping that it's still in single digit milliseconds at that size... And can sustain say 1000QPS...
We use Redis as a job queue and its great; the only limitation is being sometimes concerned about job queue size due to memory limits of the Redis server itself.
Also, when you don't want to lose a message, the Redis persistence story requires careful thought. It requires setting up RDB + AOF + appendfsync=always + backups.
Re: On SQS
#47I use Postgres SKIP LOCKED as a queue. I used to use SQS but Postgres gives me everything I want. I can also do priority queueing and sorting. I gave up on SQS when it couldn't be accessed from a VPC. AWS might have fixed that now. All the other queueing mechanisms I investigated were dramatically more complex and heavyweight than Postgres SKIP LOCKED.
Do you have an example of such a queue somewhere?
I think I have a rough idea about how it works because I implemented something similar about four years ago in PostgreSQL but kept getting locking issues I couldn't get out of:
- https://stackoverflow.com/questions/33467813/concurrent-craw...
- https://stackoverflow.com/questions/29807033/postgresql-conc...
Also, what kind of queue size / concurrency on the polling side are you able to sustain for your current hardware?
Re: On SQS
#48With 4-5 concurrent processes reading from my (FIFO?) SQS queue, I have a hard time seeing the contents of the queue from the console. SQS really doesn't handle the concurrent readers well in my experience.
Re: On SQS
#49I use Postgres SKIP LOCKED as a queue. I used to use SQS but Postgres gives me everything I want. I can also do priority queueing and sorting. I gave up on SQS when it couldn't be accessed from a VPC. AWS might have fixed that now. All the other queueing mechanisms I investigated were dramatically more complex and heavyweight than Postgres SKIP LOCKED.
Using SKIP LOCKED - do you commit the change to the dequeued item (ack it) at the point where you exit the DB call. If so what happens if the instance that dequeued the messages crashes?
Re: On SQS
#50After reading some comments in this thread im concerned about how many misconceptions people have about AWS services. Most of the stuff from "correcting comments" is plain available for anyone.