is SNS + SQS a reasonable solution for realtime irc style topic chatrooms?
On SQS
51–60 of 229 posts
Re: On SQS
#52is SNS + SQS a reasonable solution for realtime irc style topic chatrooms?
It depends, but you can safely assume neither service is intended for real-time use cases. They'll be fast, but there won't be any speed / delivery guarantees.
Re: On SQS
#53I use Postgres SKIP LOCKED as a queue. I used to use SQS but Postgres gives me everything I want. I can also do priority queueing and sorting. I gave up on SQS when it couldn't be accessed from a VPC. AWS might have fixed that now. All the other queueing mechanisms I investigated were dramatically more complex and heavyweight than Postgres SKIP LOCKED.
Do you have an example of such a queue somewhere? I think I have a rough idea about how it works because I implemented something similar about four years ago in PostgreSQL but kept getting locking issues I couldn't get out of: - https://stackoverflow.com/questions/33467813/concurrent-craw... - https://stackoverflow.com/questions/29807033/postgresql-conc... Also, what kind of queue size / concurrency on the polling si…
Here https://www.2ndquadrant.com/en/blog/what-is-select-skip-lock... is a good discussion.
Re: On SQS
#54> So what to do? In almost all cases it's better to propagate failure and backpressure. This gives a chance for the upstream to decide if it wants to retry or fail itself. It keeps things simple and easy-to-understand. It doesn't build up huge backlogs that become disasters.
What does this actually mean in practical terms?
Re: On SQS
#55Earlier quoted context omitted.
Making the processing of a message idempotent is the ideal way to handle that limitation
That's not always possible. Thus a big no-no for SQS if that's what you need. Concluding: SQS cannot replace RabbitMQ in all usecases.
Exactly once delivery =/= exactly once processing.
Re: On SQS
#56I really wish SQS had reliably lower latency, like Redis, and also supported priority levels. (Also like redis, now, with sorted sets and the https://redis.io/commands/bzpopmax command.) Has anyone measured the performance of Redis on large sorted sets, say millions of items? Hoping that it's still in single digit milliseconds at that size... And can sustain say 1000QPS...
We use Redis as a job queue and its great; the only limitation is being sometimes concerned about job queue size due to memory limits of the Redis server itself.
Always hear using redis etc for job queues, what do these "job queues" entail? I've been an amateur and have used postgres tables as queues .. am I not being efficient?
Re: On SQS
#57Earlier quoted context omitted.
That's not always possible. Thus a big no-no for SQS if that's what you need. Concluding: SQS cannot replace RabbitMQ in all usecases.
If you’re message consumer isn’t idempotent then no MQ can help you. Exactly once delivery is impossible other than with at least once delivery and an idempotent consumer. https://bravenewgeek.com/tag/amazon-sqs/
> Exactly once delivery is impossible other than with at least once delivery
Can you explain this? Don't many applications deliver once and only once via locking? It's obviously easier as an application developer to say "I will only get this once" and accept losing messages than dealing with idempotence particularly in distributed services.
Re: On SQS
#58Earlier quoted context omitted.
We use Redis as a job queue and its great; the only limitation is being sometimes concerned about job queue size due to memory limits of the Redis server itself.
Always hear using redis etc for job queues, what do these "job queues" entail? I've been an amateur and have used postgres tables as queues .. am I not being efficient?
Tables as queues is perfectly legit, one less component to worry about.
Re: On SQS
#59Earlier quoted context omitted.
We use Redis as a job queue and its great; the only limitation is being sometimes concerned about job queue size due to memory limits of the Redis server itself.
Also, when you don't want to lose a message, the Redis persistence story requires careful thought. It requires setting up RDB + AOF + appendfsync=always + backups.
This would just kill the performance on virtualized hardware like EBS. You would lose a lot of benefits of Redis at that point.
Re: On SQS
#60We've used SQS with great results (and reliability) for many years now, but I am interested to hear the author talking about 'replaying queues' to replicate faults. I never realised you could do this with SQS. Or can you? I thought once a queue item was processed and deleted, that was it, it was gone forever, but perhaps you can see historical queue data somewhere? (without having to store it yourself)
Possibly this: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQS...
We actually already use the DeadLetterQueues in our service at the moment, but these are when failed SQS deliveries happen, then they get replicated to the DLQ.
I am more interested in diagnosing successful SQS deliveries after the fact, to see what the payloads were in case there was a downstream problem.
It seems that SQS deliveries that don't get a 200 response from our service go to the DLQ, but those that get a successful 200 disappear into the ether.