Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

On SQS

tbray.org

Post reply on HN
Print view View original

61–70 of 229 posts

Re: On SQS

#61

Post by cyberferret »

ilogik wrote: post #27
cyberferret wrote: post #18

We've used SQS with great results (and reliability) for many years now, but I am interested to hear the author talking about 'replaying queues' to replicate faults. I never realised you could do this with SQS. Or can you? I thought once a queue item was processed and deleted, that was it, it was gone forever, but perhaps you can see historical queue data somewhere? (without having to store it yourself)

maybe once you've processed a message, you resend it to a different queue, just in case?

That's actually a neat idea. Since we don't have to pay for number of items in a queue, I am thinking we could create a dummy queue and just handball all incoming messages to it for safekeeping (at least for 14 days)?!?

Re: On SQS

#62

Post by cyberferret »

oceanbreeze83 wrote: post #51

is SNS + SQS a reasonable solution for realtime irc style topic chatrooms?

There is also a size limit on the messages sent via SQS - 256Kb IIRC. Might be OK for small text based messaging, but if you want to talk attachments and other payloads then you have to hook it up to other services like S3 etc. and then it becomes complicated.

Also, as another respondent replied - there is no real deliverability guarantee, although there are certain ways to handle that within SQS.

Re: On SQS

#63

Post by actuator »

ramraj07 wrote: post #56
plasma wrote: post #31

Earlier quoted context omitted.

We use Redis as a job queue and its great; the only limitation is being sometimes concerned about job queue size due to memory limits of the Redis server itself.

Always hear using redis etc for job queues, what do these "job queues" entail? I've been an amateur and have used postgres tables as queues .. am I not being efficient?

Well, they are just a mechanism to push jobs between different components where you treat Redis as a message queue. In Redis you can implement one using Lists, Sorted Sets, Stream or Pub/Sub; though they are commonly implemented through the first two. Though if you end up using first two, to guarantee at least once semantics you need to take care of acknowledgement in a not so pleasant way.

You can replace Redis with pretty much any message queue though like RabbitMQ which has a better consumption story. The main advantage of using any one of these would be the throughout you can achieve and it decouples your database at that point which a lot of people prefer.

Re: On SQS

#64

Post by actuator »

fsaintjacques wrote: post #58
ramraj07 wrote: post #56

Earlier quoted context omitted.

Always hear using redis etc for job queues, what do these "job queues" entail? I've been an amateur and have used postgres tables as queues .. am I not being efficient?

Tables as queues is perfectly legit, one less component to worry about.

One more advantage it gives is you can have pretty easy job guarantees through the database transactions.

Re: On SQS

#65

Post by actuator »

hexene wrote: post #41

One downside of SQS is that it doesn't support fan-out, for eg. S3->SQS->multiple consumers. The recommendation instead seems to be to first push to SNS, and then hookup SQS/other consumers to it. Kinesis/Kafka would appear to be better suited for this (since they support fan-out like SNS and are pull-based like SQS), but aren't as well supported as SNS/SQS (you can't push S3 events directly to Kinesis for eg.) Can s…

Generally they do GA in the next ReInvent from when as service is announced, so probably by end of year. But I won't be sure on MSK. It is extremely limited right now, last time I checked their API had no way for even changing the number of nodes.

Re: On SQS

#66

Post by StavrosK »

fsaintjacques wrote: post #58
ramraj07 wrote: post #56

Earlier quoted context omitted.

Always hear using redis etc for job queues, what do these "job queues" entail? I've been an amateur and have used postgres tables as queues .. am I not being efficient?

Tables as queues is perfectly legit, one less component to worry about.

Does anyone have any rough performance numbers on each approach? I'll only use DB tables when I don't want to deploy Redis (which I usually do, as a cache or whatever), and I'll use RabbitMQ for more "serious" projects. On small projects, I'll even forgo Redis and use the DB as a cache too, which works pretty well.

What do y'all do?

Re: On SQS

#67

Post by ejcx »

Way to go Rick! You know you've made it when people write blog posts about your hot takes on Twitter

Re: On SQS

#68

Post by staticassertion »

hexene wrote: post #41

One downside of SQS is that it doesn't support fan-out, for eg. S3->SQS->multiple consumers. The recommendation instead seems to be to first push to SNS, and then hookup SQS/other consumers to it. Kinesis/Kafka would appear to be better suited for this (since they support fan-out like SNS and are pull-based like SQS), but aren't as well supported as SNS/SQS (you can't push S3 events directly to Kinesis for eg.) Can s…

I do S3 -> SNS -> SQS. I don't see why I would use Kinesis instead. The SNS bit is totally invisible to the consumers (you can even tell SNS not to wrap the inner message with the SNS boilerplate), downstream consumers just know they have to listen to a queue.

I don't see a downside to this approach. Perhaps some increased latency?

Re: On SQS

#69

Post by cies »

nkozyra wrote: post #57
tybit wrote: post #38

Earlier quoted context omitted.

If you’re message consumer isn’t idempotent then no MQ can help you. Exactly once delivery is impossible other than with at least once delivery and an idempotent consumer. https://bravenewgeek.com/tag/amazon-sqs/

> Exactly once delivery is impossible other than with at least once delivery Can you explain this? Don't many applications deliver once and only once via locking? It's obviously easier as an application developer to say "I will only get this once" and accept losing messages than dealing with idempotence particularly in distributed services.

This is what I mean.

Re: On SQS

#70

Post by zackbloom »

nkozyra wrote: post #57
tybit wrote: post #38

Earlier quoted context omitted.

If you’re message consumer isn’t idempotent then no MQ can help you. Exactly once delivery is impossible other than with at least once delivery and an idempotent consumer. https://bravenewgeek.com/tag/amazon-sqs/

> Exactly once delivery is impossible other than with at least once delivery Can you explain this? Don't many applications deliver once and only once via locking? It's obviously easier as an application developer to say "I will only get this once" and accept losing messages than dealing with idempotence particularly in distributed services.

What if your server dies right as it's about to release that lock and mark the job done? How will you ever know if it was completed or not?
Post reply on HN