Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

On SQS

tbray.org

Post reply on HN
Print view View original

111–120 of 229 posts

Re: On SQS

#111

Post by eropple »

polskibus wrote: post #88

Does anyone know a good, low overhead out-of-process message queue, that's lightweight enough that it can be useful for communicating between processes on the same machine, but if necessary it can scale beyond it? In case of a single-machine product that comprises of several services, a message queue can sometimes be useful for pull model, but adding RabbitMQ to the stack makes installation and ops much more complex…

"Low overhead"? Redis.

Reliable and durable? Emphatically not Redis, and at that point you probably want to just start looking at SQS.

(I've had good luck shipping products as docker-compose files, these days. Even to fairly backwards clients.)

Re: On SQS

#112

Post by mabbo »

A long time ago, as new-ish developer, I was building a system that needed to take inputs, then run "pass/fail/wait and try again later" until timeout or completion. This wasn't mission-critical stuff, mind you, so a lost message would annoy someone but not cause any actual harm.

As I was figuring out how to setup a datastore, query it for running workflows and all that jazz, I happened upon an interesting SQS feature: Post with Delay.

And so, the system has no database. Instead, when new work arrives it posts the details of the work to be done to SQS. All hosts in the fleet are polling SQS for messages. When they receive one, they do the checks and if the process isn't complete they repost the message again with a 5-minute delay. In 5 minutes, a host in the fleet will receive the message and try again. The process continues as long as it needs to.

Looking back, part of me now is horrified at this design. But: that system now has thousands of users and continues to scale really well. Data loss is very rare. Costs are low. No datastore to manage. SQS is just really darned neat because it can do things like that.

Re: On SQS

#113

Post by eropple »

gizzlon wrote: post #108
polskibus wrote: post #88

Does anyone know a good, low overhead out-of-process message queue, that's lightweight enough that it can be useful for communicating between processes on the same machine, but if necessary it can scale beyond it? In case of a single-machine product that comprises of several services, a message queue can sometimes be useful for pull model, but adding RabbitMQ to the stack makes installation and ops much more complex…

Guess it depends on the definition of "queue". Potentials: - https://nsq.io/ - https://nats.io/

I am curious what definition of "queue" fits NSQ. It's a distributed messaging platform.

Re: On SQS

#114

Post by ramraj07 »

StavrosK wrote: post #66
fsaintjacques wrote: post #58

Earlier quoted context omitted.

Tables as queues is perfectly legit, one less component to worry about.

Does anyone have any rough performance numbers on each approach? I'll only use DB tables when I don't want to deploy Redis (which I usually do, as a cache or whatever), and I'll use RabbitMQ for more "serious" projects. On small projects, I'll even forgo Redis and use the DB as a cache too, which works pretty well. What do y'all do?

My job queue requirements have been very modest (less than ten jobs per hour) and no more than 2 or 3 workers. I also generally have redis deployed but it never occurred to me to use it given it's "ephemeral" nature - always remained paranoid that services on the cloud should always be assumed to go down at any time

Re: On SQS

#115

Post by staticassertion »

skybrian wrote: post #103
staticassertion wrote: post #80

Earlier quoted context omitted.

That's exactly how you do it. To me, it's the opposite of brittle - every consumer owns a queue, and is isolated from all other consumers. Clients are totally unaware of other systems, and there's no shared resource under contention.

Hmm. It seems a bit awkward if you have a variable number of consumers?

I haven't run into that myself, when would you want a variable number of consumers? Usually the way I have it is that a service, which is itself a cluster of processes, owns one queue. For example, an AWS Lambda triggered by that queue.

Then any new lambdas or other services that want to subscribe to messages will have another queue, and another, etc.

I haven't had a case where I had service groups coming up and down, I'm struggling to think of a use case.

Re: On SQS

#116

Post by StavrosK »

ramraj07 wrote: post #114
StavrosK wrote: post #66

Earlier quoted context omitted.

Does anyone have any rough performance numbers on each approach? I'll only use DB tables when I don't want to deploy Redis (which I usually do, as a cache or whatever), and I'll use RabbitMQ for more "serious" projects. On small projects, I'll even forgo Redis and use the DB as a cache too, which works pretty well. What do y'all do?

My job queue requirements have been very modest (less than ten jobs per hour) and no more than 2 or 3 workers. I also generally have redis deployed but it never occurred to me to use it given it's "ephemeral" nature - always remained paranoid that services on the cloud should always be assumed to go down at any time

I generally agree, but I have Redis as part of my main stack now, and it's been pretty solid. I'm fairly distrustful of "cloud" services, though, so I run my stuff on Hetzner and manage them myself. For my revenues, where a day's downtime might cost me $5/mo in lost revenue, I think it makes more sense.

Re: On SQS

#117

Post by stellar678 »

staticassertion wrote: post #80
hexene wrote: post #75

Earlier quoted context omitted.

If you wanted multiple pull-based consumers for the stream, wouldn't you need a separate SQS queue per consumer, with each queue hooked up to SNS? Perhaps I'm mistaken, but that seems brittle to me. With Kinesis/Kafka, you only need to register a new appName/consumer group on the single queue for fan-out. Plus, both are FIFO by default, at least within a partition.

That's exactly how you do it. To me, it's the opposite of brittle - every consumer owns a queue, and is isolated from all other consumers. Clients are totally unaware of other systems, and there's no shared resource under contention.

I feel like the create/delete queue semantics hint that a queue should be a long-lived thing that consumers are configured to connect to. When I saw suggestions to have one queue per consumer and have that consumer create/delete the queue during its execution lifecycle, the idea of one-queue-per-consumer started making more sense to me.

Re: On SQS

#118

Post by polskibus »

eropple wrote: post #111
polskibus wrote: post #88

Does anyone know a good, low overhead out-of-process message queue, that's lightweight enough that it can be useful for communicating between processes on the same machine, but if necessary it can scale beyond it? In case of a single-machine product that comprises of several services, a message queue can sometimes be useful for pull model, but adding RabbitMQ to the stack makes installation and ops much more complex…

"Low overhead"? Redis. Reliable and durable? Emphatically not Redis, and at that point you probably want to just start looking at SQS. (I've had good luck shipping products as docker-compose files, these days. Even to fairly backwards clients.)

I guess I didn't make it clear. Solution has to work in-house / in private cloud.

Re: On SQS

#119

Post by tjanks »

eropple wrote: post #111
polskibus wrote: post #88

Does anyone know a good, low overhead out-of-process message queue, that's lightweight enough that it can be useful for communicating between processes on the same machine, but if necessary it can scale beyond it? In case of a single-machine product that comprises of several services, a message queue can sometimes be useful for pull model, but adding RabbitMQ to the stack makes installation and ops much more complex…

"Low overhead"? Redis. Reliable and durable? Emphatically not Redis, and at that point you probably want to just start looking at SQS. (I've had good luck shipping products as docker-compose files, these days. Even to fairly backwards clients.)

Can you explain why Redis is not durable? Looking into it for a project but this comment worries me.

Re: On SQS

#120

Post by staticassertion »

stellar678 wrote: post #117
staticassertion wrote: post #80

Earlier quoted context omitted.

That's exactly how you do it. To me, it's the opposite of brittle - every consumer owns a queue, and is isolated from all other consumers. Clients are totally unaware of other systems, and there's no shared resource under contention.

I feel like the create/delete queue semantics hint that a queue should be a long-lived thing that consumers are configured to connect to. When I saw suggestions to have one queue per consumer and have that consumer create/delete the queue during its execution lifecycle, the idea of one-queue-per-consumer started making more sense to me.

I think the word "Consumer" here is "Consumer Group".

For example, an AWS Lambda triggered from SQS will lead to thousands of executions, each lambda pulling a new message from SQS.

But another consumer group, maybe a group of load balanced EC2 instances, will have a separate queue.

In general, I don't know of cases where you want a single message duplicated across a variable number of consumer groups - services are not ephemeral things, even if their underlying processes are. You don't build a service, deploy it, and then tear it down the next day and throw away the code.

Post reply on HN