The first architecture using two ELBs has a long list of cons, most of which are solved by using HAProxy as your internal load balancer. May want to considering adding that as another option on the matrix.
Comparing Message Queue Architectures on AWS
21–30 of 44 posts
Re: Comparing Message Queue Architectures on AWS
#22Have you also evaluated Kafka? as it is a common choice feeding Storm .
Kafka is a log, not a queue. They have every so slightly different semantics, but logs have HOL blocking per partition for processing messages because they use a watermark, compared to queues, which track on a per-message basis.
The article already includes a couple of "not queues" that they use for event stream processing, Kinesis being one of them. Kinesis consumers also maintain a watermark to keep track of how far they've read in a stream. Considering that, a Kafka comparison seems like a reasonable ask.
Re: Comparing Message Queue Architectures on AWS
#23Have you also evaluated Kafka? as it is a common choice feeding Storm .
Sure we have. Kinesis was considered because it was inspired by Kafka, and Kafka is known to work well. The thing is that any distributed stateful service, by design, requires DevOps experience and takes time to manage. We try to stay lean at this stage, and chose Kinesis over Kafka becuase it took very little time to setup, and has no "maintenance costs". Had we have chosen Kafka, we would have enjoyed much shorter latency times.
Re: Comparing Message Queue Architectures on AWS
#24Surprised they didn't mention IronMQ: http://www.iron.io/mq It's not perfect but pretty damn close. And it run on AWS et al.
You are right about IronMQ, they are said to have a very strong product. For the record, we haven't actually checked redislabs and cloudamqp either, but since they are based on opensource that is mentioned in the blog post, I added them to the notes section. I couldn't say anything intelligent about IronMQ since it is not based on an opensource offering, and so other than actually trying it, I cannot tell its pros/cons.
Re: Comparing Message Queue Architectures on AWS
#25The first architecture using two ELBs has a long list of cons, most of which are solved by using HAProxy as your internal load balancer. May want to considering adding that as another option on the matrix.
Interesting. Could you please elaborate?
Re: Comparing Message Queue Architectures on AWS
#26Out of curiosity did you look at gearman? I really like the ability to coalesce identical jobs.
Re: Comparing Message Queue Architectures on AWS
#27Earlier quoted context omitted.
For RabbitMQ you mention: No message delivery guarantee in face of RabbitMQ server failure. Wouldn't that be solved by using a persistent queue and high availability clustering (i.e. the queue is duplicated over N servers)?
Another very minor detail, but you mention that RabbitMQ can do priority based on multiple queues. While that's certainly a fine way to do it, it's worth noting that AMQP also supports per-message priorities within one queue: http://www.rabbitmq.com/amqp-0-9-1-reference.html#class.basi... > The server MUST implement at least 2 priority levels for basic messages, where priorities 0-4 and 5-9 are treated as two distinc…
Thanks! I'll update the post. A quick search found this RabbitMQ plugin https://github.com/rabbitmq/rabbitmq-priority-queue , which specifies: "In contrast to the AMQP spec, RabbitMQ queues by default do not support priorities. When creating priority queues using this plugin, you can specify as many priority levels as you like."
Re: Comparing Message Queue Architectures on AWS
#28Out of curiosity did you look at gearman? I really like the ability to coalesce identical jobs.
Not yet. gearman is a job service which sits on top of a queue (much like python celery). We did not need all that, given the fact we are using Storm.
Re: Comparing Message Queue Architectures on AWS
#29I'm surprised that SNS wasn't mentioned here, being an AWS queueing product.
Re: Comparing Message Queue Architectures on AWS
#30I'm surprised that SNS wasn't mentioned here, being an AWS queueing product.
Simple Queuing Service was mentioned (SQS)
Simple Notification Service (SNS) was not. Did you mean SNS or SQS ?