Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Comparing Message Queue Architectures on AWS

tech.forter.com

Post reply on HN
Print view View original

11–20 of 44 posts

Re: Comparing Message Queue Architectures on AWS

#11

Post by itaifrenkel »

chwahoo wrote: post #9
itaifrenkel wrote: post #2

OP here... I would be happy to discuss any comments you have.

Out of curiosity, what tool did you use to sketch the various architectures for your post?

I used Lucid Charts. It felt easy to work with.

https://www.lucidchart.com/ http://aws.amazon.com/architecture/icons/

Re: Comparing Message Queue Architectures on AWS

#12

Post by pan69 »

itaifrenkel wrote: post #5
pan69 wrote: post #4

Earlier quoted context omitted.

For RabbitMQ you mention: No message delivery guarantee in face of RabbitMQ server failure. Wouldn't that be solved by using a persistent queue and high availability clustering (i.e. the queue is duplicated over N servers)?

Your are correct. The sentence you referenced is in the section that describes RabbitMQ without clustering. The next section talks about the tradeoffs of RabbitMQ clustering.

OK. Fair enough. But that's merely a configuration choice and not really a shortcoming of RabbitMQ and even without a cluster, persistent queues are at your disposal, it's just that a HA cluster would be able to give you more of a guarantee.

Really enjoyed reading your post by the way...

Re: Comparing Message Queue Architectures on AWS

#13

Post by tylertreat »

itaifrenkel wrote: post #10
tylertreat wrote: post #7

Really nice write up. I'm curious if you guys have done any extensive throughput and latency benchmarking? I saw the note saying "ballpark figures." FWIW, I've been working on a framework for empirically testing queue performance for scaled-up, distributed deployments ( https://github.com/tylertreat/Flotilla ). Haven't gotten around to adding support for AWS services yet, but would be interesting to see how they comp…

Do you have a document showing the results of the Flotilla tests ... how beanstalkd compares with RabbitMQ for example?

Not yet. The blog post linked in the readme there provides some more background on the motivation behind the project.

I'm hoping to do some in-depth analysis of several brokers at some point, but I want to get the benchmark instrumentation right first.

Re: Comparing Message Queue Architectures on AWS

#14

Post by doublerebel »

itaifrenkel wrote: post #6
papercruncher wrote: post #3

Earlier quoted context omitted.

I know it's not the core topic of the article, but would love a deeper dive into your Kinesis implementation. What is your API & processing servers written in? How easy was it to pump events into Kinesis? What does your event processing architecture look like with respect to Kinesis? Where does the event data end up? RedShift?

The event stream processing is probably worthy of another blog post, but to be brief... we try to align all of our API servers (or event dispatchers) to be nodejs and have all of the processing in Storm (java). But reality is a bit more complicated. Specifically the API server pushing events into Kinesis is in python. It is a refactoring of some code that was written in the first days of the company. At the time it w…

I'm also very interested in the details of your event streaming. Are you using Node.js streams at all? I am a huge fan of Node streams but see few examples of them used in large-scale production, especially when combined with other architectures (as in your Java Storm example).

Re: Comparing Message Queue Architectures on AWS

#15

Post by itaifrenkel »

doublerebel wrote: post #14
itaifrenkel wrote: post #6

Earlier quoted context omitted.

The event stream processing is probably worthy of another blog post, but to be brief... we try to align all of our API servers (or event dispatchers) to be nodejs and have all of the processing in Storm (java). But reality is a bit more complicated. Specifically the API server pushing events into Kinesis is in python. It is a refactoring of some code that was written in the first days of the company. At the time it w…

I'm also very interested in the details of your event streaming. Are you using Node.js streams at all? I am a huge fan of Node streams but see few examples of them used in large-scale production, especially when combined with other architectures (as in your Java Storm example).

We are currently not using nodejs streams. The nodejs components are the event dispatchers placing the events in a queue, they do not include much even processing. Most of the event processing is done in Apache Storm. We have also contributed the Storm-Nodejs integration, but the nodejs Storm components are not using streams either. I have tried using streams to compress the information between the Java parent process in Storm, and the nodejs child process, but compression wasn't a good tradeoff, so even that little bit didn't get into production.

Re: Comparing Message Queue Architectures on AWS

#16

Post by itaifrenkel »

pan69 wrote: post #12
itaifrenkel wrote: post #5

Earlier quoted context omitted.

Your are correct. The sentence you referenced is in the section that describes RabbitMQ without clustering. The next section talks about the tradeoffs of RabbitMQ clustering.

OK. Fair enough. But that's merely a configuration choice and not really a shortcoming of RabbitMQ and even without a cluster, persistent queues are at your disposal, it's just that a HA cluster would be able to give you more of a guarantee. Really enjoyed reading your post by the way...

thanks. The Pros/Cons were supposed to refer to the complete architecture setup, and not to the Q component by itself. I can see how that could have been read that way...

Re: Comparing Message Queue Architectures on AWS

#18

Post by ekimekim »

pan69 wrote: post #4
itaifrenkel wrote: post #2

OP here... I would be happy to discuss any comments you have.

For RabbitMQ you mention: No message delivery guarantee in face of RabbitMQ server failure. Wouldn't that be solved by using a persistent queue and high availability clustering (i.e. the queue is duplicated over N servers)?

Another very minor detail, but you mention that RabbitMQ can do priority based on multiple queues. While that's certainly a fine way to do it, it's worth noting that AMQP also supports per-message priorities within one queue:

http://www.rabbitmq.com/amqp-0-9-1-reference.html#class.basi...

> The server MUST implement at least 2 priority levels for basic messages, where priorities 0-4 and 5-9 are treated as two distinct levels.

Depending on your client, it may be difficult to prioritize consumption of one queue over another, so this solution could be preferred.

Re: Comparing Message Queue Architectures on AWS

#19

Post by sargun »

yashinovsky wrote: post #17

Have you also evaluated Kafka? as it is a common choice feeding Storm .

Kafka is a log, not a queue. They have every so slightly different semantics, but logs have HOL blocking per partition for processing messages because they use a watermark, compared to queues, which track on a per-message basis.
Post reply on HN