Comparing Message Queue Architectures on AWS
tech.forter.com
Comparing Message Queue Architectures on AWS
1–10 of 44 posts
Re: Comparing Message Queue Architectures on AWS
#2OP here... I would be happy to discuss any comments you have.
Re: Comparing Message Queue Architectures on AWS
#3OP here... I would be happy to discuss any comments you have.
I know it's not the core topic of the article, but would love a deeper dive into your Kinesis implementation. What is your API & processing servers written in? How easy was it to pump events into Kinesis? What does your event processing architecture look like with respect to Kinesis? Where does the event data end up? RedShift?
Re: Comparing Message Queue Architectures on AWS
#4OP here... I would be happy to discuss any comments you have.
For RabbitMQ you mention:
No message delivery guarantee in face of RabbitMQ server failure.
Re: Comparing Message Queue Architectures on AWS
#5OP here... I would be happy to discuss any comments you have.
For RabbitMQ you mention: No message delivery guarantee in face of RabbitMQ server failure. Wouldn't that be solved by using a persistent queue and high availability clustering (i.e. the queue is duplicated over N servers)?
Your are correct. The sentence you referenced is in the section that describes RabbitMQ without clustering. The next section talks about the tradeoffs of RabbitMQ clustering.
Re: Comparing Message Queue Architectures on AWS
#6OP here... I would be happy to discuss any comments you have.
I know it's not the core topic of the article, but would love a deeper dive into your Kinesis implementation. What is your API & processing servers written in? How easy was it to pump events into Kinesis? What does your event processing architecture look like with respect to Kinesis? Where does the event data end up? RedShift?
The event stream processing is probably worthy of another blog post, but to be brief... we try to align all of our API servers (or event dispatchers) to be nodejs and have all of the processing in Storm (java). But reality is a bit more complicated. Specifically the API server pushing events into Kinesis is in python. It is a refactoring of some code that was written in the first days of the company. At the time it was based on multiple processes communicating through files... which didn't scale (as one would expect). The Kinesis implementation was done by a researcher which is a very talented coder but did not have experience with Kinesis. He was up and running after a short whiteboard session and a few hours of coding. We had to tweak a few things in the following days (such as adding more shards for higher read throughput, abandoning Kinesis shards altogether in favor of multiple stream with one shard)
Re: Comparing Message Queue Architectures on AWS
#7Really nice write up. I'm curious if you guys have done any extensive throughput and latency benchmarking? I saw the note saying "ballpark figures."
FWIW, I've been working on a framework for empirically testing queue performance for scaled-up, distributed deployments (https://github.com/tylertreat/Flotilla). Haven't gotten around to adding support for AWS services yet, but would be interesting to see how they compare.
Re: Comparing Message Queue Architectures on AWS
#8Really nice write up. I'm curious if you guys have done any extensive throughput and latency benchmarking? I saw the note saying "ballpark figures." FWIW, I've been working on a framework for empirically testing queue performance for scaled-up, distributed deployments ( https://github.com/tylertreat/Flotilla ). Haven't gotten around to adding support for AWS services yet, but would be interesting to see how they comp…
We haven't done comparative throughput performance. SaaSiness, Low Latency and use case matching was the main concern of choosing the queues. As for throughput, the main concern was for the event stream processing pipeline. Redis and Kinesis are (somewhat) SaaSed on AWS, throughput can scale out horizontally with sharding, and we haven't had any problems since :)
Re: Comparing Message Queue Architectures on AWS
#9OP here... I would be happy to discuss any comments you have.
Out of curiosity, what tool did you use to sketch the various architectures for your post?
Re: Comparing Message Queue Architectures on AWS
#10Really nice write up. I'm curious if you guys have done any extensive throughput and latency benchmarking? I saw the note saying "ballpark figures." FWIW, I've been working on a framework for empirically testing queue performance for scaled-up, distributed deployments ( https://github.com/tylertreat/Flotilla ). Haven't gotten around to adding support for AWS services yet, but would be interesting to see how they comp…
Do you have a document showing the results of the Flotilla tests ... how beanstalkd compares with RabbitMQ for example?