Have you also evaluated Kafka? as it is a common choice feeding Storm .
This is for AWS. Kafka isn't designed to work well in an environment where partitioning occurs with any frequency. AWS is such an environment.
Comparing Message Queue Architectures on AWS
31–40 of 44 posts
Re: Comparing Message Queue Architectures on AWS
#32OP here... I would be happy to discuss any comments you have.
Any reason why you don't send the data directly to the processing servers as a way to minimize complexity? If it fails to reach one, try the other.
When you mention low latency, how low are we talking? ms, seconds, minutes? The reason I am asking is because you can use s3 as an intermediate storage where you ship your compressed logs/events at a rollover interval and the processing servers discover them there. Now of course this only works if latency is not a big deal.
Re: Comparing Message Queue Architectures on AWS
#33Have you also evaluated Kafka? as it is a common choice feeding Storm .
This is for AWS. Kafka isn't designed to work well in an environment where partitioning occurs with any frequency. AWS is such an environment. https://aphyr.com/posts/293-call-me-maybe-kafka
For that matter, RabbbitMQ isn't either. see https://aphyr.com/posts/315-call-me-maybe-rabbitmq .
Re: Comparing Message Queue Architectures on AWS
#34Have you also evaluated Kafka? as it is a common choice feeding Storm .
This is for AWS. Kafka isn't designed to work well in an environment where partitioning occurs with any frequency. AWS is such an environment. https://aphyr.com/posts/293-call-me-maybe-kafka
I always wondered if Amazon would really be open about and go public when a split brain in their network caused one of the hosted data services to loose data. Maybe they just throw money at the hardware and hope for the best.
Re: Comparing Message Queue Architectures on AWS
#35I'm surprised that SNS wasn't mentioned here, being an AWS queueing product.
Simple Queuing Service was mentioned (SQS) Simple Notification Service (SNS) was not. Did you mean SNS or SQS ?
Given that I was surprised that SNS wasn't mentioned, I think I mean SNS, given they are both queuing services.
Re: Comparing Message Queue Architectures on AWS
#36Re: Comparing Message Queue Architectures on AWS
#37You give Redis a yellow rating on prioritization, noting that this can be partially achieved using multiple lists. Wouldn't it be much better to take advantage of Redis's sorted set type?
Re: Comparing Message Queue Architectures on AWS
#38OP here... I would be happy to discuss any comments you have.
Any reason why you don't send the data directly to the processing servers as a way to minimize complexity? If it fails to reach one, try the other. When you mention low latency, how low are we talking? ms, seconds, minutes? The reason I am asking is because you can use s3 as an intermediate storage where you ship your compressed logs/events at a rollover interval and the processing servers discover them there. Now of…
Our first architecture was indeed peer-to-peer where all components discovered each other. It is difficult to isolate failures on this kind of architecture. A reliable queue makes a big difference in zooming in on the root cause of the problem (a queue producer or a queue consumer problem).
Regarding latency - we are aiming for a few ms tops. preferably less. So bulk events using S3 was not considered
Re: Comparing Message Queue Architectures on AWS
#39You give Redis a yellow rating on prioritization, noting that this can be partially achieved using multiple lists. Wouldn't it be much better to take advantage of Redis's sorted set type?
Sorted Set does not have properties of a queue. For example, it does not allow duplicates (different messages with the same priority)
Re: Comparing Message Queue Architectures on AWS
#40Earlier quoted context omitted.
Simple Queuing Service was mentioned (SQS) Simple Notification Service (SNS) was not. Did you mean SNS or SQS ?
Given that I was surprised that SNS wasn't mentioned, I think I mean SNS, given they are both queuing services.
SNS is a push notification service, that focuses on various consumers outside the cloud. It can be used with web hooks (http endpoints) to publish the same message to various web servers. In that case it would guarantees at least once semantics (which means the message is delivered, but sometimes twice), but it does not guarantee FIFO.