How to use RabbitMQ in service integration
erlang-solutions.com
How to use RabbitMQ in service integration
1–10 of 44 posts
Re: How to use RabbitMQ in service integration
#2I find that one of the major drawbacks of RabbitMQ is how much CPU it requires to run efficiently. If you add clustering on top of it it can easily become the biggest CPU hog in a containerized cluster. However it does what it says, and it does quite well.
Re: How to use RabbitMQ in service integration
#3When I proposed to use RabbitMQ for service integration my colleagues didn't agree to go with it because it "won't scale" and AWS doesn't provide a managed service. Because of an AWS fetish and awful development experience the product won't launch anytime soon.
Re: How to use RabbitMQ in service integration
#4When I proposed to use RabbitMQ for service integration my colleagues didn't agree to go with it because it "won't scale" and AWS doesn't provide a managed service. Because of an AWS fetish and awful development experience the product won't launch anytime soon.
AWS released the option to have a managed RabbitMQ broker under Amazon MQ back in November (2020)
https://aws.amazon.com/about-aws/whats-new/2020/11/announcin...
Re: How to use RabbitMQ in service integration
#5When I proposed to use RabbitMQ for service integration my colleagues didn't agree to go with it because it "won't scale" and AWS doesn't provide a managed service. Because of an AWS fetish and awful development experience the product won't launch anytime soon.
At a previous employer I had what I thought was close to a slam dunk case for messaging with RabbitMQ. While I'm not the most experienced person in developing distributed messaging systems, and I expected technical objections, the disheartening response, and ultimate reason for rejection, was simply organizational inertia. In short, "you can't use RabbitMQ and messaging for that because we don't use messaging here". I'm sure there was some history that nobody mentioned or some lost institutional knowledge. The "grandma's ham" of systems design.
Re: How to use RabbitMQ in service integration
#6When I proposed to use RabbitMQ for service integration my colleagues didn't agree to go with it because it "won't scale" and AWS doesn't provide a managed service. Because of an AWS fetish and awful development experience the product won't launch anytime soon.
At a previous employer I had what I thought was close to a slam dunk case for messaging with RabbitMQ. While I'm not the most experienced person in developing distributed messaging systems, and I expected technical objections, the disheartening response, and ultimate reason for rejection, was simply organizational inertia. In short, "you can't use RabbitMQ and messaging for that because we don't use messaging here".…
I can sympathize with your colleagues. Developing the distributed messaging system is not where the bulk of the cost of running such a system goes - it's the maintenance and debugging that soak up time, money, and tears.
While my past experiences with RabbitMQ in production have been stellar, I can see why a team would be hesitant to add this complexity to their infrastructure.
Re: How to use RabbitMQ in service integration
#7I find that one of the major drawbacks of RabbitMQ is how much CPU it requires to run efficiently. If you add clustering on top of it it can easily become the biggest CPU hog in a containerized cluster. However it does what it says, and it does quite well.
Curious: what kind of load are you throwing at RabbitMQ?
Re: How to use RabbitMQ in service integration
#8When I proposed to use RabbitMQ for service integration my colleagues didn't agree to go with it because it "won't scale" and AWS doesn't provide a managed service. Because of an AWS fetish and awful development experience the product won't launch anytime soon.
Why not use SQS if you're bound to AWS requirement?
I've found RabbitMQ difficult operationally, and full of footguns (that can make you lose data), so I'm not sure why would you want to use it if you don't already have to.
Re: How to use RabbitMQ in service integration
#9Earlier quoted context omitted.
At a previous employer I had what I thought was close to a slam dunk case for messaging with RabbitMQ. While I'm not the most experienced person in developing distributed messaging systems, and I expected technical objections, the disheartening response, and ultimate reason for rejection, was simply organizational inertia. In short, "you can't use RabbitMQ and messaging for that because we don't use messaging here".…
I can sympathize with your colleagues. Developing the distributed messaging system is not where the bulk of the cost of running such a system goes - it's the maintenance and debugging that soak up time, money, and tears. While my past experiences with RabbitMQ in production have been stellar, I can see why a team would be hesitant to add this complexity to their infrastructure.
Oh here's the thing though: the company was already using RabbitMQ, as well as Kafka (for other things, mostly data science), and it was all deployed in Kubernetes and replicated across three data centers, including one in Amsterdam. The complexity was already there, but only for a very restricted set of applications.
Re: How to use RabbitMQ in service integration
#10When I proposed to use RabbitMQ for service integration my colleagues didn't agree to go with it because it "won't scale" and AWS doesn't provide a managed service. Because of an AWS fetish and awful development experience the product won't launch anytime soon.
Why not use SQS if you're bound to AWS requirement? I've found RabbitMQ difficult operationally, and full of footguns (that can make you lose data), so I'm not sure why would you want to use it if you don't already have to.
Not SQS, but Google PubSub: everything was working allright, until one day messages started arriving with horrible latency and vital parts of our service went down. We spent close to two weeks debugging our infrastructure, because _sure it could not be PubSub_. Well, _it was_. I guess nothing too big, a small bit or tweak that changed in their service that didn't propagate to the client library we were using, or god knows what. Now we run RabbitMQ for everything; at least we control our versions and we can confidently blame ourselves when something happens. But RabbitMQ, standalone and clustered, has served us well across different systems for about fifteen years.