How to Choose the Right Message Queue for Your Microservices
1–7 of 7 posts
Re: How to Choose the Right Message Queue for Your Microservices
#2> A synchronous approach is more error-prone, harder to debug, and harder to recover. Many industry professionals pass this option over.
That's the complete opposite of my experience.
Re: How to Choose the Right Message Queue for Your Microservices
#3On Kafka...
Message retrieval is push? Nope, definitely pull.
Payload is text or avro? Neither, it's just bytes, how you serialise your payload data is up to you.
> Client libraries might have a hard-time of keeping up with ever-changing protocol specification. Therefore upgrades might also be challenging.
Kafka maintains backwards compatibility for old clients for a very long time. They've only just (Kafka 3.0) deprecated the protocol released in Kafka 0.8.
Choose a client library that has active maintainers, you'll be golden. If your client lib of choose
> Message size limit is 1MB.
It defaults to this, but it can be set much higher if needed.
Lastly - Kafka's not really a message queue. Apache Pulsar offers far more MQ like semantics though.
Re: How to Choose the Right Message Queue for Your Microservices
#4A good old fashioned database queue usually does it for me. It's easy for the DBA to monitor, Ops to run reports on, not a new program, nor another service to pay for and doesn't add additional proprietary lock-in. Plus, devs already have it setup and running.
Re: How to Choose the Right Message Queue for Your Microservices
#5On Kafka... Message retrieval is push? Nope, definitely pull. Payload is text or avro? Neither, it's just bytes, how you serialise your payload data is up to you. > Client libraries might have a hard-time of keeping up with ever-changing protocol specification. Therefore upgrades might also be challenging. Kafka maintains backwards compatibility for old clients for a very long time. They've only just (Kafka 3.0) depr…
>> Message size limit is 1MB.
It defaults to this, but it can be set much higher if needed.
What is it? I was surprised to hear it was this low.
Re: How to Choose the Right Message Queue for Your Microservices
#6On Kafka... Message retrieval is push? Nope, definitely pull. Payload is text or avro? Neither, it's just bytes, how you serialise your payload data is up to you. > Client libraries might have a hard-time of keeping up with ever-changing protocol specification. Therefore upgrades might also be challenging. Kafka maintains backwards compatibility for old clients for a very long time. They've only just (Kafka 3.0) depr…
>> Message size limit is 1MB. It defaults to this, but it can be set much higher if needed. What is it? I was surprised to hear it was this low.
It's set to 1MB (and some additional bytes for log overhead) because it was found (by LinkedIn? I think?) to be a reasonable default for throughput. Technically you can set it up to anywhere near MAX_INT bytes, would be interesting to see what happens when you send a 1GiB message. :D You'd most likely need to tune other properties depending on how big you're going.
And message size, just to be clear, relates to a message from a producer which can contain 1 to N records, often compressed to reduce network contention. The 1MB limit applies to the bytes which are received by the broker, so you can send batches of records which are >1MB if you're compressing them.
Compression imposes some CPU cost on producers and consumers, but unless you've specifically configured a topic to use a mismatching compression algorithm (this defaults to "just accept whatever algorithm the producer" used), there's no cost on the broker, it's written as is, and then consumers pull it still compressed.
Re: How to Choose the Right Message Queue for Your Microservices
#7> A synchronous approach is more error-prone, harder to debug, and harder to recover. Many industry professionals pass this option over. That's the complete opposite of my experience.
Yeah I was also surprised by that line. I’ll give the author “harder to recover” but definitely not “harder to debug”