21–30 of 102 posts
Earlier quoted context omitted.
Yes. For perspective, that's about one message every ten seconds.
Sorry for the naivety/not obvious from the comments: is that too much or too little? (I've used RabbitMQ much more than Kafka.)
I agree with the author to an extent - kafka is overly complex for low traffic systems. It can however be a godsend in specific cases, but what those cases are can be hard to pinpoint.
The anti-pattern here isn't Kafka, it's using Kafka for 7,500 messages/day. Making your whole system asynchronous for that level of load is the textbook definition of over-engineering.
For example, they could have one service using CDC (his DB source connector) to propagate state out to a bunch of other systems and not know which systems are subscribing for changes.
An organization could have many such systems propagating state out to many other systems, using a single distributed log system.
Earlier quoted context omitted.
Yes. For perspective, that's about one message every ten seconds.
Sorry for the naivety/not obvious from the comments: is that too much or too little? (I've used RabbitMQ much more than Kafka.)
I'm currently maintaining a system that uses Redis as a message broker and at a rate of ~10 messages/second it marginally makes sense.
I agree that Kafka is alot of machinery for a fairly limited gain. but I really dislike this whole notion of 'antipattern', as if we can look at thing and assign a decontextualized thumbs up or down. that building systems is just a matter of assembling the right patterns, and avoiding the antipatterns.
I agree completely. Kafka, and event based architectures, are highly overused, but are also the right thing sometimes. It's FAR easier to manage an architecture where systems call into the source of truth for a given piece of data via APIs, and you should only switch to an event model if you truly need to.
I don’t want to be dismissive, but I often see these big numbers being posted, like “14M messages” or “thousands of messages” and then adding “per year” or something, which brings it down to toy level load.
Even the first “serious” example is about “thousands of messages” per minute. Say 5K a minute. That’s 83 per second, say 100. That seems .. not that interesting?
Am I being too dismissive? I think I am. I am not seeing something right. Can anybody say something to widen my perspective?
So in our case it's clearly not an anti-pattern, but the right tool for the job.
Also Avro is great but like Kafka you were probably holding it wrong.
I do prefer Protobuf in these particular scenarios as Protobufs features more closely align with svc svc RPC style communication patterns while Avro shines in longer lived scenarios where messages need to be archived and you don't want to come up with your own framing for your Protobufs.
This is because Avro has the Avro Object Container Format which is a simple block based file format, which allows for relatively efficient seeking, block based compression etc. Protobuf unfortunately doesn't define any standard file formats or even wire protocol framing. If you need to do more than simply store and scan/read in bulk you might want to use Parquet instead though.
Reading this blog post was probably a waste of my time, hopefully this comment actually helps someone though.