Kafka as an Antipattern
joshaustin.tech
Kafka as an Antipattern
1–10 of 102 posts
Re: Kafka as an Antipattern
#2Seems like a lot of what I read about Kafka really makes it sound like using it is quite, well, Kafkaesque
Why do so many engineers end up having such a struggle with an event sourcing system, yet the system itself remains highly popular I don’t know. I theorize the following:
- Its flexible enough to do things like receive events (messages) and sending downstream events from those received
- it can ingest events fast. A well tuned instance is very fast and can handle a lot of volume
- it often allows a middleware log point (or other work types) for things happening throughout your whole system
Perhaps all of these things (and more) are hard to attain using a different technology
Re: Kafka as an Antipattern
#3I agree that Kafka is alot of machinery for a fairly limited gain. but I really dislike this whole notion of 'antipattern', as if we can look at thing and assign a decontextualized thumbs up or down. that building systems is just a matter of assembling the right patterns, and avoiding the antipatterns.
Re: Kafka as an Antipattern
#4The anti-pattern here isn't Kafka, it's using Kafka for 7,500 messages/day. Making your whole system asynchronous for that level of load is the textbook definition of over-engineering.
Re: Kafka as an Antipattern
#5I agree that Kafka is alot of machinery for a fairly limited gain. but I really dislike this whole notion of 'antipattern', as if we can look at thing and assign a decontextualized thumbs up or down. that building systems is just a matter of assembling the right patterns, and avoiding the antipatterns.
Sometimes a technology or pattern can be poisonous in a very specific way that warrants a label: when they're most alluring to those least equipped to leverage them.
Just like microservices, you cross the activation energy to want Kafka very easily because it's appealing on résumés, sounds like a hedge against scale,etc.
But there's a huge asymmetry in understanding the drawbacks to them. When you spin up these systems, the drawbacks don't hit you immediately, it feels like they're solving the problem you had, and it's not until you've invested immense amounts of sweat capital (and literal capital) that you discover how badly you screwed up.
You need some way to match that low effort value prop with a low friction warning: this is not a panacea for your problems. It only seems simple, it's not simple, it will hurt you unless know it will hurt you and simply have the resources and scale to play through that hurt.
—
To me that warning is what's implied by "antipattern", it's not never use this, it's never use this unless you know why you should never use this.
Re: Kafka as an Antipattern
#6I agree that Kafka is alot of machinery for a fairly limited gain. but I really dislike this whole notion of 'antipattern', as if we can look at thing and assign a decontextualized thumbs up or down. that building systems is just a matter of assembling the right patterns, and avoiding the antipatterns.
I agree completely. Kafka, and event based architectures, are highly overused, but are also the right thing sometimes. It's FAR easier to manage an architecture where systems call into the source of truth for a given piece of data via APIs, and you should only switch to an event model if you truly need to.
Re: Kafka as an Antipattern
#7Seems like a lot of what I read about Kafka really makes it sound like using it is quite, well, Kafkaesque Why do so many engineers end up having such a struggle with an event sourcing system, yet the system itself remains highly popular I don’t know. I theorize the following: - Its flexible enough to do things like receive events (messages) and sending downstream events from those received - it can ingest events fas…
The simple approach mentioned in the article gets annoying if you have micro-services that don't share a DB. You could add a shared DB or a nosql DB but then you may as well just add Kafka. Of course the key question then shouldn't be kafka or not-kafka but if you over engineered on micro-services.
Re: Kafka as an Antipattern
#8Deceptively deep topic, and yet such a superficial post.
Re: Kafka as an Antipattern
#9It seems a lot of the complaints weren't about kafka itself, but rather seemed to stem from internal communication problems. Custom kafka message headers could very well be custom http headers, and the problem is the same. Kafka is just coincidental.
Looking at the volume though, kafka is overkill. They most likely could have just used the database and reaped the benefits of doing everything in a single transaction, with easier row level locking. The post acknowledges this.
I do think it highlights the need for a small scale kafka, though. It's conceptually great to have everything work off of logs, but kafka does add a non trivial operational burden.
Re: Kafka as an Antipattern
#10The anti-pattern here isn't Kafka, it's using Kafka for 7,500 messages/day. Making your whole system asynchronous for that level of load is the textbook definition of over-engineering.
Yes. For perspective, that's about one message every ten seconds.