Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka as an Antipattern

joshaustin.tech

Post reply on HN
Print view View original

31–40 of 102 posts

Re: Kafka as an Antipattern

#31

Post by quickthrower2 »

no_wizard wrote: post #2

Seems like a lot of what I read about Kafka really makes it sound like using it is quite, well, Kafkaesque Why do so many engineers end up having such a struggle with an event sourcing system, yet the system itself remains highly popular I don’t know. I theorize the following: - Its flexible enough to do things like receive events (messages) and sending downstream events from those received - it can ingest events fas…

- developer tries to get those jobs paying $50k/y more “what kafka, event sourcing and microservices experience do you have”

Re: Kafka as an Antipattern

#32

Post by chrsig »

imachine1980_ wrote: post #13
chrsig wrote: post #9

It seems a lot of the complaints weren't about kafka itself, but rather seemed to stem from internal communication problems. Custom kafka message headers could very well be custom http headers, and the problem is the same. Kafka is just coincidental. Looking at the volume though, kafka is overkill. They most likely could have just used the database and reaped the benefits of doing everything in a single transaction,…

>small scale kafka, though. It's conceptually great to have everything work off of logs, but kafka does add a non trivial operational burden. does something like that exist ???

Not that I'm aware of. I've been very tempted to write my own.

Re: Kafka as an Antipattern

#33

Post by x0x0 »

alexchamberlain wrote: post #14
pjscott wrote: post #10

Earlier quoted context omitted.

Yes. For perspective, that's about one message every ten seconds.

Sorry for the naivety/not obvious from the comments: is that too much or too little? (I've used RabbitMQ much more than Kafka.)

I've used kafka to process data on the order of 50mb to 5000mb / second incoming. It has complexities that are worth eating for that type of use case.

For a message every 10 seconds, its use is ... hmm. It wouldn't be in my top 50 choices.

edit: and to be clear, I'm a huge fan of kafka: it sat there and silently just worked. It was great!

Re: Kafka as an Antipattern

#34

Post by tgma »

This was not explicitly addressed in the post, but the big "Kafka antipattern" out there is building "microservice infrastructure" and using a stateful message broker between services where you should be using RPC/look-aside load balancing with deadlines and retries.

Some morons even write books and blog posts about this. The funny thing is this sort of shit is done in the name of scale, but the big folks never operate this way. Large scale infrastructures actively disdain keeping buffers and state in the middle of the request flow. They cannot afford the cost and latency of such systems. They do it the sane way[1].

[1] https://www.usenix.org/conference/osdi23/presentation/saokar

Re: Kafka as an Antipattern

#35

Post by waffletower »

Curious what language the OP and their team was using to integrate with Avro. Binary serialization can be a bit awkward, but Avro is a very stable API and it isn't difficult to find/ and or build abstractions to work with it. Perhaps I am spoiled coming from a Clojure perspective?

Re: Kafka as an Antipattern

#36

Post by ssrc »

imachine1980_ wrote: post #13
chrsig wrote: post #9

It seems a lot of the complaints weren't about kafka itself, but rather seemed to stem from internal communication problems. Custom kafka message headers could very well be custom http headers, and the problem is the same. Kafka is just coincidental. Looking at the volume though, kafka is overkill. They most likely could have just used the database and reaped the benefits of doing everything in a single transaction,…

>small scale kafka, though. It's conceptually great to have everything work off of logs, but kafka does add a non trivial operational burden. does something like that exist ???

Depending on the meaning of "small-scale kafka", both RabbitMQ and redis do support streams.

Re: Kafka as an Antipattern

#37

Post by browningstreet »

jcims wrote: post #29

I've worked at two large companies now with a mature managed Kafka offerings. The 'platform' engineering team handles all of the engineering, implementation, security and compliance, upgrades, observability etc. and have self-service onboarding with lots of recipes and sample integrations. My team moves about 5B messages a day through two topics and we're not putting a dent in the overall volume. It just enables use…

Agreed -- have used it in a bank. It was very suited for that.

Re: Kafka as an Antipattern

#38

Post by tetha »

alexchamberlain wrote: post #14
pjscott wrote: post #10

Earlier quoted context omitted.

Yes. For perspective, that's about one message every ten seconds.

Sorry for the naivety/not obvious from the comments: is that too much or too little? (I've used RabbitMQ much more than Kafka.)

At 7500 records a day, I'd even question RabbitMQ in the design.

If I was to judge that at work, my first thought would be that one of our busier postgres clusters with 2 read replicas is chugging through some 2-3k transactions per second without really needing much tuning or rather specialized hardware. The more ETL-oriented clusters are capable of processing some 100M - 200M rows per second when chugging through large queries, and these are just simple 4 core VMs again on not really specialized hardware. And postgres would parallelize these queries more if you gave it more cores and the queries aren't horrible.

At 7500 records a day or 3 million a year, you wouldn't be able to generate enough data to make one of these databases sweat over many years.

Hate me as a DBA, but write some good queries for whatever you're doing and run those in a cronjob at that scale.

Re: Kafka as an Antipattern

#39

Post by mu53 »

imachine1980_ wrote: post #13
chrsig wrote: post #9

It seems a lot of the complaints weren't about kafka itself, but rather seemed to stem from internal communication problems. Custom kafka message headers could very well be custom http headers, and the problem is the same. Kafka is just coincidental. Looking at the volume though, kafka is overkill. They most likely could have just used the database and reaped the benefits of doing everything in a single transaction,…

>small scale kafka, though. It's conceptually great to have everything work off of logs, but kafka does add a non trivial operational burden. does something like that exist ???

I think it'd be very easy to write your own. I used postgres subscribe/listen built in combined with a database table to get a distributed message system.

Writing a distributed, scalable system is really hard, and beyond the API, that is the real value for kafka

Post reply on HN