Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka as an Antipattern

joshaustin.tech

Post reply on HN
Print view View original

91–100 of 102 posts

Re: Kafka as an Antipattern

#91

Post by SanderNL »

baz00 wrote: post #72
slt2021 wrote: post #54

Earlier quoted context omitted.

you don't need "systems" for 10-20 messages a day. it all could be replaced with S3 buckets and aws-cli with even better durability and delivery latency and error handling than anything you would be able to engineer yourself

I am completely dumbfounded by this reply. You're suggesting we engineer something on S3 and aws-cli, while complaining about engineering something ourselves when AWS offers a perfectly good queue service that requires no engineering? Uff. I'm going to buy a hut in the woods and live in it.

Why does AWS even enter the conversation?

Re: Kafka as an Antipattern

#92

Post by pharmakom »

happymellon wrote: post #88
anthonyskipper wrote: post #58

Earlier quoted context omitted.

Your point is good, but that stack wouldn't win any latency awards. Many of the people I know using kafka need latencies in the millisecond range.

In all the times I've been forced to use Kafka, I have never seen single digit millisecond latency. If you need fast response Kafka is a bad choice. If you are okay going to multiple digits of milliseconds then there are simpler solutions. The only reason to use Kafka is the ability to guarantee order. For everything else it's second place at best.

what if you want to have at least once processing and durability?

Re: Kafka as an Antipattern

#93

Post by catchnear4321 »

SanderNL wrote: post #91
baz00 wrote: post #72

Earlier quoted context omitted.

I am completely dumbfounded by this reply. You're suggesting we engineer something on S3 and aws-cli, while complaining about engineering something ourselves when AWS offers a perfectly good queue service that requires no engineering? Uff. I'm going to buy a hut in the woods and live in it.

Why does AWS even enter the conversation?

same reason any vendor enters the conversation. the argument that you can get more value out of the vendor than you would have out of a full time employee, and often for less money.

sometimes it really does make sense just to pay someone else to solve the problem. not always, but not never.

Re: Kafka as an Antipattern

#94

Post by catchnear4321 »

baz00 wrote: post #89
jameshush wrote: post #86

Earlier quoted context omitted.

On top of that, if it's a big enough deal, most salespeople at other cloud providers (GCP, Oracle etc.) will gladly pay you to migrate. They'd probably even throw one of their Solutions Engineers at the problem and do it for you for free.

Selling your soul to a different crack dealer. Hmm.

crackheads will move mountains for a score.

this checks out.

Re: Kafka as an Antipattern

#95

Post by psd1 »

chrsig wrote: post #50
mu53 wrote: post #39

Earlier quoted context omitted.

I think it'd be very easy to write your own. I used postgres subscribe/listen built in combined with a database table to get a distributed message system. Writing a distributed, scalable system is really hard, and beyond the API, that is the real value for kafka

It's relatively easy -- removing any networking requirements drastically simplifies the problem. There's still some non-trivial bits that vary depending on granularity for concurrency. It's a weekend project to demonstrate the concept, maybe a few weeks to really flesh it out and iron out quirks. I imagine if you're willing to use sqlite as a backend for persistence, it gets a bit easier.

You may be right. However, consider the directorial perspective.

You have employees - you try to get and retain the best talent you can. However, every human has strengths and weaknesses, and these may not all be fully visible to you.

Rolling your own vs buying off the shelf is a gamble on future outages.

Will a third-party support and fix the issue, or have a strong community that can help you work through the issue?

If your best engineer builds something that works for long enough to become entrenched, but then carks it, will your best engineering talent be able to resolve the issue? If your rockstar quits, does the team have to pick through the halls of Cthulhu? Does your organisational ignorance of kernel networking suddenly become painfully apparent?

Remember, you need to be twice as clever to debug your code than to write it...

Re: Kafka as an Antipattern

#96

Post by happymellon »

pharmakom wrote: post #92
happymellon wrote: post #88

Earlier quoted context omitted.

In all the times I've been forced to use Kafka, I have never seen single digit millisecond latency. If you need fast response Kafka is a bad choice. If you are okay going to multiple digits of milliseconds then there are simpler solutions. The only reason to use Kafka is the ability to guarantee order. For everything else it's second place at best.

what if you want to have at least once processing and durability?

Like AWS SQS? Others provide at least once processing as well.

Kafka has been the slowest out of them that I've used and definitely more complex to use.

Re: Kafka as an Antipattern

#97

Post by Brian_K_White »

slt2021 wrote: post #75
baz00 wrote: post #72

Earlier quoted context omitted.

I am completely dumbfounded by this reply. You're suggesting we engineer something on S3 and aws-cli, while complaining about engineering something ourselves when AWS offers a perfectly good queue service that requires no engineering? Uff. I'm going to buy a hut in the woods and live in it.

I used s3 just as an example of a service with very good track record of availability for a very low cost - and perfectly reliable available service can be created with bash scripts, aws-cli and free-tier AWS account. perfectly fine with using SQS, just it will have worse availability guarantees than S3 - people should understand tradeoffs

I don't know but I think they might be suggesting that the answer to "don't need a huge complex system" is not "use someone else's huge complex system".

Re: Kafka as an Antipattern

#98

Post by slt2021 »

Brian_K_White wrote: post #97
slt2021 wrote: post #75

Earlier quoted context omitted.

I used s3 just as an example of a service with very good track record of availability for a very low cost - and perfectly reliable available service can be created with bash scripts, aws-cli and free-tier AWS account. perfectly fine with using SQS, just it will have worse availability guarantees than S3 - people should understand tradeoffs

I don't know but I think they might be suggesting that the answer to "don't need a huge complex system" is not "use someone else's huge complex system".

my understanding is that you should not roll your own expensive complex huge system, just use AWS S3 which provides eleven nines of availability at $22/tb/mo.

It is really hard to beat the cost/benefit ratio of S3.

a lot of mediocre engineers cannot swallow a pill that all their expensive work with hundreds of hours of overengineering could be replaced by a couple of AWS managed serverless services stitched together with a few mouse clicks or a single .yml file

Re: Kafka as an Antipattern

#99

Post by pharmakom »

happymellon wrote: post #96
pharmakom wrote: post #92

Earlier quoted context omitted.

what if you want to have at least once processing and durability?

Like AWS SQS? Others provide at least once processing as well. Kafka has been the slowest out of them that I've used and definitely more complex to use.

SQS has durability for up to 14 days and you can only have one consumer group.

It’s also proprietary.

Re: Kafka as an Antipattern

#100

Post by happymellon »

pharmakom wrote: post #99
happymellon wrote: post #96

Earlier quoted context omitted.

Like AWS SQS? Others provide at least once processing as well. Kafka has been the slowest out of them that I've used and definitely more complex to use.

SQS has durability for up to 14 days and you can only have one consumer group. It’s also proprietary.

Then use Rabbit.

It's still harder to get wrong than Kafka.

Post reply on HN