Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka as an Antipattern

joshaustin.tech

Post reply on HN
Print view View original

71–80 of 102 posts

Re: Kafka as an Antipattern

#71

Post by jshen »

rewmie wrote: post #49
SanderNL wrote: post #28

8000 messages a day, tops? That’s 5 a minute. Does that warrant “infrastructure”? I think a gameboy’s Z80 could handle that load. I don’t want to be dismissive, but I often see these big numbers being posted, like “14M messages” or “thousands of messages” and then adding “per year” or something, which brings it down to toy level load. Even the first “serious” example is about “thousands of messages” per minute. Say 5…

> 8000 messages a day, tops? That’s 5 a minute. Does that warrant “infrastructure”? I think a gameboy’s Z80 could handle that load. The blog post is quite clear in stating that their pain points had nothing to do with scaling or throughput. The author explicitly mentions idempotency, custom headers, and authentication. I think you're ranting about a strawman you put up.

The author's talked about their pain points, but they didn't really outline the downside of going to this approach in a comprehensive way. What I've found with event based systems is that they are much worse if your operational maturity/excellence is low. To make it worse, operational excellence can get worse over the years so you can't simply base the decision on how well you do it today.

Re: Kafka as an Antipattern

#72

Post by baz00 »

slt2021 wrote: post #54
baz00 wrote: post #47

Earlier quoted context omitted.

I think you're probably being slightly dismissive. It's not necessarily about load but various other concerns like durability, delivery latency and how failures are handled. There's a big difference between messaging and reliable messaging. I have messaging systems that take 10-20 messages a day but must deliver those messages and do it on a deadline. For that you do need infrastructure (and no that isn't a queue ins…

you don't need "systems" for 10-20 messages a day. it all could be replaced with S3 buckets and aws-cli with even better durability and delivery latency and error handling than anything you would be able to engineer yourself

I am completely dumbfounded by this reply.

You're suggesting we engineer something on S3 and aws-cli, while complaining about engineering something ourselves when AWS offers a perfectly good queue service that requires no engineering?

Uff. I'm going to buy a hut in the woods and live in it.

Re: Kafka as an Antipattern

#73

Post by baz00 »

slt2021 wrote: post #65
The_Colonel wrote: post #63

Earlier quoted context omitted.

Then you have dependency on a specific proprietary API / technology available from a single company. Doesn't look like a good trade off.

S3 API has become lingua franca and is supported by open source (minio), storage vendors (QNAP), as well as plugins that translate S3 API calls to APIs for competing cloud providers (s3proxy). all this is done because S3 provides unmatched durability and reliability at a dirt cheap cost of $22/terabyte/month of storage (with the first 50Tb/mo free!). Try to beat that reliability guarantees with whatever you handrolle…

S3 is fine until you want your data to leave AWS.

Then it costs $92 / TB to get it out again.

Also S3 has durability guarantees but it's very difficult to do a durable transactional write to S3. Try it a few million times and see. The API is a defacto shitty standard.

These two facts are rather interesting when it comes to doing a restore from your supposed backup or wonder why consistency guarantees between external metadata services (DB) and what is in S3 don't always line up.

Re: Kafka as an Antipattern

#75

Post by slt2021 »

baz00 wrote: post #72
slt2021 wrote: post #54

Earlier quoted context omitted.

you don't need "systems" for 10-20 messages a day. it all could be replaced with S3 buckets and aws-cli with even better durability and delivery latency and error handling than anything you would be able to engineer yourself

I am completely dumbfounded by this reply. You're suggesting we engineer something on S3 and aws-cli, while complaining about engineering something ourselves when AWS offers a perfectly good queue service that requires no engineering? Uff. I'm going to buy a hut in the woods and live in it.

I used s3 just as an example of a service with very good track record of availability for a very low cost - and perfectly reliable available service can be created with bash scripts, aws-cli and free-tier AWS account.

perfectly fine with using SQS, just it will have worse availability guarantees than S3 - people should understand tradeoffs

Re: Kafka as an Antipattern

#76

Post by slt2021 »

baz00 wrote: post #73
slt2021 wrote: post #65

Earlier quoted context omitted.

S3 API has become lingua franca and is supported by open source (minio), storage vendors (QNAP), as well as plugins that translate S3 API calls to APIs for competing cloud providers (s3proxy). all this is done because S3 provides unmatched durability and reliability at a dirt cheap cost of $22/terabyte/month of storage (with the first 50Tb/mo free!). Try to beat that reliability guarantees with whatever you handrolle…

S3 is fine until you want your data to leave AWS. Then it costs $92 / TB to get it out again. Also S3 has durability guarantees but it's very difficult to do a durable transactional write to S3. Try it a few million times and see. The API is a defacto shitty standard. These two facts are rather interesting when it comes to doing a restore from your supposed backup or wonder why consistency guarantees between external…

and why would you ever take raw data out of AWS ?

if it is for migration: it is one time cost that anyone can swallow easily if they decided to leave AWS for something else.

If your data is worth or just use cloudfront to download your data ($8.5/Tb)?

Re: Kafka as an Antipattern

#77

Post by slt2021 »

herbstein wrote: post #69
slt2021 wrote: post #65

Earlier quoted context omitted.

S3 API has become lingua franca and is supported by open source (minio), storage vendors (QNAP), as well as plugins that translate S3 API calls to APIs for competing cloud providers (s3proxy). all this is done because S3 provides unmatched durability and reliability at a dirt cheap cost of $22/terabyte/month of storage (with the first 50Tb/mo free!). Try to beat that reliability guarantees with whatever you handrolle…

> S3 API has become lingua franca S3 API support sounds great until your costumer builds a system with an "S3 compatible object storage" product. Soon you discover that many "S3 compatible" solutions aren't actually that compatible when pushed.

get-object and put-object is really all you need. everything else is nice to have

Re: Kafka as an Antipattern

#78

Post by karmakaze »

rewmie wrote: post #49
SanderNL wrote: post #28

8000 messages a day, tops? That’s 5 a minute. Does that warrant “infrastructure”? I think a gameboy’s Z80 could handle that load. I don’t want to be dismissive, but I often see these big numbers being posted, like “14M messages” or “thousands of messages” and then adding “per year” or something, which brings it down to toy level load. Even the first “serious” example is about “thousands of messages” per minute. Say 5…

> 8000 messages a day, tops? That’s 5 a minute. Does that warrant “infrastructure”? I think a gameboy’s Z80 could handle that load. The blog post is quite clear in stating that their pain points had nothing to do with scaling or throughput. The author explicitly mentions idempotency, custom headers, and authentication. I think you're ranting about a strawman you put up.

It seems to me that Kafka was the strawman that the article put up to take down as an antipattern. There was no part of the premise that made sense.

> While my client was in a situation where we had no choice but to use this monstrosity of Kafka, Avro, and custom message headers, I would never recommend this usage of Kafka if I had the option.

The antipattern isn't Kafka, it's the client situation.

Post reply on HN