Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka as an Antipattern

joshaustin.tech

Post reply on HN
Print view View original

61–70 of 102 posts

Re: Kafka as an Antipattern

#61

Post by slt2021 »

anthonyskipper wrote: post #58
slt2021 wrote: post #54

Earlier quoted context omitted.

you don't need "systems" for 10-20 messages a day. it all could be replaced with S3 buckets and aws-cli with even better durability and delivery latency and error handling than anything you would be able to engineer yourself

Your point is good, but that stack wouldn't win any latency awards. Many of the people I know using kafka need latencies in the millisecond range.

kafka is not for latency, it is used for high throughput. By design kafka shines in high throughput workload due to consumer and producer concurrency (consumer group), broker concurrency (multiple nodes and partitions).

for latency sensitive you will probably need redis pub/sub or something in-memory

Re: Kafka as an Antipattern

#62

Post by slowmovintarget »

chrsig wrote: post #9

It seems a lot of the complaints weren't about kafka itself, but rather seemed to stem from internal communication problems. Custom kafka message headers could very well be custom http headers, and the problem is the same. Kafka is just coincidental. Looking at the volume though, kafka is overkill. They most likely could have just used the database and reaped the benefits of doing everything in a single transaction,…

We've had "small-scale" Kafka for a long time. It's an append-only log, and there are a number of ways to implement it, but it's essentially that.

The thing that makes Kafka interesting is the technique of operating from a linux disk write-buffer. That's the trick that makes it fast and scale to huge volumes. But if you don't have the scale, you can stand up a table, or RabbitMQ, or anything that manages append-only ordered log entries. There doesn't need to be a new thing... Kafka was the new thing.

Re: Kafka as an Antipattern

#63

Post by The_Colonel »

slt2021 wrote: post #54
baz00 wrote: post #47

Earlier quoted context omitted.

I think you're probably being slightly dismissive. It's not necessarily about load but various other concerns like durability, delivery latency and how failures are handled. There's a big difference between messaging and reliable messaging. I have messaging systems that take 10-20 messages a day but must deliver those messages and do it on a deadline. For that you do need infrastructure (and no that isn't a queue ins…

you don't need "systems" for 10-20 messages a day. it all could be replaced with S3 buckets and aws-cli with even better durability and delivery latency and error handling than anything you would be able to engineer yourself

Then you have dependency on a specific proprietary API / technology available from a single company. Doesn't look like a good trade off.

Re: Kafka as an Antipattern

#64

Post by KaiserPro »

anthonyskipper wrote: post #58
slt2021 wrote: post #54

Earlier quoted context omitted.

you don't need "systems" for 10-20 messages a day. it all could be replaced with S3 buckets and aws-cli with even better durability and delivery latency and error handling than anything you would be able to engineer yourself

Your point is good, but that stack wouldn't win any latency awards. Many of the people I know using kafka need latencies in the millisecond range.

but kafka isn't fast. Most things are backed by real files, so when you hit limits or something ejected from cache, it gets slow real fast.

Kafka isn't the right choice for most things.

SQS, MQTT, NATS, rabbit if you're wanting a lot of admin are all better (plus the crap that azure and google make)

Re: Kafka as an Antipattern

#65

Post by slt2021 »

The_Colonel wrote: post #63
slt2021 wrote: post #54

Earlier quoted context omitted.

you don't need "systems" for 10-20 messages a day. it all could be replaced with S3 buckets and aws-cli with even better durability and delivery latency and error handling than anything you would be able to engineer yourself

Then you have dependency on a specific proprietary API / technology available from a single company. Doesn't look like a good trade off.

S3 API has become lingua franca and is supported by open source (minio), storage vendors (QNAP), as well as plugins that translate S3 API calls to APIs for competing cloud providers (s3proxy).

all this is done because S3 provides unmatched durability and reliability at a dirt cheap cost of $22/terabyte/month of storage (with the first 50Tb/mo free!).

Try to beat that reliability guarantees with whatever you handrolled, and I bet you will never be able to beat the cost of S3, even match the durability, reliability, availability guarantees at any reasonable cost at all

from https://aws.amazon.com/s3/storage-classes/:

  Key Features:
    Low latency and high throughput performance
    Designed for durability of 99.999999999% of objects across multiple Availability Zones
have you ever built anything with 11 nines? (as in eleven nines)

Re: Kafka as an Antipattern

#66

Post by chrsig »

slowmovintarget wrote: post #62
chrsig wrote: post #9

It seems a lot of the complaints weren't about kafka itself, but rather seemed to stem from internal communication problems. Custom kafka message headers could very well be custom http headers, and the problem is the same. Kafka is just coincidental. Looking at the volume though, kafka is overkill. They most likely could have just used the database and reaped the benefits of doing everything in a single transaction,…

We've had "small-scale" Kafka for a long time. It's an append-only log, and there are a number of ways to implement it, but it's essentially that. The thing that makes Kafka interesting is the technique of operating from a linux disk write-buffer. That's the trick that makes it fast and scale to huge volumes. But if you don't have the scale, you can stand up a table, or RabbitMQ, or anything that manages append-only…

Yes, there's nothing novel about an append only log. What's missing (or unbeknownst to me) is a library or small server that provides a good general purpose implementation.

It's not just a matter of "write to log, done!". Ensuring persistence, keeping track of consumer offsets, transparent compression, waking up consumers on new message availability, support for transactions...

It's not just a append only log that's wanted, it's a system for managing append only logs, without the complications like leadership election, replication, partitioning, etc.

Re: Kafka as an Antipattern

#67

Post by slt2021 »

opportune wrote: post #53

Oh, I’ve seen much worse than this. I truly believe system design interviews and Confluent marketing/sales have made Kafka a midwit trap: 1. You cannot just use Kafka for free. It will take dev time to set up itself, dev time to code sources and sinks, dev time to handle commonly glossed over but utterly important details like idempotency, retries, duplicate messages, consumed-but-not-committed (or whatever the term…

from your reply it seems like you have not worked with high throughput workloads that FAANG deals with daily.

your suggestion of single node rdbms as a replacement to kafka suggest you dont have experience with workloads that cannot be served by a single machine, yet you still need a single architecture.

agree that Confluent took a gread product that works for high load use case, and then tries to shove it to each and every average Fortune1000 enterprise use case with 100 users and traffic that could be well served by SQLite/Postgres on a single machine

Re: Kafka as an Antipattern

#68

Post by Zandikar »

SanderNL wrote: post #28

8000 messages a day, tops? That’s 5 a minute. Does that warrant “infrastructure”? I think a gameboy’s Z80 could handle that load. I don’t want to be dismissive, but I often see these big numbers being posted, like “14M messages” or “thousands of messages” and then adding “per year” or something, which brings it down to toy level load. Even the first “serious” example is about “thousands of messages” per minute. Say 5…

FTA's conclusion: "If you are handling thousands of messages a day, a simple database-driven queue might be better than Kafka."

They're not trying to say "thousands of messages a day" is a lot, but rather not. Or at the very least, they're saying that at that scale, it is not significant enough to merit the complexity they were dealing with.

Re: Kafka as an Antipattern

#69

Post by herbstein »

slt2021 wrote: post #65
The_Colonel wrote: post #63

Earlier quoted context omitted.

Then you have dependency on a specific proprietary API / technology available from a single company. Doesn't look like a good trade off.

S3 API has become lingua franca and is supported by open source (minio), storage vendors (QNAP), as well as plugins that translate S3 API calls to APIs for competing cloud providers (s3proxy). all this is done because S3 provides unmatched durability and reliability at a dirt cheap cost of $22/terabyte/month of storage (with the first 50Tb/mo free!). Try to beat that reliability guarantees with whatever you handrolle…

> S3 API has become lingua franca

S3 API support sounds great until your costumer builds a system with an "S3 compatible object storage" product. Soon you discover that many "S3 compatible" solutions aren't actually that compatible when pushed.

Re: Kafka as an Antipattern

#70

Post by rewmie »

SanderNL wrote: post #60
rewmie wrote: post #49

Earlier quoted context omitted.

> 8000 messages a day, tops? That’s 5 a minute. Does that warrant “infrastructure”? I think a gameboy’s Z80 could handle that load. The blog post is quite clear in stating that their pain points had nothing to do with scaling or throughput. The author explicitly mentions idempotency, custom headers, and authentication. I think you're ranting about a strawman you put up.

I think you might be right. I’ll take the hit. Still, I think talking about Kafka without load feels like Christmas without tree. I thought that was the main point of it. Handling massive loads (by distributing them).

I think that the author just leveraged the existing messaging infrastructure they were already using for other services, and also reused their know-how. Their new microservice will barely register in the overall traffic volume, but they don't need to either deploy dedicated infrastructure or onboard onto yet another technology just to have message listeners.
Post reply on HN