Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka as an Antipattern

joshaustin.tech

Post reply on HN
Print view View original

81–90 of 102 posts

Re: Kafka as an Antipattern

#81

Post by coder543 »

chrsig wrote: post #48
ssrc wrote: post #36

Earlier quoted context omitted.

Depending on the meaning of "small-scale kafka", both RabbitMQ and redis do support streams.

One of my desires would be for it to be persistent. Hopefully with the option of different storage tiers, so as logs became older they could be moved to less costly medium and transparently fetched when requested. Having an event sourced system doesn't make much sense unless you maintain messages from the start of the system. You can snapshot state and resume in order to quickly rebuild from a known good state. That…

Redis doesn't do key eviction by default. Everything lives forever, unless you tell it otherwise. Of course, it is recommended to turn on some form of persistence (and configure backups) so that things don't disappear if the server restarts.

Re: Kafka as an Antipattern

#82

Post by safetytrick »

jpgvm wrote: post #30

Anything is deceptively deep if you understanding never goes beyond skin deep. Also Avro is great but like Kafka you were probably holding it wrong. I do prefer Protobuf in these particular scenarios as Protobufs features more closely align with svc svc RPC style communication patterns while Avro shines in longer lived scenarios where messages need to be archived and you don't want to come up with your own framing fo…

I'm a big Avro fan, it isn't easy to be good at but if you follow its rules it is so much easier to evolve than plain json is.

Re: Kafka as an Antipattern

#83

Post by dilyevsky »

Kafka will be trash when used as a message bus which is exactly what author had experienced. It can work but it’s designed for stream processing not messaging so it will always be inferior when used this way

Re: Kafka as an Antipattern

#84

Post by atomicnumber3 »

jcims wrote: post #29

I've worked at two large companies now with a mature managed Kafka offerings. The 'platform' engineering team handles all of the engineering, implementation, security and compliance, upgrades, observability etc. and have self-service onboarding with lots of recipes and sample integrations. My team moves about 5B messages a day through two topics and we're not putting a dent in the overall volume. It just enables use…

The fact that you had an entire team operating it is the key, I think. I've seen various big buzzwordy techs used at various shops I've worked at and whether it was nice to work with totally came down to whether there was a team behind it operating it for us.

K8s with a k8s team running it - fabulous. Without a team and everyone kind of just needs to know enough to get by, except for the one guy who set it up? Dreadful.

Airflow when there's a team running it? Great. Luigi when it's just you, another dude, and the one guy who set it up? Not great.

Even RDS is like eh. Still had RDS tip over and we had to do manual vacuuming or something with compacting tables with dead tuples. Annoying when we were paying for RDS to ostensibly not have to do this.

Re: Kafka as an Antipattern

#85

Post by opportune »

slt2021 wrote: post #67
opportune wrote: post #53

Oh, I’ve seen much worse than this. I truly believe system design interviews and Confluent marketing/sales have made Kafka a midwit trap: 1. You cannot just use Kafka for free. It will take dev time to set up itself, dev time to code sources and sinks, dev time to handle commonly glossed over but utterly important details like idempotency, retries, duplicate messages, consumed-but-not-committed (or whatever the term…

from your reply it seems like you have not worked with high throughput workloads that FAANG deals with daily. your suggestion of single node rdbms as a replacement to kafka suggest you dont have experience with workloads that cannot be served by a single machine, yet you still need a single architecture. agree that Confluent took a gread product that works for high load use case, and then tries to shove it to each an…

Not only have I worked on some of the highest throughput workloads in FAANG, the ones I’ve worked on involve consuming from async queues (including Kafka) as a first class feature and use additional queueing under the hood to make the system work.

Kafka is overkill for the vast majority of users, including many in FAANG, and at companies in FAANG that I’ve been at, usually either an internal async queue or a bespoke solution for the problem is used instead. But more importantly, async queueing via Kafka-like technology itself is way over-applied. Polling a single node RDBMs with a horizontally scaling cache in front of it is a fine way to propagate eg config changes at massive scale. And many of the “real-time ingestion” use cases that people love using Kafka for are much better off simply being synchronous, with engineering effort instead focused on the ability to rapidly scale consumers up and down.

Re: Kafka as an Antipattern

#86

Post by jameshush »

slt2021 wrote: post #76
baz00 wrote: post #73

Earlier quoted context omitted.

S3 is fine until you want your data to leave AWS. Then it costs $92 / TB to get it out again. Also S3 has durability guarantees but it's very difficult to do a durable transactional write to S3. Try it a few million times and see. The API is a defacto shitty standard. These two facts are rather interesting when it comes to doing a restore from your supposed backup or wonder why consistency guarantees between external…

and why would you ever take raw data out of AWS ? if it is for migration: it is one time cost that anyone can swallow easily if they decided to leave AWS for something else. If your data is worth or just use cloudfront to download your data ($8.5/Tb)?

On top of that, if it's a big enough deal, most salespeople at other cloud providers (GCP, Oracle etc.) will gladly pay you to migrate. They'd probably even throw one of their Solutions Engineers at the problem and do it for you for free.

Re: Kafka as an Antipattern

#87

Post by ImPostingOnHN »

rewmie wrote: post #49
SanderNL wrote: post #28

8000 messages a day, tops? That’s 5 a minute. Does that warrant “infrastructure”? I think a gameboy’s Z80 could handle that load. I don’t want to be dismissive, but I often see these big numbers being posted, like “14M messages” or “thousands of messages” and then adding “per year” or something, which brings it down to toy level load. Even the first “serious” example is about “thousands of messages” per minute. Say 5…

> 8000 messages a day, tops? That’s 5 a minute. Does that warrant “infrastructure”? I think a gameboy’s Z80 could handle that load. The blog post is quite clear in stating that their pain points had nothing to do with scaling or throughput. The author explicitly mentions idempotency, custom headers, and authentication. I think you're ranting about a strawman you put up.

As someone who helps administer several kafka clusters which carry millions of messages per second, and works with both DBs and DBAs, the shallowness of the author's complaints seemed to belie a lack of experience and understanding of either technology. If they gave even a small amount of detail, it might have been useful:

"We couldn’t create a Kafka source connector from the database holding the messages" – Why couldn't they?

"we had extreme difficulty troubleshooting when their system refused our best understanding of said custom headers" – What made this so extremely difficult for them?

"Authentication with Kafka, and authentication to encrypt confidential data in Kafka, and getting the Avro schema from the schema registry were all separate and painful steps" – These are 3 totally unrelated features, so of course they each have settings. What made the steps so painful for them?

"We would never tolerate this level of complexity with a database" the author says of configuring kafka features, apparently missing that a database setup with all those features (external authentication, encryption, custom serialization) also requires "separate" configuration for each one.

Re: Kafka as an Antipattern

#88

Post by happymellon »

anthonyskipper wrote: post #58
slt2021 wrote: post #54

Earlier quoted context omitted.

you don't need "systems" for 10-20 messages a day. it all could be replaced with S3 buckets and aws-cli with even better durability and delivery latency and error handling than anything you would be able to engineer yourself

Your point is good, but that stack wouldn't win any latency awards. Many of the people I know using kafka need latencies in the millisecond range.

In all the times I've been forced to use Kafka, I have never seen single digit millisecond latency.

If you need fast response Kafka is a bad choice.

If you are okay going to multiple digits of milliseconds then there are simpler solutions.

The only reason to use Kafka is the ability to guarantee order. For everything else it's second place at best.

Re: Kafka as an Antipattern

#89

Post by baz00 »

jameshush wrote: post #86
slt2021 wrote: post #76

Earlier quoted context omitted.

and why would you ever take raw data out of AWS ? if it is for migration: it is one time cost that anyone can swallow easily if they decided to leave AWS for something else. If your data is worth or just use cloudfront to download your data ($8.5/Tb)?

On top of that, if it's a big enough deal, most salespeople at other cloud providers (GCP, Oracle etc.) will gladly pay you to migrate. They'd probably even throw one of their Solutions Engineers at the problem and do it for you for free.

Selling your soul to a different crack dealer. Hmm.

Re: Kafka as an Antipattern

#90

Post by iudqnolq »

local_crmdgeon wrote: post #55
mu53 wrote: post #39

Earlier quoted context omitted.

I think it'd be very easy to write your own. I used postgres subscribe/listen built in combined with a database table to get a distributed message system. Writing a distributed, scalable system is really hard, and beyond the API, that is the real value for kafka

>I used postgres subscribe/listen built in combined with a database table to get a distributed message system. Every single person I know who's done this says it was a fantastic decision, and the "eventually I'll have to migrate to X" never came.

how do deal with connection limits? you can't listen through a pooled connection.
Post reply on HN