Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka is dead, long live Kafka

warpstream.com

Post reply on HN
Print view View original

151–160 of 295 posts

Re: Kafka is dead, long live Kafka

#151

Post by morelisp »

benjaminwootton wrote: post #128
parhamn wrote: post #88

Earlier quoted context omitted.

Kafka is a closer to a persistent WAL than a message queue. If your work doesn't need a WAL, it's almost certainly overkill and you will hate it. If your work needs a WAL then it'll be your favorite tool ever.

Why? Its quite easy to use Kafka as a messaging queue without even thinking about the write ahead log semantics. It’s there if you need it, but Kafka scales down to being a message broker fairly well in my opinion.

Besides the operational concerns mentioned in the sibling comment, Kafka is simply not a great queue. You can't work-steal, you can't easily retry out-of-order, you can't size retention based on "is it processed yet", and you may need to manually implement DLQ behavior.

If you already have Kafka for other (more WAL-y, or maybe older log-shippy) reasons it can be an OK queue, especially if you've got a team that can use Kafka as a WAL they can easily work around using most of the downsides of using it as queue. But I wouldn't take it as a first choice.

Re: Kafka is dead, long live Kafka

#152

Post by EdwardDiego »

snapetom wrote: post #106
carlio wrote: post #78

Earlier quoted context omitted.

I work as a contractor so I move between places. I have found a few companies trying to introduce kafka, and every time it has been a solution in search of a problem. I don't doubt that it has a good use case but I have so far only encountered the zealots who crowbar it into any situation and that has left a residual bad taste in my mouth. So I fall into the "hate it" side.

> and every time it has been a solution in search of a problem. More refined to this, in my experience at the last two jobs, the queue problem is there, but the Kafka solution is based solely on "enterpriseyness" of Kafka, not any practical reason. RabbitMQ is highly performant, SQS is really easy. Both are great queues. Kafka is muuch more, yet, Kafka is chosen because "it's enterprise."

Kafka isn't even a queue. I've done consulting on Kafka, and several times my recommendation is "You don't want or need Kafka".

A classic sign of "you wanted an MQ" is when a consumer writes a message to a topic to let the producer know it read the message the producer wrote...

Re: Kafka is dead, long live Kafka

#153

Post by EdwardDiego »

mdaniel wrote: post #143
benjaminwootton wrote: post #119

Earlier quoted context omitted.

I’m not sure how anyone can hate Kafka? It does what it says on the tin - move data from A to B with publish/subscribe semantics. It’s quite easy to just use it as a dumb message broker with no retention if that’s all you need but if you do want to do something funky with persistence then go down that route. I’m not sure how anyone could have a negative feeling towards a vanilla, but rock solid and wildly popular ope…

> I’m not sure how anyone can hate Kafka? My experience has been that such a question has two implied audiences in it: those who consume Kafka and those who have to keep the PoS alive and healthy The whole ambiguity around whether ZK is really still needed or not https://kafka.apache.org/documentation/#zk > makes keeping two distributed systems alive and healthy, but don't worry you can't move your production cluster…

ZooKeeper is rock solid. Moving off it is a mistake, IMO.

My tinfoil hat theory is that the whole impetus for KRaft is Confluent Cloud's multi-tenanted clusters have so many partitions that it starts to exceed ZK's capacities, so Confluent have built KRaft for Confluent.

And yeah, the migration approach is nutso. Also very annoying, the KRaft metadata topics being changed to be super-secret for... ...some good reason, I'm sure.

But it entirely removes the ability to respond to changed cluster metadata that you have with ZK, where you can watch znodes.

I'm not at all a fan tbh.

Re: Kafka is dead, long live Kafka

#154

Post by EdwardDiego »

richdougherty wrote: post #101
ram_rar wrote: post #77

Earlier quoted context omitted.

how do you design avoiding message queues? Or do you use other alternatives around kafka for these things?

They may still use some sort of event/message system. Kafka is lower level than other sorts message queue systems and requires more work to get correct (dedupe, ordering, retry logic), but has great performance. It's often easier to choose a different messaging system though.

Kafka ain't a MQ, it's a distributed log.

Pulsar is a Kafkaesque system that can act as distributed log _or_ an MQ.

Re: Kafka is dead, long live Kafka

#155

Post by EdwardDiego »

crimsonpowder wrote: post #41
slantedview wrote: post #14

This article seems to be spreading some FUD, in particular this comment: > Cloud economics – by design, Kafka’s replication strategy will rack up massive inter AZ bandwidth costs. You're no more or less forced to put Kafka replicas in different AZs than you would be with an alternative.

Of course, as soon as an AZ goes down that had all of your Kafka nodes in it, you get eviscerated on HN for running such a setup.

3 AZ Kafka is the safest, but I've run several 2.5 clusters just fine. (cluster stretched across 2 AZs with a tiebreaker ZK in a 3rd AZ to maintain quorum in the event that you lose an AZ)

Re: Kafka is dead, long live Kafka

#156

Post by EdwardDiego »

rad_gruchalski wrote: post #147
Sphax wrote: post #79

Earlier quoted context omitted.

Just fyi, tiered storage is coming at some point in Kafka: https://cwiki.apache.org/confluence/plugins/servlet/mobile?c...

It’s been like that for a while.

The Uber devs are working hard at it.

Re: Kafka is dead, long live Kafka

#158

Post by pfannkuchen »

I feel like people misuse the X is dead, long live X construction, but maybe I’m missing something.

I’ve always taken it to be a reference to the stability and near-programmatic nature of British royal succession. So while “the king is dead” in some places historically would be a disaster with conflict surely following, in Britain it automatically and instantly becomes “long live the king” for the next king and there is no panic.

Re: Kafka is dead, long live Kafka

#159

Post by tofuahdude »

EdwardDiego wrote: post #150
carlio wrote: post #78

Earlier quoted context omitted.

I work as a contractor so I move between places. I have found a few companies trying to introduce kafka, and every time it has been a solution in search of a problem. I don't doubt that it has a good use case but I have so far only encountered the zealots who crowbar it into any situation and that has left a residual bad taste in my mouth. So I fall into the "hate it" side.

It's really good when you're producing a lot of data fast that you don't want to lose, and you want multiple consumers to read. It's a complex tool that solves a complicated problem. But if you don't actually have that problem, then that's a whole lot of complexity for no gain.

Hypothetical use case I've been thinking about:

Say I want to log all http requests to the server (I know I said a keyword of log) and then process those logs into aggregates, stick them in a time series.

Would it be insane to "log" everything into kafka? Or what would be the more "correct" tool for that job?

Re: Kafka is dead, long live Kafka

#160

Post by EdwardDiego »

Dear richieartoul, your blog post is a tad oversalted.

Kafka doesn't inherently require dedicated teams of experts and millions of dollars until you're running very large clusters.

But fully agree that 3-AZ stretch clusters suck money through those inter-AZ xfer fees. Which is how AWS sells MSK, the inter-AZ xfer is "free"! That is, already priced in...

Post reply on HN