Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka is dead, long live Kafka

warpstream.com

Post reply on HN
Print view View original

281–290 of 295 posts

Re: Kafka is dead, long live Kafka

#281

Post by cbsmith »

snapetom wrote: post #106
carlio wrote: post #78

Earlier quoted context omitted.

I work as a contractor so I move between places. I have found a few companies trying to introduce kafka, and every time it has been a solution in search of a problem. I don't doubt that it has a good use case but I have so far only encountered the zealots who crowbar it into any situation and that has left a residual bad taste in my mouth. So I fall into the "hate it" side.

> and every time it has been a solution in search of a problem. More refined to this, in my experience at the last two jobs, the queue problem is there, but the Kafka solution is based solely on "enterpriseyness" of Kafka, not any practical reason. RabbitMQ is highly performant, SQS is really easy. Both are great queues. Kafka is muuch more, yet, Kafka is chosen because "it's enterprise."

I always find the "it's enterprise" statement so humourous, given how much time I've had to invest in convincing enterprises that Kafka wasn't some weird fly-by-night technology that couldn't provide for the demanding enterprise.

Re: Kafka is dead, long live Kafka

#282

Post by cbsmith »

relay23 wrote: post #164
snapetom wrote: post #106

Earlier quoted context omitted.

> and every time it has been a solution in search of a problem. More refined to this, in my experience at the last two jobs, the queue problem is there, but the Kafka solution is based solely on "enterpriseyness" of Kafka, not any practical reason. RabbitMQ is highly performant, SQS is really easy. Both are great queues. Kafka is muuch more, yet, Kafka is chosen because "it's enterprise."

Maybe "it's enterprise" means that's what the enterprise standardized on. There are a couple of practical reasons that come to mind on why that's the case - a) it's more resilient and durable than messaging platforms, and b) it is a platform of dumb pipes, so to make it a central data bus managed by platform teams means that they don't have to get into the detail of which queues perform what functions, have what char…

That is, oddly, precisely the reason I've pushed for Kafka in certain environments... and the push back was that "Kafka isn't enterprise". ;-)

Re: Kafka is dead, long live Kafka

#283

Post by lmm »

zbentley wrote: post #252
lmm wrote: post #197

Earlier quoted context omitted.

To me a technology company is not just a company that uses tech (every company does that) but one whose core value proposition is fundamentally technical. And I think most serious companies doing that have a need for highly available data storage, for which Kafka is the least bad option. What are the alternatives? Cassandra is just as operationally complex and harder to fit your dataflow into. The various efforts to…

> As far as I can tell there are mostly companies that use Kafka and companies that have a SPOF PostgreSQL/MySQL database I haven't seen that at all, across the many companies I've worked at, consulted with, and talked with others about. Kafka is usually an ancillary system added to companies with a strong culture around one or more pre-existing datastores (from PG/MySQL to Dynamo/Cassandra to Mongo/Elastic). When Ka…

> Are you really seeing companies use Kafka for their main persistence layer?

I'm seeing kafka-streams-style event processing as the primary data layer used by most business operations, although only in the last couple of years.

> As in, like, KQL or the equivalent for all/most business operations?

> Even the CQRS/ES zealots are still consuming from Kafka topics into (usually relational) databases for reads.

Yeah, I'm not seeing KQL, and I'm still seeing relational databases used for a lot of secondary views and indices. But the SQL database is populated from the Kafka, not vice versa, and can be wiped and regenerated if needed, and at least in theory it can't be used for live processing (so an SQL outage would take down the management UI and mean customers couldn't change their settings, it would be a big deal and need fixing quickly, but it wouldn't be an outage in the primary system).

Re: Kafka is dead, long live Kafka

#284

Post by AtlasBarfed »

bit_flipper wrote: post #278
AtlasBarfed wrote: post #271

Earlier quoted context omitted.

I have a dim view of almost all inherently single-node datastores that advertise a clustered hack (and they are hacks) as a patch-on (yes, even PostgreSQL). Sure it will work in most cases, but the failure modes are scary for all of them. A distributed database will have network failures, will have conflicting writes, will have to either pick between being down if any of the network is down (CP) or you need a "hard/c…

Not your main point, but MongoDB didn't commission Kyle to do that report as they had in the past, he did it on his own time. That's why his report doesn't mention repeat testing. They do actually run his tests in their CI and those new tests were used to isolate that specific bug. Moreover, some of the complaints about weak durability defaults for writing were later fixed: https://www.mongodb.com/blog/post/default-m…

Do you have a source for this? I got the impression at the time that there was some commissioning of his services, but that they didn't like the report. But he publishes work, and released the report, which forced them to deal with it.

Every distributed tech fails when he test it, but the tenor and nature of the report for MongoDB was different. It basically said between the lines "do not use this product".

MongoDB has a history of really crappy persistence decisions and silently failed writes, and as soon as it gets publicized saying "we fixed it in the next release". The same thing happened here of course. I simply don't trust the software or the company.

Mysql has the same annoying pattern in its history, although I have more confidence in the software because of the sheer number of users.

Still, I would probably pick PostgreSQL for both relation and document stores.

Re: Kafka is dead, long live Kafka

#285

Post by EdwardDiego »

cbsmith wrote: post #280
EdwardDiego wrote: post #152

Earlier quoted context omitted.

Kafka isn't even a queue. I've done consulting on Kafka, and several times my recommendation is "You don't want or need Kafka". A classic sign of "you wanted an MQ" is when a consumer writes a message to a topic to let the producer know it read the message the producer wrote...

Right. Kafka is a database. ;-)

Anything is a database if you're brave enough :D

Re: Kafka is dead, long live Kafka

#287

Post by regularfry »

valenterry wrote: post #108
whartung wrote: post #104

Earlier quoted context omitted.

There was recently an article about distributed systems that showed up here. (Harry Doyle: Christ, I can't find it. To hell with it!) And the author made a very interesting point about message queues. Simply, any problem that could be resolved by a message queue could be resolved by load balancing or persistence, and, therefore, messages queues were actually kind of a bad idea. There were two basic issues. The first…

> It's also a mechanically simpler thing to send out a request and wait for the response than to do the dance via a message queue. Yeah, this is simpler for the requester, but not for the counterpart that has to respond. Because now, the responder has to have 100% uptime and better not fail during the request, otherwise things get lost. Let's take sending emails as an example. You have a server A that can send emails…

This is where the "or persistence" bit comes in, but the trade-off isn't as you describe. One way to approach it would be for B to set a flag against the user in the database they must already have, because they have users so that A can process password resets on its own schedule. It's add a column versus add a message queue. Databases already come with locking primitives and well understood patterns, no reinvention needed.

Re: Kafka is dead, long live Kafka

#288

Post by regularfry »

valenterry wrote: post #110
dopidopHN wrote: post #98

Earlier quoted context omitted.

RabbitMQ, ActiveMQ, MQtt All of those are fine is you only need pub/sub

How good does RabbitMQ do in terms of availability nowadays? Because one thing a message queue should offer is high availability - otherwie it loses one of it's most compelling benefits.

The one thing that made me pull back from RabbitMQ years ago was that using it between datacentres was a bad plan, because all the clustering was based on Erlang's underlying cluster implementation and the advice on that was not to use it between geographically distinct locations. I don't know if it's since improved or if that advice no longer holds, but working under an environment where we needed cross-DC redundancy made it impossible to select, for that reason.

Re: Kafka is dead, long live Kafka

#289

Post by valenterry »

regularfry wrote: post #287
valenterry wrote: post #108

Earlier quoted context omitted.

> It's also a mechanically simpler thing to send out a request and wait for the response than to do the dance via a message queue. Yeah, this is simpler for the requester, but not for the counterpart that has to respond. Because now, the responder has to have 100% uptime and better not fail during the request, otherwise things get lost. Let's take sending emails as an example. You have a server A that can send emails…

This is where the "or persistence" bit comes in, but the trade-off isn't as you describe. One way to approach it would be for B to set a flag against the user in the database they must already have, because they have users so that A can process password resets on its own schedule. It's add a column versus add a message queue. Databases already come with locking primitives and well understood patterns, no reinvention…

You are now assuming that they indeed have a database nd store users in there, and not use a 3rd party system that might not even allow to attach meta data (or at least make it difficult).

Re: Kafka is dead, long live Kafka

#290

Post by regularfry »

valenterry wrote: post #289
regularfry wrote: post #287

Earlier quoted context omitted.

This is where the "or persistence" bit comes in, but the trade-off isn't as you describe. One way to approach it would be for B to set a flag against the user in the database they must already have, because they have users so that A can process password resets on its own schedule. It's add a column versus add a message queue. Databases already come with locking primitives and well understood patterns, no reinvention…

You are now assuming that they indeed have a database nd store users in there, and not use a 3rd party system that might not even allow to attach meta data (or at least make it difficult).

In which case why is the password reset my problem at all?

But even if I've handed off identity management entirely, I almost certainly do have some per-user state. Otherwise... what on earth am I doing with users in the first place?

Post reply on HN