Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka is dead, long live Kafka

warpstream.com

Post reply on HN
Print view View original

251–260 of 295 posts

Re: Kafka is dead, long live Kafka

#251

Post by zbentley »

EdwardDiego wrote: post #152
snapetom wrote: post #106

Earlier quoted context omitted.

> and every time it has been a solution in search of a problem. More refined to this, in my experience at the last two jobs, the queue problem is there, but the Kafka solution is based solely on "enterpriseyness" of Kafka, not any practical reason. RabbitMQ is highly performant, SQS is really easy. Both are great queues. Kafka is muuch more, yet, Kafka is chosen because "it's enterprise."

Kafka isn't even a queue. I've done consulting on Kafka, and several times my recommendation is "You don't want or need Kafka". A classic sign of "you wanted an MQ" is when a consumer writes a message to a topic to let the producer know it read the message the producer wrote...

> A classic sign of "you wanted an MQ" is when a consumer writes a message to a topic to let the producer know it read the message the producer wrote...

Oof. Queued RPC is such a siren song; so many developers either stumble into this pattern or seek it out. And it's such a pain. Suddenly the latency of (often user-sensitive) operations is contingent on the latency of a queue consumer plus the time it takes to process everything in the queue before the RPC was issued. Goodbye, predictable turnaround times.

Re: Kafka is dead, long live Kafka

#252

Post by zbentley »

lmm wrote: post #197
slotrans wrote: post #68

> In our experience, Kafka is one of the most polarizing technologies in the data space. Some people hate it, some people swear by it, but almost every technology company uses it. (emphasis added) Surely that's false? Or, I mean, neither of us are providing any evidence here... For my part, 0 of the last 6 companies I've worked for used it. The company before that did (I drove its adoption), but we later abandoned it…

To me a technology company is not just a company that uses tech (every company does that) but one whose core value proposition is fundamentally technical. And I think most serious companies doing that have a need for highly available data storage, for which Kafka is the least bad option. What are the alternatives? Cassandra is just as operationally complex and harder to fit your dataflow into. The various efforts to…

> As far as I can tell there are mostly companies that use Kafka and companies that have a SPOF PostgreSQL/MySQL database

I haven't seen that at all, across the many companies I've worked at, consulted with, and talked with others about.

Kafka is usually an ancillary system added to companies with a strong culture around one or more pre-existing datastores (from PG/MySQL to Dynamo/Cassandra to Mongo/Elastic). When Kafka's actually needed, it handles things those pre-existing stores can't do efficiently at high volumes.

Are you really seeing companies use Kafka for their main persistence layer? As in, like, KQL or the equivalent for all/most business operations?

Even the CQRS/ES zealots are still consuming from Kafka topics into (usually relational) databases for reads.

Re: Kafka is dead, long live Kafka

#253

Post by zbentley »

whartung wrote: post #104
ram_rar wrote: post #77

Earlier quoted context omitted.

how do you design avoiding message queues? Or do you use other alternatives around kafka for these things?

There was recently an article about distributed systems that showed up here. (Harry Doyle: Christ, I can't find it. To hell with it!) And the author made a very interesting point about message queues. Simply, any problem that could be resolved by a message queue could be resolved by load balancing or persistence, and, therefore, messages queues were actually kind of a bad idea. There were two basic issues. The first…

> If the request is Important, you certainly don't want to trust it to a message queue, a device not really designed for the storage of messages.

That's false. Virtually all MQ systems are designed to persist (often with replication/redundancy) and store data. Most MQs also support non-persistent delivery, with the cost/benefit (ephemerality/performance) that entails, but that doesn't mean that durable storage is any less well-supported.

Sure, folks have plenty of operational war stories regarding failures of persistence in their MQ broker/cluster/whatnot. Same as the DBAs who manage relational databases.

Re: Kafka is dead, long live Kafka

#255

Post by zbentley »

valenterry wrote: post #110
dopidopHN wrote: post #98

Earlier quoted context omitted.

RabbitMQ, ActiveMQ, MQtt All of those are fine is you only need pub/sub

How good does RabbitMQ do in terms of availability nowadays? Because one thing a message queue should offer is high availability - otherwie it loses one of it's most compelling benefits.

Rabbit's quorum queues are an improvement on the extremely poor HA/clustering system they provided previously. Users can now choose between both.

Rabbit's defaults are still unfortunate, in my opinion: queues and messages are not disk-persisted by default, though this can easily be enabled. As a result, many folks run and benchmark a "high availability rabbit" only to discover that they're benchmarking distributed state stored in memory, not disk.

https://www.rabbitmq.com/quorum-queues.html

Re: Kafka is dead, long live Kafka

#256

Post by richieartoul »

benbuick wrote: post #242

Hey Richie, As you seem to support the Kafka + Kinesis(?) API: We have developed Kadeck (kadeck.com), a control plane and collaboration layer for Kafka & Kinesis teams. I have looked at the documentation and reckon we should be supporting a lot of Warpstream already. Is there any quick way to test this out? Would love to share this with our users! Whether dead or alive, there's no denying that Kafka, or data streamin…

(WarpStream Cofounder)

https://docs.WarpStream.com is the best document we have right now. This sounds interesting though, can you jump in our slack or shoot me an email at founders@warpstreamlabs.com ? Happy to support you any way we can!

Re: Kafka is dead, long live Kafka

#257

Post by bob1029 »

CyanLite4 wrote: post #230
bob1029 wrote: post #199

Earlier quoted context omitted.

We use a table called "Messages" in our SQL Server database. Everyone talks to the same database. Turns out we don't really need to push extreme message rates or meet aggressive single-digit millisecond budgets, so this works out well in practice. It is also the easiest thing on earth to develop & debug, because you can monitor the table/log and instantly understand the state of the whole system and how it got there.…

fyi, SQL Server has a message broker built-in. https://learn.microsoft.com/en-us/sql/database-engine/config...

Not available in SQL Server Hyperscale, unfortunately. We think we can ride this one all the way.

Re: Kafka is dead, long live Kafka

#258

Post by banditelol »

richieartoul wrote: post #203
smrtinsert wrote: post #196

Earlier quoted context omitted.

Yeah really kind of surprised the free tier isn't the self hosted kind. That will keep me looking.

(WarpStream Cofounder) FWIW we’re considering a version where you can host the metadata yourself for enterprise users. For the free tier though we didn’t think it made sense since for a workload that could fit into our free tier, it didn’t seem like anyone would want to be responsible for the metadata layer themselves. Would love your feedback on that.

I'm also quite surprised there's no option for self hosting metadata on the free tier. To be fair, in my experience having managed metadata server for prefect (while they have similar Bring Your Own Server model) is quite hard to get right. But at the end we decided to keep maintaining it because our company prefer having all the data in our servers (including metadata).

But I'm still excited to try this, since at least now I could play around (and learn) with a partially kafka compatible system, without the burden of maintaining all of Kafka parts (and costs). Thanks!

Re: Kafka is dead, long live Kafka

#260

Post by chaotic-good »

ryanworl wrote: post #124
poorman wrote: post #121

Earlier quoted context omitted.

It sounds like there's a sweet spot here. If you are not ACKing Produce requests for 100ms then there's a huge amount latency. If the user want's to reduce that latency from 100ms to say 1ms then their S3 GET requests cost just went up by 100x.

[WarpStream co-founder here] We've done lots of customer research here and, combined with the experience my co-founder and I have, we can confidently say most Kafka users (especially high-throughput users) would happily make a trade off of increased end-to-end latency in exchange for a massive cost reduction and the operational simplicity provided by WarpStream.

It could be easy to operate when everything is fine but what's about incidents? If I understand correctly, there is a metadata database (BTW, is it multi-AZ as well?). But what if there is a data loss incident and some metadata was lost? Is it possible to recover from S3? If this is possible, then I guess that can't be very simple and should require a lot of time because S3 is not that easy to scan to find all the artefacts needed for recovery.

Also, this metadata database looks like a bottleneck. All writes and reads should go through it so it could be a point of failure. It's probably distributed and in this case it has its own complex failure modes and it has to be operated somehow.

Also, putting things from different partitions into one object is also something I'm not very keen about. You're introducing a lot of read amplification and S3 bills for egress. So if the object/file has data from 10 partitions and I only need 1, I'm paying for 10x more egress than I need to. The doc mentions fanout reads from multiple agents to satisfy a fetch request. I guess this is the price to pay for this. This is also affects the metadata database. If every object stores data from one partition the metadata can be easily partitioned. But if the object could have data from many partitions it's probably difficult to partition. One reason why Kafka/Redpanda/Pulsar scale very well is that the data and metadata can be easily partitioned and these systems do not have to handle as much metadata as I think WarpStream have to.

Post reply on HN