Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka is dead, long live Kafka

warpstream.com

Post reply on HN
Print view View original

201–210 of 295 posts

Re: Kafka is dead, long live Kafka

#201

Post by lmm »

whalesalad wrote: post #168
whatscooking wrote: post #52

Earlier quoted context omitted.

What? It’s pretty easy to set up and get going

The last time I was running a cluster it was 2014-2015 and it was not too bad to get up and running, but it felt so heavy handed for what it was. The zookeeper requirement was frustrating. The fact that consumers needed to maintain offset state was frustrating (although... understandable). For the right use case I suppose it is 100% worthwhile but more often than not I have seen it used in places where other tools wo…

They've made Zookeeper optional now, although they're being understandably conservative about that (e.g. only declared it production-ready a year or two ago).

Re: Kafka is dead, long live Kafka

#202

Post by Jasper_ »

whartung wrote: post #104
ram_rar wrote: post #77

Earlier quoted context omitted.

how do you design avoiding message queues? Or do you use other alternatives around kafka for these things?

There was recently an article about distributed systems that showed up here. (Harry Doyle: Christ, I can't find it. To hell with it!) And the author made a very interesting point about message queues. Simply, any problem that could be resolved by a message queue could be resolved by load balancing or persistence, and, therefore, messages queues were actually kind of a bad idea. There were two basic issues. The first…

Was it this one? https://ferd.ca/queues-don-t-fix-overload.html

Re: Kafka is dead, long live Kafka

#203

Post by richieartoul »

smrtinsert wrote: post #196
bothra90 wrote: post #49

Exciting work! Some questions: 1. Any plans on open-sourcing this? 2. Why not have a tiered architecture that can provide lower latency? p99 of 1s can be too high for some use-cases. 3. Related to 2, how does WarpStream compare to tiered storage in Pulsar? [Edit 1] Added (3)

Yeah really kind of surprised the free tier isn't the self hosted kind. That will keep me looking.

(WarpStream Cofounder)

FWIW we’re considering a version where you can host the metadata yourself for enterprise users. For the free tier though we didn’t think it made sense since for a workload that could fit into our free tier, it didn’t seem like anyone would want to be responsible for the metadata layer themselves. Would love your feedback on that.

Re: Kafka is dead, long live Kafka

#204

Post by otabdeveloper4 »

lmm wrote: post #197
slotrans wrote: post #68

> In our experience, Kafka is one of the most polarizing technologies in the data space. Some people hate it, some people swear by it, but almost every technology company uses it. (emphasis added) Surely that's false? Or, I mean, neither of us are providing any evidence here... For my part, 0 of the last 6 companies I've worked for used it. The company before that did (I drove its adoption), but we later abandoned it…

To me a technology company is not just a company that uses tech (every company does that) but one whose core value proposition is fundamentally technical. And I think most serious companies doing that have a need for highly available data storage, for which Kafka is the least bad option. What are the alternatives? Cassandra is just as operationally complex and harder to fit your dataflow into. The various efforts to…

We standardized on Clickhouse for everything. (With its own set of surprising and/or horrifying ops issues.) But at least it is a proper high-load, high-availablity solution, unlike Kafka, Cassandra, et al.

Re: Kafka is dead, long live Kafka

#205

Post by AtlasBarfed »

parhamn wrote: post #195
AtlasBarfed wrote: post #181

Earlier quoted context omitted.

1) scala/java ... is that fundamentally difficult? 2) zookeeper is being eliminated as a dependenct from kafka 3) durable disk management ... I mean, it's data, and it goes on a disk. Look, do you want a distributed fault-tolerant system that doesn't run on specialized / expensive hardware? Well, sorry, those systems are hard. I get this a lot for Cassandra. You either have the stones for it as a technical org to run…

> 1) scala/java ... is that fundamentally difficult? It's certainly at least more so as you have a highly configurable VM in-between where you're forced to learn java-isms to manage (can't just lean on your unix skills) > 3) durable disk management ... I mean, it's data, and it goes on a disk. Most MQ don't store things to disk besides memory flushing to recovery from crash, in most cases the data is cleared as soon…

But... go is gc as well. Most JVM gripes are about the knobs on GC, but Go is still a fundamentally GC'd language, so you'd have issues with that.

So... Go was the rewrite? Scylla at least rewrote Cassandra in C++ with some nice low-to-hardware improvements. Rust? ok. C++? ok. Avoid the GC pauses and get thread-per-core and userspace networking to bypass syscall boundaries.

And look, this thing is not going to steal the market share of Kafka. Kafka will continue to get supported, patched, and whenever the next API version of AWS comes out (it needs one), will this get updated for that?

Yeah, Kafka is "enterprisey" because ... it's java? Well no, Kafka is scalable, flexibly deployable (there's a reason big companies like the JVM), has a company behind it, is tunable, has support options, can be SaaS'd, has a knowledge database (REEEAAALLLLY important for distributed systems).

All those SQLite/RocksDB projects that slapped a raft protocol on top of them are in the same boat compared to Scylla or Cassandra or Dynamo. Distributed systems are HARD and need a mindshare of really smart experienced people that sustain them over time. Because when Kafka/Cassandra type systems get properly implemented, they are important systems moving / storing / processing a ton of data. I've seen hundred node Cassandra systems, those things aren't supposed to go down, ever. They are million dollar a year (maybe month) systems.

The big administration lifts in them like moving clouds, upgrading a cluster, recovering from region losses or intercontinental network outages are known quantities. Is some Go binary adhoc rewrite going to have all that? Documented with many people that know how to do it?

Re: Kafka is dead, long live Kafka

#206

Post by lmm »

otabdeveloper4 wrote: post #204
lmm wrote: post #197

Earlier quoted context omitted.

To me a technology company is not just a company that uses tech (every company does that) but one whose core value proposition is fundamentally technical. And I think most serious companies doing that have a need for highly available data storage, for which Kafka is the least bad option. What are the alternatives? Cassandra is just as operationally complex and harder to fit your dataflow into. The various efforts to…

We standardized on Clickhouse for everything. (With its own set of surprising and/or horrifying ops issues.) But at least it is a proper high-load, high-availablity solution, unlike Kafka, Cassandra, et al.

> We standardized on Clickhouse for everything. (With its own set of surprising and/or horrifying ops issues.)

Clickhouse I admittedly haven't personally seen quite as much operational unpleasantness as Greenplum or Galera, but at this point I'm dubious of anything in that bucket.

> But at least it is a proper high-load, high-availablity solution, unlike Kafka, Cassandra, et al.

What went wrong with those for you? In my experience the setup stage is cumbersome, but once you've got them running they work well and do what you expect; most complaints you see come down to they're not relational/not SQL/not ACID (true, but IME more of an advantage than a disadvantage).

Re: Kafka is dead, long live Kafka

#207

Post by EdwardDiego »

jitl wrote: post #182
EdwardDiego wrote: post #153

Earlier quoted context omitted.

ZooKeeper is rock solid. Moving off it is a mistake, IMO. My tinfoil hat theory is that the whole impetus for KRaft is Confluent Cloud's multi-tenanted clusters have so many partitions that it starts to exceed ZK's capacities, so Confluent have built KRaft for Confluent. And yeah, the migration approach is nutso. Also very annoying, the KRaft metadata topics being changed to be super-secret for... ...some good reason…

> ZooKeeper is rock solid lol. lmao. never falls over or causes incidents, in the same way c is rock solid and never SIGSEGV or causes security problems

Fair, that's just been in my experience. It's still better than KRaft though.

Re: Kafka is dead, long live Kafka

#209

Post by benjaminwootton »

richieartoul wrote: post #131
benjaminwootton wrote: post #119

Earlier quoted context omitted.

I’m not sure how anyone can hate Kafka? It does what it says on the tin - move data from A to B with publish/subscribe semantics. It’s quite easy to just use it as a dumb message broker with no retention if that’s all you need but if you do want to do something funky with persistence then go down that route. I’m not sure how anyone could have a negative feeling towards a vanilla, but rock solid and wildly popular ope…

(WarpStream co-founder) I'm not sure what you mean. Message persistence is a fundamental feature of Kafka that almost everyone using it relies on, its not some esoteric feature no one uses. We're each coming from our own network bias here, but in my experience a lot of people are really unhappy with the operational toil associated with running Kafka at scale in production.

Yes it’s fundamental, but it’s not generally that significant to users of Kafka.

As a developer, Kafka is a place to publish and subscribe to data with reliability and performance.

As a developer, the fact that messages are persistent is nothing more than a cool feature in that I can replay messages if I need to.

Things like consumer groups and offsets are features of the API, but they aren’t complex. Every similar tool whether it be RabbitMQ or IBM MQ has its own API abstractions and features. Likewise, I need to learn about failover semantics, but that’s the same with any API dependency.

It seems that you and the other posters here have a concensus that it’s hard to operate. Rather than saying that Kafka is dead or a polarising technology, a better line of argument is that it’s simply hard or expensive to operate at scale. (I personally think that’s par for the course with a technology like this, but that’s an aside.)

You have to remember that for everyone operating Kafka, there will be on average tens or hundreds of developers using it. And the vast, vast majority of those will not find it to be particularly polarising. Instead, they’ll find it a de-facto choice.

Re: Kafka is dead, long live Kafka

#210

Post by rupurt »

This is super interesting @richieartoul. I specced out something similar myself and was going to implement it in Zig https://github.com/fremantle-industries/transit.

FWIW I came to a similar conclusion that a lot of the power in Kafka comes from the API and that eventually much of the complexity of managing the cluster will eventually be abstracted away with multiple implementations. I also felt that if I could implement Kafka persistent over the S3 keyspace then I could start with persistence direct to S3 like you've done with warpstream and then layer on a faster hot disk and in memory tiering mechanism to eventually lower end to end latencies.

I love where you're going with this so hit me up on twitter if you ever want to chat more in-depth https://twitter.com/rupurt.

Post reply on HN