Kafka's been dead almost 100 years.
Kafka is dead, long live Kafka
191–200 of 295 posts
Re: Kafka is dead, long live Kafka
#192I didn't check the site. I have nightmares debugging kafka from years ago and it's only huger now. No thank you, give me Apache MQ or RabbitMQ, even IBM's MQ and it's all good and don't try to tell me I need a streaming thingamajig.
Re: Kafka is dead, long live Kafka
#193> In our experience, Kafka is one of the most polarizing technologies in the data space. Some people hate it, some people swear by it, but almost every technology company uses it. (emphasis added) Surely that's false? Or, I mean, neither of us are providing any evidence here... For my part, 0 of the last 6 companies I've worked for used it. The company before that did (I drove its adoption), but we later abandoned it…
What do you use instead? Polling APIs? A queue instead of an event stream?
Event based architectures definitely add infrastructural overhead, but are positive at a certain scale and/or architectural complexity (multiple decoupled subscribers).
Re: Kafka is dead, long live Kafka
#194Earlier quoted context omitted.
ZooKeeper is rock solid. Moving off it is a mistake, IMO. My tinfoil hat theory is that the whole impetus for KRaft is Confluent Cloud's multi-tenanted clusters have so many partitions that it starts to exceed ZK's capacities, so Confluent have built KRaft for Confluent. And yeah, the migration approach is nutso. Also very annoying, the KRaft metadata topics being changed to be super-secret for... ...some good reason…
> ZooKeeper is rock solid lol. lmao. never falls over or causes incidents, in the same way c is rock solid and never SIGSEGV or causes security problems
Some people say "well, it could be worse" but my go-to version is "at least it's not fucking etcd"
Re: Kafka is dead, long live Kafka
#195Earlier quoted context omitted.
Because operationalizing Kafka is difficult from a infrastructure (scala/java, zookeper, durable disk management, lots of moving parts), learning and a code perspective (pointer tracking, partition management, delegation, etc) relative to the other pubsub/mq tools. So if you don't have it operationalized and your use case is simple, it makes most sense to use a simpler tool (rmq/ampq, cloud pubsub, nsq, etc, perhaps…
1) scala/java ... is that fundamentally difficult? 2) zookeeper is being eliminated as a dependenct from kafka 3) durable disk management ... I mean, it's data, and it goes on a disk. Look, do you want a distributed fault-tolerant system that doesn't run on specialized / expensive hardware? Well, sorry, those systems are hard. I get this a lot for Cassandra. You either have the stones for it as a technical org to run…
> 1) scala/java ... is that fundamentally difficult?
It's certainly at least more so as you have a highly configurable VM in-between where you're forced to learn java-isms to manage (can't just lean on your unix skills)
> 3) durable disk management ... I mean, it's data, and it goes on a disk.
Most MQ don't store things to disk besides memory flushing to recovery from crash, in most cases the data is cleared as soon as the message is acked/expired.
Look, I'm not saying not to use Kafka, I'm just pointing out the evaluation criteria. There are certainly better options if you just want a MQ, especially if you want to support MQ patterns like fanout.
The reality is if you're doing <20k TPS on a MQ (most are) and don't need replay/persistance, then ./redis-server will suffice and operationally it will be much much easier.
Re: Kafka is dead, long live Kafka
#196Exciting work! Some questions: 1. Any plans on open-sourcing this? 2. Why not have a tiered architecture that can provide lower latency? p99 of 1s can be too high for some use-cases. 3. Related to 2, how does WarpStream compare to tiered storage in Pulsar? [Edit 1] Added (3)
Yeah really kind of surprised the free tier isn't the self hosted kind. That will keep me looking.
Re: Kafka is dead, long live Kafka
#197> In our experience, Kafka is one of the most polarizing technologies in the data space. Some people hate it, some people swear by it, but almost every technology company uses it. (emphasis added) Surely that's false? Or, I mean, neither of us are providing any evidence here... For my part, 0 of the last 6 companies I've worked for used it. The company before that did (I drove its adoption), but we later abandoned it…
To me a technology company is not just a company that uses tech (every company does that) but one whose core value proposition is fundamentally technical. And I think most serious companies doing that have a need for highly available data storage, for which Kafka is the least bad option.
What are the alternatives? Cassandra is just as operationally complex and harder to fit your dataflow into. The various efforts to built proper master-master HA on MySQL or PostgreSQL or similar tend to be flaky, expensive, and vendor-lockined. BigTable can work if you're all-in on Google Cloud, but that's quite a risk.
As far as I can tell there are mostly companies that use Kafka and companies that have a SPOF PostgreSQL/MySQL database (with some read replicas, and maybe some untested perl scripts that are supposed to be able to promote a replica to master) and stick their fingers in their ears.
Re: Kafka is dead, long live Kafka
#198Earlier quoted context omitted.
how do you design avoiding message queues? Or do you use other alternatives around kafka for these things?
There was recently an article about distributed systems that showed up here. (Harry Doyle: Christ, I can't find it. To hell with it!) And the author made a very interesting point about message queues. Simply, any problem that could be resolved by a message queue could be resolved by load balancing or persistence, and, therefore, messages queues were actually kind of a bad idea. There were two basic issues. The first…
You might be thinking of:
https://blog.nelhage.com/post/systems-at-capacity/
Re: Kafka is dead, long live Kafka
#199> In our experience, Kafka is one of the most polarizing technologies in the data space. Some people hate it, some people swear by it, but almost every technology company uses it. (emphasis added) Surely that's false? Or, I mean, neither of us are providing any evidence here... For my part, 0 of the last 6 companies I've worked for used it. The company before that did (I drove its adoption), but we later abandoned it…
What do you use instead? Polling APIs? A queue instead of an event stream? Event based architectures definitely add infrastructural overhead, but are positive at a certain scale and/or architectural complexity (multiple decoupled subscribers).
We use a table called "Messages" in our SQL Server database. Everyone talks to the same database. Turns out we don't really need to push extreme message rates or meet aggressive single-digit millisecond budgets, so this works out well in practice. It is also the easiest thing on earth to develop & debug, because you can monitor the table/log and instantly understand the state of the whole system and how it got there. Oh - it is also magically included in our DR strategy. No extra infra. We don't have to have a totally separate procedure to recover this thing.
We primarily use it as a backhaul between parts of our infrastructure in order to perform RPC. The approach is for the users of the broker (our services) to poll it at whatever rate is required. This is actually a little bit clever if you think about it - Users that don't really care about liveliness can poll for their messages every minute or so. Users that are in the hot path of a web UI could poll every 50~100ms.
Polling sounds kinda shitty (at least to me) but I argue it's the best default engineering solution until proven otherwise (assuming its not somehow harder than the other magic async event bubbling things). We don't have a lot of services doing this so contention isn't really a problem for us. Even if it did get to that point, I would reach for a read replica before I refactored how all of messaging worked. Most of polling is just a read operation that does nothing, so we can go horizontal on that part pretty easily.
Re: Kafka is dead, long live Kafka
#200Earlier quoted context omitted.
ZooKeeper is rock solid. Moving off it is a mistake, IMO. My tinfoil hat theory is that the whole impetus for KRaft is Confluent Cloud's multi-tenanted clusters have so many partitions that it starts to exceed ZK's capacities, so Confluent have built KRaft for Confluent. And yeah, the migration approach is nutso. Also very annoying, the KRaft metadata topics being changed to be super-secret for... ...some good reason…
> ZooKeeper is rock solid lol. lmao. never falls over or causes incidents, in the same way c is rock solid and never SIGSEGV or causes security problems
This thread is an excellent example of the author's point: Kafka is polarizing.
Personally, in my experience with Kafka and Zookeeper at Airbnb back in the day (we also used ZK for general-purpose service discovery), they both were... temperamental. They'd chug along just fine for a bit, seemingly handling outages that e.g. RDS would have thrown a fit over, and then suddenly they'd be cataclysmically down in extremely complicated ways and be very difficult to bring back up. Even just using them required teaching a more complex mental model than most cloud-hosted offerings of similar things, and you ended up in this path dependency trap of "we already invested so much in Kafka, so if you want to send a message, use Kafka" when for like 95+% of use cases something easy like SQS would've been fine and simpler. TBQH I don't think either Kafka or ZK ever quite paid back their operational overhead cost, and personally I wouldn't recommend using either unless you absolutely need to.
Warpstream looks really cool in that light!