Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka is dead, long live Kafka

warpstream.com

Post reply on HN
Print view View original

181–190 of 295 posts

Re: Kafka is dead, long live Kafka

#181

Post by AtlasBarfed »

parhamn wrote: post #146
benjaminwootton wrote: post #128

Earlier quoted context omitted.

Why? Its quite easy to use Kafka as a messaging queue without even thinking about the write ahead log semantics. It’s there if you need it, but Kafka scales down to being a message broker fairly well in my opinion.

Because operationalizing Kafka is difficult from a infrastructure (scala/java, zookeper, durable disk management, lots of moving parts), learning and a code perspective (pointer tracking, partition management, delegation, etc) relative to the other pubsub/mq tools. So if you don't have it operationalized and your use case is simple, it makes most sense to use a simpler tool (rmq/ampq, cloud pubsub, nsq, etc, perhaps…

1) scala/java ... is that fundamentally difficult?

2) zookeeper is being eliminated as a dependenct from kafka

3) durable disk management ... I mean, it's data, and it goes on a disk.

Look, do you want a distributed fault-tolerant system that doesn't run on specialized / expensive hardware? Well, sorry, those systems are hard. I get this a lot for Cassandra.

You either have the stones for it as a technical org to run software like that, or you pay SAAS overhead for it. A Go binary is not going to magically solve this.

EVEN IF you go SaaS, you still need monitoring and a host of other aspects (perf testing, metrics, etc) to keep abreast of your overall system.

And what's with pretending that S3 doesn't have ingress/egress charges? Last I checked those were more or less in like with EBS networking charges and inter-region costs, but I haven't looked in like a year.

And if this basically ties you to AWS, then why not just ... pay for AWS managed Kafka from Confluent?

The big fake sell from this is that it magically makes Kafka easy because it ... uses Go and uses S3. From my experience, those and "disk management" aren't the big headaches with Kafka and Cassandra masterless distributed systems. They are maybe 5% of the headaches or less.

Re: Kafka is dead, long live Kafka

#182

Post by jitl »

EdwardDiego wrote: post #153
mdaniel wrote: post #143

Earlier quoted context omitted.

> I’m not sure how anyone can hate Kafka? My experience has been that such a question has two implied audiences in it: those who consume Kafka and those who have to keep the PoS alive and healthy The whole ambiguity around whether ZK is really still needed or not https://kafka.apache.org/documentation/#zk > makes keeping two distributed systems alive and healthy, but don't worry you can't move your production cluster…

ZooKeeper is rock solid. Moving off it is a mistake, IMO. My tinfoil hat theory is that the whole impetus for KRaft is Confluent Cloud's multi-tenanted clusters have so many partitions that it starts to exceed ZK's capacities, so Confluent have built KRaft for Confluent. And yeah, the migration approach is nutso. Also very annoying, the KRaft metadata topics being changed to be super-secret for... ...some good reason…

> ZooKeeper is rock solid

lol. lmao. never falls over or causes incidents, in the same way c is rock solid and never SIGSEGV or causes security problems

Re: Kafka is dead, long live Kafka

#183

Post by kbumsik »

ryanworl wrote: post #2

I'm Ryan Worl, co-founder and CTO of WarpStream. We're super excited to announce our Developer Preview of our Kafka protocol compatible streaming system built directly on top of S3 with no stateful disks/nodes to run, no rebalancing data, no ZooKeeper, and 5-10x cheaper because of no cross-AZ bandwidth charges. If you have any questions about WarpStream, my co-founder (richieartoul) and I will be here to answer them.

Does it support S3-compatible services, notably Cloudflare R2? I heard that there might be special handling for each S3-compatible providers, due the slightly different API behavior and different consistency models, and etc.

If it support Cloudflare R2 then it would be great for multi-cloud too.

Re: Kafka is dead, long live Kafka

#184

Post by AtlasBarfed »

snapetom wrote: post #106
carlio wrote: post #78

Earlier quoted context omitted.

I work as a contractor so I move between places. I have found a few companies trying to introduce kafka, and every time it has been a solution in search of a problem. I don't doubt that it has a good use case but I have so far only encountered the zealots who crowbar it into any situation and that has left a residual bad taste in my mouth. So I fall into the "hate it" side.

> and every time it has been a solution in search of a problem. More refined to this, in my experience at the last two jobs, the queue problem is there, but the Kafka solution is based solely on "enterpriseyness" of Kafka, not any practical reason. RabbitMQ is highly performant, SQS is really easy. Both are great queues. Kafka is muuch more, yet, Kafka is chosen because "it's enterprise."

RabbitMQ is "highly performant" is a handwave. The words tell me nothing, just like any other tech/software that is described as "powerful".

In my last two major gigs, RabbitMQ was already being run in a clustered config, and it was not going well. Both places were in the process of doing arch changes to do a change to Kafka.

It seems like something that works great in a big scaled node and you can go to big nodes these days, but I don't think it is ready for cloud/distributed durability.

I'm not aware of Jepsen testing of RabbitMQ in distributed mode for example, and I wouldn't consider any distributed/clustered product that hasn't let Jepsen embarass it yet.

Cassandra and Kafka are frequent examples of YAGNI overengineering (although the fault tolerance can be nice without scale), the reality is that pumping up single-node solutions for too long is a big trap. Projects that start to stress single-nodes (I'm thinking like a 4xlarge anything on aws) should probably get to thinking about the need for jumping to dynamodb/cassandra/bigtable/kafka/etc.

RabbitMQ --> Kafka is a pretty easy lift if your messaging has good abstractions.

relational DB --> Cassandra is a lot bigger headache because of the lack of joins.

Re: Kafka is dead, long live Kafka

#185

Post by siliconc0w »

It's a neat idea for a SaaS to just host the control-plane/metadata and allow customers to write to their own data store via S3. I can't think of too many other systems work that way - it certainly is operationally simpler but I wonder if it also gets you some compliance wins.

I wonder if we'll eventually standardize on a cloud-agnostic S3-like protocol but for locking and fast transactional writes so we can just have companies building pure software again but using standardized Cloud instead of POSIX APIs.

Re: Kafka is dead, long live Kafka

#186

Post by edude03 »

Okay, I hate to be that guy but I'm given two options for pricing

> Free but with only 24 Hours of Retention

or

> Contact us.

Pricing is hard for sure, but if I'm being honest neither of these options make me want to try the product.

Re: Kafka is dead, long live Kafka

#187

Post by twelve40 »

benjaminwootton wrote: post #119
slotrans wrote: post #68

> In our experience, Kafka is one of the most polarizing technologies in the data space. Some people hate it, some people swear by it, but almost every technology company uses it. (emphasis added) Surely that's false? Or, I mean, neither of us are providing any evidence here... For my part, 0 of the last 6 companies I've worked for used it. The company before that did (I drove its adoption), but we later abandoned it…

I’m not sure how anyone can hate Kafka? It does what it says on the tin - move data from A to B with publish/subscribe semantics. It’s quite easy to just use it as a dumb message broker with no retention if that’s all you need but if you do want to do something funky with persistence then go down that route. I’m not sure how anyone could have a negative feeling towards a vanilla, but rock solid and wildly popular ope…

i work at a place that has a whole devops department maintaining it (so not even my pain), and i still hate how overengineered the overall system ended up, it's more busywork programming for it, debugging it, etc. No reason not to use sqs or rabbit or whatever unless you have a very special use case, or begin to hit X messages/second. Or just like to spend lots of time writing boilerplate and configuring.

Re: Kafka is dead, long live Kafka

#188

Post by richieartoul »

edude03 wrote: post #186

Okay, I hate to be that guy but I'm given two options for pricing > Free but with only 24 Hours of Retention or > Contact us. Pricing is hard for sure, but if I'm being honest neither of these options make me want to try the product.

(WarpStream co-founder)

I know that is super frustrating, I’m sorry. We’re just really early and unsure how to begin even discussing public pricing.

We are 100% committed to that free tier. We designed our entire architecture around making that free tier so cost effective that we could offer it for free.

We want to get to a point where we can have transparent pricing on our website, it will just take us some time to figure that out after speaking with initial customers. We would love to hear your feedback though. You won’t get handed off to a sales guy or anything, it’ll just be a direct conversation with me (CEO) and my cofounder Ryan (CTO)

Re: Kafka is dead, long live Kafka

#189

Post by pronoiac »

whartung wrote: post #104
ram_rar wrote: post #77

Earlier quoted context omitted.

how do you design avoiding message queues? Or do you use other alternatives around kafka for these things?

There was recently an article about distributed systems that showed up here. (Harry Doyle: Christ, I can't find it. To hell with it!) And the author made a very interesting point about message queues. Simply, any problem that could be resolved by a message queue could be resolved by load balancing or persistence, and, therefore, messages queues were actually kind of a bad idea. There were two basic issues. The first…

If I'm thinking of the same article: was it about going from message queues to state machines for more operational robustness?

Re: Kafka is dead, long live Kafka

#190

Post by richieartoul »

guru4consulting wrote: post #179
poorman wrote: post #121

Earlier quoted context omitted.

It sounds like there's a sweet spot here. If you are not ACKing Produce requests for 100ms then there's a huge amount latency. If the user want's to reduce that latency from 100ms to say 1ms then their S3 GET requests cost just went up by 100x.

won't that be a problem for high-traffic topics? Kafka latency is usually in single digit milliseconds. For a topic with high throughput, a typical java client instance can send thousands of messages per second. When the acknowledgement latency increases to 1000ms, then the producer client would need to have multiple threads to handle the blocking calls. Either producer will have to scale to multiple instances, or el…

(WarpStream cofounder)

Yeah you have to produce in parallel and use batching, but it works well in practice. We’ve tested it up to 1GiB/s in throughout without issue

Post reply on HN