Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What If We Could Rebuild Kafka from Scratch?

morling.dev

Post reply on HN
Print view View original

111–120 of 229 posts

Re: What If We Could Rebuild Kafka from Scratch?

#111

Post by dheera »

mike_hearn wrote: post #103
wvh wrote: post #37

Earlier quoted context omitted.

The problem is that you don't know who's listening. You don't want all possible interested parties to hammer the database. Hence the events in between. Arguably, I'd not use Kafka to store actual data, just to notify in-flight.

In some databases that's not a problem. Oracle has a built in horizontally scalable message queue engine that's transactional with the rest of the database. You can register a series of SELECT queries and be notified when the results have (probably) changed, either via direct TCP server push or via a queued message for pickup later. It's not polling based, the transaction engine knows what query predicates to keep an…

I know as an Oracle employee you don't want to hear this, but part of the problem is that you are no longer database-agnostic if you do this.

The messaging tech being separate from the database tech means the architects can swap out the database if needed in the future without needing to rewrite the producers and consumers.

Re: What If We Could Rebuild Kafka from Scratch?

#112

Post by openplatypus »

peanut-walrus wrote: post #13

Object storage for Kafka? Wouldn't this 10x the latency and cost? I feel like Kafka is a victim of it's own success, it's excellent for what it was designed, but since the design is simple and elegant, people have been using it for all sorts of things for which it was not designed. And well, of course it's not perfect for these use cases.

> Object storage for Kafka? Wouldn't this 10x the latency and cost?

It will become slower. It will become costlier (to maintain). And we will end up with local replicas for performance.

If only people looked outside AWS bubble and realised they are SEVERELY overcharged for storage, this would be mute point.

I would welcome getting partition dropped in favour of multi-tenancy ... but for my use cases this is often equivalent.

Storage is not the problem though.

Re: What If We Could Rebuild Kafka from Scratch?

#113

Post by bjornsing »

cogman10 wrote: post #87
galeaspablo wrote: post #80

Agreed. The head of line problem is worth solving for certain use cases. But today, all streaming systems (or workarounds) with per message key acknowledgements incur O(n^2) costs in either computation, bandwidth, or storage per n messages. This applies to Pulsar for example, which is often used for this feature. Now, now, this degenerate time/space complexity might not show up every day, but when it does, you’re toa…

> streaming system will process n messages in O (n log n) I'm guessing this is mostly around how backed up the stream is. n isn't the total number of messages but rather the current number of unacked messages. Would a radix structure work better here? If you throw something like a UUID7 on the messages and store them in a radix structure you should be able to get O(n) performance here correct? Or am I not understandi…

I think the problem is that if you want quick access to all messages with a particular key then you have to maintain some kind of index over all persisted messages. So n would be total number of persisted messages as I read it, which can be quite large. But even storing them in the first place is O(n), so O(n log n) might not be so bad.

Re: What If We Could Rebuild Kafka from Scratch?

#114

Post by bjornsing »

peanut-walrus wrote: post #13

Object storage for Kafka? Wouldn't this 10x the latency and cost? I feel like Kafka is a victim of it's own success, it's excellent for what it was designed, but since the design is simple and elegant, people have been using it for all sorts of things for which it was not designed. And well, of course it's not perfect for these use cases.

The weird thing driving this thinking is that cross-AZ network data transfer between EC2 instances on AWS is more expensive than shuffling the same data through S3 (which has free data transfer to/from EC2). It’s just stupid, but that’s how it is.

Re: What If We Could Rebuild Kafka from Scratch?

#115

Post by radiator »

DrFalkyn wrote: post #107
munksbeer wrote: post #58

Earlier quoted context omitted.

I'm not a fan or an anti-fan of kafka, but I do wonder about the hate it gets. We use it for streaming tick data, system events, order events, etc, into kdb. We write to kafka and forget. The messages are persisted, and we don't have to worry if kdb has an issue. Out of band consumers read from the topics and persist to kdb. In several years of doing this we haven't really had any major issues. It does the job we wan…

What happens if the Kafka node fails ?

"the" node? Kafka is a cluster of multiple nodes.

Re: What If We Could Rebuild Kafka from Scratch?

#116

Post by [deleted] »

frklem wrote: post #71
gunnarmorling wrote: post #69

Earlier quoted context omitted.

For each of the items discussed I explicitly mention why they would be desirable to have. How is this engineering for the sake of engineering?

True, for each of the points discussed, there is an explicit mention on why it is desirable. But those are technical solutions, to technical problems. There is nothing wrong with that. The issue is, that the whole article is about technicalities because of technicalities, hence the 'engineering for the cause of engineering' (which is different from '.. for the sake of...'). It is at this point that the 'idea of rebui…

[deleted]

Re: What If We Could Rebuild Kafka from Scratch?

#117

Post by dangoodmanUT »

axegon_ wrote: post #35
elvircrn wrote: post #31

Surprised there's no mention of Redpanda here.

Having used both Kafka and Redpanda on several occasions, I'd pick Redpanda any day of the week without a second thought. Easier to setup, easier to maintain, a lot less finicky ans uses a fraction of the resources.

same

Re: What If We Could Rebuild Kafka from Scratch?

#118

Post by mike_hearn »

dheera wrote: post #111
mike_hearn wrote: post #103

Earlier quoted context omitted.

In some databases that's not a problem. Oracle has a built in horizontally scalable message queue engine that's transactional with the rest of the database. You can register a series of SELECT queries and be notified when the results have (probably) changed, either via direct TCP server push or via a queued message for pickup later. It's not polling based, the transaction engine knows what query predicates to keep an…

I know as an Oracle employee you don't want to hear this, but part of the problem is that you are no longer database-agnostic if you do this. The messaging tech being separate from the database tech means the architects can swap out the database if needed in the future without needing to rewrite the producers and consumers.

I don't work on the database itself, so it's neither here nor there to me. Still, the benefits of targeting the LCD must be weighed against the costs. Not having scalable transactions imposes a huge drain on the engineering org that sucks up time and imposes opportunity costs.

Re: What If We Could Rebuild Kafka from Scratch?

#119

Post by dangoodmanUT »

hardwaresofton wrote: post #2

See also: Warpstream, which was so good it got acquired by Confluent. Feels like there is another squeeze in that idea if someone “just” took all their docs and replicated the feature set. But maybe that’s what S2 is already aiming at. Wonder how long warpstream docs, marketing materials and useful blogs will stay up.

i wouldn't say it was so good it got acquired by them, rather confluent had no s3-backed play and it was easier for them to acquire warpstream than to add it to kafka directly

warpstream has latency issues, which downstream turn into cost issues

Re: What If We Could Rebuild Kafka from Scratch?

#120

Post by raverbashing »

vim-guru wrote: post #16

https://nats.io is easier to use than Kafka and already solves several of the points in this post I believe, like removing partitions, supporting key-based streams, and having flexible topic hierarchies.

Honestly that website has the least amount of information per text I've seen in multiple websites

I had to really dig (outside of that website) to understand even what NATS is and/or does

It goes too hard on the keyword babbling and too little on the "what does this actually do"

> Services can live anywhere and are easily discoverable - decentralized, zerotrust security

Ok cool, this tells me absolutely nothing. What service? Who to whom? Discovering what?

Post reply on HN