Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What If We Could Rebuild Kafka from Scratch?

morling.dev

Post reply on HN
Print view View original

161–170 of 229 posts

Re: What If We Could Rebuild Kafka from Scratch?

#161

Post by singron »

Misdicorl wrote: post #135
singron wrote: post #108

Earlier quoted context omitted.

Check out the parallel consumer: https://github.com/confluentinc/parallel-consumer It processes unrelated keys in parallel within a partition. It has to track what offsets have been processed between the last committed offset of the partition and the tip (i.e. only what's currently processed out of order). When it commits, it saves this state in the commit metadata highly compressed. Most of the time, it was only pro…

I suppose it depends on your message volume. To me, processing 100k messages and then getting a page however long later as the broker (or whatever) falls apart sounds much worse than head of line blocking and seeing the problem directly in my consumer. If I need to not do head of line blocking, I can build whatever failsafe mechanisms I need for the problematic data and defer to some other queueing system (typically,…

It's interesting you say that, since this turned an infra scaling problem into a worker problem for us. Previously, we would get terrible head-of-line throughput issues, so we would use an egregious number of partitions to try to alleviate that. Lots of partitions is hard to manage since resizing topics is operationally tedious and it puts a lot of strain on brokers. But no matter how many partitions you have, the head-of-line still blocks. Even cases where certain keys had slightly slower throughput would clog up the whole partition with normal consumers.

The parallel consumer nearly entirely solved this problem. Only the most egregious cases where keys were ~3000 times slower than other keys would cause an issue, and then you could solve it by disabling that key for a while.

Re: What If We Could Rebuild Kafka from Scratch?

#164

Post by nicksnyder »

peanut-walrus wrote: post #125
gunnarmorling wrote: post #68

Earlier quoted context omitted.

It can increase latency (which can be somewhat mitigated though by having a write buffer e.g. on EBS volumes), but it substantially _reduces_ cost: all cross-AZ traffic (which is $$$) is handled by the object storage layer, where it doesn't get charged. This architecture has been tremendously popular recently, championed by Warpstream and also available by Confluent (Freight clusters), AutoMQ, BufStream, etc. The KIP…

So it's cheaper *on AWS*. Any cloud provider where cross-AZ traffic is not $$$, I can't imagine this architecture being cheaper. Engineering solutions which only exist because AWS pricing is whack are...well, certainly a choice. I can also think of lots of cases where whatever you're running is fine to just run in a single AZ since it's not critical.

The other clouds have fees like this too.

Even if this were to change, using object storage results in a lot of operational simplicity as well compared to managing a bunch of disks. You can easily and quickly scale to zero or scale up to handle bursts in traffic.

An architecture like this also makes it possible to achieve a truly active-active multi-region Kafka cluster that has real SLAs.

See: https://buf.build/blog/bufstream-multi-region

(disclosure: I work at Buf)

Re: What If We Could Rebuild Kafka from Scratch?

#165

Post by jjk7 »

dangoodmanUT wrote: post #119
hardwaresofton wrote: post #2

See also: Warpstream, which was so good it got acquired by Confluent. Feels like there is another squeeze in that idea if someone “just” took all their docs and replicated the feature set. But maybe that’s what S2 is already aiming at. Wonder how long warpstream docs, marketing materials and useful blogs will stay up.

i wouldn't say it was so good it got acquired by them, rather confluent had no s3-backed play and it was easier for them to acquire warpstream than to add it to kafka directly warpstream has latency issues, which downstream turn into cost issues

Confluent has "Freight" which is their integrated s3-backed play.

Re: What If We Could Rebuild Kafka from Scratch?

#166

Post by throwaway7783 »

Spivak wrote: post #70
wvh wrote: post #37

Earlier quoted context omitted.

The problem is that you don't know who's listening. You don't want all possible interested parties to hammer the database. Hence the events in between. Arguably, I'd not use Kafka to store actual data, just to notify in-flight.

For read only queries, hammer away I can scale reads nigh infinitely horizontally. There's no secret sauce that makes it so that only kafka can do this.

It's not about scaling reads, but coordinating consumers so no more than one consumer processes same messages. That means some kind of locking, that means scaling issues.

Re: What If We Could Rebuild Kafka from Scratch?

#167

Post by gitroom »

honestly, kafka always felt like way more moving parts than my brain wants to track, but at the same time, its kinda impressive how the ecosystem just keeps growing - you think the reason people stick to it is hype, laziness, or just not enough real pain yet to switch?

Re: What If We Could Rebuild Kafka from Scratch?

#168

Post by bjornsing »

thesimon wrote: post #154
bjornsing wrote: post #129

> Key-centric access: instead of partition-based access, efficient access and replay of all the messages with one and the same key would be desirable. I’ve been working on a datastore that’s perfect for this [1], but I’m getting very little traction. Does anyone have any ideas why that is? Is my marketing just bad, or is this feature just not very useful after all? 1. https://www.haystackdb.dev/

> HaystackDB is accessed through a RESTful HTTPS API. No client library necessary. That's cool, but but I would prefer to not reinvent the wheel. If you have a simple library, that would already be useful. Some simple code or request examples would be convenient as well. I really don't know how easy or difficult your interface design is. It would be cool to see the API docs.

Yeah, it’s a bit of a chicken and egg problem. Since I don’t have a way to find potential customers I feel it’s too risky investing in stuff like client libraries and good API docs. But I can definitely understand you’d like to see more.

Re: What If We Could Rebuild Kafka from Scratch?

#169

Post by SkipperCat »

EdwardDiego wrote: post #92
elvircrn wrote: post #31

Surprised there's no mention of Redpanda here.

Redpanda isn't FOSS. Apache Kafka is.

Redpanda community edition operates under the BSL license - https://www.redpanda.com/blog/open-source

IANAL, but it looks pretty open source to me.

Re: What If We Could Rebuild Kafka from Scratch?

#170

Post by klooney »

amazingman wrote: post #153
benjaminwootton wrote: post #130

Earlier quoted context omitted.

That’s my experience too. I’ve deployed it more than ten times as a consultant and never really understood the reputation for complexity. It “just works.”

I've deployed it a bunch of times and, crucially, maintained it thereafter. It's very complex, especially when troubleshooting pathological behavior or recovering from failures, and I don't see why anyone with significant experience with Kafka could reasonably claim otherwise. Kafka is perhaps the most aptly named software I've ever used. That said, it's rock solid and I continue to recommend it for cases where it ma…

I know a team that had their Kafka cluster fall over, and they couldn't get it to stay up until eventually their LOB got shut down for being unreliable. I don't know if they were especially bad at their jobs, or their volumes were unreasonable, or what, but it seemed like a bad time for all.
Post reply on HN