Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What If We Could Rebuild Kafka from Scratch?

morling.dev

Post reply on HN
Print view View original

71–80 of 229 posts

Re: What If We Could Rebuild Kafka from Scratch?

#71

Post by frklem »

gunnarmorling wrote: post #69
frklem wrote: post #52

"Faced with such a marked defensive negative attitude on the part of a biased culture, men who have knowledge of technical objects and appreciate their significance try to justify their judgment by giving to the technical object the only status that today has any stability apart from that granted to aesthetic objects, the status of something sacred. This, of course, gives rise to an intemperate technicism that is not…

For each of the items discussed I explicitly mention why they would be desirable to have. How is this engineering for the sake of engineering?

True, for each of the points discussed, there is an explicit mention on why it is desirable. But those are technical solutions, to technical problems. There is nothing wrong with that. The issue is, that the whole article is about technicalities because of technicalities, hence the 'engineering for the cause of engineering' (which is different from '.. for the sake of...'). It is at this point that the 'idea of rebuilding Kafka' becomes a pure technical one, detached from the intention of having something like Kafka. Other commenters in the thread also pointed out to the fact of Kafka not having a clear intention. I agree that a lot of software nowadays suffer from the same problem.

Re: What If We Could Rebuild Kafka from Scratch?

#72

Post by gunnarmorling »

carlmr wrote: post #10
Ozzie_osman wrote: post #4

I feel like everyone's journey with Kafka ends up being pretty similar. Initially, you think "oh, an append-only log that can scale, brilliant and simple" then you try it out and realize it is far, far, from being simple.

I'm wondering how much of that is bad developer UX and defaults, and how much of that is inherent complexity in the problem space. Like the article outlines, partitions are not that useful for most people. Instead of removing them, how about having them behind a feature flag, i.e. not on by default. That would ease 99% of users problems. The next point in the article which to me resonates is the lack of proper schema…

> why is there no in-memory kafka server that I can use for simple testing purposes.

Take a look at Debezium's KafkaCluster, which is exactly that: https://github.com/debezium/debezium/blob/main/debezium-core....

It's used within Debezium's test suite. Check out the test for this class itself to see how it's being used: https://github.com/debezium/debezium/blob/main/debezium-core...

Re: What If We Could Rebuild Kafka from Scratch?

#74

Post by Hamuko »

Ozzie_osman wrote: post #4

I feel like everyone's journey with Kafka ends up being pretty similar. Initially, you think "oh, an append-only log that can scale, brilliant and simple" then you try it out and realize it is far, far, from being simple.

>Initially, you think "oh, an append-only log that can scale, brilliant and simple"

Really? I got scared by Kafka by just reading through the documentation.

Re: What If We Could Rebuild Kafka from Scratch?

#75

Post by bushbaba »

vergessenmir wrote: post #64
immibis wrote: post #36

Earlier quoted context omitted.

Why do you need to queue the writes?

some writes might fail, you may need to retry, the data store may be temporarily available etc. There may be many things that go wrong and how you handle this depends on your data guarantees and consistency requirements. If you're not queuing what are you doing when a write fails, throwing away the data?

Some Kafka writes might fail. Hence the Kafka client having a queue with retries.

Re: What If We Could Rebuild Kafka from Scratch?

#76

Post by salomonk_mur »

nitwit005 wrote: post #17

> When producing a record to a topic and then using that record for materializing some derived data view on some downstream data store, there’s no way for the producer to know when it will be able to "see" that downstream update. For certain use cases it would be helpful to be able to guarantee that derived data views have been updated when a produce request gets acknowledged, allowing Kafka to act as a log for a tru…

Yeah... Not happening when you have scores of clients running down your database.

The reason message queue systems exist is scale. Good luck sending a notification at 9am to your 3 million users and keeping your database alive in the sudden influx of activity. You need to queue that load.

Re: What If We Could Rebuild Kafka from Scratch?

#77

Post by [deleted] »

gunnarmorling wrote: post #72
carlmr wrote: post #10

Earlier quoted context omitted.

I'm wondering how much of that is bad developer UX and defaults, and how much of that is inherent complexity in the problem space. Like the article outlines, partitions are not that useful for most people. Instead of removing them, how about having them behind a feature flag, i.e. not on by default. That would ease 99% of users problems. The next point in the article which to me resonates is the lack of proper schema…

> why is there no in-memory kafka server that I can use for simple testing purposes. Take a look at Debezium's KafkaCluster, which is exactly that: https://github.com/debezium/debezium/blob/main/debezium-core... . It's used within Debezium's test suite. Check out the test for this class itself to see how it's being used: https://github.com/debezium/debezium/blob/main/debezium-core...

[deleted]

Re: What If We Could Rebuild Kafka from Scratch?

#79

Post by gunnarmorling »

supermatt wrote: post #56

> "Do away with partitions" > "Key-level streams (... of events)" When you are leaning on the storage backend for physical partitioning (as per the cloud example, where they would literally partition based on keys), doesnt this effectively just boil down to renaming partitions to keys, and keys to events?

That's one way to look at this, yes. The difference being that keys actually have a meaning to clients (as providers of ordering and also a failure domain), whereas partitions in their current form don't.

Re: What If We Could Rebuild Kafka from Scratch?

#80

Post by galeaspablo »

Agreed. The head of line problem is worth solving for certain use cases.

But today, all streaming systems (or workarounds) with per message key acknowledgements incur O(n^2) costs in either computation, bandwidth, or storage per n messages. This applies to Pulsar for example, which is often used for this feature.

Now, now, this degenerate time/space complexity might not show up every day, but when it does, you’re toast, and you have to wait it out.

My colleagues and I have studied this problem in depth for years, and our conclusion is that a fundamental architectural change is needed to support scalable per message key acknowledgements. Furthermore, the architecture will fundamentally require a sorted index, meaning that any such a queuing / streaming system will process n messages in O (n log n).

We’ve wanted to blog about this for a while, but never found the time. I hope this comment helps out if you’re thinking of relying on per message key acknowledgments; you should expect sporadic outages / delays.

Post reply on HN