Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What If We Could Rebuild Kafka from Scratch?

morling.dev

Post reply on HN
Print view View original

131–140 of 229 posts

Re: What If We Could Rebuild Kafka from Scratch?

#131

Post by 1oooqooq »

hbogert wrote: post #50
1oooqooq wrote: post #39

Earlier quoted context omitted.

systemd knows very well what it wants to be, they just don't tell anyone. it's real goal is to make Linux administration as useless as windows so RH can sell certifications. tell me the output of systemctl is not as awful as opening the windows service panel.

Tell me systemctl output isn't more beneficial than per distro bash-mess

not really. both requires that you know obscure and badly documented stuff.

systemd whole premise is "people will not read the distro or bash scripting manual"...

then nobody read systemd's (you have even less reason, since it's badly written, ever changing in conflicting ways, and a single use tool)

so you went from complaining your coworkers can't write bash to complaining they don't know they have to use EXEC= EXEC=/bin/x

because random values, without any hint, are lists of commands instead of string values.

Re: What If We Could Rebuild Kafka from Scratch?

#132

Post by dpflan »

galeaspablo wrote: post #82
vermon wrote: post #26

Interesting, if partitioning is not a useful concept of Kafka, what are some of the better alternatives for controlling consumer concurrency?

It is useful, but it is not generally applicable. Given an arbitrary causality graph between n messages, it would be ideal if you could consume your messages in topological order. And that you could do so in O(n log n). No queuing system in the world does arbitrary causality graphs without O(n^2) costs. I dream of the day where this changes. And because of this, we’ve adapted our message causality topologies to cope…

Can you elaborate on how you have “adapted…message causality topologies to cope with consuming mechanisms” in relation to the example of a bank account? The causality topology being what here, couldn’t one day MoneyIn should come before else there can be now true MoneyOut?

Re: What If We Could Rebuild Kafka from Scratch?

#133

Post by galeaspablo »

singron wrote: post #108
galeaspablo wrote: post #80

Agreed. The head of line problem is worth solving for certain use cases. But today, all streaming systems (or workarounds) with per message key acknowledgements incur O(n^2) costs in either computation, bandwidth, or storage per n messages. This applies to Pulsar for example, which is often used for this feature. Now, now, this degenerate time/space complexity might not show up every day, but when it does, you’re toa…

Check out the parallel consumer: https://github.com/confluentinc/parallel-consumer It processes unrelated keys in parallel within a partition. It has to track what offsets have been processed between the last committed offset of the partition and the tip (i.e. only what's currently processed out of order). When it commits, it saves this state in the commit metadata highly compressed. Most of the time, it was only pro…

Disclosure (given this is from Confluent): I'm ex MSK (Managed Streaming for Kafka at AWS) and my current company was competing with Confluent before we pivoted.

Yup, this is one more example, just like Pulsar. There are definitely great optimizations to be made on the average case. In the case of parallel consumer, if you'd like to keep ordering guarantees, you retain O(n^2) processing time in the worst case.

The issues arise when you try to traverse arbitrary dependency topologies in your messages. So you're left with two options:

1. Make damn sure that causal dependencies don't exhibit O(n^2) behavior, which requires formal models to be 100% sure. 2. Give up ordering or make some other nasty tradeoff.

At a high level the problem boils down to traversing a DAG in topological order. From computer science theory, we know that this requires a sorted index. And if you're implementing an index on top of Kafka, you might as well embed your data into and consume directly from the index. Of course, this is easier said than done, and that's why no one has cracked this problem yet. We were going to try, but alas we pivoted :)

Edit: Topological sort does not required a sorted index (or similar) if you don't care about concurrency. But then you've lost the advantages of your queue.

Re: What If We Could Rebuild Kafka from Scratch?

#134

Post by galeaspablo »

bjornsing wrote: post #113
cogman10 wrote: post #87

Earlier quoted context omitted.

> streaming system will process n messages in O (n log n) I'm guessing this is mostly around how backed up the stream is. n isn't the total number of messages but rather the current number of unacked messages. Would a radix structure work better here? If you throw something like a UUID7 on the messages and store them in a radix structure you should be able to get O(n) performance here correct? Or am I not understandi…

I think the problem is that if you want quick access to all messages with a particular key then you have to maintain some kind of index over all persisted messages. So n would be total number of persisted messages as I read it, which can be quite large. But even storing them in the first place is O(n), so O(n log n) might not be so bad.

That's correct. And keep in mind that you might have new consumers starting from the beginning come into play, so you have to permanently retain the indexes.

And yes, O(n log n ) is not bad at all. Sorted database indexes (whether SQL, NoSQL, or AcmeVendorSQL, etc.) already take O(n log n) to insert n elements into data storage or to read n elements from data storage.

Re: What If We Could Rebuild Kafka from Scratch?

#135

Post by Misdicorl »

singron wrote: post #108
galeaspablo wrote: post #80

Agreed. The head of line problem is worth solving for certain use cases. But today, all streaming systems (or workarounds) with per message key acknowledgements incur O(n^2) costs in either computation, bandwidth, or storage per n messages. This applies to Pulsar for example, which is often used for this feature. Now, now, this degenerate time/space complexity might not show up every day, but when it does, you’re toa…

Check out the parallel consumer: https://github.com/confluentinc/parallel-consumer It processes unrelated keys in parallel within a partition. It has to track what offsets have been processed between the last committed offset of the partition and the tip (i.e. only what's currently processed out of order). When it commits, it saves this state in the commit metadata highly compressed. Most of the time, it was only pro…

I suppose it depends on your message volume. To me, processing 100k messages and then getting a page however long later as the broker (or whatever) falls apart sounds much worse than head of line blocking and seeing the problem directly in my consumer. If I need to not do head of line blocking, I can build whatever failsafe mechanisms I need for the problematic data and defer to some other queueing system (typically, just add an attempt counter and replay the message to the same kafka topic and then if attempts > X, send it off to wherever)

I'd rather debug a worker problem than an infra scaling problem every day of the week and twice on Sundays.

Re: What If We Could Rebuild Kafka from Scratch?

#136

Post by Misdicorl »

Misdicorl wrote: post #135
singron wrote: post #108

Earlier quoted context omitted.

Check out the parallel consumer: https://github.com/confluentinc/parallel-consumer It processes unrelated keys in parallel within a partition. It has to track what offsets have been processed between the last committed offset of the partition and the tip (i.e. only what's currently processed out of order). When it commits, it saves this state in the commit metadata highly compressed. Most of the time, it was only pro…

I suppose it depends on your message volume. To me, processing 100k messages and then getting a page however long later as the broker (or whatever) falls apart sounds much worse than head of line blocking and seeing the problem directly in my consumer. If I need to not do head of line blocking, I can build whatever failsafe mechanisms I need for the problematic data and defer to some other queueing system (typically,…

Follow on: If you're using kafka to publish messages to multiple consumers, this is even worse as now you're infecting every consumer with data processing issues from every other consumer. Bad juju

Re: What If We Could Rebuild Kafka from Scratch?

#137

Post by galeaspablo »

dpflan wrote: post #132
galeaspablo wrote: post #82

Earlier quoted context omitted.

It is useful, but it is not generally applicable. Given an arbitrary causality graph between n messages, it would be ideal if you could consume your messages in topological order. And that you could do so in O(n log n). No queuing system in the world does arbitrary causality graphs without O(n^2) costs. I dream of the day where this changes. And because of this, we’ve adapted our message causality topologies to cope…

Can you elaborate on how you have “adapted…message causality topologies to cope with consuming mechanisms” in relation to the example of a bank account? The causality topology being what here, couldn’t one day MoneyIn should come before else there can be now true MoneyOut?

Right on, great question. Some examples:

Example Option 1

You give up on the guarantees across partition keys (bank accounts), and you accept that balances will not reflect a causally consistent state of the past.

E.g., Bob deposits 100, Bob sends 50 to Alice.

Balances: Bob 0 Alice 50 # the source system was never in this state Bob 100 Alice 50 # the source system was never in this state Bob 50 Alice 50 # eventually consistent final state

Example Option 2

You give up on parallelism, and consume in total order (i.e., one single partition / unit of parallelism - e.g., in Kafka set a partitioner that always hashes to the same value).

Example Option 3

In the consumer you "wait" whenever you get a message that violates causal order.

E.g., Bob deposits 100 Bob sends 50 to Alice (Bob-MoneyOut 50 -> Alice-MoneyIn 50).

If we attempt to consume Alice-MoneyIn before Bob-MoneyOut, we exponentially back off from the partition containing Alice-MoneyIn.

(Option 3 is terrible because of O(n^2) processing times in the worst case and the possibility for deadlocks (two partitions are waiting for one another))

Re: What If We Could Rebuild Kafka from Scratch?

#138

Post by hardwaresofton »

dangoodmanUT wrote: post #119
hardwaresofton wrote: post #2

See also: Warpstream, which was so good it got acquired by Confluent. Feels like there is another squeeze in that idea if someone “just” took all their docs and replicated the feature set. But maybe that’s what S2 is already aiming at. Wonder how long warpstream docs, marketing materials and useful blogs will stay up.

i wouldn't say it was so good it got acquired by them, rather confluent had no s3-backed play and it was easier for them to acquire warpstream than to add it to kafka directly warpstream has latency issues, which downstream turn into cost issues

That's a good point -- I assumed there were other choices, but now that I look, warpstream may have been the only already-kafka-compatible option.

That said, they were at least "good enough" to make "buy" more appealing than "build"

Re: What If We Could Rebuild Kafka from Scratch?

#139

Post by galeaspablo »

bjornsing wrote: post #129

> Key-centric access: instead of partition-based access, efficient access and replay of all the messages with one and the same key would be desirable. I’ve been working on a datastore that’s perfect for this [1], but I’m getting very little traction. Does anyone have any ideas why that is? Is my marketing just bad, or is this feature just not very useful after all? 1. https://www.haystackdb.dev/

Some input from previously working on a superset of this problem. And being in a similar position.

Mature projects have too much bureacracy, and even spending time talking to you = opportunity cost. So making a case for why you're going to solve a problem for them is tough.

New projects (whether at big companies or small companies) have 20 other things to worry about, so the problem isn't big enough.

I wrote about this in our blog if you're curious: https://ambar.cloud/blog/a-new-path-for-ambar

Re: What If We Could Rebuild Kafka from Scratch?

#140

Post by nitwit005 »

salomonk_mur wrote: post #76
nitwit005 wrote: post #17

> When producing a record to a topic and then using that record for materializing some derived data view on some downstream data store, there’s no way for the producer to know when it will be able to "see" that downstream update. For certain use cases it would be helpful to be able to guarantee that derived data views have been updated when a produce request gets acknowledged, allowing Kafka to act as a log for a tru…

Yeah... Not happening when you have scores of clients running down your database. The reason message queue systems exist is scale. Good luck sending a notification at 9am to your 3 million users and keeping your database alive in the sudden influx of activity. You need to queue that load.

Kafka is itself a database. Sending a message requires what is essentially a database insert. You're still doing a DB commit either way.
Post reply on HN