Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What If We Could Rebuild Kafka from Scratch?

morling.dev

Post reply on HN
Print view View original

191–200 of 229 posts

Re: What If We Could Rebuild Kafka from Scratch?

#191

Post by galeaspablo »

singron wrote: post #159
galeaspablo wrote: post #155

Earlier quoted context omitted.

Apologies, we've been so deep into this problem that we take our slang for granted :) A graphical representation might be worth a thousand words, keeping in mind it's just one example. Imagine you're traversing the following. A1 -> A2 -> A3... | v B1 -> B2 -> B3... | v C1 -> C2 -> C3... | v D1 -> D2 -> D3... | v E1 -> E2 -> E3... | v F1 -> F2 -> F3... | v ... Efficient concurrent consumption of these messages (while…

Do you have an example use case for this? This does seem like something unsuited to kafka, but I'm having a hard time imagining why you would structure something like this.

Great follow up question, thank you. I could talk about this "topic" for days, so I appreciate the opportunity to expand. :)

Let's imagine ourselves as a couple of engineers at Acme Foreign Exchange House. We'd like to track Acme's net cash position across multiple currencies, and execute trades accordingly (e.g., heding). And we'd like to retrospectively analyze our hedges, to assess their effectiveness.

Let's say I have this set of transactions (for accounts A, B, C, D, E, F, etc.)

A1 -> A2 -> A3 -> A4

B1 -> B2 -> B3 -> B4

C1-> C2

D1 -> D2 -> D3 -> D4

E1 -> E2

F1

Let's say that that:

- E1 was a deposit made into account E for $2M USD.

- E2 was an outgoing transfer of $2M USD sent to account F (incoming £1.7M GBP at F1).

If we consume our transactions and partiton our consumption by account id, we could get into a state where E1 and F1 are reflected in our net position, but E2 isn't. That is, our calculation has both $2M USD and £1.7M GBP, when in reality we only ever held either $2M USD or £1.7M GBP.

So what could we do?

1. Make sure that we respect causality order. I.e., there's no F1 reflected in our net position if we haven't processed E2.

2. Make sure that pairs of transactions (e.g., E2 and F1) update our net position atomically.

This is otherwise known as a "consistent cut" (see slide 25 here https://www.cs.cornell.edu/courses/cs6410/2011fa/lectures/19...).

Opinion: the world is causally ordered in arbitrary ways as above. But the tools, frameworks, and infrastructure more readily available to us struggle at modeling arbitrary partially ordered causality graphs. So we shrug our shoulders, and we learn to live with the edge cases. But it doesn't have to be so.

Re: What If We Could Rebuild Kafka from Scratch?

#192

Post by galeaspablo »

pas wrote: post #188
galeaspablo wrote: post #137

Earlier quoted context omitted.

Right on, great question. Some examples: Example Option 1 You give up on the guarantees across partition keys (bank accounts), and you accept that balances will not reflect a causally consistent state of the past. E.g., Bob deposits 100, Bob sends 50 to Alice. Balances: Bob 0 Alice 50 # the source system was never in this state Bob 100 Alice 50 # the source system was never in this state Bob 50 Alice 50 # eventually…

hmmm... could this be solved by "vector clocks"? if producers are emitting something that depends on a previous event they send the id of the previous event. (so like capabilities, you need proof of "data access".) or the problem is that again this is O(n^2)? (because then the consumers now need to buffer [potentially] n key streams (and then search for them every time - so "n" times)?

Great intuition. Exactly right. More details in one of my other comments, here: https://news.ycombinator.com/item?id=43796688

Re: What If We Could Rebuild Kafka from Scratch?

#193

Post by galeaspablo »

dpflan wrote: post #180
galeaspablo wrote: post #137

Earlier quoted context omitted.

Right on, great question. Some examples: Example Option 1 You give up on the guarantees across partition keys (bank accounts), and you accept that balances will not reflect a causally consistent state of the past. E.g., Bob deposits 100, Bob sends 50 to Alice. Balances: Bob 0 Alice 50 # the source system was never in this state Bob 100 Alice 50 # the source system was never in this state Bob 50 Alice 50 # eventually…

Thanks. With these of examples of messages appearance in time and in physical location in Kafka, how have you adapted your consumers? Which scenario / architectural decision (one of the examples?) have you moved forward with and creating support to yield your desired causality handling?

Option 1, but after so many years banging our heads against the wall reasoning about this, we hoped someone would eventually give us a queue that supports arbitrary causal dependency graphs.

We thought about building it ourselves, because we know the data structures, high level algorithms, and disk optimizations required. BUT we pivoted our company, so we've postponed this for the foreseeable future. After all, theory is relatively easy, but a true production grade implementation takes years.

Re: What If We Could Rebuild Kafka from Scratch?

#194

Post by sgarland »

nchmy wrote: post #178
raverbashing wrote: post #120

Earlier quoted context omitted.

Honestly that website has the least amount of information per text I've seen in multiple websites I had to really dig (outside of that website) to understand even what NATS is and/or does It goes too hard on the keyword babbling and too little on the "what does this actually do" > Services can live anywhere and are easily discoverable - decentralized, zerotrust security Ok cool, this tells me absolutely nothing. What…

Agreed. It is amazing though, and actually so simple that you don't need much docs. Their YouTube channel is excellent.

In no way are videos a substitute for docs, WTF.

Re: What If We Could Rebuild Kafka from Scratch?

#195

Post by sgarland »

EdwardDiego wrote: post #88
mrweasel wrote: post #25

Earlier quoted context omitted.

The worst part of Kafka, for me, is managing the cluster. I don't really like the partitioning and the almost hopelessness that ensues when something goes wrong. Recovery is really tricky. Granted it doesn't happen often, if you plan correctly, but the possibility of going wrong in the partitioning and replication makes updates and upgrades nightmare fuel.

Have a look at Strimzi, a K8s operator, gives you a mostly-managed Kafka experience.

Now you have two problems.

Re: What If We Could Rebuild Kafka from Scratch?

#196

Post by bjornsing »

bjornsing wrote: post #158
tinix wrote: post #149

Earlier quoted context omitted.

what can you do that redis can't? I'm also skeptical of the graph on your front page that claims S3 cost as much as DynamoDB. that alone makes it look like total nonsense. as someone else said, extraordinary claims require extraordinary evidence.

> what can you do that redis can't? Keep the data in S3 for 0.023 USD per GB-month. If you have a billion keys that can be useful. > I'm also skeptical of the graph on your front page that claims S3 cost as much as DynamoDB. Good point. Could have put a bit more work into that.

> Good point. Could have put a bit more work into that.

On second thought, and after looking at my cost estimates, the reason DynamoDB ends up costing about the same as S3 for this kind of use case is storage costs. DynamoDB is a lot cheaper than S3 to write to, but 5-10x more expensive to keep data stored in. So after about 16-32 months you reach break even.

Re: What If We Could Rebuild Kafka from Scratch?

#197

Post by Bombthecat »

munksbeer wrote: post #58
Ozzie_osman wrote: post #4

I feel like everyone's journey with Kafka ends up being pretty similar. Initially, you think "oh, an append-only log that can scale, brilliant and simple" then you try it out and realize it is far, far, from being simple.

I'm not a fan or an anti-fan of kafka, but I do wonder about the hate it gets. We use it for streaming tick data, system events, order events, etc, into kdb. We write to kafka and forget. The messages are persisted, and we don't have to worry if kdb has an issue. Out of band consumers read from the topics and persist to kdb. In several years of doing this we haven't really had any major issues. It does the job we wan…

How long is your Kafka down when you cut the cable to Kafka and it needs to fail over?

Re: What If We Could Rebuild Kafka from Scratch?

#198

Post by tarasglek »

1oooqooq wrote: post #131
hbogert wrote: post #50

Earlier quoted context omitted.

Tell me systemctl output isn't more beneficial than per distro bash-mess

not really. both requires that you know obscure and badly documented stuff. systemd whole premise is "people will not read the distro or bash scripting manual"... then nobody read systemd's (you have even less reason, since it's badly written, ever changing in conflicting ways, and a single use tool) so you went from complaining your coworkers can't write bash to complaining they don't know they have to use EXEC= EXE…

Bash scripts are write-only software. one read systemd c

Re: What If We Could Rebuild Kafka from Scratch?

#199

Post by zaphirplane »

EdwardDiego wrote: post #89
vkazanov wrote: post #32

Earlier quoted context omitted.

Yeah... It took 4 years to properly integrate Kafka into our pipelines. Everything, like everything is complicated with it: cluster management, numerous semi-tested configurations, etc. My final conclusion with it is that the project just doesn't really know what it wants to be. Instead it tries to provide everything for everybody, and ends up being an unbelievably complicated mess. You know, there are systems that k…

> that the project just doesn't really know what it wants to be It's a distributed log? What else is it trying to do?

Calling it a distributed log may just be a Reductio ad absurdum

Re: What If We Could Rebuild Kafka from Scratch?

#200

Post by zaphirplane »

hbogert wrote: post #50
1oooqooq wrote: post #39

Earlier quoted context omitted.

systemd knows very well what it wants to be, they just don't tell anyone. it's real goal is to make Linux administration as useless as windows so RH can sell certifications. tell me the output of systemctl is not as awful as opening the windows service panel.

Tell me systemctl output isn't more beneficial than per distro bash-mess

I have no dog in the systemd wars. The bash output is what? Distro use other open source software.
Post reply on HN