Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What If We Could Rebuild Kafka from Scratch?

morling.dev

Post reply on HN
Print view View original

201–210 of 229 posts

Re: What If We Could Rebuild Kafka from Scratch?

#201

Post by nitwit005 »

lamp_book wrote: post #187
nitwit005 wrote: post #140

Earlier quoted context omitted.

Kafka is itself a database. Sending a message requires what is essentially a database insert. You're still doing a DB commit either way.

It's more of a commit log/write-ahead log/replication stream than a DBMS - consider that DBMSs typically include these in addition to their primary storage.

It's clearly not a full blown database product, but it's still got the core elements of a database. Your data is getting replicated to multiple instances, written to disk, and an index is created for quick lookup.

It's just the table that's getting is essentially append only (excepting the cleanup processes it supports).

Re: What If We Could Rebuild Kafka from Scratch?

#202

Post by nchmy »

sgarland wrote: post #194
nchmy wrote: post #178

Earlier quoted context omitted.

Agreed. It is amazing though, and actually so simple that you don't need much docs. Their YouTube channel is excellent.

In no way are videos a substitute for docs, WTF.

I literally agreed that the docs were shit. I just happened to say the videos were great.

Re: What If We Could Rebuild Kafka from Scratch?

#203

Post by wvh »

immibis wrote: post #85
wvh wrote: post #37

Earlier quoted context omitted.

The problem is that you don't know who's listening. You don't want all possible interested parties to hammer the database. Hence the events in between. Arguably, I'd not use Kafka to store actual data, just to notify in-flight.

But you do know who's listening, because you were the one who installed all the listeners. ("you" can be plural.) This reminds me of the OOP vs DOD debate again. OOP adherents say they don't know all the types of data their code operates on; DOD adherents say they actually do, since their program contains a finite number of classes, and a finite subset of those classes can be the ones called in this particular virtua…

I have worked for financial institutions where random departments have random interests in certain data transactions. You (as in a dev team in one such department) have no say in who touches the data, from where, and how it's used. Kafka is used as a corporate message bus to let e.g. the accountant department know something happened in another department. Those "listening" departments don't have devs and are not involved in development, they operate more on the MS BI level of PowerPoint.

So yes, in large companies, your development team is just a small cog, you don't set policy for what happens to the data you gather. And in some sectors, like finances, you are an especially small cog with little power, which might sound strange if you only ever worked for a software startup.

Re: What If We Could Rebuild Kafka from Scratch?

#204

Post by voxic11 »

nicksnyder wrote: post #164
peanut-walrus wrote: post #125

Earlier quoted context omitted.

So it's cheaper *on AWS*. Any cloud provider where cross-AZ traffic is not $$$, I can't imagine this architecture being cheaper. Engineering solutions which only exist because AWS pricing is whack are...well, certainly a choice. I can also think of lots of cases where whatever you're running is fine to just run in a single AZ since it's not critical.

The other clouds have fees like this too. Even if this were to change, using object storage results in a lot of operational simplicity as well compared to managing a bunch of disks. You can easily and quickly scale to zero or scale up to handle bursts in traffic. An architecture like this also makes it possible to achieve a truly active-active multi-region Kafka cluster that has real SLAs. See: https://buf.build/blog…

I think Azure eliminated their inter-availability zone data transfer fees last year. https://azure.microsoft.com/en-us/updates?id=update-on-inter...

Re: What If We Could Rebuild Kafka from Scratch?

#205

Post by bluecarbuncle »

singron wrote: post #108
galeaspablo wrote: post #80

Agreed. The head of line problem is worth solving for certain use cases. But today, all streaming systems (or workarounds) with per message key acknowledgements incur O(n^2) costs in either computation, bandwidth, or storage per n messages. This applies to Pulsar for example, which is often used for this feature. Now, now, this degenerate time/space complexity might not show up every day, but when it does, you’re toa…

Check out the parallel consumer: https://github.com/confluentinc/parallel-consumer It processes unrelated keys in parallel within a partition. It has to track what offsets have been processed between the last committed offset of the partition and the tip (i.e. only what's currently processed out of order). When it commits, it saves this state in the commit metadata highly compressed. Most of the time, it was only pro…

There's also a similar project from Line https://github.com/line/decaton.

Re: What If We Could Rebuild Kafka from Scratch?

#206

Post by lamp_book »

nitwit005 wrote: post #201
lamp_book wrote: post #187

Earlier quoted context omitted.

It's more of a commit log/write-ahead log/replication stream than a DBMS - consider that DBMSs typically include these in addition to their primary storage.

It's clearly not a full blown database product, but it's still got the core elements of a database. Your data is getting replicated to multiple instances, written to disk, and an index is created for quick lookup. It's just the table that's getting is essentially append only (excepting the cleanup processes it supports).

My point was more about the two having separate use cases - but Kafka handles write throughput much easier compared to an RDBMS due to lack of transactions and not needing to manipulate the primary table data structures.

Re: What If We Could Rebuild Kafka from Scratch?

#207

Post by dxxvi »

carlmr wrote: post #10
Ozzie_osman wrote: post #4

I feel like everyone's journey with Kafka ends up being pretty similar. Initially, you think "oh, an append-only log that can scale, brilliant and simple" then you try it out and realize it is far, far, from being simple.

I'm wondering how much of that is bad developer UX and defaults, and how much of that is inherent complexity in the problem space. Like the article outlines, partitions are not that useful for most people. Instead of removing them, how about having them behind a feature flag, i.e. not on by default. That would ease 99% of users problems. The next point in the article which to me resonates is the lack of proper schema…

> why is there no in-memory kafka server that I can use for simple testing purposes https://github.com/embeddedkafka/embedded-kafka It's for scala. I'm trying to do something similar in Java but haven't got time yet.

Re: What If We Could Rebuild Kafka from Scratch?

#208

Post by taurath »

galeaspablo wrote: post #155
taurath wrote: post #146

Earlier quoted context omitted.

> traverse arbitrary dependency topologies Is there another way to state this? It’s very difficult for me to grok. > DAG Directed acyclic graph right?

Apologies, we've been so deep into this problem that we take our slang for granted :) A graphical representation might be worth a thousand words, keeping in mind it's just one example. Imagine you're traversing the following. A1 -> A2 -> A3... | v B1 -> B2 -> B3... | v C1 -> C2 -> C3... | v D1 -> D2 -> D3... | v E1 -> E2 -> E3... | v F1 -> F2 -> F3... | v ... Efficient concurrent consumption of these messages (while…

Wanted to say thanks so much for writing this all out - I've always thought of ordering as being sort of inherently against the point of parallel streams, so its interesting to hear about the state of the art and the benefits that are trying to be gleaned! I'm not thinking in stream processors terribly often so I wasn't aware of how dependencies are mapped.

If you don't mind another followup (and your patience with my ignorance hasn't run out :P), wouldn't the efficient concurrent consumption imply knowing the dependency graph before the events are processed? IE, is it possible in any instance to get to O(w+h) in a stream?

Re: What If We Could Rebuild Kafka from Scratch?

#209

Post by atombender »

raverbashing wrote: post #120
vim-guru wrote: post #16

https://nats.io is easier to use than Kafka and already solves several of the points in this post I believe, like removing partitions, supporting key-based streams, and having flexible topic hierarchies.

Honestly that website has the least amount of information per text I've seen in multiple websites I had to really dig (outside of that website) to understand even what NATS is and/or does It goes too hard on the keyword babbling and too little on the "what does this actually do" > Services can live anywhere and are easily discoverable - decentralized, zerotrust security Ok cool, this tells me absolutely nothing. What…

The documentation site might be a little better: https://docs.nats.io/

NATS is mainly two things:

1. Core NATS, an ephemeral message broker. It's a lightweight pub/sub system where routing of messages is based on wildcard paths. All in memory, lightning fast, extremely lightweight. You can use it for RPC, queues, broadcasting, file transfer, anything.

2. JetStream, which is a Kafka/Pulsar-like log built on top of Core NATS. Streams are indexed, meaning there's no strong need for partitioning or consumer groups, since readers can efficiently filter the stream by interest (you can still partition for write performance). Supports both durable and ephemeral consumers, in-memory streams, ack/nack, priority groups, deduplication, exactly-once delivery, hierarchical clusters, offline clusters ("leaf clusters"), mirroring, etc.

I often find it difficult to explain the magic of NATS. It's a communication model that doesn't really exist anywhere else, as far as I've seen. The closest might be ZeroMQ.

JetStream could be explained as "Kafka for people who don't want to administer Kafka". It's very low-maintenance, very easy to use, feels super lightweight, and still offers much of the performance and reliability of Kafka, as well as a much richer feature set that maps better to what people may want from a streaming log.

Re: What If We Could Rebuild Kafka from Scratch?

#210

Post by kcexn »

galeaspablo wrote: post #80

Agreed. The head of line problem is worth solving for certain use cases. But today, all streaming systems (or workarounds) with per message key acknowledgements incur O(n^2) costs in either computation, bandwidth, or storage per n messages. This applies to Pulsar for example, which is often used for this feature. Now, now, this degenerate time/space complexity might not show up every day, but when it does, you’re toa…

> Furthermore, the architecture will fundamentally require a sorted index, meaning that any such a queuing / streaming system will process n messages in O (n log n).

Would using a sorted index have an impact on the measured servicing time of each message? (Not worst-case, something more like average-cass). It's made extremely clear in the Kafka docs that Kafka's relies heavily on the operating systems filesystem cache for performance, and that seeking through events on disk turns out to be very slow compared to just processing events in-order.

Post reply on HN