Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka Is Not a Database

materialize.com

Post reply on HN
Print view View original

121–130 of 172 posts

Re: Kafka Is Not a Database

#121

Post by vvern »

gunnarmorling wrote: post #117
vvern wrote: post #107

Earlier quoted context omitted.

> 1. Log "intents to write" rather than writes themselves in Topic A 2. Have a separate denormalization computed and kept in a separate Topic B, which can be read from. This denormalization needs to be read until the intent propagates from Topic A. 3. Convert those intents into commits. 4. Deal with all the failure cases in a distributed system, e.g. cleaning up abandoned intents, etc. People do do this. I have done…

>> If you use an OLTP database, and generate events into Kafka via CDC, you get the best of both worlds. > This is the next chapter in the gospel of the distributed transaction. Actually, it's the opposite. CDC helps to avoid distributed transaction; apps write to a single database only, and other resources (Kafka, other databases, etc.) based on that are updated asychronously, eventually consistent.

I mean, I hear you that people do that and it does allow you to avoid needing distributed transactions. When you're stuck with an underlying NoSQL database without transactions, this is the best thing you can do. This is the whole "invert the database" idea that ends up with [1].

I'm arguing that that usage pattern was painful and that if you can have horizontally scalable transaction processing and a stream of events due to committed transactions downstream, you can use that to power correct and easy to reason about materializations as well as asynchronous task processing.

Transact to change state. React to state changes. Prosper.

[1] https://youtu.be/5ZjhNTM8XU8?t=2075

Re: Kafka Is Not a Database

#122

Post by andrewem »

AndrewKemendo wrote: post #32

Alternatively from Jay Krebs [1] a much more thorough and nuanced discussion that is probably the best send-up on this topic. "So is it crazy to do this? The answer is no, there’s nothing crazy about storing data in Kafka: it works well for this because it was designed to do it. Data in Kafka is persisted to disk, checksummed, and replicated for fault tolerance. Accumulating more stored data doesn’t make it slower. T…

*Kreps, not Krebs

Re: Kafka Is Not a Database

#123

Post by wdb »

arafsheikh wrote: post #97
theptip wrote: post #90

Earlier quoted context omitted.

Short answer: you write your consumer's state into the same DB as you're writing the side-effects to, in the same transaction. Long answer: say your consumer is a service with a SQL DB -- if you want to process Event(offset=123), you need to 1. start a transaction, 2. write a record in your DB logging that you've consumed offset=123, 3. write your data for the side-effect, 4. commit your transaction. (Reverse 2 and 3…

Persisting message offsets in DB has its own challenges. The apps become tightly coupled with a specific Kafka cluster and that makes it difficult to swap clusters in case of a failover event. If you expect apps to persist offsets then it’s important to have a mechanism/process to safely reset the app state in DB when the stored offset doesn’t make sense.

Interesting, do you have any resources about how to best handle side effects with message queues (e.g. GCP PubSub)? Trying to find out if its worth the effort, or good practice to allow replay-ability (like from backup) of message and get back to the same state

Re: Kafka Is Not a Database

#124

Post by shay_ker »

frankmcsherry wrote: post #93
shay_ker wrote: post #83

This is maybe a silly question, but what's the difference between the timely dataflow that Materialize uses and Spark's execution engine? From my understanding they're doing very similar things - break down a sequence of functions on a stream of data, parallelize them on several machines, and then gather the results. I understand that the feature set of timely dataflow is more flexible than Spark - I just don't under…

There are a few differences, the main one between Spark and timely dataflow is that TD operators can be stateful, and so can respond to new rounds of input data in time proportional to the new input data, rather than that plus accumulated state. So, streaming one new record in and seeing how this changes the results of a multi-way join with many other large relations can happen in milliseconds in TD, vs batch systems…

That's very helpful, thanks! I think I still have to wrap my head around _why_ being stateful allows TD to respond faster, but maybe I just gotta dig deeper and see for myself.

Re: Kafka Is Not a Database

#125

Post by zaphar »

IfOnlyYouKnew wrote: post #113
zaphar wrote: post #101

Earlier quoted context omitted.

What is your distinction? Is mongodb a database? What about leveldb? Whether we want it to be so or not the term database is much more encompassing thank it used to be. You can try to fight that change if you want to but it means you'll be speaking a different language that most of the rest of us as a result.

You're using the term "traditional database" two levels up. Whatever your definition of that is, it's their definition of "database". Your initial post says "I have a different definition of 'database'", which is different than anyone else's, as you acknowledge when you refer to that common definition as a 'traditional database'. Then, you fault them for making an argument that is invalid for your non-standard defini…

No my post said that the term database has evolved beyond their definition. If you want to fight that trend then you will be swimming against the current.

Re: Kafka Is Not a Database

#126

Post by frankmcsherry »

shay_ker wrote: post #124
frankmcsherry wrote: post #93

Earlier quoted context omitted.

There are a few differences, the main one between Spark and timely dataflow is that TD operators can be stateful, and so can respond to new rounds of input data in time proportional to the new input data, rather than that plus accumulated state. So, streaming one new record in and seeing how this changes the results of a multi-way join with many other large relations can happen in milliseconds in TD, vs batch systems…

That's very helpful, thanks! I think I still have to wrap my head around _why_ being stateful allows TD to respond faster, but maybe I just gotta dig deeper and see for myself.

It just comes down to something as simple as: "if I have shown you 1M different things, and now show you one more thing, what do you have to do to tell me whether that one thing is new or not?"

If you can keep a hash map of the things you've seen, then it is easy to respond quickly to that one new thing. If you are not allowed to maintain any state, then you don't have a lot of options to efficiently respond to the new thing, and most likely need to re-read the 1M things.

That's the benefit of being stateful. There is a cost too, which is that you need to be able to reconstruct your state in the case of a failure, but fortunately things like differential dataflow (built on TD) are effectively deterministic.

Also, I suspect "Spark" is a moving target. The original paper described something that was very much a batch processor; they've been trying to fix that since, and perhaps they've made some progress in the intervening years.

Re: Kafka Is Not a Database

#127

Post by shay_ker »

frankmcsherry wrote: post #126
shay_ker wrote: post #124

Earlier quoted context omitted.

That's very helpful, thanks! I think I still have to wrap my head around _why_ being stateful allows TD to respond faster, but maybe I just gotta dig deeper and see for myself.

It just comes down to something as simple as: "if I have shown you 1M different things, and now show you one more thing, what do you have to do to tell me whether that one thing is new or not?" If you can keep a hash map of the things you've seen, then it is easy to respond quickly to that one new thing. If you are not allowed to maintain any state, then you don't have a lot of options to efficiently respond to the n…

I see. To my small brain it sounds like TD can intelligently memoize or cache the outputs of each "step" so that it only recalculates when it needs to as the inputs change.

I think Spark does that sometimes these days, but I don't know much about the specifics of how and when Spark does it.

Does TD have to keep _everything_ in memory, or can it be strategic in what it keeps and what it evicts?

Re: Kafka Is Not a Database

#128

Post by astn-austin »

shay_ker wrote: post #124
frankmcsherry wrote: post #93

Earlier quoted context omitted.

There are a few differences, the main one between Spark and timely dataflow is that TD operators can be stateful, and so can respond to new rounds of input data in time proportional to the new input data, rather than that plus accumulated state. So, streaming one new record in and seeing how this changes the results of a multi-way join with many other large relations can happen in milliseconds in TD, vs batch systems…

That's very helpful, thanks! I think I still have to wrap my head around _why_ being stateful allows TD to respond faster, but maybe I just gotta dig deeper and see for myself.

You should check out Apache Flink. It does a bunch of those things that Spark doesn't, though it's also missing a few things that Spark has. https://flink.apache.org/

Re: Kafka Is Not a Database

#129

Post by frankmcsherry »

shay_ker wrote: post #127
frankmcsherry wrote: post #126

Earlier quoted context omitted.

It just comes down to something as simple as: "if I have shown you 1M different things, and now show you one more thing, what do you have to do to tell me whether that one thing is new or not?" If you can keep a hash map of the things you've seen, then it is easy to respond quickly to that one new thing. If you are not allowed to maintain any state, then you don't have a lot of options to efficiently respond to the n…

I see. To my small brain it sounds like TD can intelligently memoize or cache the outputs of each "step" so that it only recalculates when it needs to as the inputs change. I think Spark does that sometimes these days, but I don't know much about the specifics of how and when Spark does it. Does TD have to keep _everything_ in memory, or can it be strategic in what it keeps and what it evicts?

TD lets you write whatever logic you want (it is fairly unopinionated on your logic and state).

Differential dataflow plugs in certain logic there, and it does indeed maintain a synopsis of what data have gone past, sufficient to respond to future updates but not necessarily the entirety of data that it has seen.

It would be tricky to implement DD over classic Spark, as DD relies on these synopses for its performance. There are some changes to Spark proposed in recent papers where it can pull in immutable LSM layers w/o reading them (e.g. just mmapping them) that might improve things, but until that happens there will be a gap.

Re: Kafka Is Not a Database

#130

Post by shay_ker »

frankmcsherry wrote: post #129
shay_ker wrote: post #127

Earlier quoted context omitted.

I see. To my small brain it sounds like TD can intelligently memoize or cache the outputs of each "step" so that it only recalculates when it needs to as the inputs change. I think Spark does that sometimes these days, but I don't know much about the specifics of how and when Spark does it. Does TD have to keep _everything_ in memory, or can it be strategic in what it keeps and what it evicts?

TD lets you write whatever logic you want (it is fairly unopinionated on your logic and state). Differential dataflow plugs in certain logic there, and it does indeed maintain a synopsis of what data have gone past, sufficient to respond to future updates but not necessarily the entirety of data that it has seen. It would be tricky to implement DD over classic Spark, as DD relies on these synopses for its performance…

Gotcha. Thanks for answering all my q's!
Post reply on HN