Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka Is Not a Database

materialize.com

Post reply on HN
Print view View original

81–90 of 172 posts

Re: Kafka Is Not a Database

#81

Post by georgewfraser »

j-pb wrote: post #26

Tbh, It's a weird blog post coming from the materialize folks, considering they know better. The "event sourced" arch they sketched is missing pieces. Normaly you'd have single writer instances that are locked to the corresponding kafka partition, which ensure strong transactional guarantees, IF you need them. Throwing shade for maketings sake is something that they should be above. I mean c'mon, I'd argue that Postg…

We're trying to address a real problem that is happening in our industry: VPs of eng and principal engineers at startups are adopting the "Kappa Architecture" / "Turning the Database Inside Out", without realizing how much functionality from traditional database systems they are leaving behind. This has led to a barrage of consistency bugs in everything from food-delivery apps to the "unread message count" in LinkedIn. We're at the peak of the hype cycle for Kafka, and it's being used in all kinds of places it doesn't belong. For 99% of companies, a traditional DBMS is the right foundation.

Re: Kafka Is Not a Database

#82

Post by Kalium »

EamonnMR wrote: post #72
Kalium wrote: post #15

As recently as last year, I worked for a company where the Chief Architect, in his infinite wisdom, had decided that a database was a silly legacy thing. The future looked like Kafka streams, with each service being a function against Kafka streams, and data retention set to infinite. Predictably, this setup ran into an interesting assortment of issues. There were no real transactions, no ensured consistency, and no…

At our company we use a ton of services that operate essentially as as functions on a Kafka stream (well, they tend to read/write in batches for efficiency) but we write event streams we want to query later into a regular database for later query. It works out very well. The idea of our poor Kafka cluster having to field queries in addition to the load of acting as a transport layer is frightening. The 'superpower' K…

This particular org did its level best to think of Kafka as the regular database for queries.

Re: Kafka Is Not a Database

#83

Post by shay_ker »

This is maybe a silly question, but what's the difference between the timely dataflow that Materialize uses and Spark's execution engine? From my understanding they're doing very similar things - break down a sequence of functions on a stream of data, parallelize them on several machines, and then gather the results.

I understand that the feature set of timely dataflow is more flexible than Spark - I just don't understand why (I couldn't figure it out from the paper, academic papers really go over my head).

Re: Kafka Is Not a Database

#84

Post by sixdimensional »

Kalium wrote: post #20
JMTQp8lwXL wrote: post #16

Earlier quoted context omitted.

Instead of single architects, companies need architect boards. And they can vote on these ideas before a single individual becomes a single point of failure. Expecting 1 person to make 100% correct decisions all the time is too much expectation for one person. People go down rabbit holes and they have weird takeaways, like replace all the databases with queues.

I agree in abstract, but in practice it's quite difficult to set up a successful democratic architecture board. You need teams or departments that all have architects, and an engineering organization where both managers and engineers accept a degree of centralized technical leadership. Getting there is, in my opinion, the work of years. It's especially challenging because spinning up such a board requires a person wh…

Thinking in terms of building physical buildings.. architects are often the visionaries but engineers are the realists. You know, architects come up with these crazy incredible building designs based on their engineering understanding, but ultimately it has to be vetted, proven and implemented by engineers.

I have, however, always wondered why people should be seen as "only architects" and "only engineers". While the separation of duties is critical to ensure the overall construction is sound, people can be visionary engineers, and people can be knowledgeable in both how to do something in real terms as well as dreaming on how to go beyond.

Re: Kafka Is Not a Database

#85

Post by theptip »

This is a bit dumbed down, and ignores the domain terminology required to properly discuss the trade-offs here (which is puzzling given that it links to a post by Aphyr, where you can find incredibly thorough discussions around isolation levels and anomalies).

> The fundamental problem with using Kafka as your primary data store is it provides no isolation.

This is false. I can only assume the author doesn't know about the Kafka transactions feature?

To be specific, Kafka's transaction machinery offers read-committed isolation, and you get read-uncommitted by default if you don't opt-in to use that transaction machinery (the docs: https://kafka.apache.org/0110/javadoc/index.html?org/apache/...). Depending on your workload, read-committed might be sufficient for correctness, in which case you can absolutely use Kafka as your database.

Of course, proving that your application is sound with just read-committed isolation is can be challenging, not to mention testing that your application continues to be sound as new features are added.

Because of that, in general I think that the underlying point of this article is probably correct, in that you probably shouldn't use Kafka as your database -- but for certain applications / use-cases it's a completely valid system design choice.

More generally this is an area that many applications get wrong by using the wrong isolation levels, because most frameworks encourage incorrect implementations by their unsafe defaults; e.g. see the classic "Feral concurrency control" paper http://www.bailis.org/papers/feral-sigmod2015.pdf. So I think the general message of "don't use Kafka as your DB unless you know enough about consistency to convince yourself that read-committed isolation is and will always be sufficient for your usecase" would be more appropriate (though it's certainly a less snappy title).

Re: Kafka Is Not a Database

#86

Post by Kalium »

sixdimensional wrote: post #79
Kalium wrote: post #15

As recently as last year, I worked for a company where the Chief Architect, in his infinite wisdom, had decided that a database was a silly legacy thing. The future looked like Kafka streams, with each service being a function against Kafka streams, and data retention set to infinite. Predictably, this setup ran into an interesting assortment of issues. There were no real transactions, no ensured consistency, and no…

It sounds like this would have made a better proof of concept than an commitment to the architecture. The idea on the face of it is not per se a bad one, quite interesting, but the implementations are perhaps not there yet to back such an idea. It's important to know as an architect when your vision for the architecture is outpacing reality, to know your time horizon, and match the vision with the tools that help you…

It was definitely an interesting proof of concept that needed some refinement. The core idea was functional services against nicely organized data streams on a flat network. Which is a really cool approach that works quite well for a lot of things.

Several of these points fell apart when credit card handling and PCI-DSS entered the picture.

Re: Kafka Is Not a Database

#87

Post by mamon »

shay_ker wrote: post #83

This is maybe a silly question, but what's the difference between the timely dataflow that Materialize uses and Spark's execution engine? From my understanding they're doing very similar things - break down a sequence of functions on a stream of data, parallelize them on several machines, and then gather the results. I understand that the feature set of timely dataflow is more flexible than Spark - I just don't under…

There's no difference really. All "Big Data" (tm) tools are trying to capitalize on the hype, so Kafka adds database capabilities, while Spark adds Streaming. At some point they will reach feature parity.

Re: Kafka Is Not a Database

#89

Post by georgewfraser »

theptip wrote: post #85

This is a bit dumbed down, and ignores the domain terminology required to properly discuss the trade-offs here (which is puzzling given that it links to a post by Aphyr, where you can find incredibly thorough discussions around isolation levels and anomalies). > The fundamental problem with using Kafka as your primary data store is it provides no isolation. This is false. I can only assume the author doesn't know abo…

"Read-committed isolation" is not a meaningful implementation of transactions. If you can't do read, then a write, while guaranteeing the database didn't change in between, then you don't really have transactions.

Re: Kafka Is Not a Database

#90

Post by theptip »

jashmatthews wrote: post #78
cfontes wrote: post #66

Earlier quoted context omitted.

It actually does with |Exactly-once Semantics| in fact I've been using as the single source of truth in a cash management system for almost 2 years without a single issue related to transactions.

How do you deal with side effects outside of the Kafka cluster?

Short answer: you write your consumer's state into the same DB as you're writing the side-effects to, in the same transaction.

Long answer: say your consumer is a service with a SQL DB -- if you want to process Event(offset=123), you need to 1. start a transaction, 2. write a record in your DB logging that you've consumed offset=123, 3. write your data for the side-effect, 4. commit your transaction. (Reverse 2 and 3 if you prefer; it shouldn't make a difference). If your side-effect write fails (say your DB goes down) then your transaction will be broken, your side-effect won't be readable outside the transaction, and the update to the consumer offset pointer also won't get persisted. Next loop around on your consumer's event loop, you'll start at the same offset and retry the same transaction.

Post reply on HN