Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka Is Not a Database

materialize.com

Post reply on HN
Print view View original

161–170 of 172 posts

Re: Kafka Is Not a Database

#161

Post by ben509 »

lmm wrote: post #150
ben509 wrote: post #141

Earlier quoted context omitted.

The ACID system can guarantee nobody is billed for an item that you can't deliver. If you want a more user-friendly guarantee, you can reserve it when it's added to the cart.

> If you want a more user-friendly guarantee, you can reserve it when it's added to the cart. If you open an ACID transaction when the user adds something to the cart and don't close it until they check out, you'll find your database gets locked up pretty quickly. So you can't actually use the ACID transactions to implement the behaviour you want - you have to implement some kind of reserve/commit semantics in usersp…

You don't need to hold the transaction open the entire time, you just need the inventory count to be correct.

You track the inventory and reservations. Taking a reservation checks that inventory is available. With row-level locking, only that inventory item is locked. If that fails, it can search for timed out reservations, update the inventory and try again.

If it succeeds, it decrements the inventory then adds a reservation. At that point, the transaction can close, and in the common case, you only held locks long enough to update a row in inventory and add a row to reservations.

Re: Kafka Is Not a Database

#162

Post by meditative »

johnx123-up wrote: post #152
AndrewKemendo wrote: post #32

Alternatively from Jay Krebs [1] a much more thorough and nuanced discussion that is probably the best send-up on this topic. "So is it crazy to do this? The answer is no, there’s nothing crazy about storing data in Kafka: it works well for this because it was designed to do it. Data in Kafka is persisted to disk, checksummed, and replicated for fault tolerance. Accumulating more stored data doesn’t make it slower. T…

OT: Are there any Kafka alternatives in Rust or Go or C? (for a non-JVM stack)

https://docs.nats.io/nats-streaming-concepts/intro

https://github.com/liftbridge-io/liftbridge

Re: Kafka Is Not a Database

#163

Post by bastawhiz »

lmm wrote: post #151
bastawhiz wrote: post #145

Earlier quoted context omitted.

The A in ACID literally stands for atomicity. If you're using an ACID database that can't guarantee that an item is available for purchase at the time it's purchased, you're using a bad database. The user having stale data in their browser and finding an item has already been purchased is very different from the database, within a transaction, allowing an item which has already been purchased to be purchased again.

> The user having stale data in their browser and finding an item has already been purchased is very different from the database, within a transaction, allowing an item which has already been purchased to be purchased again. This is like "the operation was a success, but the patient died". What matters is the user-facing behaviour of your whole system; a transaction that can't actually cover the parts the user cares…

You're misunderstanding the problem. If I press "buy now" and it says "success", I don't want to receive an email later saying "oops, we didn't actually have any in stock, we'll send you a refund" because another order was sitting in a queue. That's the failure mode here. The staleness of what's in the browser is an orthogonal concern unrelated to the database or use of Kafka.

Re: Kafka Is Not a Database

#164

Post by theptip »

LgWoodenBadger wrote: post #110
georgewfraser wrote: post #89

Earlier quoted context omitted.

"Read-committed isolation" is not a meaningful implementation of transactions. If you can't do read, then a write, while guaranteeing the database didn't change in between, then you don't really have transactions.

This sounds like "serializable" which is (in my experience) rarely useful for a meaningful system.

If you read the "Feral concurrency control" paper I linked above, particularly section 7 on conclusions, they make the case that serializable is the only isolation level that's actually safe with naive coding styles on frameworks like Rails and Django which do application-level validation. If you do validation in your application and don't use serializable isolation, then you have to be careful about manually locking, OR just be sure that your usage pattern isn't vulnerable to the anomalies that you're introducing by using a weaker isolation level.

If you're building a financial ledger, you absolutely must use serializable isolation. If you're building a Twitter clone, sure, use something weaker that will gain you some performance.

I'd make the case that we should be recommending the use of serializable by default unless you have a reason why you think it's OK to use something weaker, rather than having the default be better-performing-but-unsafe. The sort of concurrency validation errors that you get if you needed Serializable and used Read-Committed instead are really, really hard to reproduce, debug, and diagnose.

Re: Kafka Is Not a Database

#165

Post by UK-Al05 »

jacques_chester wrote: post #74
UK-Al05 wrote: post #10

Earlier quoted context omitted.

That's why you partition by some id. Say stock SKU id for stock control. Then you can handle other SKUs in parallel. It's only in serial for a single SKU. That's probably the maximum performance potential your going to get in a traditional db anyway.

This strikes me as mixing the physical and logical models.

There's logical and physical partitions.

Logical partitions are always handled by the same physical partition. But physical partitions can handle multiple logical partitions.

Re: Kafka Is Not a Database

#166

Post by lmm »

bastawhiz wrote: post #163
lmm wrote: post #151

Earlier quoted context omitted.

> The user having stale data in their browser and finding an item has already been purchased is very different from the database, within a transaction, allowing an item which has already been purchased to be purchased again. This is like "the operation was a success, but the patient died". What matters is the user-facing behaviour of your whole system; a transaction that can't actually cover the parts the user cares…

You're misunderstanding the problem. If I press "buy now" and it says "success", I don't want to receive an email later saying "oops, we didn't actually have any in stock, we'll send you a refund" because another order was sitting in a queue. That's the failure mode here. The staleness of what's in the browser is an orthogonal concern unrelated to the database or use of Kafka.

It's very much related. You have one item in stock. Two users see the item as available in their browsers and click "buy now". That's the problem that you actually have to solve, and database transactions don't help you solve it: whether you're using a transactional datastore or not you have to do pretty much the same thing when the user clicks "buy now": issue an attempt to buy it, wait for that attempt to be confirmed/denied, and handle both cases.

And once you've solved that problem you don't need or want ACID transactions because they don't actually do anything for you. Order confirmation emails have the same problem as the browser: you can't (or at least shouldn't) actually hold a database-level transaction open while you connect to an email server, so you have to do something like recording a queue of email confirmations that are ready to be sent - exactly the same thing you do when using Kafka.

Re: Kafka Is Not a Database

#167

Post by lmm »

ben509 wrote: post #161
lmm wrote: post #150

Earlier quoted context omitted.

> If you want a more user-friendly guarantee, you can reserve it when it's added to the cart. If you open an ACID transaction when the user adds something to the cart and don't close it until they check out, you'll find your database gets locked up pretty quickly. So you can't actually use the ACID transactions to implement the behaviour you want - you have to implement some kind of reserve/commit semantics in usersp…

You don't need to hold the transaction open the entire time, you just need the inventory count to be correct. You track the inventory and reservations. Taking a reservation checks that inventory is available. With row-level locking, only that inventory item is locked. If that fails, it can search for timed out reservations, update the inventory and try again. If it succeeds, it decrements the inventory then adds a re…

You still have to handle the case where your transaction fails though. So you actually end up writing the same thing you'd do if you didn't have transactions: you issue an attempt to reserve, wait for the response to that attempt (whether that's transaction commit succeeding/failing or a queue processor processing), and handle both possible results. The transaction support doesn't actually help you because it's at the wrong level to be useful.

Re: Kafka Is Not a Database

#168

Post by lmm »

Kalium wrote: post #146
lmm wrote: post #138

Earlier quoted context omitted.

> There were no real transactions, no ensured consistency Which is the right way to do it, because transactions don't extend into the real world. If you need to wait for the consequences of a given event, wait for the consequences of that event. Otherwise, all you really care about is all events happening in a consistent order. It's a much more practical consistency model. > and no referential integrity The problem w…

> Really? I've found the exact opposite - teams that used an RDBMS had to throw away their customer data under GDPR, because even though they had an entry in their database saying that the customer had agreed, they couldn't tell you what the customer had agreed to or when. Whereas teams using Kafka in the way you describe had an event record for the original agreement, and could tell you where any given piece of data…

Yeah fair enough. I'd argue that this is kind of a double standard (a traditional RDBMS may well be copies of "deleted" data on dirty pages, and may well leave that data on the physical disk indefinitely, for much the same reasons as Kafka does - it just makes it a bit fiddlier for you to access it), but your legal team may decide that it's required.

I don't think your overall scorn is warranted - there are bigger problems that are endemic to RDBMS deployments, and the advantages of a stream-first architecture are very real - but there are genuine difficulties around handling data obliteration and it's something you need to design for carefully if you're using an immutable-first architecture and have that requirement.

Re: Kafka Is Not a Database

#169

Post by vladsanchez »

rowland66 wrote: post #116
vladsanchez wrote: post #108

I want to Upvote this more than once. So much facts into a condensed into a small essay. Good job! Money quote: "Event-sourced architectures like these suffer many such isolation anomalies, which constantly gaslight users with “time travel” behavior that we’re all familiar with."

Maybe I am just old because I had to Google what gaslighting meant, but as best I can tell getting gaslighted by your system architecture is really stretching the meaning of the term gaslighting to the point of absurdity.

I concur, but we both got it's a metaphor. ;)

Re: Kafka Is Not a Database

#170

Post by bastawhiz »

lmm wrote: post #166
bastawhiz wrote: post #163

Earlier quoted context omitted.

You're misunderstanding the problem. If I press "buy now" and it says "success", I don't want to receive an email later saying "oops, we didn't actually have any in stock, we'll send you a refund" because another order was sitting in a queue. That's the failure mode here. The staleness of what's in the browser is an orthogonal concern unrelated to the database or use of Kafka.

It's very much related. You have one item in stock. Two users see the item as available in their browsers and click "buy now". That's the problem that you actually have to solve, and database transactions don't help you solve it: whether you're using a transactional datastore or not you have to do pretty much the same thing when the user clicks "buy now": issue an attempt to buy it, wait for that attempt to be confir…

You're still missing the point. You're saying:

> issue an attempt to buy it, wait for that attempt to be confirmed/denied, and handle both cases.

Only one user should receive a success. The article describes a setup where both users receive a success because of write skew. This is the very problem that transactions avoid: locking the row storing the number of items in stock so that it cannot be decremented below zero.

You can still use Kafka. What the article is saying is that you can't just check the DB to see if items are in stock and write your order to a queue. Because there might already be an order for the same item in the queue which makes your new order invalid.

Literally the whole point of ACID databases is that multiple clients can operate on the same set of data and avoid putting it into a bad state. If I'm a bank storing an account balance of $20 and you withdraw $20 and I withdraw $20, only one of us should get $20. This is the same problem.

Post reply on HN