Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka Is Not a Database

materialize.com

Post reply on HN
Print view View original

141–150 of 172 posts

Re: Kafka Is Not a Database

#141

Post by ben509 »

lmm wrote: post #137

> The problem we now have is called write skew. Our reads from the inventory view can be out of date by the time the checkout event is processed. If two users try to buy the same item at nearly the same time, they will both succeed, and we won’t have enough inventory for them both. And you'll have exactly the same problem if you're using a traditional ACID database: the user saw the item as being available, clicked b…

The ACID system can guarantee nobody is billed for an item that you can't deliver. If you want a more user-friendly guarantee, you can reserve it when it's added to the cart.

Re: Kafka Is Not a Database

#142

Post by victor106 »

Hamuko wrote: post #24
yumaikas wrote: post #21

Earlier quoted context omitted.

I think I'd differentiate between a database and a data store. I'd argue that a filesystem is a data store, rather than a database.

Some say that Microsoft Excel is the world's most popular database engine.

The most popular database where the number of concurrent users = 1

Re: Kafka Is Not a Database

#143

Post by arafsheikh »

wdb wrote: post #123
arafsheikh wrote: post #97

Earlier quoted context omitted.

Persisting message offsets in DB has its own challenges. The apps become tightly coupled with a specific Kafka cluster and that makes it difficult to swap clusters in case of a failover event. If you expect apps to persist offsets then it’s important to have a mechanism/process to safely reset the app state in DB when the stored offset doesn’t make sense.

Interesting, do you have any resources about how to best handle side effects with message queues (e.g. GCP PubSub)? Trying to find out if its worth the effort, or good practice to allow replay-ability (like from backup) of message and get back to the same state

Not sure about PubSub, but Kafka doesn’t store messages on disk forever by default. So hydrating the state from a message bus may not be a good idea depending on what you’re doing. I’d say DB is the way to go here.

Don’t have any resources though. I’m only speaking from experience.

Re: Kafka Is Not a Database

#144

Post by bastawhiz »

Hamuko wrote: post #24
yumaikas wrote: post #21

Earlier quoted context omitted.

I think I'd differentiate between a database and a data store. I'd argue that a filesystem is a data store, rather than a database.

Some say that Microsoft Excel is the world's most popular database engine.

Excel allows you to perform queries for almost arbitrary data on a spreadsheet. I wouldn't doubt for a minute that with sufficient time and motivation you could write a full sql engine with excel functions.

Re: Kafka Is Not a Database

#145

Post by bastawhiz »

lmm wrote: post #137

> The problem we now have is called write skew. Our reads from the inventory view can be out of date by the time the checkout event is processed. If two users try to buy the same item at nearly the same time, they will both succeed, and we won’t have enough inventory for them both. And you'll have exactly the same problem if you're using a traditional ACID database: the user saw the item as being available, clicked b…

The A in ACID literally stands for atomicity. If you're using an ACID database that can't guarantee that an item is available for purchase at the time it's purchased, you're using a bad database.

The user having stale data in their browser and finding an item has already been purchased is very different from the database, within a transaction, allowing an item which has already been purchased to be purchased again.

Re: Kafka Is Not a Database

#146

Post by Kalium »

lmm wrote: post #138
Kalium wrote: post #15

As recently as last year, I worked for a company where the Chief Architect, in his infinite wisdom, had decided that a database was a silly legacy thing. The future looked like Kafka streams, with each service being a function against Kafka streams, and data retention set to infinite. Predictably, this setup ran into an interesting assortment of issues. There were no real transactions, no ensured consistency, and no…

> There were no real transactions, no ensured consistency Which is the right way to do it, because transactions don't extend into the real world. If you need to wait for the consequences of a given event, wait for the consequences of that event. Otherwise, all you really care about is all events happening in a consistent order. It's a much more practical consistency model. > and no referential integrity The problem w…

> Really? I've found the exact opposite - teams that used an RDBMS had to throw away their customer data under GDPR, because even though they had an entry in their database saying that the customer had agreed, they couldn't tell you what the customer had agreed to or when. Whereas teams using Kafka in the way you describe had an event record for the original agreement, and could tell you where any given piece of data came from.

This is absolutely wonderful! Unfortunately, this team decided to store data subject to GDPR deletion requests in Kafka, where deletion is quite difficult. It was a problem, when trying to do deletion programmatically, across many teams using the same set of topics.

The real nightmare came when this team, obsessed with the power of infinite retention periods, encountered PCI-DSS. You see, the business wanted to move away from Stripe and similar to dealing with a processor directly, in order to save on transaction fees. So obviously they could just put credit card data into Kafka...

Re: Kafka Is Not a Database

#147

Post by CyberDildonics »

victor106 wrote: post #142
Hamuko wrote: post #24

Earlier quoted context omitted.

Some say that Microsoft Excel is the world's most popular database engine.

The most popular database where the number of concurrent users = 1

Depends on how many people are looking over your shoulder.

Re: Kafka Is Not a Database

#148

Post by AndrewKemendo »

andrewem wrote: post #122
AndrewKemendo wrote: post #32

Alternatively from Jay Krebs [1] a much more thorough and nuanced discussion that is probably the best send-up on this topic. "So is it crazy to do this? The answer is no, there’s nothing crazy about storing data in Kafka: it works well for this because it was designed to do it. Data in Kafka is persisted to disk, checksummed, and replicated for fault tolerance. Accumulating more stored data doesn’t make it slower. T…

*Kreps, not Krebs

Yes sorry! Unfortunately too late to edit :(

Re: Kafka Is Not a Database

#149

Post by fierro »

georgewfraser wrote: post #81
j-pb wrote: post #26

Tbh, It's a weird blog post coming from the materialize folks, considering they know better. The "event sourced" arch they sketched is missing pieces. Normaly you'd have single writer instances that are locked to the corresponding kafka partition, which ensure strong transactional guarantees, IF you need them. Throwing shade for maketings sake is something that they should be above. I mean c'mon, I'd argue that Postg…

We're trying to address a real problem that is happening in our industry: VPs of eng and principal engineers at startups are adopting the "Kappa Architecture" / "Turning the Database Inside Out", without realizing how much functionality from traditional database systems they are leaving behind. This has led to a barrage of consistency bugs in everything from food-delivery apps to the "unread message count" in LinkedI…

what is the Kappa architecture?

Re: Kafka Is Not a Database

#150

Post by lmm »

ben509 wrote: post #141
lmm wrote: post #137

> The problem we now have is called write skew. Our reads from the inventory view can be out of date by the time the checkout event is processed. If two users try to buy the same item at nearly the same time, they will both succeed, and we won’t have enough inventory for them both. And you'll have exactly the same problem if you're using a traditional ACID database: the user saw the item as being available, clicked b…

The ACID system can guarantee nobody is billed for an item that you can't deliver. If you want a more user-friendly guarantee, you can reserve it when it's added to the cart.

> If you want a more user-friendly guarantee, you can reserve it when it's added to the cart.

If you open an ACID transaction when the user adds something to the cart and don't close it until they check out, you'll find your database gets locked up pretty quickly. So you can't actually use the ACID transactions to implement the behaviour you want - you have to implement some kind of reserve/commit semantics in userspace, whether you're using an ACID database or not.

Post reply on HN