Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka Is Not a Database

materialize.com

Post reply on HN
Print view View original

111–120 of 172 posts

Re: Kafka Is Not a Database

#111

Post by jacques_chester »

AndyPa32 wrote: post #109
jacques_chester wrote: post #96

Earlier quoted context omitted.

The CAP theorem line was smart-arsery. The thing here is that the database can update the cart and the inventory in one logical step, to the exclusion of others. The Kafka approach doesn't guarantee that out of the box, leading to the creation of de facto locking protocols (write cart intent, read cart intent, write inventory intent ...). A traditional database does that for you with selectable levels of guarantees.

Yeah, if you are lucky enough that your cart and inventory is stored in the same database.

Of course. That's why Materialize are interesting, they are apparently able to give a greatly improved compute:result ratio than the previous state of the art.

Re: Kafka Is Not a Database

#112

Post by UK-Al05 »

mrkeen wrote: post #50
UK-Al05 wrote: post #7

Earlier quoted context omitted.

This is lack of abstraction. You can certainly fix this in kafka using various hacks, but its implementation of an abstraction you can get in a standard db for free. Funnily enough a list of events is pretty much what a transaction log is in a standard db. Although the events have more of a business meaning. In many ways event sourcing is removing a lot of abstraction databases give you.

> a standard db Yes, ACID works for one database. Many databases? Not so much. > Funnily enough a list of events is pretty much what a transaction log is in a standard db. When I "SELECT Balance WHERE user = 12345", I usually just get back a balance, I don't get back the transaction log. If nothing else, adopting the Kafka model gets your teammates to append updates to your ledger, rather than changing values in-plac…

Yes that my point. You get half of a database. The transaction log. Not the up to date state.

Re: Kafka Is Not a Database

#113

Post by IfOnlyYouKnew »

zaphar wrote: post #101
yumaikas wrote: post #21

Earlier quoted context omitted.

I think I'd differentiate between a database and a data store. I'd argue that a filesystem is a data store, rather than a database.

What is your distinction? Is mongodb a database? What about leveldb? Whether we want it to be so or not the term database is much more encompassing thank it used to be. You can try to fight that change if you want to but it means you'll be speaking a different language that most of the rest of us as a result.

You're using the term "traditional database" two levels up. Whatever your definition of that is, it's their definition of "database".

Your initial post says "I have a different definition of 'database'", which is different than anyone else's, as you acknowledge when you refer to that common definition as a 'traditional database'.

Then, you fault them for making an argument that is invalid for your non-standard definition of a database!

You would like Kafka, the author.

Re: Kafka Is Not a Database

#114

Post by atmosx »

arthurcolle wrote: post #76

I had an issue with RabbitMQ where I didn't know how my consumer was going to use the data that I was writing to a queue yet (from a producer that was listening on a SocketIO or WebSockets stream), and I was kind of just going to figure it out in an hour or something. Eventually, my buffer ran out of memory and I couldn't write anything else to it, and it was dropping lots of messages. I was bummed. Is there a way to…

RabbitMQ and Kafka serve different needs.

RabbitMQ is the most widely used open source implementation of the AMQP protocol. It is slower but can support complex routing scenarios internally and handle situations were at-least-once-delivery guarantees are important. RabbitMQ supports on-disk persistent queues, which you can tune if you like. Compared to Kafka, RabbitMQ is slow in terms of volume that can be managed per queue.

Kafka is fast because it is horizontally scalable and you have parallel producers and consumers per topic. You can tune the speed and move needle where you need between consistency and availability. However, if you want things like at-least-once-delivery and such, you'll have to use the building blocks kafka gives you, but ultimately you'll have to handle this on the application side.

Regarding storage, by default kafka stores data for 7 days. IIRC the NY Times stores articles from 1970 onwards on kafka clusters. The storage is horizontally scalable and durable. This is a common use case. As many have pointed out, the cluster setup depends highly on you needs. We store data for 7 days in kafka as well and it's in the order of 500GB or more per node.

Looks like you have a configuration issue. You can configure rabbitMQ to store queues on the hard disk and with a quick calculation you can make sure you have enough space for 10 or 150 hours of data. I don't see any reason to switch to kafka, a different tool with different characteristics, just because you need more storage.

Re: Kafka Is Not a Database

#115

Post by yumaikas »

zaphar wrote: post #101
yumaikas wrote: post #21

Earlier quoted context omitted.

I think I'd differentiate between a database and a data store. I'd argue that a filesystem is a data store, rather than a database.

What is your distinction? Is mongodb a database? What about leveldb? Whether we want it to be so or not the term database is much more encompassing thank it used to be. You can try to fight that change if you want to but it means you'll be speaking a different language that most of the rest of us as a result.

Honestly, where I'd draw the distinction between a database and a datastore is the ability to do multi-statement transactions. So, a filesystem? A datastore. Some support fsync on a single file, but multi-file sync isn't usually supported.

MongoDB and LevelDB both support transactions, so I'd err on calling them databases myself.

Re: Kafka Is Not a Database

#116

Post by rowland66 »

vladsanchez wrote: post #108

I want to Upvote this more than once. So much facts into a condensed into a small essay. Good job! Money quote: "Event-sourced architectures like these suffer many such isolation anomalies, which constantly gaslight users with “time travel” behavior that we’re all familiar with."

Maybe I am just old because I had to Google what gaslighting meant, but as best I can tell getting gaslighted by your system architecture is really stretching the meaning of the term gaslighting to the point of absurdity.

Re: Kafka Is Not a Database

#117

Post by gunnarmorling »

vvern wrote: post #107
arjunnarayan wrote: post #69

Earlier quoted context omitted.

Hi! I'm one of the two authors here. At Materialize, we're definitely of the 'we are a bunch of voices, we are people rather than corp-speak, and you get our largely unfiltered takes' flavor. This is my (and George's from Fivetran) take. In particular this is not Frank's take, as you attribute below :) > SQL is declarative, reactive Materialize streams are declarative on a whole new level. Thank you for the kind word…

> 1. Log "intents to write" rather than writes themselves in Topic A 2. Have a separate denormalization computed and kept in a separate Topic B, which can be read from. This denormalization needs to be read until the intent propagates from Topic A. 3. Convert those intents into commits. 4. Deal with all the failure cases in a distributed system, e.g. cleaning up abandoned intents, etc. People do do this. I have done…

>> If you use an OLTP database, and generate events into Kafka via CDC, you get the best of both worlds.

> This is the next chapter in the gospel of the distributed transaction.

Actually, it's the opposite. CDC helps to avoid distributed transaction; apps write to a single database only, and other resources (Kafka, other databases, etc.) based on that are updated asychronously, eventually consistent.

Re: Kafka Is Not a Database

#118

Post by skyde »

hodgesrm wrote: post #104
AndrewKemendo wrote: post #32

Alternatively from Jay Krebs [1] a much more thorough and nuanced discussion that is probably the best send-up on this topic. "So is it crazy to do this? The answer is no, there’s nothing crazy about storing data in Kafka: it works well for this because it was designed to do it. Data in Kafka is persisted to disk, checksummed, and replicated for fault tolerance. Accumulating more stored data doesn’t make it slower. T…

Thanks, this is a great article. The money quote for me is: > I think it makes more sense to think of your datacenter as a giant database, in that database Kafka is the commit log, and these various storage systems are kinds of derived indexes or views. My company supports analytic systems and we see this pattern constantly. It's also sort of a Pat Helland view of the world that subsumes a large fraction of data mana…

The problem is that Kafka API and Commit log API are very different.

If you wanted to literally use Kafka for your commit log the same way the Amazon aurora are using a distributed commit log. You would find that a lot of feature a commit log need are missing and impossible to add to kafka.

Re: Kafka Is Not a Database

#119

Post by skyde »

arjunnarayan wrote: post #69
j-pb wrote: post #26

Tbh, It's a weird blog post coming from the materialize folks, considering they know better. The "event sourced" arch they sketched is missing pieces. Normaly you'd have single writer instances that are locked to the corresponding kafka partition, which ensure strong transactional guarantees, IF you need them. Throwing shade for maketings sake is something that they should be above. I mean c'mon, I'd argue that Postg…

Hi! I'm one of the two authors here. At Materialize, we're definitely of the 'we are a bunch of voices, we are people rather than corp-speak, and you get our largely unfiltered takes' flavor. This is my (and George's from Fivetran) take. In particular this is not Frank's take, as you attribute below :) > SQL is declarative, reactive Materialize streams are declarative on a whole new level. Thank you for the kind word…

> If you use an OLTP database, and generate events into Kafka via CDC, you get the best of both worlds.

100% agree this is the way to go instead of rolling your own transaction support you get the "ACID" for free from the DB and use KAFKA to archive changes and subscribe to them.

Re: Kafka Is Not a Database

#120

Post by Spivak »

jacques_chester wrote: post #96
Spivak wrote: post #94

Earlier quoted context omitted.

None of this shows up as user-facing any differently than a relational database. No CAP theorem at all. Kafka: User clicks buy and it shows “processing” which behind the scenes posts the buy message and waits for a “confirmed” message. When it’s confirmed user is directed to success! If someone else posts the buy before them they get back a “failed: sold out” message. Relational: User clicks buy and it shows “process…

The CAP theorem line was smart-arsery. The thing here is that the database can update the cart and the inventory in one logical step, to the exclusion of others. The Kafka approach doesn't guarantee that out of the box, leading to the creation of de facto locking protocols (write cart intent, read cart intent, write inventory intent ...). A traditional database does that for you with selectable levels of guarantees.

You're absolutely right but I think the advantage is that to the users of the database it doesn't really feel like there's any locking going on and it's this implicit thing that's constructed from the history.

Like for a hypothetical ticket selling platform you just get a log like

    00:00 RESERVE AAA FOR xxx, 5 min
    00:02 BUY     BBB FOR qqq
    00:15 RESERVE AAA FOR yyy, 5 min
    00:23 CONFIRM AAA FOR xxx
    00:25 CONFIRM BBB for qqq
    00:27 REJECT  AAA FOR yyy, "already reserved"
    05:16 RESERVE AAA FOR zzz, 5 min
    05:34 CONFIRM AAA FOR zzz
So although there's "locking" going on here enforced by the consumer sending the confirms the producer just sends intents/events and sees whether they're successful or not and both producer and consumer are stateless.

I guess it just depends on which model you think is more a PITA to work with.

Post reply on HN