Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka Is Not a Database

materialize.com

Post reply on HN
Print view View original

101–110 of 172 posts

Re: Kafka Is Not a Database

#101

Post by zaphar »

yumaikas wrote: post #21
zaphar wrote: post #17

If it stores data it's a database. Filesystems are databases, MongoDB is a database. LevelDB is a database. Postgres and MySQL are databases. Kafka is a database. They are all very different in features and functionality though. What the authors mean is that kafka is not a traditional database and doesn't solve the same problems that traditional databases solve. Which is a useful distinction to make but is not the di…

I think I'd differentiate between a database and a data store. I'd argue that a filesystem is a data store, rather than a database.

What is your distinction? Is mongodb a database? What about leveldb?

Whether we want it to be so or not the term database is much more encompassing thank it used to be. You can try to fight that change if you want to but it means you'll be speaking a different language that most of the rest of us as a result.

Re: Kafka Is Not a Database

#102

Post by Baeocystin »

Hamuko wrote: post #24
yumaikas wrote: post #21

Earlier quoted context omitted.

I think I'd differentiate between a database and a data store. I'd argue that a filesystem is a data store, rather than a database.

Some say that Microsoft Excel is the world's most popular database engine.

Honestly, I'd agree with that, too, at least in a colloquial sense.

Re: Kafka Is Not a Database

#103

Post by jkarneges »

ivalm wrote: post #63
jkarneges wrote: post #38

Earlier quoted context omitted.

I'd agree there is an arguable difference between sharing a server vs sharing data within the server. Bottleneck issues aside, letting two microservices connect to the same Postgres cluster but access different "databases" (collection of tables) within that cluster could be considered an acceptable data separation. Certainly with multi-tenant DBaaS systems there may be some server sharing by unrelated microservices/c…

> Whereas letting two microservices access the same database tables would probably be frowned upon. > Nevertheless, sharing the same Kafka topics between microservices seems to be a common thing to do. I think if it is part of one whole isn’t this fine? You have one service that generates customer facing output, you may have another service that powers analytics/dashboards you may have yet another service that ETLs d…

Sure. I believe microservice boundaries are more about development agility rather than scalability. By limiting each microservice to a minimal API surface and a "2 pizza" team, everyone can iterate faster. And if a particular microservice is implemented as multiple sub-services sharing database tables only known by the team in charge, that seems fine.

Re: Kafka Is Not a Database

#104

Post by hodgesrm »

AndrewKemendo wrote: post #32

Alternatively from Jay Krebs [1] a much more thorough and nuanced discussion that is probably the best send-up on this topic. "So is it crazy to do this? The answer is no, there’s nothing crazy about storing data in Kafka: it works well for this because it was designed to do it. Data in Kafka is persisted to disk, checksummed, and replicated for fault tolerance. Accumulating more stored data doesn’t make it slower. T…

Thanks, this is a great article. The money quote for me is:

> I think it makes more sense to think of your datacenter as a giant database, in that database Kafka is the commit log, and these various storage systems are kinds of derived indexes or views.

My company supports analytic systems and we see this pattern constantly. It's also sort of a Pat Helland view of the world that subsumes a large fraction of data management under a relatively simple idea. [1] What's interesting is that Pat also sees it as a way to avoid sticky coordination problems as well.

[1] http://cidrdb.org/cidr2015/Papers/CIDR15_Paper16.pdf

Re: Kafka Is Not a Database

#105

Post by skybrian »

arjunnarayan wrote: post #69
j-pb wrote: post #26

Tbh, It's a weird blog post coming from the materialize folks, considering they know better. The "event sourced" arch they sketched is missing pieces. Normaly you'd have single writer instances that are locked to the corresponding kafka partition, which ensure strong transactional guarantees, IF you need them. Throwing shade for maketings sake is something that they should be above. I mean c'mon, I'd argue that Postg…

Hi! I'm one of the two authors here. At Materialize, we're definitely of the 'we are a bunch of voices, we are people rather than corp-speak, and you get our largely unfiltered takes' flavor. This is my (and George's from Fivetran) take. In particular this is not Frank's take, as you attribute below :) > SQL is declarative, reactive Materialize streams are declarative on a whole new level. Thank you for the kind word…

It seems like Kafka could be used to record requests and responses (upstream of the database) and approved changes (downstream of the database), but isn't good for the approval process itself?

Re: Kafka Is Not a Database

#107

Post by vvern »

arjunnarayan wrote: post #69
j-pb wrote: post #26

Tbh, It's a weird blog post coming from the materialize folks, considering they know better. The "event sourced" arch they sketched is missing pieces. Normaly you'd have single writer instances that are locked to the corresponding kafka partition, which ensure strong transactional guarantees, IF you need them. Throwing shade for maketings sake is something that they should be above. I mean c'mon, I'd argue that Postg…

Hi! I'm one of the two authors here. At Materialize, we're definitely of the 'we are a bunch of voices, we are people rather than corp-speak, and you get our largely unfiltered takes' flavor. This is my (and George's from Fivetran) take. In particular this is not Frank's take, as you attribute below :) > SQL is declarative, reactive Materialize streams are declarative on a whole new level. Thank you for the kind word…

> 1. Log "intents to write" rather than writes themselves in Topic A 2. Have a separate denormalization computed and kept in a separate Topic B, which can be read from. This denormalization needs to be read until the intent propagates from Topic A. 3. Convert those intents into commits. 4. Deal with all the failure cases in a distributed system, e.g. cleaning up abandoned intents, etc.

People do do this. I have done this. I wish I had been more principled with the error paths. It got there _eventually_.

It was a lot of code and complexity to ship a feature which in retrospect could have been nearly trivial with a transactional database. I'd say months rather than days. I won't get those years of my life back.

The products were build on top of Kafka, Cassandra, and Elasticsearch where, over time, there was a desire to maintain some amount of referential integrity. The only reason we bought into this architecture at the time was horizontal scalability (not even multi-region). Kafka, sagas, 2PC at the "application layer" can work, but you're going to spend a heck of a lot on engineering.

It was this experience that drove me to Cockroach and I've been spreading the good word ever since.

> If you use an OLTP database, and generate events into Kafka via CDC, you get the best of both worlds.

This is the next chapter in the gospel of the distributed transaction.

Re: Kafka Is Not a Database

#108

Post by vladsanchez »

I want to Upvote this more than once. So much facts into a condensed into a small essay. Good job!

Money quote: "Event-sourced architectures like these suffer many such isolation anomalies, which constantly gaslight users with “time travel” behavior that we’re all familiar with."

Re: Kafka Is Not a Database

#109

Post by AndyPa32 »

jacques_chester wrote: post #96
Spivak wrote: post #94

Earlier quoted context omitted.

None of this shows up as user-facing any differently than a relational database. No CAP theorem at all. Kafka: User clicks buy and it shows “processing” which behind the scenes posts the buy message and waits for a “confirmed” message. When it’s confirmed user is directed to success! If someone else posts the buy before them they get back a “failed: sold out” message. Relational: User clicks buy and it shows “process…

The CAP theorem line was smart-arsery. The thing here is that the database can update the cart and the inventory in one logical step, to the exclusion of others. The Kafka approach doesn't guarantee that out of the box, leading to the creation of de facto locking protocols (write cart intent, read cart intent, write inventory intent ...). A traditional database does that for you with selectable levels of guarantees.

Yeah, if you are lucky enough that your cart and inventory is stored in the same database.

Re: Kafka Is Not a Database

#110

Post by LgWoodenBadger »

georgewfraser wrote: post #89
theptip wrote: post #85

This is a bit dumbed down, and ignores the domain terminology required to properly discuss the trade-offs here (which is puzzling given that it links to a post by Aphyr, where you can find incredibly thorough discussions around isolation levels and anomalies). > The fundamental problem with using Kafka as your primary data store is it provides no isolation. This is false. I can only assume the author doesn't know abo…

"Read-committed isolation" is not a meaningful implementation of transactions. If you can't do read, then a write, while guaranteeing the database didn't change in between, then you don't really have transactions.

This sounds like "serializable" which is (in my experience) rarely useful for a meaningful system.
Post reply on HN