Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka Is Not a Database

materialize.com

Post reply on HN
Print view View original

31–40 of 172 posts

Re: Kafka Is Not a Database

#31

Post by Spivak »

I feel like the inventory thing is a bit of a straw-man because the situation is set out in such a way that you need transactions for it to work. If you find yourself wishing you had a global write-lock on a topic to then of course it won't work. Modeling your data for Kafka is work just the same as it is for MySQL. Of course it might not be the best tool for the job but you should at least give it a fair shake.

You should be able to post "buy" messages to a topic without fear that it messes up your data integrity. Who cares if two people are fighting over the last item? You have a durable log. Post both "buys" and wait for the "confirm" message from a consumer that's reading the log at that point in time, validates, and confirms or rejects the buys. At the point that the buy reaches a consumer there is enough information to know for sure whether it's valid or not. Both of the buy events happened and should be recorded whether they can be fulfilled or not.

Re: Kafka Is Not a Database

#32

Post by AndrewKemendo »

Alternatively from Jay Krebs [1] a much more thorough and nuanced discussion that is probably the best send-up on this topic.

"So is it crazy to do this? The answer is no, there’s nothing crazy about storing data in Kafka: it works well for this because it was designed to do it. Data in Kafka is persisted to disk, checksummed, and replicated for fault tolerance. Accumulating more stored data doesn’t make it slower. There are Kafka clusters running in production with over a petabyte of stored data."

[1] https://www.confluent.io/blog/okay-store-data-apache-kafka/

Re: Kafka Is Not a Database

#33

Post by georgewfraser »

Cojen wrote: post #29

Jim Gray disagrees: https://arxiv.org/ftp/cs/papers/0701/0701158.pdf

It's funny that you use that example, we actually cited that in an earlier draft of this post. Despite the seemingly opposite title "Queues are Databases", that note actually makes many of the same arguments, that message brokers are missing much of the functionality of database management systems and this is a problem.

Re: Kafka Is Not a Database

#34

Post by dkhenry »

j-pb wrote: post #26

Tbh, It's a weird blog post coming from the materialize folks, considering they know better. The "event sourced" arch they sketched is missing pieces. Normaly you'd have single writer instances that are locked to the corresponding kafka partition, which ensure strong transactional guarantees, IF you need them. Throwing shade for maketings sake is something that they should be above. I mean c'mon, I'd argue that Postg…

I am not terribly surprised. The materialize team was previously at CockroachDB which also had a habit of putting out marketing material like this.

Re: Kafka Is Not a Database

#35

Post by edbrown23 »

UK-Al05 wrote: post #10
_y5hn wrote: post #8

Earlier quoted context omitted.

Sure, but that defeats the quest for horizontal scalability. You can build highly performant systems based on serial execution, but not sure this is an area where Kafka excels particularly.

That's why you partition by some id. Say stock SKU id for stock control. Then you can handle other SKUs in parallel. It's only in serial for a single SKU. That's probably the maximum performance potential your going to get in a traditional db anyway.

This definitely seems like the "Kafka" way to solve this problem, but I fear there are implications to this partitioning scheme I'd love to see answered. For example, partition counts aren't infinite, and aren't easily adjusted after the fact. So if you choose, say, 10 partitions originally, for a SKU space that is nearly infinite, then in reality you can only handle 10 parallel streams of work. Any SKU that is partitioned behind a bit of slow work is then blocked by that work.

It's doable to repartition to 100 partitions or more, but you basically need to replay the work kept in the log based on 10 partitions onto the new 100 partitions, and that operation gets more expensive over time. Then of course you're basically stuck again once your traffic increases to a high enough level that the original problem returns. If the unit of horizontal scaling is the partition, but the partition count can't be easily changed, consumers eventually lose their horizontal scalability in Kafka, from my perspective.

Re: Kafka Is Not a Database

#36

Post by edgyquant »

yumaikas wrote: post #21
zaphar wrote: post #17

If it stores data it's a database. Filesystems are databases, MongoDB is a database. LevelDB is a database. Postgres and MySQL are databases. Kafka is a database. They are all very different in features and functionality though. What the authors mean is that kafka is not a traditional database and doesn't solve the same problems that traditional databases solve. Which is a useful distinction to make but is not the di…

I think I'd differentiate between a database and a data store. I'd argue that a filesystem is a data store, rather than a database.

I’d say initially file systems were data stores but once they developed hierarchies they became more akin to a database. I’m not sure there’s a huge difference but it seems a database is a collection of data stores (though there or probably a more technical and correct definition.)

Re: Kafka Is Not a Database

#37

Post by j-pb »

dkhenry wrote: post #34
j-pb wrote: post #26

Tbh, It's a weird blog post coming from the materialize folks, considering they know better. The "event sourced" arch they sketched is missing pieces. Normaly you'd have single writer instances that are locked to the corresponding kafka partition, which ensure strong transactional guarantees, IF you need them. Throwing shade for maketings sake is something that they should be above. I mean c'mon, I'd argue that Postg…

I am not terribly surprised. The materialize team was previously at CockroachDB which also had a habit of putting out marketing material like this.

Maybe I'm biased because I'm such a huge Frank McSherry fanboy. The differential dataflow work he does in rust is simply awesome, and he writes great papers too!

Little known fun fact, the rust type- and borrow-checker uses a datalog engine internally to express typing rules, and that engine was written and improved by Frank McSherry. So whenever you hit compile on a rust program, you're using a tiny bit of Materialize tech.

Re: Kafka Is Not a Database

#38

Post by jkarneges »

mrweasel wrote: post #22
ivalm wrote: post #12

Earlier quoted context omitted.

Wait, what? Isn’t the whole point of having multiple publishers/subscribers?

I think the point was about using a single cluster for multiple topics, for different services. Depending on the scenario I can see the point. If the micro services are all part of the larger overall solution, having a single cluster is perfectly fine. Using the same cluster for multiple "product" is a little like having one central database server for a number of different solutions. You can do it, but it potentiall…

I'd agree there is an arguable difference between sharing a server vs sharing data within the server.

Bottleneck issues aside, letting two microservices connect to the same Postgres cluster but access different "databases" (collection of tables) within that cluster could be considered an acceptable data separation. Certainly with multi-tenant DBaaS systems there may be some server sharing by unrelated microservices/customers. Whereas letting two microservices access the same database tables would probably be frowned upon.

Nevertheless, sharing the same Kafka topics between microservices seems to be a common thing to do.

Re: Kafka Is Not a Database

#39

Post by revertts »

dgb23 wrote: post #5
fouc wrote: post #2

Any sufficiently complex software will end up implementing a database.

The article links to this talk[0], which has a funny and interesting sounding title: "Did you accidentally build a database?" [0] https://www.oreilly.com/library/view/strata-hadoop/978149194...

That link's a 3min clip for non-subscribers, but the full talk is here https://www.youtube.com/watch?v=Bz2EXg0Fy98

Re: Kafka Is Not a Database

#40

Post by je42 »

Ok. I admit using Kafka as DB is not straight forward but just stating it doesn't provide ACID functionality is not enough.

The example they give is very simplistic. With the correct design of kafka topics and events the problem of the example can be fixed.

And according to oracle https://www.oracle.com/database/what-is-database/ :

> A database is an organized collection of structured information, or data, typically stored electronically in a computer system.

So Kafka clearly fits that definition.

Post reply on HN