Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka Is Not a Database

materialize.com

Post reply on HN
Print view View original

21–30 of 172 posts

Re: Kafka Is Not a Database

#21

Post by yumaikas »

zaphar wrote: post #17

If it stores data it's a database. Filesystems are databases, MongoDB is a database. LevelDB is a database. Postgres and MySQL are databases. Kafka is a database. They are all very different in features and functionality though. What the authors mean is that kafka is not a traditional database and doesn't solve the same problems that traditional databases solve. Which is a useful distinction to make but is not the di…

I think I'd differentiate between a database and a data store.

I'd argue that a filesystem is a data store, rather than a database.

Re: Kafka Is Not a Database

#22

Post by mrweasel »

ivalm wrote: post #12
jkarneges wrote: post #9

Another potential misuse of Kafka I've been wondering about is how a single Kafka instance/cluster is often shared by multiple microservices. On one hand the ability to connect multiple microservices to a central message broker is convenient, but on the the other hand this goes against the microservice philosophy of not sharing subcomponents (databases, etc). I wonder where the lines should be drawn.

Wait, what? Isn’t the whole point of having multiple publishers/subscribers?

I think the point was about using a single cluster for multiple topics, for different services.

Depending on the scenario I can see the point. If the micro services are all part of the larger overall solution, having a single cluster is perfectly fine. Using the same cluster for multiple "product" is a little like having one central database server for a number of different solutions. You can do it, but it potentially become a bottleneck or a central point for your different solutions to impact performance of each other.

Re: Kafka Is Not a Database

#23

Post by wongarsu »

mumblemumble wrote: post #13
jkarneges wrote: post #9

Another potential misuse of Kafka I've been wondering about is how a single Kafka instance/cluster is often shared by multiple microservices. On one hand the ability to connect multiple microservices to a central message broker is convenient, but on the the other hand this goes against the microservice philosophy of not sharing subcomponents (databases, etc). I wonder where the lines should be drawn.

I would argue that, if it's being used properly, the message broker itself is a service. It runs as a separate process, you communicate with it over an API, and its subcomponents (e.g., the database) are encapsulated. It's all about framing and perspective, of course. But that's how I'd want to try and frame it from a system architecture point of view.

By that same reasoning Postgres is it's own micro service. It runs as a separate process, you communicate with it over a well defined API, and it's subcomponents (data store, query optimizer etc) are encapsulated.

With enough framing everything is possible, and in some contexts it will even make sense.

Re: Kafka Is Not a Database

#24

Post by Hamuko »

yumaikas wrote: post #21
zaphar wrote: post #17

If it stores data it's a database. Filesystems are databases, MongoDB is a database. LevelDB is a database. Postgres and MySQL are databases. Kafka is a database. They are all very different in features and functionality though. What the authors mean is that kafka is not a traditional database and doesn't solve the same problems that traditional databases solve. Which is a useful distinction to make but is not the di…

I think I'd differentiate between a database and a data store. I'd argue that a filesystem is a data store, rather than a database.

Some say that Microsoft Excel is the world's most popular database engine.

Re: Kafka Is Not a Database

#25

Post by fredliu »

Kafka is essentially commit logs, which are at the core of any traditional database engines. Streaming is just turning the gut of DB inside out (mostly for scalability reasons), while DB is wrapped up commit logs that provides higher level functionalities (ACID, Transactions, etc.). It's two sides of the same coin, yin and yang of the same thing... But on the practical side of things, yes, if what you needed more are indeed what's described in this article, your life would be easier with a traditional DB.

Re: Kafka Is Not a Database

#26

Post by j-pb »

Tbh, It's a weird blog post coming from the materialize folks, considering they know better.

The "event sourced" arch they sketched is missing pieces. Normaly you'd have single writer instances that are locked to the corresponding kafka partition, which ensure strong transactional guarantees, IF you need them.

Throwing shade for maketings sake is something that they should be above.

I mean c'mon, I'd argue that Postgres enhanced with Materialize isn't a database anymore either, but in a good sense!

It's building material. A hybrid between MQ, DB, backend logic & frontend logic.

The reduction in application logic and the increase in reliability you can get from reactive systems is insane.

SQL is declarative, reactive Materialize streams are declarative on a whole new level.

Once that tech makes it into other parts of computing like the frontend, development will be so much better, less code, less bug, a lot more fun.

Imagine that your react component could simply declare all the data it needs from a db, and the system will figure out all the caching and rerendering.

So yeah, they have awesome tech with many advantages, so I don't get why they bad-mouth other architectures.

Re: Kafka Is Not a Database

#27

Post by cirego »

tutfbhuf wrote: post #11

Well, then you have never heard of ksqlDB. It adds SQL and DB features to Kafka. It is backed by Confluent (LinkedIn) same company that developed Kafka initially. https://ksqldb.io

My understanding is that ksqlDB is a read-only interface on top of streams and only helps people write better consumers. The problems mentioned in the blog post relate to producers.

Re: Kafka Is Not a Database

#28

Post by mumblemumble »

wongarsu wrote: post #23
mumblemumble wrote: post #13

Earlier quoted context omitted.

I would argue that, if it's being used properly, the message broker itself is a service. It runs as a separate process, you communicate with it over an API, and its subcomponents (e.g., the database) are encapsulated. It's all about framing and perspective, of course. But that's how I'd want to try and frame it from a system architecture point of view.

By that same reasoning Postgres is it's own micro service. It runs as a separate process, you communicate with it over a well defined API, and it's subcomponents (data store, query optimizer etc) are encapsulated. With enough framing everything is possible, and in some contexts it will even make sense.

It has to do with how it functions in practice, IMO. PostgreSQL itself is arguably a service, but the database probably is not - you're probably crawling all over its implementation details and data model.

You could take a stand and say, "All access is through stored procedures. They are the API." And, if that API operates as the same semantic level as a well-crafted REST API, then you could make an argument that that particular database is a microservice that just happens to have been implemented on top of PostgreSQL. But I don't think I've ever seen such a thing happen in the wild. It's much more popular to use an ORM to get things nice and tightly coupled.

Post reply on HN