Well, then you have never heard of ksqlDB. It adds SQL and DB features to Kafka. It is backed by Confluent (LinkedIn) same company that developed Kafka initially.
Kafka Is Not a Database
11–20 of 172 posts
Re: Kafka Is Not a Database
#12Another potential misuse of Kafka I've been wondering about is how a single Kafka instance/cluster is often shared by multiple microservices. On one hand the ability to connect multiple microservices to a central message broker is convenient, but on the the other hand this goes against the microservice philosophy of not sharing subcomponents (databases, etc). I wonder where the lines should be drawn.
Wait, what? Isn’t the whole point of having multiple publishers/subscribers?
Re: Kafka Is Not a Database
#13Another potential misuse of Kafka I've been wondering about is how a single Kafka instance/cluster is often shared by multiple microservices. On one hand the ability to connect multiple microservices to a central message broker is convenient, but on the the other hand this goes against the microservice philosophy of not sharing subcomponents (databases, etc). I wonder where the lines should be drawn.
I would argue that, if it's being used properly, the message broker itself is a service. It runs as a separate process, you communicate with it over an API, and its subcomponents (e.g., the database) are encapsulated.
It's all about framing and perspective, of course. But that's how I'd want to try and frame it from a system architecture point of view.
Re: Kafka Is Not a Database
#14Well, then you have never heard of ksqlDB. It adds SQL and DB features to Kafka. It is backed by Confluent (LinkedIn) same company that developed Kafka initially. https://ksqldb.io
We’re very aware of ksqlDB. I would recommend this video from last week where Matthias talks about some of the strengths and weaknesses of ksql: https://youtu.be/KUQuegJ4do8
Re: Kafka Is Not a Database
#15As recently as last year, I worked for a company where the Chief Architect, in his infinite wisdom, had decided that a database was a silly legacy thing. The future looked like Kafka streams, with each service being a function against Kafka streams, and data retention set to infinite.
Predictably, this setup ran into an interesting assortment of issues. There were no real transactions, no ensured consistency, and no referential integrity. There was also no authentication or authorization, because a default-configured deployment of Kafka from Confluent happily neglects such trivial details.
To say this was a vast mess would be to put it lightly. It was a nightmare to code against once you left the fantasy world of functional programming nirvana and encountered real requirements. It meant pushing a whole series of concerns that isolation addresses into application code... or not addressing them at all. Teams routinely relied on one another's internal kafka streams. It was a GDPR nightmare.
Kafka Connect was deployed to bridge between Kafka and some real databases. This was its own mess.
Kafka, I have learned, is a very powerful tool. And like all shiny new tools, deeply prone to misuse.
Re: Kafka Is Not a Database
#16As recently as last year, I worked for a company where the Chief Architect, in his infinite wisdom, had decided that a database was a silly legacy thing. The future looked like Kafka streams, with each service being a function against Kafka streams, and data retention set to infinite. Predictably, this setup ran into an interesting assortment of issues. There were no real transactions, no ensured consistency, and no…
Instead of single architects, companies need architect boards. And they can vote on these ideas before a single individual becomes a single point of failure.
Expecting 1 person to make 100% correct decisions all the time is too much expectation for one person. People go down rabbit holes and they have weird takeaways, like replace all the databases with queues.
Re: Kafka Is Not a Database
#17If it stores data it's a database. Filesystems are databases, MongoDB is a database. LevelDB is a database. Postgres and MySQL are databases. Kafka is a database. They are all very different in features and functionality though.
What the authors mean is that kafka is not a traditional database and doesn't solve the same problems that traditional databases solve. Which is a useful distinction to make but is not the distinction they make.
The reality is that database is now a very general term and for many usecases you can choose to special purpose databases for what you need.
Re: Kafka Is Not a Database
#18using the any tool for correct problem requires skills.
Re: Kafka Is Not a Database
#19Relevant to the discussion:
Martin Kleppmann | Kafka Summit SF 2018 Keynote (Is Kafka a Database?) [1]
Re: Kafka Is Not a Database
#20As recently as last year, I worked for a company where the Chief Architect, in his infinite wisdom, had decided that a database was a silly legacy thing. The future looked like Kafka streams, with each service being a function against Kafka streams, and data retention set to infinite. Predictably, this setup ran into an interesting assortment of issues. There were no real transactions, no ensured consistency, and no…
Instead of single architects, companies need architect boards. And they can vote on these ideas before a single individual becomes a single point of failure. Expecting 1 person to make 100% correct decisions all the time is too much expectation for one person. People go down rabbit holes and they have weird takeaways, like replace all the databases with queues.
I agree in abstract, but in practice it's quite difficult to set up a successful democratic architecture board. You need teams or departments that all have architects, and an engineering organization where both managers and engineers accept a degree of centralized technical leadership. Getting there is, in my opinion, the work of years. It's especially challenging because spinning up such a board requires a person who can run it single-handedly.
In this particular company, the Chief Architect in theory had a group around him. They did nothing to check his poor decisions, and from the outside seemed primarily interested in living in the functional programming nirvana he promised them.