Kafka Is Not a Database
materialize.com
Kafka Is Not a Database
1–10 of 172 posts
Re: Kafka Is Not a Database
#2Any sufficiently complex software will end up implementing a database.
Re: Kafka Is Not a Database
#3neither has to be.
Re: Kafka Is Not a Database
#4I think because software engineers tend to excel at pattern recognition, oftentimes solutions to different problems appear so similar that it seems like with a small amount of abstraction, they can be reused. But it's a trap!
Everything abstracted to the highest level is the same, but problems aren't solved at the highest level.
The devil, as they say, is in the details.
Re: Kafka Is Not a Database
#5Any sufficiently complex software will end up implementing a database.
The article links to this talk[0], which has a funny and interesting sounding title: "Did you accidentally build a database?"
[0] https://www.oreilly.com/library/view/strata-hadoop/978149194...
Re: Kafka Is Not a Database
#6One way around this is to make sure your kafka command streams are processed in order, in serial partitioned by an id where you want the concurrency control.
Normally you only want concurrency control within certain boundaries.
By figuring out the minimum amount transaction and concurrency boundaries you can inch out quite a bit of performance.
Re: Kafka Is Not a Database
#7I think because software engineers tend to excel at pattern recognition, oftentimes solutions to different problems appear so similar that it seems like with a small amount of abstraction, they can be reused. But it's a trap! Everything abstracted to the highest level is the same, but problems aren't solved at the highest level. The devil, as they say, is in the details.
This is lack of abstraction. You can certainly fix this in kafka using various hacks, but its implementation of an abstraction you can get in a standard db for free.
Funnily enough a list of events is pretty much what a transaction log is in a standard db. Although the events have more of a business meaning. In many ways event sourcing is removing a lot of abstraction databases give you.
Re: Kafka Is Not a Database
#8One way around this is to make sure your kafka command streams are processed in order, in serial partitioned by an id where you want the concurrency control. Normally you only want concurrency control within certain boundaries. By figuring out the minimum amount transaction and concurrency boundaries you can inch out quite a bit of performance.
Sure, but that defeats the quest for horizontal scalability. You can build highly performant systems based on serial execution, but not sure this is an area where Kafka excels particularly.
Re: Kafka Is Not a Database
#9Another potential misuse of Kafka I've been wondering about is how a single Kafka instance/cluster is often shared by multiple microservices.
On one hand the ability to connect multiple microservices to a central message broker is convenient, but on the the other hand this goes against the microservice philosophy of not sharing subcomponents (databases, etc). I wonder where the lines should be drawn.
Re: Kafka Is Not a Database
#10One way around this is to make sure your kafka command streams are processed in order, in serial partitioned by an id where you want the concurrency control. Normally you only want concurrency control within certain boundaries. By figuring out the minimum amount transaction and concurrency boundaries you can inch out quite a bit of performance.
Sure, but that defeats the quest for horizontal scalability. You can build highly performant systems based on serial execution, but not sure this is an area where Kafka excels particularly.
That's why you partition by some id. Say stock SKU id for stock control. Then you can handle other SKUs in parallel. It's only in serial for a single SKU. That's probably the maximum performance potential your going to get in a traditional db anyway.