Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka Is Not a Database

materialize.com

Post reply on HN
Print view View original

41–50 of 172 posts

Re: Kafka Is Not a Database

#41

Post by onefuncman »

jkarneges wrote: post #9

Another potential misuse of Kafka I've been wondering about is how a single Kafka instance/cluster is often shared by multiple microservices. On one hand the ability to connect multiple microservices to a central message broker is convenient, but on the the other hand this goes against the microservice philosophy of not sharing subcomponents (databases, etc). I wonder where the lines should be drawn.

this isn't any different than microservices getting a deathball dependency on a user service or logging service or security service or ...

you either don't allow microservices to consume from others' topics, or you publish event schemas so they can still iterate independently.

the move to a 2nd kafka cluster in my experience has always been driven by isolation and fault tolerance concerns, not scalability.

Re: Kafka Is Not a Database

#42

Post by megachucky »

Here is my answer if Kafka is a database:

(the answer is yes, but you should still not try to replace every other database)

https://www.kai-waehner.de/blog/2020/03/12/can-apache-kafka-...

Would be curious what the people here think about my post, too. Disclaimer: I work for Confluent - and I am also happy about critical feedback.

Re: Kafka Is Not a Database

#43

Post by hasanic »

I mean, duh? Does Apache Kafka ever made the claim that it is a database?

Other things that are not a database: Apache Traffic Server, Apache Mahout, Apache Jakarta, Apache ActiveMQ... hundreds of these exist.

Re: Kafka Is Not a Database

#44

Post by halbritt »

je42 wrote: post #40

Ok. I admit using Kafka as DB is not straight forward but just stating it doesn't provide ACID functionality is not enough. The example they give is very simplistic. With the correct design of kafka topics and events the problem of the example can be fixed. And according to oracle https://www.oracle.com/database/what-is-database/ : > A database is an organized collection of structured information, or data, typically…

The author presumes that every use case requires a transactional database. ACID is nice, especially if it's needed, but generally not needed, especially in many streaming data applications for which Kafka is most suitable.

Re: Kafka Is Not a Database

#45

Post by halbritt »

AndrewKemendo wrote: post #32

Alternatively from Jay Krebs [1] a much more thorough and nuanced discussion that is probably the best send-up on this topic. "So is it crazy to do this? The answer is no, there’s nothing crazy about storing data in Kafka: it works well for this because it was designed to do it. Data in Kafka is persisted to disk, checksummed, and replicated for fault tolerance. Accumulating more stored data doesn’t make it slower. T…

I recommend his book, "I heart logs". It's a short read, but changed my perspective.

Re: Kafka Is Not a Database

#46

Post by satisfaction »

je42 wrote: post #40

Ok. I admit using Kafka as DB is not straight forward but just stating it doesn't provide ACID functionality is not enough. The example they give is very simplistic. With the correct design of kafka topics and events the problem of the example can be fixed. And according to oracle https://www.oracle.com/database/what-is-database/ : > A database is an organized collection of structured information, or data, typically…

I'm guilty of using it as a DB, my home weather station writes to kafka topics, sometimes the postgres instance is down for months, no problems letting the kafka topic store the data until i get around to rebooting pg and restarting the connector.

Re: Kafka Is Not a Database

#47

Post by dathinab »

je42 wrote: post #40

Ok. I admit using Kafka as DB is not straight forward but just stating it doesn't provide ACID functionality is not enough. The example they give is very simplistic. With the correct design of kafka topics and events the problem of the example can be fixed. And according to oracle https://www.oracle.com/database/what-is-database/ : > A database is an organized collection of structured information, or data, typically…

Didn't some newspaper use Kafka to store the newspapers they released in order or something similar? (I think it was the New York Times, maybe??).

Honestly as long as you don't use it as a general purpose database, it might very well be the best choice for your use-case.

Re: Kafka Is Not a Database

#49

Post by selfhoster11 »

mumblemumble wrote: post #28
wongarsu wrote: post #23

Earlier quoted context omitted.

By that same reasoning Postgres is it's own micro service. It runs as a separate process, you communicate with it over a well defined API, and it's subcomponents (data store, query optimizer etc) are encapsulated. With enough framing everything is possible, and in some contexts it will even make sense.

It has to do with how it functions in practice, IMO. PostgreSQL itself is arguably a service, but the database probably is not - you're probably crawling all over its implementation details and data model. You could take a stand and say, "All access is through stored procedures. They are the API." And, if that API operates as the same semantic level as a well-crafted REST API, then you could make an argument that tha…

Such implementations exist and have been discussed a few days ago here on HN. There are also REST adapters for Postgres: https://github.com/PostgREST/postgrest

Re: Kafka Is Not a Database

#50

Post by mrkeen »

UK-Al05 wrote: post #7
tacitusarc wrote: post #4

I think because software engineers tend to excel at pattern recognition, oftentimes solutions to different problems appear so similar that it seems like with a small amount of abstraction, they can be reused. But it's a trap! Everything abstracted to the highest level is the same, but problems aren't solved at the highest level. The devil, as they say, is in the details.

This is lack of abstraction. You can certainly fix this in kafka using various hacks, but its implementation of an abstraction you can get in a standard db for free. Funnily enough a list of events is pretty much what a transaction log is in a standard db. Although the events have more of a business meaning. In many ways event sourcing is removing a lot of abstraction databases give you.

> a standard db

Yes, ACID works for one database. Many databases? Not so much.

> Funnily enough a list of events is pretty much what a transaction log is in a standard db.

When I "SELECT Balance WHERE user = 12345", I usually just get back a balance, I don't get back the transaction log.

If nothing else, adopting the Kafka model gets your teammates to append updates to your ledger, rather than changing values in-place.

Post reply on HN