Elasticsearch is also not a database.
Kafka Is Not a Database
51–60 of 172 posts
Re: Kafka Is Not a Database
#52Tbh, It's a weird blog post coming from the materialize folks, considering they know better. The "event sourced" arch they sketched is missing pieces. Normaly you'd have single writer instances that are locked to the corresponding kafka partition, which ensure strong transactional guarantees, IF you need them. Throwing shade for maketings sake is something that they should be above. I mean c'mon, I'd argue that Postg…
I am not terribly surprised. The materialize team was previously at CockroachDB which also had a habit of putting out marketing material like this.
I ignored them for a long time because of it. I just assumed they were lightweights being annoying because they didn't have anything else.
It serves as anti-marketing, at least to me.
Re: Kafka Is Not a Database
#53[deleted]
Re: Kafka Is Not a Database
#54Earlier quoted context omitted.
That's why you partition by some id. Say stock SKU id for stock control. Then you can handle other SKUs in parallel. It's only in serial for a single SKU. That's probably the maximum performance potential your going to get in a traditional db anyway.
This definitely seems like the "Kafka" way to solve this problem, but I fear there are implications to this partitioning scheme I'd love to see answered. For example, partition counts aren't infinite, and aren't easily adjusted after the fact. So if you choose, say, 10 partitions originally, for a SKU space that is nearly infinite, then in reality you can only handle 10 parallel streams of work. Any SKU that is parti…
On the other hand Kafka partitions are relatively cheap on both broker and client side; 100 partitions does not require 100 parallel consumers so over-provisioning is not so risky.
Re: Kafka Is Not a Database
#55Tbh, It's a weird blog post coming from the materialize folks, considering they know better. The "event sourced" arch they sketched is missing pieces. Normaly you'd have single writer instances that are locked to the corresponding kafka partition, which ensure strong transactional guarantees, IF you need them. Throwing shade for maketings sake is something that they should be above. I mean c'mon, I'd argue that Postg…
Totally agree with this. "Kafka is terrible if you use it with poor practices" should be the title of this. It's totally clickbait targeting the "Database Inside Out" articles / concept, which if you read the most common 3-page high level it explicitly states you should be materializing up into other data sources / structures / systems / etc
To date, I've never written code that reads from a Kafka topic that wasn't taking data and transforming + materializing it into domain intelligence
Re: Kafka Is Not a Database
#56Is this really a thing? Do people really try to use Kafka as the system of record for financial transactions or similar data?
Hell yes. Best thing I ever did.
Updating balances using an RDBMS is like managing your finances with pencils and erasers. Unless you somehow ban the UPDATE statement.
Updating balances with Kafka is like working in pen. You can't[1] change the ledger lines, you can only add corrections after the fact.
[1] Yes, Kafka records can be updated/deleted depending on configuration. But when your codebase is written around append-only operations, in-place mutations are hard and corrections are easy, so your fellow programmers fall into the 'pit of success'.
Re: Kafka Is Not a Database
#57I mean, duh? Does Apache Kafka ever made the claim that it is a database? Other things that are not a database: Apache Traffic Server, Apache Mahout, Apache Jakarta, Apache ActiveMQ... hundreds of these exist.
The title sounds generic, but the article makes it clear that it's responding to specific proposed uses of Kafka.
Re: Kafka Is Not a Database
#58Is this really a thing? Do people really try to use Kafka as the system of record for financial transactions or similar data?
Hell yes. Best thing I ever did. Updating balances using an RDBMS is like managing your finances with pencils and erasers. Unless you somehow ban the UPDATE statement. Updating balances with Kafka is like working in pen. You can't[1] change the ledger lines, you can only add corrections after the fact. [1] Yes, Kafka records can be updated/deleted depending on configuration. But when your codebase is written around a…
I would just put the ledger in a database table if it's that important and maintain the current state of the account in a separate table. ACID transactions and database constraints make this kind of consistency easier to achieve than many alternatives. It's also easier to prove correctness since you can run queries that return consistent results thanks to the isolation guaranteed by ACID. (Modulo some corner cases that are not hard to work around.)
Just my $0.02.
Re: Kafka Is Not a Database
#59Ok. I admit using Kafka as DB is not straight forward but just stating it doesn't provide ACID functionality is not enough. The example they give is very simplistic. With the correct design of kafka topics and events the problem of the example can be fixed. And according to oracle https://www.oracle.com/database/what-is-database/ : > A database is an organized collection of structured information, or data, typically…
Didn't some newspaper use Kafka to store the newspapers they released in order or something similar? (I think it was the New York Times, maybe??). Honestly as long as you don't use it as a general purpose database, it might very well be the best choice for your use-case.
Indeed. Known the tools and apply the one that has least cons and most pros ;).
Also, a good starter to for knowing how to use a Streaming Store like Kafka as DB is the video Database inside-out. https://www.youtube.com/watch?v=fU9hR3kiOK0
Re: Kafka Is Not a Database
#60Alternatively from Jay Krebs [1] a much more thorough and nuanced discussion that is probably the best send-up on this topic. "So is it crazy to do this? The answer is no, there’s nothing crazy about storing data in Kafka: it works well for this because it was designed to do it. Data in Kafka is persisted to disk, checksummed, and replicated for fault tolerance. Accumulating more stored data doesn’t make it slower. T…
Log compaction definitely isn’t problem free. I’d say it was crazy when he wrote that.
We ran a cluster with lots of compacted topics, hundreds of terabytes of data. At the time it would make broker startup insanely slow. An unclean startup could literally take an hour to go through all the compacted partitions. It was awful.