Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka Is Not a Database

materialize.com

Post reply on HN
Print view View original

131–140 of 172 posts

Re: Kafka Is Not a Database

#131

Post by Aqueous »

skyde wrote: post #118
hodgesrm wrote: post #104

Earlier quoted context omitted.

Thanks, this is a great article. The money quote for me is: > I think it makes more sense to think of your datacenter as a giant database, in that database Kafka is the commit log, and these various storage systems are kinds of derived indexes or views. My company supports analytic systems and we see this pattern constantly. It's also sort of a Pat Helland view of the world that subsumes a large fraction of data mana…

The problem is that Kafka API and Commit log API are very different. If you wanted to literally use Kafka for your commit log the same way the Amazon aurora are using a distributed commit log. You would find that a lot of feature a commit log need are missing and impossible to add to kafka.

Yes. as the article points out there's no way to reject an event based on some criteria that keeps the events internally consistent. for instance, two people can't check out the same item of inventory = 1. you need to first validate this event on the materialized view of the commit log to make sure the item is still available. however, what happens when your validation goes out of date? For example, what if there's an event checking the last inventory in the event log but the materialized view doesn't reflect that yet? you end up checking out the same inventory twice and promising it to both customers! not good.

DBs can solve this through optimistic locking (atomically making sure my new event applies to the version number of the previous event + 1, otherwise failing) but there's no way to do this in Kafka as far as I know (however, as this is a problem i'm currently facing, please feel free to let me know if there is)

Re: Kafka Is Not a Database

#133

Post by fuckadtech »

morelisp wrote: post #54
edbrown23 wrote: post #35

Earlier quoted context omitted.

This definitely seems like the "Kafka" way to solve this problem, but I fear there are implications to this partitioning scheme I'd love to see answered. For example, partition counts aren't infinite, and aren't easily adjusted after the fact. So if you choose, say, 10 partitions originally, for a SKU space that is nearly infinite, then in reality you can only handle 10 parallel streams of work. Any SKU that is parti…

On the other hand Kafka partitions are relatively cheap on both broker and client side; 100 partitions does not require 100 parallel consumers so over-provisioning is not so risky.

Yes, but this assumes that users have the foresight to over-provision (enough). It becomes an operational pain in reality.

There is also no getting around the variation of the noisy (or slow) neighbor problem slowing down an entire partition. That also becomes an operational pain.

Re: Kafka Is Not a Database

#134

Post by lmm »

dathinab wrote: post #47
je42 wrote: post #40

Ok. I admit using Kafka as DB is not straight forward but just stating it doesn't provide ACID functionality is not enough. The example they give is very simplistic. With the correct design of kafka topics and events the problem of the example can be fixed. And according to oracle https://www.oracle.com/database/what-is-database/ : > A database is an organized collection of structured information, or data, typically…

Didn't some newspaper use Kafka to store the newspapers they released in order or something similar? (I think it was the New York Times, maybe??). Honestly as long as you don't use it as a general purpose database, it might very well be the best choice for your use-case.

I'd actually say Kafka makes a better general-purpose database. SQL databases are only appropriate for a pretty narrow range of problems.

Re: Kafka Is Not a Database

#135

Post by isbvhodnvemrwvn »

LgWoodenBadger wrote: post #110
georgewfraser wrote: post #89

Earlier quoted context omitted.

"Read-committed isolation" is not a meaningful implementation of transactions. If you can't do read, then a write, while guaranteeing the database didn't change in between, then you don't really have transactions.

This sounds like "serializable" which is (in my experience) rarely useful for a meaningful system.

It's not strictly speaking serializable, what they described is a lost update anomaly, to avoid which repeatable read or snapshot isolation is sufficient.

Serializable (or serializable snapshot isolation) is stronger, it doesn't allow anomalies such as write skew, but also a lot more expensive since you need to keep track of changes in rows matched by predicates to avoid phantom reads (as compared to just keeping track of the rows returned by the query with predicates in this particular transaction).

Also worth noting that some DBs such as Oracle actually lie about implementing serializable (in this case they only offer snapshot isolation), so it's worth keeping that in mind and use locks if necessary.

Re: Kafka Is Not a Database

#136

Post by lmm »

jacques_chester wrote: post #91
mrkeen wrote: post #65

Earlier quoted context omitted.

> ACID transactions and database constraints make this kind of consistency easier to achieve than many alternatives. If your company only runs one database. > It's also easier to prove correctness RDBMSs are wonderful and I don't consider them unreliable at all. But I can't prove the correctness of my teammate's actions. I want them to show their working by putting their updates onto an append-only ledger.

> If your company only runs one database. I think the same argument can be made with "only one Kafka cluster" and "only one blockchain".

> I think the same argument can be made with "only one Kafka cluster"

No it can't. Kafka events for the same partition will always be processed in order. If you have a stream transformation that reads from a topic that lives in one cluster and writes to a topic that lives in another cluster, then (as long as you chose the right partition key) everything will work correctly. Even if you have a sequence of transformations that zig-zag between two different clusters, it will do the right thing. You can't achieve that with traditional databases.

Re: Kafka Is Not a Database

#137

Post by lmm »

> The problem we now have is called write skew. Our reads from the inventory view can be out of date by the time the checkout event is processed. If two users try to buy the same item at nearly the same time, they will both succeed, and we won’t have enough inventory for them both.

And you'll have exactly the same problem if you're using a traditional ACID database: the user saw the item as being available, clicked buy, but it was unavailable by the they went to get it. Using an ACID database doesn't gain you anything; you might as well just use Kafka for everything.

Re: Kafka Is Not a Database

#138

Post by lmm »

Kalium wrote: post #15

As recently as last year, I worked for a company where the Chief Architect, in his infinite wisdom, had decided that a database was a silly legacy thing. The future looked like Kafka streams, with each service being a function against Kafka streams, and data retention set to infinite. Predictably, this setup ran into an interesting assortment of issues. There were no real transactions, no ensured consistency, and no…

> There were no real transactions, no ensured consistency

Which is the right way to do it, because transactions don't extend into the real world. If you need to wait for the consequences of a given event, wait for the consequences of that event. Otherwise, all you really care about is all events happening in a consistent order. It's a much more practical consistency model.

> and no referential integrity

The problem with enforcing referential integrity is how you handle violations of it. Usually you don't really want to outright reject something because it refers to something else that doesn't exist yet, so you end up solving the same problem either way.

> There was also no authentication or authorization, because a default-configured deployment of Kafka from Confluent happily neglects such trivial details.

Pretty common in the database world - both MySQL and PostgreSQL use plaintext protocols by default. Properly-configured kafka uses TLS and/or SASL and has a good ACL system and is as secure as anything else.

> It was a nightmare to code against once you left the fantasy world of functional programming nirvana and encountered real requirements. It meant pushing a whole series of concerns that isolation addresses into application code... or not addressing them at all.

My experience is just the opposite - ACID isolation sounds great until you actually use it in the real world, and then you find it doesn't address your problems and doesn't give you enough control to fix it yourself. It's like when you use one of those magical do-everything frameworks - it works great until you need to customise something slightly, then it's a nightmare. Kafka pushes more of the work onto you upfront - you have to understand your dataflow and design it explicitly - but that pays off immensely.

> It was a GDPR nightmare.

Really? I've found the exact opposite - teams that used an RDBMS had to throw away their customer data under GDPR, because even though they had an entry in their database saying that the customer had agreed, they couldn't tell you what the customer had agreed to or when. Whereas teams using Kafka in the way you describe had an event record for the original agreement, and could tell you where any given piece of data came from.

Re: Kafka Is Not a Database

#139

Post by hinkley »

hodgesrm wrote: post #104
AndrewKemendo wrote: post #32

Alternatively from Jay Krebs [1] a much more thorough and nuanced discussion that is probably the best send-up on this topic. "So is it crazy to do this? The answer is no, there’s nothing crazy about storing data in Kafka: it works well for this because it was designed to do it. Data in Kafka is persisted to disk, checksummed, and replicated for fault tolerance. Accumulating more stored data doesn’t make it slower. T…

Thanks, this is a great article. The money quote for me is: > I think it makes more sense to think of your datacenter as a giant database, in that database Kafka is the commit log, and these various storage systems are kinds of derived indexes or views. My company supports analytic systems and we see this pattern constantly. It's also sort of a Pat Helland view of the world that subsumes a large fraction of data mana…

I’d really love for this to be formalized.a standard format streaming the data through several Susie,s and keeping them eventually consistent.

Re: Kafka Is Not a Database

#140

Post by arduanika »

skyde wrote: post #118
hodgesrm wrote: post #104

Earlier quoted context omitted.

Thanks, this is a great article. The money quote for me is: > I think it makes more sense to think of your datacenter as a giant database, in that database Kafka is the commit log, and these various storage systems are kinds of derived indexes or views. My company supports analytic systems and we see this pattern constantly. It's also sort of a Pat Helland view of the world that subsumes a large fraction of data mana…

The problem is that Kafka API and Commit log API are very different. If you wanted to literally use Kafka for your commit log the same way the Amazon aurora are using a distributed commit log. You would find that a lot of feature a commit log need are missing and impossible to add to kafka.

Greenspun's 11th law: Any sufficiently stateful program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of SQL / relational databases.
Post reply on HN