Kafka Is Not a Database

That is a valid theory when we talk about readers which look at recent data or when you are trying to append data to the existing system.

But in practice, the accumulation of cold data on a local disk is where this starts to hurt, particularly if that has to serve read traffic which starts from the beginning of time (i.e your queries don't start with a timestamp range).

KSQL transforms does help reduce the depth of the traversal, by building flatter versions of the data set, but you need to repartition the same data on every lookup key you want - so if you had a video game log trace, you'd need multiple materializations for (user) , (user,game), (game) etc.

And on this local storage part, EBS is expensive to just hold cold data, but then replicate it to maintain availability during a node recovery - EBS is like a 1.5x redundant store, better than a single node. I liked the Druid segment model of shoving it off to S3 and still being to read off it (i.e not just stream to S3 as a dumping ground).

When Pravega came out, I liked it a lot for the same - but it hasn't gained enough traction.

I'm not selling any particular tool but Amazon's QLDB is an interesting example of a blockchain-based database. I am interested to see how things like Kafka and this might come together somehow.

In my opinion, we have evolved to a point where storage is not a concern for temporal use cases - i.e. we can now store every change in an immutable fashion. When you think about this being an append only transaction log that you never have to purge, and you make observable events on that log (which is what most CDC systems do)... yeah it works. Now you have every change, cryptographically secure, with events to trigger downstream consumers and you can really rethink the whole architecture of monolithic databases vs. data platforms.

It is an exciting time we are in, in my opinion.

The original sin of Kafka is that it begins with only memory.

To me the right middle way is relational temporal tables[0]. You get both the memoryless query/update convenience and the ability to travel through time.

[0] SQL:2011 introduced temporal data, but in a slightly janky just-so-happens-to-fit-Oracle's-preference kind of way.

The idea on the face of it is not per se a bad one, quite interesting, but the implementations are perhaps not there yet to back such an idea.

It's important to know as an architect when your vision for the architecture is outpacing reality, to know your time horizon, and match the vision with the tools that help you implement the actual use cases you have in your hand right now.

It sounds like this person might have had an interesting idea but not a working system. In another light, this could have been a good idea if all the technology was in place to support it.. but the timing and implementation doesn't sound like it was right, perhaps.

The old saying "use the right tool for the job" comes to mind, but that can be hard to see when the tools are changing so fast, and there is a risk to going too far onto the bleeding edge. Perhaps the saying should have been, "use the rightest tool you can find at the time, that gives you some room to grow, for the job"...