What If We Could Rebuild Kafka from Scratch?

That depends on how you are using partitions. A partition per topic is effectively going to give you exactly that. What you call a key is then just a topic, and hierarchy (including multitenancy and other forms of namespacing) can be implemented via topic naming convention. This isn't even a novel way to use kafka - its quite a common approach in practice.

Obviously this then comes at the cost of throughput - which is exactly why systems that use these approaches are often much slower than partitioned topics in kafka. Even in your object store example there needs to be synchronization across the storage partitions, and that overhead will give you substantially reduced throughput - as you are effectively using a distributed lock for each write to the complete quorum.

1. Sequential IO when reading from disk.

2. Use of disk cache (instead of reading from disk) when re-reading recently read events.

#2 helps when you have many consumer groups reading from the tail. And this advantage would extend to index-based streaming.

But #1 would not fully extend to index-based streaming.

When does this matter? When adding a new consumer group you would lose the speed advantage of sequential IO, because it consumes from the beginning (which isn’t in disk cache).

BUT this has become less important now that SSDs are so prevalent and affordable. Additionally, in practice, the bottleneck isn’t in disk IO. Consumers tend to perform IO in other systems that incur O(log n) per insert. Or network cards can get saturated way before disk IO is the limiting factor.

I speculate that we got Kafka et al because we didn’t have such abundance of SSDs in the early 2010’s.

So, returning to your question, you wouldn’t notice the difference in the average case, as long as there are SSDs under the hood.

Java hasn't focused on developing the standard libraries to keep up with the modern world. There is no standard http server or API libraries. A lot of the "standard" libraries in software projects are from 3d party sources like Apache, which obviously can't be trusted if they make decisions like above.

Lombok is still heavily used, and its almost a sin to write getters and setters manually instead of using it, but it works by hacking the AST, and the language maintainers never thought to include that as part of the language features.

Furthermore, the biggest use of java in android app development has shifted to use Kotlin instead, because Kotlin aims to address a lot of the pain points of Java. Yet language maintainers of Java don't seem to care.

So overall, if you look at the java ecosystem, its clear that the people working in it have no connection to use in the real world, and are taking the language into some direction that fits their convoluted ideas of what a language should be.

From a more functional perspective, in the modern age of the cloud, ARM chips, and cheap memory, developer cost outweighs infrastructure cost by orders of magnitude. If you write your service in python for MVP, you end up saving a shitload of time, through things like massive amounts of 3d party libraries, not having to wait for compile step, being able to develop during debugging session, much more flexibility in how you write code, and so on. Then as you move from MVP to a product, its much easier to refactor the code with MyPy type checking, convert collections of functions into Classes if need be, and so on. Finally, with PyPy running your code much faster than than CPython, you still will on the average be slower than Java, but not by much, and the added infra cost will be much less than a single developer salary per month.