Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What If We Could Rebuild Kafka from Scratch?

morling.dev

Post reply on HN
Print view View original

51–60 of 229 posts

Re: What If We Could Rebuild Kafka from Scratch?

#51

Post by buster »

carlmr wrote: post #30
ahoka wrote: post #14

Earlier quoted context omitted.

I think it's just horrible software built on great ideas sold on a false premise (this is a generic message queue and if you don't use this you cannot "scale").

I kind of agree on the horrible software bit, but what do you use instead? And can you convince your company to use that, too?

I find that many such systems really just need a scalable messaging system. Use RabbitMQ, Nats, Pub/Sub, ... There are plenty.

Confluent has rather good marketing and when you need messaging but can also gain a persistent, super scalable data store and more, why not use that instead? The obvious answer is: Because there is no one-size-fits-all-solution with no drawbacks.

Re: What If We Could Rebuild Kafka from Scratch?

#52

Post by frklem »

"Faced with such a marked defensive negative attitude on the part of a biased culture, men who have knowledge of technical objects and appreciate their significance try to justify their judgment by giving to the technical object the only status that today has any stability apart from that granted to aesthetic objects, the status of something sacred. This, of course, gives rise to an intemperate technicism that is nothing other than idolatry of the machine and, through such idolatry, by way of identification, it leads to a technocratic yearning for unconditional power. The desire for power confirms the machine as a way to supremacy and makes of it the modern philtre (love-potion)." Gilbert Simondon, On the mode of existence of technical objects.

This is exactly what I interpret from these kind of articles: engineering just for the cause of engineering. I am not saying we should not investigate on how to improve our engineered artifacts, or that we should not improve them. But I see a generalized lack of reflection on why we should do it, and I think it is related to a detachment from the domains we create software for. The article suggests uses of the technology that come from so different ways of using it, that it looses coherence as a technical item.

Re: What If We Could Rebuild Kafka from Scratch?

#53

Post by Spivak »

alpaca128 wrote: post #44
Spivak wrote: post #11

Earlier quoted context omitted.

The fact that this is so common I think yearns for a language that is basically python and rust smashed together where a project would have some code in the python side and some code in the rust side intermixed fluidly like how you can drop to asm in C. Don't even really try to make the two halves of the language similar. An embedded interpreter and JIT in rust basically but jostled around a bit to make it more cohes…

You can already do that with a macro: https://crates.io/crates/inline-python But I'm doubtful that it's going to make things simpler if one can't even decide on a language.

It's not really a "decision" thing, more that the pattern of write most of the code in python, rewrite the performance critical bits in rust as a cpython extension would be nice if it were made first class.

Re: What If We Could Rebuild Kafka from Scratch?

#54

Post by smitty1e »

immibis wrote: post #36
menzoic wrote: post #34

Earlier quoted context omitted.

Writing directly to the datastore ignores the need for queuing the writes. How do you solve for that need?

Why do you need to queue the writes?

A proficient coder can write a program to accomplish a task in the singular.

In the plural, accomplishing that task in a performant way at enterprise scale seems to involve turning every function call into an asynchronous, queued service of some sort.

Which then begets additional deployment and monitoring services.

A queued problem requires a cute solution, bringing acute pain.

Re: What If We Could Rebuild Kafka from Scratch?

#55

Post by vkazanov »

hbogert wrote: post #50
1oooqooq wrote: post #39

Earlier quoted context omitted.

systemd knows very well what it wants to be, they just don't tell anyone. it's real goal is to make Linux administration as useless as windows so RH can sell certifications. tell me the output of systemctl is not as awful as opening the windows service panel.

Tell me systemctl output isn't more beneficial than per distro bash-mess

Well, systemd IS useful, the same way Kafka is. I don't want to back to crappy bash for service management, and Kafka is a de fact standard event streaming solution.

But both are kind of hard to understand end-to-end, especially for an occasional user.

Re: What If We Could Rebuild Kafka from Scratch?

#56

Post by supermatt »

> "Do away with partitions"

> "Key-level streams (... of events)"

When you are leaning on the storage backend for physical partitioning (as per the cloud example, where they would literally partition based on keys), doesnt this effectively just boil down to renaming partitions to keys, and keys to events?

Re: What If We Could Rebuild Kafka from Scratch?

#57

Post by LunaSea »

raducu wrote: post #46
LunaSea wrote: post #28

Earlier quoted context omitted.

But it's also the reason for why these Apache projects systematically get displaced by better and faster C, C++, Rust or Go alternatives.

It would make sense for a highly succesful but stable java project to be replaced like that, but since I'm in the java world, it's usually replaced with another java project. I could provide examples myself, but I'm not convinced it's about java vs c++ or go: hadoop, cassandra, zookeeper

As an outsider, Java looks like a language that can be very fast but it seems like certain idiomatic practices or patterns lead to over-engineered and thus sometimes also slow projects. The Factory> joke comes to mind.

Re: What If We Could Rebuild Kafka from Scratch?

#58

Post by munksbeer »

Ozzie_osman wrote: post #4

I feel like everyone's journey with Kafka ends up being pretty similar. Initially, you think "oh, an append-only log that can scale, brilliant and simple" then you try it out and realize it is far, far, from being simple.

I'm not a fan or an anti-fan of kafka, but I do wonder about the hate it gets.

We use it for streaming tick data, system events, order events, etc, into kdb. We write to kafka and forget. The messages are persisted, and we don't have to worry if kdb has an issue. Out of band consumers read from the topics and persist to kdb.

In several years of doing this we haven't really had any major issues. It does the job we want. Of course, we use the aws managed service, so that simplifies quite a few things.

I read all the hate comments and wonder what we're missing.

Re: What If We Could Rebuild Kafka from Scratch?

#60

Post by gherkinnn »

Mistletoe wrote: post #27

I know it’s not what the article is about but I really wish we could rebuild Franz Kafka and hear what he thought about the tech dystopia we are in. >I cannot make you understand. I cannot make anyone understand what is happening inside me. I cannot even explain it to myself. -Franz Kafka, The Metamorphosis

Both der Proceß and das Schloß apply remarkably well to our times. No extra steps necessary.
Post reply on HN