How did you pick the name?
Rochefort – Poor Man's Kafka
11–20 of 26 posts
Re: Rochefort – Poor Man's Kafka
#12Earlier quoted context omitted.
I'm assuming he means money as in time, to install, configure, optimize, and monitor those distributed systems.
As opposed to the time required to implement, debug, and support a custom solution?
I love the concepts Kafka defines so clearly, but the software is too complex and have dozens of knobs you have to adjust.
Simple code with "obviously no bugs" vs complex code with "no obvious bugs".
Re: Rochefort – Poor Man's Kafka
#13I've run Kafka at large scale. I've also seen even larger scale attempts to replace it. Just use Kafka. Seriously, it's rock solid and is practically lingua franca in backend architecture these days. Everyone understands it and every data processing framework or service supports it. Kafka is much, much more than just distributed pub/sub. It's disk cache optimizations alone make rolling your own a terrible idea.
This is a principle our industry implements poorly. It often seems like each new generation (for very small values of the word generation) must reinvent the same thing. Perhaps because the old thing was too complex to understand immediately... complexity driven by the needs of the underlying problem... complexity which the new implementation will inevitably obtain if it survives long enough and becomes popular enough that anyone cares.
(That said, I'm highly in favor of innovation of most any kind; building new things is great, if the new thing has some plausible innovation over the old thing!)
Re: Rochefort – Poor Man's Kafka
#14If you're looking for a log that's not Kafka, also worthwhile to check out Redis Streams, which will be included in the upcoming Redis 5.0 release.
http://antirez.com/news/114 (I think the API has changed a bit since this blog post, but the concepts and capabilities are the same.)
Re: Rochefort – Poor Man's Kafka
#15How did you pick the name?
"Rochefort Trappistes 10 is my favorite beer and I was drinking it while doing the initial implementation at sunday night"
Re: Rochefort – Poor Man's Kafka
#16Earlier quoted context omitted.
I'm scratching my head about "we dont have money to pay for kafka+zk+monitoring(kafka,zk)". Kafka and Zookeeper are both open source. As are monitoring and alerting tools such as Prometheus. Surely the hosting and storage costs are similar. So what does this project offer its creator, other than a great deal of infrastructural debt and all the latent bugs of a roll-your-own solution that lacks a community?
I'm assuming he means money as in time, to install, configure, optimize, and monitor those distributed systems.
I'm a self-taught, have a single dedicated server, I have a single-instance Kafka running on top of ZK. Yes, I lose the benefits of replication, failover, etc. I don't need that though. The whole installation took me half an hour, learning Kafka took maybe 3 hours, and as long as my server's been up, Kafka's been up.
Granted, I am not monitoring Kafka, but I do other processes.
The other nice thing is that now that I have ZK, other software that need it can just reuse the same process.
I think using the maintenance cost as a reason to write your own tool, is a short-sighted decision.
Re: Rochefort – Poor Man's Kafka
#17I've run Kafka at large scale. I've also seen even larger scale attempts to replace it. Just use Kafka. Seriously, it's rock solid and is practically lingua franca in backend architecture these days. Everyone understands it and every data processing framework or service supports it. Kafka is much, much more than just distributed pub/sub. It's disk cache optimizations alone make rolling your own a terrible idea.
This is a principle our industry implements poorly. It often seems like each new generation (for very small values of the word generation) must reinvent the same thing. Perhaps because the old thing was too complex to understand immediately... complexity driven by the needs of the underlying problem... complexity which the new implementation will inevitably obtain if it survives long enough and becomes popular enough…
The industry itself feeds this cycle by rewarding people for creating new projects and demanding shiny github projects.
Re: Rochefort – Poor Man's Kafka
#18I've run Kafka at large scale. I've also seen even larger scale attempts to replace it. Just use Kafka. Seriously, it's rock solid and is practically lingua franca in backend architecture these days. Everyone understands it and every data processing framework or service supports it. Kafka is much, much more than just distributed pub/sub. It's disk cache optimizations alone make rolling your own a terrible idea.
This is a principle our industry implements poorly. It often seems like each new generation (for very small values of the word generation) must reinvent the same thing. Perhaps because the old thing was too complex to understand immediately... complexity driven by the needs of the underlying problem... complexity which the new implementation will inevitably obtain if it survives long enough and becomes popular enough…
It’s true not just in software , but in many aspects of modern society. We value innovation (even when it is bot innovative) a lot more than maintenance. Just look at physical infrastructure as an example.
Re: Rochefort – Poor Man's Kafka
#19this repository appears to be just a hair over a week old, so i am skeptical even of "I use the service in production and it works very nice". fun project i'm sure, but if i felt like breaking the rules and engaging in a little NIH of this sort - i'm not sure i'd choose HTTP (or any other network protocol) as the hub to build it around
Re: Rochefort – Poor Man's Kafka
#20this repository appears to be just a hair over a week old, so i am skeptical even of "I use the service in production and it works very nice". fun project i'm sure, but if i felt like breaking the rules and engaging in a little NIH of this sort - i'm not sure i'd choose HTTP (or any other network protocol) as the hub to build it around
I don’t disagree with you, buts it’s also possible it was modeled/extracted from something non-public.