Rochefort – Poor Man's Kafka
github.com
Rochefort – Poor Man's Kafka
1–10 of 26 posts
Re: Rochefort – Poor Man's Kafka
#2I'm always very curious about the backstory of projects like this. Without that backstory there is very little chance I'd try out something like this.
Ideally the read me would explain why Kafka didn't cut it, why the trade offs the authors made were worth it (in this case), and why I did consider using a this system.
Sadly I don't have enough time to read an entire repo of code to try and figure these things out.
Re: Rochefort – Poor Man's Kafka
#3Backstory and caveats:
> Losing Data and NIH
> You can lose data on crash and there is no replication, so you have to orchestrate that yourself doing double writes or something.
> The super simple architecture allows for all kidnds of hacks to do backups/replication/sharding but you have to do those yourself.
> My usecase is ok with losing some data, and we dont have money to pay for kafka+zk+monitoring(kafka,zk), nor time to learn how to optimize it for our quite big write and very big multi-read load.
> Keep in mind that there is some not-invented-here syndrome involved into making it, but I use the service in production and it works very nice :)
Re: Rochefort – Poor Man's Kafka
#4I'm always very curious about the backstory of projects like this. Without that backstory there is very little chance I'd try out something like this. Ideally the read me would explain why Kafka didn't cut it, why the trade offs the authors made were worth it (in this case), and why I did consider using a this system. Sadly I don't have enough time to read an entire repo of code to try and figure these things out.
Atleast read the bottom of the README where some of this is covered
Re: Rochefort – Poor Man's Kafka
#5Backstory and caveats: > Losing Data and NIH > You can lose data on crash and there is no replication, so you have to orchestrate that yourself doing double writes or something. > The super simple architecture allows for all kidnds of hacks to do backups/replication/sharding but you have to do those yourself. > My usecase is ok with losing some data, and we dont have money to pay for kafka+zk+monitoring(kafka,zk), no…
I'm scratching my head about "we dont have money to pay for kafka+zk+monitoring(kafka,zk)". Kafka and Zookeeper are both open source. As are monitoring and alerting tools such as Prometheus. Surely the hosting and storage costs are similar. So what does this project offer its creator, other than a great deal of infrastructural debt and all the latent bugs of a roll-your-own solution that lacks a community?
Re: Rochefort – Poor Man's Kafka
#6Backstory and caveats: > Losing Data and NIH > You can lose data on crash and there is no replication, so you have to orchestrate that yourself doing double writes or something. > The super simple architecture allows for all kidnds of hacks to do backups/replication/sharding but you have to do those yourself. > My usecase is ok with losing some data, and we dont have money to pay for kafka+zk+monitoring(kafka,zk), no…
I'm scratching my head about "we dont have money to pay for kafka+zk+monitoring(kafka,zk)". Kafka and Zookeeper are both open source. As are monitoring and alerting tools such as Prometheus. Surely the hosting and storage costs are similar. So what does this project offer its creator, other than a great deal of infrastructural debt and all the latent bugs of a roll-your-own solution that lacks a community?
I'm assuming he means money as in time, to install, configure, optimize, and monitor those distributed systems.
Re: Rochefort – Poor Man's Kafka
#7Earlier quoted context omitted.
I'm scratching my head about "we dont have money to pay for kafka+zk+monitoring(kafka,zk)". Kafka and Zookeeper are both open source. As are monitoring and alerting tools such as Prometheus. Surely the hosting and storage costs are similar. So what does this project offer its creator, other than a great deal of infrastructural debt and all the latent bugs of a roll-your-own solution that lacks a community?
I'm assuming he means money as in time, to install, configure, optimize, and monitor those distributed systems.
As opposed to the time required to implement, debug, and support a custom solution?
Re: Rochefort – Poor Man's Kafka
#8Earlier quoted context omitted.
I'm scratching my head about "we dont have money to pay for kafka+zk+monitoring(kafka,zk)". Kafka and Zookeeper are both open source. As are monitoring and alerting tools such as Prometheus. Surely the hosting and storage costs are similar. So what does this project offer its creator, other than a great deal of infrastructural debt and all the latent bugs of a roll-your-own solution that lacks a community?
I'm assuming he means money as in time, to install, configure, optimize, and monitor those distributed systems.
Even if that's the case, deploying and scaling a Kafka cluster is something that hundreds of companies have figured out and publicly written about. It's something that you can hire an experienced engineer to fix. When this thing runs into problems, they will be all new ones.
Re: Rochefort – Poor Man's Kafka
#9I've run Kafka at large scale. I've also seen even larger scale attempts to replace it.
Just use Kafka. Seriously, it's rock solid and is practically lingua franca in backend architecture these days. Everyone understands it and every data processing framework or service supports it.
Kafka is much, much more than just distributed pub/sub. It's disk cache optimizations alone make rolling your own a terrible idea.
Re: Rochefort – Poor Man's Kafka
#10On a somewhat related note there is a link implementation of Kafka written in Go called Jocko.[0]