Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rochefort – Poor Man's Kafka

github.com

Post reply on HN
Print view View original

21–26 of 26 posts

Re: Rochefort – Poor Man's Kafka

#21

Post by booknomads »

549362-30499 wrote: post #5
jitl wrote: post #3

Backstory and caveats: > Losing Data and NIH > You can lose data on crash and there is no replication, so you have to orchestrate that yourself doing double writes or something. > The super simple architecture allows for all kidnds of hacks to do backups/replication/sharding but you have to do those yourself. > My usecase is ok with losing some data, and we dont have money to pay for kafka+zk+monitoring(kafka,zk), no…

I'm scratching my head about "we dont have money to pay for kafka+zk+monitoring(kafka,zk)". Kafka and Zookeeper are both open source. As are monitoring and alerting tools such as Prometheus. Surely the hosting and storage costs are similar. So what does this project offer its creator, other than a great deal of infrastructural debt and all the latent bugs of a roll-your-own solution that lacks a community?

Just setting that up will cost at least 2-4gb of ram, and we are streched thin as it is, 2gb of ram would mean we have to get one more node for our kubernetes cluster in gcloud.

Me and my team understands the 300 lines of code that go into rochefort and can twist and modify it for our needs.

Performance will make or break our startup, which deals with real time user behaviour analytics, and doing high performant java for a while, I know very well how much time I will have to spend looking at G1 logs to fine tune it.

I am sure we wont use rochefort after we scale up, but for now I think gives us greather velocity than kafka (just because if we want to modify kafka we have to spend a week on a simple change).

I want to be able to add more meta information in the header, or read the files from another process, rsync and read them to my laptop, add custom reducers etc, all those things will take me minutes with rochefort and days with kafka.

Re: Rochefort – Poor Man's Kafka

#22

Post by tbrock »

voxadam wrote: post #10

On a somewhat related note there is a link implementation of Kafka written in Go called Jocko.[0] [0] https://github.com/travisjeffery/jocko

This is actually interesting because Kakfka’s is great and the protocol makes sense but I want a binary to run so I don’t have to deal with the java ecosystem.

Re: Rochefort – Poor Man's Kafka

#23

Post by bonesss »

549362-30499 wrote: post #8
matt_wulfeck wrote: post #6

Earlier quoted context omitted.

I'm assuming he means money as in time, to install, configure, optimize, and monitor those distributed systems.

Even if that's the case, deploying and scaling a Kafka cluster is something that hundreds of companies have figured out and publicly written about. It's something that you can hire an experienced engineer to fix. When this thing runs into problems, they will be all new ones.

These days you can also find kubernetes scripts that handle the Kafka installation, setup, routing, etc.

That just leaves the "simple work" of administering and tuning which, as you pointed out, is competence that's steadily growing in the industry.

Re: Rochefort – Poor Man's Kafka

#24

Post by bonesss »

saryant wrote: post #9

I've run Kafka at large scale. I've also seen even larger scale attempts to replace it. Just use Kafka. Seriously, it's rock solid and is practically lingua franca in backend architecture these days. Everyone understands it and every data processing framework or service supports it. Kafka is much, much more than just distributed pub/sub. It's disk cache optimizations alone make rolling your own a terrible idea.

> Kafka is much, much more than just distributed pub/sub

In between 0% Kafka and 100% Kafka: Kafkas REST proxy gives a minimalistic API surface for using the system. Recreating that API in another language with a simpler backend is highly achievable...

My Kafka installation has a parallel relational backend that provides an onboarding story for smaller apps and groups, for example. It provides about 13% of Kafkas functionality and can't scale meaningfully, but provides baseline data streaming in a pinch and is API compatible with how we use Kafka most of the time.

Re: Rochefort – Poor Man's Kafka

#26

Post by smugworth »

ckocagil wrote: post #12
jdormit wrote: post #7

Earlier quoted context omitted.

As opposed to the time required to implement, debug, and support a custom solution?

I love the concepts Kafka defines so clearly, but the software is too complex and have dozens of knobs you have to adjust. Simple code with "obviously no bugs" vs complex code with "no obvious bugs".

> I love the concepts Kafka defines so clearly, but the software is too complex and have dozens of knobs you have to adjust.

This is one of our biggest headaches, and it's not even that a Kafka server itself is so configurable. We have hundreds of teams writing client applications, and jumping on bridges because Kafka clients have poor configurations is getting old. Too many knobs to twiddle, but I guess that's what happens if you're expecting to be able to tweak for high performance.

Post reply on HN