Live data from Hacker News

Twitter open-sources a high-performance replicated log service

github.com

51–60 of 121 posts

Re: Twitter open-sources a high-performance replicated log service

#51

I was interested until I saw the Zookeeper dependency. I have had too many deployment nightmares with Zookeeper. I would prefer to avoid it as much as possible, plus systems software in Java, sigh.

'plus systems software in Java', mind sharing an explanation?

Java has garbage collection and is about two times slower than C

Re: Twitter open-sources a high-performance replicated log service

#53
post #31
post #3

This could be an interesting competitor to Apache Kafka, which is singularly unique in this space as far as I'm aware. On another note, I find it somewhat funny that these are called "log" services, logging is probably the least interesting use case for these things I can think of. A better description in my mind would be as a distributed event processing framework, since what they are really doing is distributing di…

I'm pretty sure that this post inspired DL. It was written by Jay, one of the 3 founders of kafka (Jay, Jun, and Neha) and should be recommended reading for every software engineer if you've not read it: https://engineering.linkedin.com/distributed-systems/log-wha... An ordered append only datastructure is rightfully called a log. The fact that text based files are called logs is just an annoying feature in common en…

IIRC, we started working on Distributed Log at least a year before that post.

Re: Twitter open-sources a high-performance replicated log service

#54
post #43

I was interested until I saw the Zookeeper dependency. I have had too many deployment nightmares with Zookeeper. I would prefer to avoid it as much as possible, plus systems software in Java, sigh.

Would also be interested to hear more about the "deployment nightmares with Zookeeper". For us, it has to be one of the most stable pieces of 3rd party server software we run.

General things: when your session fails to migrate, when log compaction can't keep up with writes, when logs fill up your disk (ZK used to require external cron jobs to prune logs), when watcher notifications arrive late, when the leader's GC exceeds its heartbeat timeout, and so on.

Lots of things can and do go wrong with Zookeeper. I suspect it depends on the use case, but building a Zookeeper dependency into any system is potentially asking a lot of users/operators.

Re: Twitter open-sources a high-performance replicated log service

#56

No, actually Java is a bane to the database world. Cassandra doesn't work, and Hadoop is a complete waste of hosts for most companies (hence the move to Spark.)

> hence the move to Spark

...which also runs on the Java Virtual Machine and is subject to the same pros and cons.

Re: Twitter open-sources a high-performance replicated log service

#58

No, actually Java is a bane to the database world. Cassandra doesn't work, and Hadoop is a complete waste of hosts for most companies (hence the move to Spark.)

Disagree on two fronts : - By hadoop I assume you mean Map Reduce ? There are other Engines like SAMSA and FLINK and Kafka makes a great event store. Anyway, MR is super for massive throughput batch jobs, for example huge HIVE queries or Pig jobs and if you are reading, breaking the heap size, doing one thing and then writing there is no bonus from doing it in SPARK. - SPARK is written in Scala, which runs on the JVM. And it has a nice Java API as well !

Re: Twitter open-sources a high-performance replicated log service

#59

No, actually Java is a bane to the database world. Cassandra doesn't work, and Hadoop is a complete waste of hosts for most companies (hence the move to Spark.)

> hence the move to Spark ...which also runs on the Java Virtual Machine and is subject to the same pros and cons.

heh - you beat me too it!

Re: Twitter open-sources a high-performance replicated log service

#60
post #51

Earlier quoted context omitted.

'plus systems software in Java', mind sharing an explanation?

Java has garbage collection and is about two times slower than C

For good C, which is hard to write - for me and other mortal humans.

On the other hand, as a mortal, I can write a-grade-above-code-that-an-idiot-would-write-just code in Java at about 10 times the speed I can write dire-useless-risible C code.

The comparison is pointless though, good modern languages like Rust and Julia are developing and LLVM is enabling further development.

Post reply on HN