Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Twitter open-sources a high-performance replicated log service

github.com

Post reply on HN
Print view View original

61–70 of 121 posts

Re: Twitter open-sources a high-performance replicated log service

#61

Post by mioelnir »

i0exception wrote: post #53
SEJeff wrote: post #31

Earlier quoted context omitted.

I'm pretty sure that this post inspired DL. It was written by Jay, one of the 3 founders of kafka (Jay, Jun, and Neha) and should be recommended reading for every software engineer if you've not read it: https://engineering.linkedin.com/distributed-systems/log-wha... An ordered append only datastructure is rightfully called a log. The fact that text based files are called logs is just an annoying feature in common en…

IIRC, we started working on Distributed Log at least a year before that post.

The Twitter blog post in other comments says that "At design time we had concerns about Kafka’s I/O model...", which means that at the least Kafka was extensively studied during the design phase of DistributedLog, which may have started earlier than Kafka's release of course.

Re: Twitter open-sources a high-performance replicated log service

#62

Post by growse »

throwaway_xx9 wrote: post #55

No, actually Java is a bane to the database world. Cassandra doesn't work, and Hadoop is a complete waste of hosts for most companies (hence the move to Spark.)

I'd be interested to know how you can assert that Hadoop is a 'complete waste of hosts for most companies'. Also, don't underestimate the many, many people successfully running Spark on YARN at scale. Hadoop is actually quite helpful to some workloads.

Re: Twitter open-sources a high-performance replicated log service

#63

Post by growse »

slantedview wrote: post #54
gshx wrote: post #43

Earlier quoted context omitted.

Would also be interested to hear more about the "deployment nightmares with Zookeeper". For us, it has to be one of the most stable pieces of 3rd party server software we run.

General things: when your session fails to migrate, when log compaction can't keep up with writes, when logs fill up your disk (ZK used to require external cron jobs to prune logs), when watcher notifications arrive late, when the leader's GC exceeds its heartbeat timeout, and so on. Lots of things can and do go wrong with Zookeeper. I suspect it depends on the use case, but building a Zookeeper dependency into any s…

Well, like many things, if you deploy it on sensible hardware and put some thought into how its operated, zookeeper isn't particularly problematic. It's certainly not a 'nightmare'.

Re: Twitter open-sources a high-performance replicated log service

#64

Post by nicobn »

throwaway_xx9 wrote: post #55

No, actually Java is a bane to the database world. Cassandra doesn't work, and Hadoop is a complete waste of hosts for most companies (hence the move to Spark.)

Blanket statements like "Cassandra doesn't work" and "Hadoop is a complete waste of hosts for most companies" are unproductive and contribute nothing, unless you can back them with data and real world examples.

So, what data do you base these assertions on ? Also, not to burst your bubble but a lot of businesses (if not the majority) run Spark on YARN. And Spark is built on the JVM.

Re: Twitter open-sources a high-performance replicated log service

#65

Post by manigandham »

tcoppi wrote: post #3

This could be an interesting competitor to Apache Kafka, which is singularly unique in this space as far as I'm aware. On another note, I find it somewhat funny that these are called "log" services, logging is probably the least interesting use case for these things I can think of. A better description in my mind would be as a distributed event processing framework, since what they are really doing is distributing di…

Kafka isn't that unique. There are several hosted options as well as commercial software that can do what Kafka does.

Kafka has the open-source/java community and ecosystem that fits in well with the rest of the current big data processing stuff though.

Re: Twitter open-sources a high-performance replicated log service

#66

Post by manigandham »

tcoppi wrote: post #23
goejoegg wrote: post #17

Earlier quoted context omitted.

If you read the CS papers surrounding distributed systems, you will often see the notion of a 'journal' or a 'log', meaning an append-only structure, which typically contains numerous agreed-upon facts.

I agree that term describes the foundational data structure these services provide, but in common parlance that is not really what it means at all and it is confounded by a common(and boring) use case being to synchronize processing of actual, text-based, logs.

What does "boring" have to do with any use-case?

Re: Twitter open-sources a high-performance replicated log service

#67

Post by kinkdr »

ryanobjc wrote: post #36

I was interested until I saw the Zookeeper dependency. I have had too many deployment nightmares with Zookeeper. I would prefer to avoid it as much as possible, plus systems software in Java, sigh.

Exactly this.

When I saw the header I thought to myself, "Yay! Now I can run something like Kafka, without depending on Zookeeper or having to install JVM!! Give me stable drivers for Python and Go and I am sold!"

If I have to install Zookeeper and JVM, why not use Kafka?

Re: Twitter open-sources a high-performance replicated log service

#68

Post by vruiz »

ryanobjc wrote: post #36

I was interested until I saw the Zookeeper dependency. I have had too many deployment nightmares with Zookeeper. I would prefer to avoid it as much as possible, plus systems software in Java, sigh.

I have similar worries with kafka, that’s why I’ve been working on and off in my own log based message queue for while now [0]. So far is nothing but a humble beginning to something targeting very simple use cases, but AFAIK there aren't any lightweight solutions in this space.

[0] https://github.com/ninibe/netlog

Re: Twitter open-sources a high-performance replicated log service

#69

Post by ryanobjc »

jlisam13 wrote: post #49
ryanobjc wrote: post #36

I was interested until I saw the Zookeeper dependency. I have had too many deployment nightmares with Zookeeper. I would prefer to avoid it as much as possible, plus systems software in Java, sigh.

'plus systems software in Java', mind sharing an explanation?

Java at first is great. Once you get in to large and complex loads, the GC tuning becomes load specific. While tuning GC can be fun (for the first dozen times), the bigger problem happens when your load shifts and your GC tuning becomes subtly wrong.

Also even IF you are tuned properly, there is a world of 99-percentile you'll never get to.

Additionally, the difference between openJDK and Oracle JDK becomes something admins learn to hate you for. Complex shell scripts to invoke java, and lots of standard unix tools just don't work super well. eg: pgrep and pkill. you can tweak it, but it takes a little while to learn the many tips and tricks.

It's not all bad, most Java programs are deployed in a static-all-batteries-included fashion, so you rarely worry about system-installed library versions. So that makes deployment a little less hassle. You never have to recompile for cross platform. The profiling tooling and other stuff is pretty good, and the more you're willing to pay the better the tools get.

Re: Twitter open-sources a high-performance replicated log service

#70

Post by ryanobjc »

asimuvPR wrote: post #42
ryanobjc wrote: post #36

I was interested until I saw the Zookeeper dependency. I have had too many deployment nightmares with Zookeeper. I would prefer to avoid it as much as possible, plus systems software in Java, sigh.

Mind sharing a bit more?

The classic problem of ZooKeeper is the client and the server upon initial startup will DNS resolve and permanently cache the IP addresses of the ZK cluster. Thus any cluster migrations require you to restart _EVERYTHING_.

So running zookeeper in a dynamic environment becomes very risky. Aka AWS. Perhaps on GCE it's less dangerous, because the host migration is pretty good.

There are other operational issues people have noted above as well.

Post reply on HN