Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Twitter open-sources a high-performance replicated log service

github.com

Post reply on HN
Print view View original

71–80 of 121 posts

Re: Twitter open-sources a high-performance replicated log service

#71

Post by pc86 »

nicobn wrote: post #64
throwaway_xx9 wrote: post #55

No, actually Java is a bane to the database world. Cassandra doesn't work, and Hadoop is a complete waste of hosts for most companies (hence the move to Spark.)

Blanket statements like "Cassandra doesn't work" and "Hadoop is a complete waste of hosts for most companies" are unproductive and contribute nothing, unless you can back them with data and real world examples. So, what data do you base these assertions on ? Also, not to burst your bubble but a lot of businesses (if not the majority) run Spark on YARN. And Spark is built on the JVM.

If they had data and examples they would almost certainly have enough experience not to say things like "______ doesn't work" and "______ is a complete waste of hosts."

Re: Twitter open-sources a high-performance replicated log service

#72

Post by btym »

throwaway_xx9 wrote: post #55

No, actually Java is a bane to the database world. Cassandra doesn't work, and Hadoop is a complete waste of hosts for most companies (hence the move to Spark.)

Spark, Kafka, Flink, Storm, YARN, Samza, etc... Good luck staying out of the JVM. "The database world" is a bane to the big data processing world.

Re: Twitter open-sources a high-performance replicated log service

#73

Post by btym »

d0vs wrote: post #12
DannyBee wrote: post #9

Earlier quoted context omitted.

Which says " At design time we had concerns about Kafka’s I/O model and its lack of strong durability guarantees‐a non-starter for an application like a distributed transaction log[3]" Seems reasonable, right? Except "[3] Kafka addressed these durability concerns in version 0.8" So they built a whole thing, because they didn't bother to ask or say "hey, if we help fix the durability, would that we welcome?" or even "…

Seriously this seems like a common pattern in open source: [big company] could just improve [X] but instead builds something from the ground up.

Or forks it and maintains an incompatible, non-contributable internal version based on a release from five years ago.

Re: Twitter open-sources a high-performance replicated log service

#74

Post by mgodave »

trungonnews wrote: post #52

Does DL has a limit on the number of partitions like Kafka?

A DL stream is a single partition if you are thinking in terms of kafka. We stitch them together as a partitioned stream a la Kafka using a different system. So, to answer your question, no, there is no limit to the number of DL streams you can put together into a bigger logical stream, we have some very large ones.

Re: Twitter open-sources a high-performance replicated log service

#75

Post by markbnj »

btym wrote: post #72
throwaway_xx9 wrote: post #55

No, actually Java is a bane to the database world. Cassandra doesn't work, and Hadoop is a complete waste of hosts for most companies (hence the move to Spark.)

Spark, Kafka, Flink, Storm, YARN, Samza, etc... Good luck staying out of the JVM. "The database world" is a bane to the big data processing world.

... logstash, kibana, elasticsearch, lucene, solr ... yeah pretty hard to not run java if you're doing distributed, scalable systems.

Re: Twitter open-sources a high-performance replicated log service

#76

Post by boredandroid »

Several people asked how this compares to Kafka (I'm one of the people who created Kafka at LinkedIn). Here's my take:

I think the motivations they list are: 1. Different I/O model 2. Was started before Kafka had replication (the first release of Kafka with replication was in late 2013 I think)

The I/O model I'm less sure about, we looked at similar things for Kafka and they didn't seem worth it (basically you're doing a ton of stuff at the app level that the OS does pretty well--namely caching and buffering linear I/O), we'd have to look at actual benchmarks to know.

Here is my take on the pros and cons of the core tech.

Pros: - Seems to have better built in support for fencing/idempotence - Better geo placement?

Cons: - Lots more moving pieces. Already people are irritated that there are both Kafka nodes and ZK to set up. This system seems to split this over separate physical tiers for serving, core, storage, and zookeeper. My experience has been lot's of tiers is generally a big headache.

Neutral: - There seems to be a built in achival to HDFS. I think if the consumer is fast and efficient then you don't need to reach around your consumer api which will be high latency (since you have to wait for files to be closed out).

There is also a bunch of stuff Kafka does that I'm just not sure about how complete it is in DistributedLog: - Clients in a bunch of languages - Integration with all the major stream processing frameworks - Log compaction http://kafka.apache.org/documentation.html#compaction - Connector management http://www.confluent.io/blog/announcing-kafka-connect-buildi... - Quotas/throttling - Security/ACLs

Re: Twitter open-sources a high-performance replicated log service

#77

Post by eropple »

ryanobjc wrote: post #70
asimuvPR wrote: post #42

Earlier quoted context omitted.

Mind sharing a bit more?

The classic problem of ZooKeeper is the client and the server upon initial startup will DNS resolve and permanently cache the IP addresses of the ZK cluster. Thus any cluster migrations require you to restart _EVERYTHING_. So running zookeeper in a dynamic environment becomes very risky. Aka AWS. Perhaps on GCE it's less dangerous, because the host migration is pretty good. There are other operational issues people h…

You're right in that Zookeeper can be a hassle in a dynamic environment, but in my experience Exhibitor largely removes that concern.

Re: Twitter open-sources a high-performance replicated log service

#78

Post by rad_gruchalski »

ryanobjc wrote: post #70
asimuvPR wrote: post #42

Earlier quoted context omitted.

Mind sharing a bit more?

The classic problem of ZooKeeper is the client and the server upon initial startup will DNS resolve and permanently cache the IP addresses of the ZK cluster. Thus any cluster migrations require you to restart _EVERYTHING_. So running zookeeper in a dynamic environment becomes very risky. Aka AWS. Perhaps on GCE it's less dangerous, because the host migration is pretty good. There are other operational issues people h…

Then configure it using IP addresses. I've been using Zookeeper like this for 4 years in public EC2, swapping machines in and out and it always worked. What's the problem.

Re: Twitter open-sources a high-performance replicated log service

#79

Post by frugalmail »

ryanobjc wrote: post #36

I was interested until I saw the Zookeeper dependency. I have had too many deployment nightmares with Zookeeper. I would prefer to avoid it as much as possible, plus systems software in Java, sigh.

Rails, node? Web scalable, eh?
Post reply on HN