Earlier quoted context omitted.
'plus systems software in Java', mind sharing an explanation?
Java has garbage collection and is about two times slower than C
C dont have garbage collection. In some cases Java can outperform C.
Twitter open-sources a high-performance replicated log service
81–90 of 121 posts
Re: Twitter open-sources a high-performance replicated log service
#82Does DL has a limit on the number of partitions like Kafka?
A DL stream is a single partition if you are thinking in terms of kafka. We stitch them together as a partitioned stream a la Kafka using a different system. So, to answer your question, no, there is no limit to the number of DL streams you can put together into a bigger logical stream, we have some very large ones.
Ah! That actually explains the strange syntax I was seeing in the tutorials:
// Create Stream `basic-stream-{3-7}`
// dlog tool create -u ${distributedlog-uri} -r ${stream-prefix} -e ${stream-regex}
./distributedlog-core/bin/dlog tool create -u distributedlog://127.0.0.1:7000/messaging/distributedlog -r basic-stream- -e 3-7
Re: Twitter open-sources a high-performance replicated log service
#83People talk about Zookeeper as a negative, but in a quorum it's been one of the most stable and reliable pieces of software I've deployed, despite being a bit frustrating to set up / configure. Netflix's Exhibitor[1] is an indispensable addition to it.
Also, once you're on the Big Data™ train, a lot of things like to plug into Zookeeper, so it becomes more of a convenience.
Kafka, and presumably DL, are at their most useful when you're pushing the limits of NIC and/or HDD performance for throughput. Zookeeper's configuration is a footnote in the complexity of managing one of these systems, and lets them avoid implementing their own byzantine coordination system. Also, folks seem to appreciate Aphyr's opinion, and he states it pretty plainly: Use Zookeeper. It’s mature, well-designed, and battle-tested. [2]
Re: Twitter open-sources a high-performance replicated log service
#84Earlier quoted context omitted.
The classic problem of ZooKeeper is the client and the server upon initial startup will DNS resolve and permanently cache the IP addresses of the ZK cluster. Thus any cluster migrations require you to restart _EVERYTHING_. So running zookeeper in a dynamic environment becomes very risky. Aka AWS. Perhaps on GCE it's less dangerous, because the host migration is pretty good. There are other operational issues people h…
Then configure it using IP addresses. I've been using Zookeeper like this for 4 years in public EC2, swapping machines in and out and it always worked. What's the problem.
i think you do know the problem, which is not everyone has that set up. Yes, if you are cloud, and can roll with static IPs then great for you. Some people still have to be on-prem and reusing IPs isn't always an option.
Re: Twitter open-sources a high-performance replicated log service
#85I am not that into this topic, but could you compare DL to Soundclouds Roshi[0], and if not, whats different? Thanks!
[0] https://github.com/soundcloud/roshi
Re: Twitter open-sources a high-performance replicated log service
#86This looks potentially fantastic. If I could beg one wish from the developers of this (and almost every other project anywhere near this space), though, it would be one tiny piece of documentation: What's your unique ID scheme? Let's say I'm willing to believe[1] that you've got Durable and Consistent down, once messages make it committed in to the system. What's the story for messages on their way in? My application…
I stumbled upon this page while reading the docs: https://twitter.github.io/distributedlog/html/design/main.ht...
It looks like they use fencing and a two-phase commit to prevent duplicate writes. Whether that covers all failure scenarios I'm not sure.
Re: Twitter open-sources a high-performance replicated log service
#87Re: Twitter open-sources a high-performance replicated log service
#88Earlier quoted context omitted.
'plus systems software in Java', mind sharing an explanation?
Java at first is great. Once you get in to large and complex loads, the GC tuning becomes load specific. While tuning GC can be fun (for the first dozen times), the bigger problem happens when your load shifts and your GC tuning becomes subtly wrong. Also even IF you are tuned properly, there is a world of 99-percentile you'll never get to. Additionally, the difference between openJDK and Oracle JDK becomes something…
I find that 'jps' works as a good replacement for pgrep.
Re: Twitter open-sources a high-performance replicated log service
#89No, actually Java is a bane to the database world. Cassandra doesn't work, and Hadoop is a complete waste of hosts for most companies (hence the move to Spark.)
I'd be interested to know how you can assert that Hadoop is a 'complete waste of hosts for most companies'. Also, don't underestimate the many, many people successfully running Spark on YARN at scale. Hadoop is actually quite helpful to some workloads.
Most companies simply don't have the data volume to make Hadoop worthwhile. You can process tens of TB in an RDBMS on a beefy machine cheaper than a Hadoop cluster.
Hadoop is slow, but on huge data volume the overheads are dwarfed by the parallelism gained. Most companies don't have huge volume though.
For example recently I saw someone propose using Hadoop for a sub-TB dataset...
Re: Twitter open-sources a high-performance replicated log service
#90Earlier quoted context omitted.
> This is an archived project and is no longer supported or updated by Facebook
Yes but you can still use it, or fork it, it's actually a really cool federated logging system.
I hear, from people that would know, that it has a lot of bugs that were fixed internally but never made it into an open-source release.