Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Baidu File System – A distributed file system for real-time applications

github.com

Post reply on HN
Print view View original

141–150 of 169 posts

Re: Baidu File System – A distributed file system for real-time applications

#141

Post by justinsb »

imafatboy wrote: post #107
justinsb wrote: post #77

Looks nice. I know Raft better than most of the other pieces, so that's where I started; I didn't see code for dynamic membership changes nor log truncation. I can understand getting by with a fixed membership, but log truncation seems like a requirement for a production system. Would be interested to hear whether this is planned or whether there is a clever way around it!

Well, there's another project in the same organization named iNexus achieved in log truncation. It uses leveldb as underlying storage and the leveldb is slightly modified to clean the outdated data when compacting. Maybe BFS will do something similar. For the source code, please refer to https://github.com/baidu/ins And I'm sorry for the lack of English documents in this repo. We are working on it.

Thank you - excited to see this! I do think there is a lack of a C++ library for Raft that stands alone (and you have two projects just within baidu that could share code). I'd be excited to help with a standalone project! And I'm sorry for my lack of non-English, but it seems that the variable names are still in english so I can follow the code :-)

(It is a pity that Chrome doesn't automatically translate github pages that contain different languages - not sure why that isn't happening.)

Re: Baidu File System – A distributed file system for real-time applications

#142

Post by notacoward »

usgroup wrote: post #140
notacoward wrote: post #119

Earlier quoted context omitted.

Oh, sorry, didn't realize we were playing the "move the goalposts" game. If you were to google for "gluster" and "containers" you'd get everything from slick marketing stuff to a presentation at the recent Gluster developer summit in Berlin. I have no idea if any of those would meet your next set of standards but, frankly, meh.

Container hosting with a homogeneous cluster constraint was a real requirement for me that I could not find a solution for amongst existing options but since you're a gluster dev you'd probably know better whether its possible; so happy to stand corrected. Thanks for correcting; and no offence intended.

It is certainly possible. The first user I know of who did this was using Mesos. Nowadays the push is more around doing it with Kubernetes and OpenShift; I know there was at least one presentation on it at Red Hat Summit. I'm a core-infrastructure guy, so that's kind of not my bailiwick, but if there's nothing in Gluster's own documentation about such things there might be something in one of those other communities.

Re: Baidu File System – A distributed file system for real-time applications

#143

Post by otterley »

pkolaczk wrote: post #113
otterley wrote: post #109

Earlier quoted context omitted.

During non-GC periods, probably true. But having a realtime filesystem service that is prone to stop-the-world GC pauses is a showstopper for many applications. Also, a C++ implementation is likelier to use far less memory than a Java implementation, assuming the skills of both programmers are roughly equal.

The underlying local filesystem on each node is not truly realtime, so a "realtime distributed file system" is already quite a stretch. Also JVM is perfectly fine with pause times below a few tens of ms worst-case (when using properly tuned G1, CMS GC), which is lower than worst-case latency induced by network + I/O. As for using less memory - you don't allocate buffers for file data on the JVM heap. You allocate the…

> As for using less memory - you don't allocate buffers for file data on the JVM heap.

I meant the code size and heap allocations for data structures, not file buffers.

And 100MB is huge compared to many C++ programs. And that's on top of the Java runtime!

Re: Baidu File System – A distributed file system for real-time applications

#144

Post by pkolaczk »

GauntletWizard wrote: post #139
pkolaczk wrote: post #113

Earlier quoted context omitted.

The underlying local filesystem on each node is not truly realtime, so a "realtime distributed file system" is already quite a stretch. Also JVM is perfectly fine with pause times below a few tens of ms worst-case (when using properly tuned G1, CMS GC), which is lower than worst-case latency induced by network + I/O. As for using less memory - you don't allocate buffers for file data on the JVM heap. You allocate the…

You're using the academic version of realtime, not the one that anybody cares about. HDFS's biggest problem is, and has always been, that it's literally impossible to tune it to give anything like reliable performance, mostly because the nameserver is a single point of lag for the entire system. "Worst case network and IO" latency is a huge stretch. Network performance is predictably sub-ms if you're using a network…

HDFS biggest problem is its SPOF master-slave architecture, not JVM nor GC. With a truly distributed shared nothing system Java Gc would not be a problem, because servers can now run with no major Gc for hours or days. So two servers or clients doing Gc at the same time are very unlikely. And even if some of them do, the pauses from Gc are much more predictable than the pauses from I/O which on a loaded system can take seconds, not milliseconds.

Also if GC was such a huge problem, exchanges or HFT companies wouldn't use Java for their low latency stuff, and there definitely are companies which do.

Re: Baidu File System – A distributed file system for real-time applications

#146

Post by pkolaczk »

otterley wrote: post #143
pkolaczk wrote: post #113

Earlier quoted context omitted.

The underlying local filesystem on each node is not truly realtime, so a "realtime distributed file system" is already quite a stretch. Also JVM is perfectly fine with pause times below a few tens of ms worst-case (when using properly tuned G1, CMS GC), which is lower than worst-case latency induced by network + I/O. As for using less memory - you don't allocate buffers for file data on the JVM heap. You allocate the…

> As for using less memory - you don't allocate buffers for file data on the JVM heap. I meant the code size and heap allocations for data structures, not file buffers. And 100MB is huge compared to many C++ programs. And that's on top of the Java runtime!

Sure and this DFS in C++ memory use is probably huge compared to many hand-crafted assembly or C programs from 1980s. But who cares? 100 MB or even 1GB is really tiny for today's server hardware. And Java runtime itself is a few MB really. What takes most memory in many Java programs (e.g. IDEs) is code and libraries.

Re: Baidu File System – A distributed file system for real-time applications

#147

Post by kuschku »

ergo14 wrote: post #95
kuschku wrote: post #79

Earlier quoted context omitted.

Leader/Follower is actually something entirely different, but the linked chinese document talks about a master/client approach. Sadly, in the past years, due to some political movements, the term "master/slave" has been declared problematic, and GitHub actively warns that projects using such language can and will be excluded from the service. There have been previous discussions about this on HN.

Github says that? Can you share a link? I thought moderation is up to users unless someone is actually abusing the service.

Read this case, for example: https://news.ycombinator.com/item?id=9966118

Re: Baidu File System – A distributed file system for real-time applications

#148

Post by [deleted] »

gcr wrote: post #103
kuschku wrote: post #79

Earlier quoted context omitted.

Leader/Follower is actually something entirely different, but the linked chinese document talks about a master/client approach. Sadly, in the past years, due to some political movements, the term "master/slave" has been declared problematic, and GitHub actively warns that projects using such language can and will be excluded from the service. There have been previous discussions about this on HN.

Where has Github warned this? I can't find any official documentation about this.

[deleted]

Re: Baidu File System – A distributed file system for real-time applications

#149

Post by kuschku »

gcr wrote: post #103
kuschku wrote: post #79

Earlier quoted context omitted.

Leader/Follower is actually something entirely different, but the linked chinese document talks about a master/client approach. Sadly, in the past years, due to some political movements, the term "master/slave" has been declared problematic, and GitHub actively warns that projects using such language can and will be excluded from the service. There have been previous discussions about this on HN.

Where has Github warned this? I can't find any official documentation about this.

I can’t find the specific case of "master/slave", but you can find a similar case on "retard" here: https://news.ycombinator.com/item?id=9966118

Re: Baidu File System – A distributed file system for real-time applications

#150

Post by kuschku »

Cyph0n wrote: post #81
kuschku wrote: post #79

Earlier quoted context omitted.

Leader/Follower is actually something entirely different, but the linked chinese document talks about a master/client approach. Sadly, in the past years, due to some political movements, the term "master/slave" has been declared problematic, and GitHub actively warns that projects using such language can and will be excluded from the service. There have been previous discussions about this on HN.

Wow, that's very interesting. I thought GitHub delegates moderation to the repo owners. There was actually a huge debate about this on Reddit caused by Swift merging a rename change PR into master. The Swift team was so excited about the change for some reason that they didn't even run tests before the merge...

As I mentioned in other subthreads (sadly I can’t edit the original comment anymore, so I have to duplicate content), there is this very famous example of several repos getting banned, and another getting threatened to be banned, for using the word "retard": https://news.ycombinator.com/item?id=9966118
Post reply on HN