Wasn't there already a file system named BFS? This might get confusing.
BFS was part of BeOS which is defunct. The creator went on to work at Apple.
Baidu File System – A distributed file system for real-time applications
101–110 of 169 posts
Re: Baidu File System – A distributed file system for real-time applications
#102Re: Baidu File System – A distributed file system for real-time applications
#103Earlier quoted context omitted.
what is the leader/follower pattern? Something like master/slave approach?
Leader/Follower is actually something entirely different, but the linked chinese document talks about a master/client approach. Sadly, in the past years, due to some political movements, the term "master/slave" has been declared problematic, and GitHub actively warns that projects using such language can and will be excluded from the service. There have been previous discussions about this on HN.
Where has Github warned this? I can't find any official documentation about this.
Re: Baidu File System – A distributed file system for real-time applications
#104Earlier quoted context omitted.
Sssshhh don't point out the hypocrisy.
Is it hypocrisy? English is not my native language but I consider it the default CS language. There must be a way for us to share knowledge, and that turned out to be English.
Yes, but it is a problem. Unlike Computer languages, human languages do not only convey pure meanings, i.e. pure descriptions of relations between entities. They embed a full baggage of culture, so even if it is convenient and pragmatic to use English in CS as main language, it is not neutral, it is both an effect and a cause of the Anglo-saxon cultural, economic and military hegenomy over the world.
Re: Baidu File System – A distributed file system for real-time applications
#105Earlier quoted context omitted.
PS: if your DFS works within a docker container you'll have a very strong differentiator since the rest don't. You'd also possibly solve the "how to do storage in a container cloud without resorting to NAS or separate clusters" problem.
> if your DFS works within a docker container you'll have a very strong differentiator since the rest don't. Untrue. Gluster is already deployed that way in many places. Yes, in production and at scale.
Do you mean hackery of this sort:
http://blog.xebia.com/persistence-with-docker-containers-tea...
Or do you know of clean, container only (no plugins or special external tools) solution ?
Re: Baidu File System – A distributed file system for real-time applications
#106I wonder how it compares to quantcast's qfs, anyone know?
Re: Baidu File System – A distributed file system for real-time applications
#107Looks nice. I know Raft better than most of the other pieces, so that's where I started; I didn't see code for dynamic membership changes nor log truncation. I can understand getting by with a fixed membership, but log truncation seems like a requirement for a production system. Would be interested to hear whether this is planned or whether there is a clever way around it!
Well, there's another project in the same organization named iNexus achieved in log truncation. It uses leveldb as underlying storage and the leveldb is slightly modified to clean the outdated data when compacting. Maybe BFS will do something similar.
For the source code, please refer to https://github.com/baidu/ins
And I'm sorry for the lack of English documents in this repo. We are working on it.
Re: Baidu File System – A distributed file system for real-time applications
#108More like a C++ clone of HDFS than most people are likely hoping. While you seem to be able to mount it with FUSE I imagine it's primarily meant to be programmed against directly. Using Raft over a dependency on an external consensus system is nice. Definitely makes the namenode architecture much better.
It looks like a faster version of HDFS since it's written in C++ (vs Java). Another important aspect is that is using SSD + SATA(I suppose) , which could be a better option than standard SATA/SSD or LV cache using SATA + SSD. Even if it's just a new thing, if it proves to be faster it may be implemented in Hadoop ecosystem in the future. HDFS has a lot of features being a mature piece of software but it lacks on the…
"It looks like a faster version of HDFS since it's written in C++ (vs Java)."
This is non sequitur. The conclusion does not follow from the premise.
Re: Baidu File System – A distributed file system for real-time applications
#109Earlier quoted context omitted.
It looks like a faster version of HDFS since it's written in C++ (vs Java). Another important aspect is that is using SSD + SATA(I suppose) , which could be a better option than standard SATA/SSD or LV cache using SATA + SSD. Even if it's just a new thing, if it proves to be faster it may be implemented in Hadoop ecosystem in the future. HDFS has a lot of features being a mature piece of software but it lacks on the…
"It looks like a faster version of HDFS since it's written in C++ (vs Java)." This is non sequitur. The conclusion does not follow from the premise.
During non-GC periods, probably true. But having a realtime filesystem service that is prone to stop-the-world GC pauses is a showstopper for many applications.
Also, a C++ implementation is likelier to use far less memory than a Java implementation, assuming the skills of both programmers are roughly equal.
Re: Baidu File System – A distributed file system for real-time applications
#110Earlier quoted context omitted.
Wow, that's very interesting. I thought GitHub delegates moderation to the repo owners. There was actually a huge debate about this on Reddit caused by Swift merging a rename change PR into master. The Swift team was so excited about the change for some reason that they didn't even run tests before the merge...
Do you have a link for that? I don't follow swift development and I'm unsure what they actually changed. Rename of what?
Changing variable names using master/slave to leader/follower.
Here's my comment from the thread:
https://reddit.com/r/ProgrammerHumor/comments/3veu2t/comment...
The rest of the discussion is a good read too.