Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Baidu File System – A distributed file system for real-time applications

github.com

Post reply on HN
Print view View original

121–130 of 169 posts

Re: Baidu File System – A distributed file system for real-time applications

#121

Post by placeybordeaux »

lylei wrote: post #51
usgroup wrote: post #8

Looking through the code it supports fuse, but the documentation in ENG is sparse. It also looks to underpin Tera: the Baidu distributed DB. I think a low read/write latency dfs suitable for real time applications would be a game changer. I'm hoping they up the documentation from here and engage the English speaking community.

Thanks for your advice. We are working on translating all the documents :)

I spent a good bit of time yesterday just throwing all the docs into google translate. Tera looks really interesting, but currently there is no way I can use it unless there is documentation in my native language :/

Re: Baidu File System – A distributed file system for real-time applications

#122

Post by muddyrivers »

khc wrote: post #115

What I really want to know: "Once your code has passed the code-review and merged, it will be run on thousands of servers" And the Chinese text below says tens of thousands of servers, which is it? :-)

Considering Baidu's scale, it would be tens of thousands.

There are several other discrepancies in the doc between the Chinese version and the English one. Some technical proofreading is needed.

Re: Baidu File System – A distributed file system for real-time applications

#123

Post by pkolaczk »

00k wrote: post #47
anilgulecha wrote: post #15

Can a distributed storage expert comment in what ways this differs from hadoop?

first of all, impl in C++ (JVM/GC is pain in the ass) - clear arch (only master and dataserver) - very concise config file and easy to deploy - most important, 10k nodes scalability without federation design of namespace

Lack of good documentation, no tests and possibly undefined behaviour in a few places. The code also doesn't look any cleaner than HDFS and uses some weird mix of C (*printf, error codes) and C++ (vectors, smart pointers, RAII etc).

Re: Baidu File System – A distributed file system for real-time applications

#124

Post by idealpersona »

omginternets wrote: post #82
ergo14 wrote: post #76

Earlier quoted context omitted.

what is the leader/follower pattern? Something like master/slave approach?

It's the PC version of master/slave. See [0] for the original madness... [0] https://github.com/antirez/redis/issues/3185

Good fucking god. This is insane. And anyone who opposed the the proposal, even while pointing out the fallacy of the core idea, got downvoted to hell too. This gives me a lot of context for what I saw in the last season of South Park.

Re: Baidu File System – A distributed file system for real-time applications

#125

Post by jstimpfle »

pkolaczk wrote: post #123
00k wrote: post #47

Earlier quoted context omitted.

first of all, impl in C++ (JVM/GC is pain in the ass) - clear arch (only master and dataserver) - very concise config file and easy to deploy - most important, 10k nodes scalability without federation design of namespace

Lack of good documentation, no tests and possibly undefined behaviour in a few places. The code also doesn't look any cleaner than HDFS and uses some weird mix of C (*printf, error codes) and C++ (vectors, smart pointers, RAII etc).

> weird mix of C (*printf, error codes) and C++ (vectors, smart pointers, RAII etc).

Haven't looked at any code, but what you describe is very common usage.

Re: Baidu File System – A distributed file system for real-time applications

#126

Post by bluejekyll »

usgroup wrote: post #8

Looking through the code it supports fuse, but the documentation in ENG is sparse. It also looks to underpin Tera: the Baidu distributed DB. I think a low read/write latency dfs suitable for real time applications would be a game changer. I'm hoping they up the documentation from here and engage the English speaking community.

Speaking of documentation, I see almost no comments in the code.

It all looks reasonable, but this takes self-documenting to an extreme.

Re: Baidu File System – A distributed file system for real-time applications

#128

Post by joelwilliamson »

StreamBright wrote: post #111
panglott wrote: post #80

Earlier quoted context omitted.

The hegemony of English is unquestionable.

Last time I checked English was the 3rd most spoken language on the planet. - Chinese - Spanish - English You mean in tech?

That's ordering by native speakers. If you count non-natives, English has far more speakers than Spanish (but far less than Mandarin).

Re: Baidu File System – A distributed file system for real-time applications

#129

Post by puzzle »

chubot wrote: post #127

Interesting: Google flags, protocol buffers, and Google C++ style.

If you look at Baidu's infrastructure, it's almost like a parallel universe where the names are identical or almost identical to Google's: BFE, GTC, GSLB. And BFS does look a lot like GFS2 aka Colossus.

Re: Baidu File System – A distributed file system for real-time applications

#130

Post by Monotoko »

lylei wrote: post #51
usgroup wrote: post #8

Looking through the code it supports fuse, but the documentation in ENG is sparse. It also looks to underpin Tera: the Baidu distributed DB. I think a low read/write latency dfs suitable for real time applications would be a game changer. I'm hoping they up the documentation from here and engage the English speaking community.

Thanks for your advice. We are working on translating all the documents :)

Hey, don't suppose you know if the Baidu Maps team will be translating their docs anytime soon? (specifically the android API docs)
Post reply on HN