Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Baidu File System – A distributed file system for real-time applications

github.com

Post reply on HN
Print view View original

151–160 of 169 posts

Re: Baidu File System – A distributed file system for real-time applications

#151

Post by kuschku »

idealpersona wrote: post #124
omginternets wrote: post #82

Earlier quoted context omitted.

It's the PC version of master/slave. See [0] for the original madness... [0] https://github.com/antirez/redis/issues/3185

Good fucking god. This is insane. And anyone who opposed the the proposal, even while pointing out the fallacy of the core idea, got downvoted to hell too. This gives me a lot of context for what I saw in the last season of South Park.

Even worse, look at GitHub banning repos for using offensive words: https://news.ycombinator.com/item?id=9966118

Re: Baidu File System – A distributed file system for real-time applications

#152

Post by p1mrx »

sshb wrote: post #68

Does it support ipv6?

The underlying socket libraries might in theory, but they're using them poorly. Example from nameserver_main.cc:

    std::string listen_addr = std::string("0.0.0.0") + server_addr.substr(server_addr.rfind(':'));

Re: Baidu File System – A distributed file system for real-time applications

#153

Post by knorker »

GoToRO wrote: post #33
knorker wrote: post #25

Earlier quoted context omitted.

Not to be rude, but I'm glad I don't work where you work. How would that be consistent with your own classes? "Oh no, you can't just use a 'Tree' object, you need to explicitly set that there are no leaves yet, no branches yet, no squirrels yet, etc… etc…" Do you .clear() your vectors before you use them? This sounds like newbies that do: #define TRUE (1 == 1)

It's not about initialization, it's more about specifying clearly what happens in all cases. Anyway the reason for which it would no pass review is that today you use one compiler, tomorrow you have to use another and then you have to review all these little details again. It's about saving money more than anything and you do that by not relying on compiler behavior.

Looks like I can't edit, so I'll create another comment.

I ran into this article that puts quite nicely why the problem isn't the "else", but the "if" itself:

https://medium.com/@bartobri/applying-the-linus-tarvolds-goo...

This is what I meant in the other comment by preferring the non-branching.

Re: Baidu File System – A distributed file system for real-time applications

#154

Post by int_handler »

kuschku wrote: post #79
ergo14 wrote: post #76

Earlier quoted context omitted.

what is the leader/follower pattern? Something like master/slave approach?

Leader/Follower is actually something entirely different, but the linked chinese document talks about a master/client approach. Sadly, in the past years, due to some political movements, the term "master/slave" has been declared problematic, and GitHub actively warns that projects using such language can and will be excluded from the service. There have been previous discussions about this on HN.

Are these conceptually/semantically different from master/worker?

Re: Baidu File System – A distributed file system for real-time applications

#155

Post by int_handler »

chubot wrote: post #127

Interesting: Google flags, protocol buffers, and Google C++ style.

Seems that most of Baidu's C++ open source projects have this pattern as well, albeit with minor variations on the Google C++ style such as 4 spaces for indentation.

Re: Baidu File System – A distributed file system for real-time applications

#156

Post by otterley »

pkolaczk wrote: post #146
otterley wrote: post #143

Earlier quoted context omitted.

> As for using less memory - you don't allocate buffers for file data on the JVM heap. I meant the code size and heap allocations for data structures, not file buffers. And 100MB is huge compared to many C++ programs. And that's on top of the Java runtime!

Sure and this DFS in C++ memory use is probably huge compared to many hand-crafted assembly or C programs from 1980s. But who cares? 100 MB or even 1GB is really tiny for today's server hardware. And Java runtime itself is a few MB really. What takes most memory in many Java programs (e.g. IDEs) is code and libraries.

Size can lead to a tremendous difference in performance on modern CPUs, particularly if you can take advantage of L2/L3 instruction and data caches. It still matters, even on modern "big memory" systems where gigabytes of installed RAM are the norm.

Re: Baidu File System – A distributed file system for real-time applications

#157

Post by otterley »

pkolaczk wrote: post #144
GauntletWizard wrote: post #139

Earlier quoted context omitted.

You're using the academic version of realtime, not the one that anybody cares about. HDFS's biggest problem is, and has always been, that it's literally impossible to tune it to give anything like reliable performance, mostly because the nameserver is a single point of lag for the entire system. "Worst case network and IO" latency is a huge stretch. Network performance is predictably sub-ms if you're using a network…

HDFS biggest problem is its SPOF master-slave architecture, not JVM nor GC. With a truly distributed shared nothing system Java Gc would not be a problem, because servers can now run with no major Gc for hours or days. So two servers or clients doing Gc at the same time are very unlikely. And even if some of them do, the pauses from Gc are much more predictable than the pauses from I/O which on a loaded system can ta…

> Also if GC was such a huge problem, exchanges or HFT companies wouldn't use Java for their low latency stuff, and there definitely are companies which do.

Can you name one?

Re: Baidu File System – A distributed file system for real-time applications

#158

Post by techolic »

stomato wrote: post #29
techolic wrote: post #7

Earlier quoted context omitted.

Care to explain the awesomeness you found?

I see many positive comments that are not downvoted, so when you downvote someone saying "awesome", I suspect it is because you disagree, not because it was a low value post, which would be the reason why you would downvote. Also, your response was "explain why"; again, I don't see people usually questioning each acclaimation.

For the record I didn't downvote GP, at the time I asked there were only two comments and I was in the mood for learning as this isn't my area.

Re: Baidu File System – A distributed file system for real-time applications

#159

Post by pkolaczk »

otterley wrote: post #157
pkolaczk wrote: post #144

Earlier quoted context omitted.

HDFS biggest problem is its SPOF master-slave architecture, not JVM nor GC. With a truly distributed shared nothing system Java Gc would not be a problem, because servers can now run with no major Gc for hours or days. So two servers or clients doing Gc at the same time are very unlikely. And even if some of them do, the pauses from Gc are much more predictable than the pauses from I/O which on a loaded system can ta…

> Also if GC was such a huge problem, exchanges or HFT companies wouldn't use Java for their low latency stuff, and there definitely are companies which do. Can you name one?

LMAX, New York Exchange.

Re: Baidu File System – A distributed file system for real-time applications

#160

Post by pkolaczk »

otterley wrote: post #156
pkolaczk wrote: post #146

Earlier quoted context omitted.

Sure and this DFS in C++ memory use is probably huge compared to many hand-crafted assembly or C programs from 1980s. But who cares? 100 MB or even 1GB is really tiny for today's server hardware. And Java runtime itself is a few MB really. What takes most memory in many Java programs (e.g. IDEs) is code and libraries.

Size can lead to a tremendous difference in performance on modern CPUs, particularly if you can take advantage of L2/L3 instruction and data caches. It still matters, even on modern "big memory" systems where gigabytes of installed RAM are the norm.

Technically correct, but filesystems are mostly about I/O. For example this Baidu filesystem copies blocks of data into userland memory and transfers them in RPC messages - any system using proper zero copy approach would easily beat it even if coded in Python or JS. Baidu also seems to use threads, locks and SEDA instead of more efficient (but much harder to code) thread-per-core async architecture. Threadpools and lock based synchronization are terrible for latency.

The fact that something is in C++ doesn't make it automatically efficient. And particularly, if we're talking about milliseconds, not nanoseconds here, in Java or C# you can do just everything what you can do in C++, performance-wise.

Post reply on HN