Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Why are supercomputers all running Linux and not BSD?

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 82 posts

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#31

Post by ska »

8fingerlouie wrote: post #4

Some will say better hardware support. While Linux has better hardware support, i usually find this to be in the more exotic direction. I think it's simply down to Linux being where the money is. The big players (IBM, Dell, etc) are all actively promoting Linux, and trained personel is also somewhat easy to find. So Linux is "the beast you know". As for FreeBSD, it might be a technically better platform, but it is li…

  While Linux has better hardware support, i usually find this to be in the more exotic direction.
HPC, and supercomputing in specific, tends to the exotic (particularly with busses)

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#32

Post by pgtan »

jacquesm wrote: post #30

Because of one man: Donald Becker. At the beginning of the commodity super computer era Donald did an absolutely amazing job squeezing out every last bit of performance from commodity networking hardware for 'Beowulf' style clusters. This gave Linux a head start and the self-reinforcing effects of such a head start did the rest, it made answering the question 'for which OS should we start writing drivers?' for specia…

~ $ dmesg | grep Beck

3c59x: Donald Becker and others. www.scyld.com/network/vortex.html

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#33

Post by KaiserPro »

profquail wrote: post #15
KaiserPro wrote: post #11

Earlier quoted context omitted.

I think netflix use BSD because they wanted to use BSD. Sure some flavours of BSD have ZFS built in, but thats a pretty rare corner case. Linux has two things that are extremely useful compared to BSD: 1) commercial backing (should one choose it) 2) first class support for inifinband, top end ethernet (should they use it) and storage controllers

FreeBSD has commercial support from iXsystems. FreeBSD (and perhaps some other BSDs) support “top-end” Ethernet as well. There was a great post on the Netflix blog a couple of months ago (discussed on HN) about how Netflix optimized their systems to serve video at 100Gbps. FreeBSD does not support Infiniband, afaik.

https://medium.com/netflix-techblog/serving-100-gbps-from-an... Is the article. I'd not actually seen that.

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#34

Post by KaiserPro »

smilesnd wrote: post #23
nl wrote: post #18

Earlier quoted context omitted.

But all the supercomputers use their own custom linux. So no commerical backing. This is just wrong. Yes, they use custom Linux, but it is highly highly supported. You buy a Cray or a BlueGene and you get dedicated kernel engineers as well as on site support etc etc. They cut networking and storage to a minimal because those are bottlenecks. These things are just massive ram/cpu/gpu boxes connected properly through p…

I don't consider it commercial linux because they are not competing with other options. The companies that do build these supercomputers have to provide technical support because nothing out there exist for it. Just a different view of what commercial linux is vs building hardware specific software.

Its very much commercial linux, because you are paying for a service, that's linux based.

Sure with how cheap inifiband is (especially compared to 40/100 gig ethernet) one _could_ cobble together a system your self.

Where the magic sauce comes in, and where the like of cray really make things shine is the software they provide to allow end users _easily_ do multi-machine scaling.

libraries for just in time delivery of data directly into ram? yup. location aware job dispatchers that co-locate jobs near each other logically? yup.

All of those hard things are solved for you.

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#35

Post by underwoodley »

smilesnd wrote: post #12
KaiserPro wrote: post #11

Earlier quoted context omitted.

I think netflix use BSD because they wanted to use BSD. Sure some flavours of BSD have ZFS built in, but thats a pretty rare corner case. Linux has two things that are extremely useful compared to BSD: 1) commercial backing (should one choose it) 2) first class support for inifinband, top end ethernet (should they use it) and storage controllers

Netflix uses bsd because it has great IO handling. https://www.quora.com/Why-did-Netflix-choose-FreeBSD-over-Li... But all the supercomputers use their own custom linux. So no commerical backing. Also these computers are not your standard data center. They cut networking and storage to a minimal because those are bottlenecks. These things are just massive ram/cpu/gpu boxes connected properly through pci. Edit: I was…

The Quora discussion doesn't seem to add much. Just a guy saying 'FreeBSD is rock-solid' and '[for] raw performance, .. nothing beats FreeBSD', without giving any technical details.

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#36

Post by renchap »

jpgvm wrote: post #9

HPC may look like COTS gear but it's not. BSD doesn't have drivers for Infiniband and other HPC interconnects. Nor does it have client drivers (let alone server implementation) for Lustre which is the distributed filesystem used by most super computers. I imagine MPI support on BSD is also likely non-existent. Then there is the matter of accelerator support, i.e NVidia GPUs and Intel Xeon Phi. It's not to say that so…

FreeBSD does support Infiniband: https://wiki.freebsd.org/InfiniBand

This page mentions that Mellanox has provided work on this. Also storage vendors uses external infiniband stacks on FreeBSD for many years (I got my first Isilon cluster something like 8 years ago, and it was using an IB backend, with a forked FreeBSD 7 kernel iirc) and are stable and widely deployed.

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#37

Post by X86BSD »

renchap wrote: post #36
jpgvm wrote: post #9

HPC may look like COTS gear but it's not. BSD doesn't have drivers for Infiniband and other HPC interconnects. Nor does it have client drivers (let alone server implementation) for Lustre which is the distributed filesystem used by most super computers. I imagine MPI support on BSD is also likely non-existent. Then there is the matter of accelerator support, i.e NVidia GPUs and Intel Xeon Phi. It's not to say that so…

FreeBSD does support Infiniband: https://wiki.freebsd.org/InfiniBand This page mentions that Mellanox has provided work on this. Also storage vendors uses external infiniband stacks on FreeBSD for many years (I got my first Isilon cluster something like 8 years ago, and it was using an IB backend, with a forked FreeBSD 7 kernel iirc) and are stable and widely deployed.

AFAIR infiniband support in FreeBSD has been there for a very long time.

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#38

Post by erikj »

VSpike wrote: post #5

I wonder if this is another validation of the "worse is better" philosophy as described recently in an HN article http://minnie.tuhs.org/pipermail/tuhs/2017-May/009935.html and also discussed at https://www.jwz.org/doc/worse-is-better.html

JWZ seems to dislike Hacker News for some reason, judging by the redirect.

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#39

Post by kelnage »

pgtan wrote: post #32
jacquesm wrote: post #30

Because of one man: Donald Becker. At the beginning of the commodity super computer era Donald did an absolutely amazing job squeezing out every last bit of performance from commodity networking hardware for 'Beowulf' style clusters. This gave Linux a head start and the self-reinforcing effects of such a head start did the rest, it made answering the question 'for which OS should we start writing drivers?' for specia…

~ $ dmesg | grep Beck 3c59x: Donald Becker and others. www.scyld.com/network/vortex.html

Sadly that link is long dead, but the Wayback Machine comes to the rescue: https://web.archive.org/web/20000619092736/www.scyld.com/net...

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#40

Post by hendzen »

tatersolid wrote: post #25
KaiserPro wrote: post #11

Earlier quoted context omitted.

I think netflix use BSD because they wanted to use BSD. Sure some flavours of BSD have ZFS built in, but thats a pretty rare corner case. Linux has two things that are extremely useful compared to BSD: 1) commercial backing (should one choose it) 2) first class support for inifinband, top end ethernet (should they use it) and storage controllers

Netflix uses BSD for OpenConnect because asynchronous disk I/O-which is critical for a CDN-remains a tire fire on Linux after more than 20 years. On Linux you basically have to use blocking threads to emulate async disk I/O, which means tons of threads and overhead when you’re handling 10k-100k concurrent connections per box.

That's not really true anymore. Linux AIO works fine on XFS.

http://man7.org/linux/man-pages/man2/io_submit.2.html

Post reply on HN