Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Why are supercomputers all running Linux and not BSD?

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 82 posts

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#61

Post by na85 »

a012 wrote: post #27
nl wrote: post #16

Earlier quoted context omitted.

This is the correct answer (especially Infiniband - and Aries on Crays) Also NUMA is very important on supercomputers, and it works well on Linux. The other thing worth noting is the much better support IBM has for Linux on PowerPC (2 in the top 10). I think Sunway (most powerful in the world) is a Linux shop too.

Is it egg and chicken matter? Vendors don't write driver for BSD and BSDs lack of users because lacks of drivers. Honestly I hope I can run an OpenBSD and install whatever driver for my plugged in devices, both for my personal and production servers.

OpenBSD's performance is atrocious. Scrolling the browser laggy and unresponsive on my ThinkPad x220, and closing tabs sometimes results in multiple-second freezes.

That's maybe good for a router but simply not HPC material.

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#62

Post by znpy »

renchap wrote: post #36
jpgvm wrote: post #9

HPC may look like COTS gear but it's not. BSD doesn't have drivers for Infiniband and other HPC interconnects. Nor does it have client drivers (let alone server implementation) for Lustre which is the distributed filesystem used by most super computers. I imagine MPI support on BSD is also likely non-existent. Then there is the matter of accelerator support, i.e NVidia GPUs and Intel Xeon Phi. It's not to say that so…

FreeBSD does support Infiniband: https://wiki.freebsd.org/InfiniBand This page mentions that Mellanox has provided work on this. Also storage vendors uses external infiniband stacks on FreeBSD for many years (I got my first Isilon cluster something like 8 years ago, and it was using an IB backend, with a forked FreeBSD 7 kernel iirc) and are stable and widely deployed.

No matter what your argument is, someone will always come up with the counter argument for a particular example, missing the general point.

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#63

Post by eighthnate »

jabl wrote: post #56

I think, largely, the same reasons apply to Linux vs. BSD in supercomputers as Linux vs. BSD generally. You might as well ask why Linux and not *BSD is used in Android, on servers generally, or by large technical knowledgeable organizations such as Google, Amazon, Facebook, etc. So, in no particular order: - Linux came on the scene when BSD's were mired in legal uncertainty. After the legal issues were settled, Linux…

> The GPL license meant that improvements were shared rather than squirreled away in various proprietary spin-offs and thus lost when whatever company was behind them folded (generally, exceptions going both ways surely exist!).

I've always debated this. You would think that BSD licenses would be more attractive to corporations like google, amazon, facebook, etc and GPL licenses were more attractive to researchers and one would have thought that the BSD systems ( freebsd, netbsd, openbsd, etc ) would be the dominant unix-style OSes. Instead the GPL linux based OSes became dominant.

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#64

Post by CyberFonic »

eighthnate wrote: post #63
jabl wrote: post #56

I think, largely, the same reasons apply to Linux vs. BSD in supercomputers as Linux vs. BSD generally. You might as well ask why Linux and not *BSD is used in Android, on servers generally, or by large technical knowledgeable organizations such as Google, Amazon, Facebook, etc. So, in no particular order: - Linux came on the scene when BSD's were mired in legal uncertainty. After the legal issues were settled, Linux…

> The GPL license meant that improvements were shared rather than squirreled away in various proprietary spin-offs and thus lost when whatever company was behind them folded (generally, exceptions going both ways surely exist!). I've always debated this. You would think that BSD licenses would be more attractive to corporations like google, amazon, facebook, etc and GPL licenses were more attractive to researchers an…

The question is about supercomputers specifically, which are mostly used by researchers and some applications like weather forecasting, aerodynamic simulations, etc. The infrastructures used by Google, Facebook, Amazon are massive clusters of computers, but they are not supercomputers.

In the research space peer-review and reproducible results are critical. So GPL does fit in well. The makers of supercomputers have to accommodate their clients' requirements.

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#65

Post by kev009 »

SEJeff wrote: post #44

Because BSD's SMP support has traditionally been pretty terrible compared to Linux's. They still have a SLAB memory allocator (compared with Linux's default of SLUB which is much better for heavily SMP systems). Many of the vendors for HPC (I'm looking at you Mellanox) primarily develop and certify their products on Linux. While they might work on BSD ok, you're not going to get the full performance and all of the fe…

Your final line is 100% correct but all your supporting details are not.

HPC is generally a "softball" workload because the code is going to be more sympathetic to the hardware than many other computer usages. Processes will batch allocate a lot of RAM and peg runnable state for a long time.

SMP.. "it depends", again a parallel vector matrix multiply is just going to sit in the runnable state on all the cores and the kernel is pretty irrelevant. There is a lot of junior job stuff left in FreeBSD to move locks around. The VFS is quite bad. In an HPC type workload these things probably wont matter that much unless you see a lot of "system %". They will show up in profiles and are generally also easy to fix. But it's not hard to construct a microbenchmark showing Linux > $else in those areas.

SLUB.. no. What kind of HPC workload is going to care much about this? The Linux allocators are pretty awful at contig kernel memory allocation (see ZFS on Linux). I don't see why UMA would architecturally flop here.

NUMA is a sore point on FreeBSD. It should be usable in 12.0. Isilon and Netflix are paying Jeff Roberson to work on it. Some folks on my team are also doing minor NUMA and locking work, but for commercial CDN workloads.

Mellanox does a pretty stellar job on FreeBSD Ethernet and Infiniband support. Unfair dig at them. I generally prefer Chelsio, but Mellanox has lowest latency which is relevant for HPC.

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#66

Post by kev009 »

mkj wrote: post #19

Intel compilers aren't available for bsd. The improved optimisation versus gcc is worthwhile.

They actually are, as is VTune and some other commercial stuff from intel and the open source libraries like ISA-L and IPP and frameworks like DPDK, SPDK and NV-DIMM stuff.. all work on FreeBSD.

Last I heard from my rep, intel was discontinuing icc altogether because it didn't make a lot of sense to not put the optimizations in the compilers most people use.. gcc, llvm, vcpp.

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#67

Post by nl »

smilesnd wrote: post #55
KaiserPro wrote: post #34

Earlier quoted context omitted.

Its very much commercial linux, because you are paying for a service, that's linux based. Sure with how cheap inifiband is (especially compared to 40/100 gig ethernet) one _could_ cobble together a system your self. Where the magic sauce comes in, and where the like of cray really make things shine is the software they provide to allow end users _easily_ do multi-machine scaling. libraries for just in time delivery o…

Redhat is a commercial linux because they are competing with other os/distro in this market. If I pay Joe $5 a month to keep my ubuntu up to date it doesn't make ubuntu a commercial linux even though I am paying for a linux service. These companies building supercomputers are competing in producing supercomputers. Not in providing a linux disto and providing a service for said linux. I very much doubt I could get acc…

This is pretty much exactly how every HPC OS has been sold since the Cray X-MP. It's like if you buy Isilon - it is a software, hardware and support you buy. No one argues that isn't commercial.

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#68

Post by oliv__ »

znpy wrote: post #62
renchap wrote: post #36

Earlier quoted context omitted.

FreeBSD does support Infiniband: https://wiki.freebsd.org/InfiniBand This page mentions that Mellanox has provided work on this. Also storage vendors uses external infiniband stacks on FreeBSD for many years (I got my first Isilon cluster something like 8 years ago, and it was using an IB backend, with a forked FreeBSD 7 kernel iirc) and are stable and widely deployed.

No matter what your argument is, someone will always come up with the counter argument for a particular example, missing the general point.

I need to print this on a T-shirt

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#69

Post by SEJeff »

kev009 wrote: post #65
SEJeff wrote: post #44

Because BSD's SMP support has traditionally been pretty terrible compared to Linux's. They still have a SLAB memory allocator (compared with Linux's default of SLUB which is much better for heavily SMP systems). Many of the vendors for HPC (I'm looking at you Mellanox) primarily develop and certify their products on Linux. While they might work on BSD ok, you're not going to get the full performance and all of the fe…

Your final line is 100% correct but all your supporting details are not. HPC is generally a "softball" workload because the code is going to be more sympathetic to the hardware than many other computer usages. Processes will batch allocate a lot of RAM and peg runnable state for a long time. SMP.. "it depends", again a parallel vector matrix multiply is just going to sit in the runnable state on all the cores and the…

Awesome response, thanks for taking the time to write it.

SLUB was written by Christoph Lameter when he was at Silicon Graphics for their monster Altix machines. It took Linux hours to boot (with SLAB) on that machine. He wrote SLUB in a fit of brilliance to make Linux suck less on these, of which HPC workloads can most certainly be ran. Just like some of the crazy Cray computers, SGI machines used to own HPC. Note that I work with Christoph in the same office and have discussed this with him in person. Regarding contiguous memory allocation, a lot of serious HPC workloads use huge pages set at boot to defeat this, so that part of Linux's fail is a non-issue (You're entirely right btw). Really awesome to hear about NUMA bits in FreeBSD being improved, and I sufficiently feel hit with a cluebat on it.

The bit from Mellanox was from their engineers (in their Haifa, Israel office before lunch) telling me they build their products for Linux first, and then port to everything else. They care deeply that it works on Linux, and it is nice if it works on other systems but not as important. It wasn't a dig at them, it was what the engineer said to me.

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#70

Post by mkj »

kev009 wrote: post #66
mkj wrote: post #19

Intel compilers aren't available for bsd. The improved optimisation versus gcc is worthwhile.

They actually are, as is VTune and some other commercial stuff from intel and the open source libraries like ISA-L and IPP and frameworks like DPDK, SPDK and NV-DIMM stuff.. all work on FreeBSD. Last I heard from my rep, intel was discontinuing icc altogether because it didn't make a lot of sense to not put the optimizations in the compilers most people use.. gcc, llvm, vcpp.

Ah released in 2015, cool.

I'd assumed Intel kept their compilers as a competitive advantage even if they weren't profitable by themselves. Could certainly see it happening though.

Post reply on HN