Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Why are supercomputers all running Linux and not BSD?

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 82 posts

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#21

Post by scardine »

alvil wrote: post #7
8fingerlouie wrote: post #4

Some will say better hardware support. While Linux has better hardware support, i usually find this to be in the more exotic direction. I think it's simply down to Linux being where the money is. The big players (IBM, Dell, etc) are all actively promoting Linux, and trained personel is also somewhat easy to find. So Linux is "the beast you know". As for FreeBSD, it might be a technically better platform, but it is li…

> While Linux has better hardware support Linux has wider, not better hardware support. If OpenBSD supports some hardware it simply works out of the box and is stable and rock solid.

From the famous poster hanging on Facebook's office:

"Done is better than perfect".

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#22

Post by xxxdarrenxxx »

scardine wrote: post #21
alvil wrote: post #7

Earlier quoted context omitted.

> While Linux has better hardware support Linux has wider, not better hardware support. If OpenBSD supports some hardware it simply works out of the box and is stable and rock solid.

From the famous poster hanging on Facebook's office: "Done is better than perfect".

Yes the snowball effect, exponential.

Let's take this example.

Today I woke up and said to myself. I'm gonna make a supercomputer.

I will research and research, and come to a point where I could choose Linux or BSD for example.

I read about one thing, and another and it all already has real examples running Linux (the top 500).

So I can re-invent the wheel for god knows how long, or use what works for the top 500, which seems to be Linux.

Decision easily made.

Now the top 501 super computers run Linux.

Who is brave enough to not actively accept the wheel that is.

I probably wouldn't, especially with the fact that I have * no insurance * that a non-Linux super computer can hold it's own in real world applicable problem solving.

Theoretically one might get there, but the pragmatism is strong.

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#23

Post by smilesnd »

nl wrote: post #18
smilesnd wrote: post #12

Earlier quoted context omitted.

Netflix uses bsd because it has great IO handling. https://www.quora.com/Why-did-Netflix-choose-FreeBSD-over-Li... But all the supercomputers use their own custom linux. So no commerical backing. Also these computers are not your standard data center. They cut networking and storage to a minimal because those are bottlenecks. These things are just massive ram/cpu/gpu boxes connected properly through pci. Edit: I was…

But all the supercomputers use their own custom linux. So no commerical backing. This is just wrong. Yes, they use custom Linux, but it is highly highly supported. You buy a Cray or a BlueGene and you get dedicated kernel engineers as well as on site support etc etc. They cut networking and storage to a minimal because those are bottlenecks. These things are just massive ram/cpu/gpu boxes connected properly through p…

I don't consider it commercial linux because they are not competing with other options. The companies that do build these supercomputers have to provide technical support because nothing out there exist for it. Just a different view of what commercial linux is vs building hardware specific software.

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#24

Post by lostboys67 »

KaiserPro wrote: post #11
smilesnd wrote: post #6

A comment I found while looking at linux OS that are run on super computers. "Originally, the top 500 list was populated entirely by proprietary Unix systems from vendors like Cray research, SGI, etc. In June 1998, the first Linux system entered the top 500 list. By June 2003, Linux systems passed the 25% mark, accounting for 139 of the top 500. By November of 2003, Linux systems comprised over 56% of the top 500. By…

I think netflix use BSD because they wanted to use BSD. Sure some flavours of BSD have ZFS built in, but thats a pretty rare corner case. Linux has two things that are extremely useful compared to BSD: 1) commercial backing (should one choose it) 2) first class support for inifinband, top end ethernet (should they use it) and storage controllers

Most modern HPC clusters use Infiniband and the more exotic Ethernet types - having done courses on classic structured Ethernet setups seeing some of the challenges building HPC clusters are fascinating.

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#25

Post by tatersolid »

KaiserPro wrote: post #11
smilesnd wrote: post #6

A comment I found while looking at linux OS that are run on super computers. "Originally, the top 500 list was populated entirely by proprietary Unix systems from vendors like Cray research, SGI, etc. In June 1998, the first Linux system entered the top 500 list. By June 2003, Linux systems passed the 25% mark, accounting for 139 of the top 500. By November of 2003, Linux systems comprised over 56% of the top 500. By…

I think netflix use BSD because they wanted to use BSD. Sure some flavours of BSD have ZFS built in, but thats a pretty rare corner case. Linux has two things that are extremely useful compared to BSD: 1) commercial backing (should one choose it) 2) first class support for inifinband, top end ethernet (should they use it) and storage controllers

Netflix uses BSD for OpenConnect because asynchronous disk I/O-which is critical for a CDN-remains a tire fire on Linux after more than 20 years.

On Linux you basically have to use blocking threads to emulate async disk I/O, which means tons of threads and overhead when you’re handling 10k-100k concurrent connections per box.

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#27

Post by a012 »

nl wrote: post #16
jpgvm wrote: post #9

HPC may look like COTS gear but it's not. BSD doesn't have drivers for Infiniband and other HPC interconnects. Nor does it have client drivers (let alone server implementation) for Lustre which is the distributed filesystem used by most super computers. I imagine MPI support on BSD is also likely non-existent. Then there is the matter of accelerator support, i.e NVidia GPUs and Intel Xeon Phi. It's not to say that so…

This is the correct answer (especially Infiniband - and Aries on Crays) Also NUMA is very important on supercomputers, and it works well on Linux. The other thing worth noting is the much better support IBM has for Linux on PowerPC (2 in the top 10). I think Sunway (most powerful in the world) is a Linux shop too.

Is it egg and chicken matter? Vendors don't write driver for BSD and BSDs lack of users because lacks of drivers. Honestly I hope I can run an OpenBSD and install whatever driver for my plugged in devices, both for my personal and production servers.

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#28

Post by Annatar »

jpgvm wrote: post #9

HPC may look like COTS gear but it's not. BSD doesn't have drivers for Infiniband and other HPC interconnects. Nor does it have client drivers (let alone server implementation) for Lustre which is the distributed filesystem used by most super computers. I imagine MPI support on BSD is also likely non-existent. Then there is the matter of accelerator support, i.e NVidia GPUs and Intel Xeon Phi. It's not to say that so…

Okay, if that is the case, illumos and therefore SmartOS has long had stable Infiniband and MPI support and coming from Solaris is famous for his excellent scalability on very large number of processors, as well as long tradition of HPC. Why isn’t it used for HPC then?

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#29

Post by jo909 »

8fingerlouie wrote: post #4

Some will say better hardware support. While Linux has better hardware support, i usually find this to be in the more exotic direction. I think it's simply down to Linux being where the money is. The big players (IBM, Dell, etc) are all actively promoting Linux, and trained personel is also somewhat easy to find. So Linux is "the beast you know". As for FreeBSD, it might be a technically better platform, but it is li…

And hardware support does not only mean that it "somehow" works, but that the hardware vendor and the operating system vendor have certified this combination and will support you with all your problems for many many years.

Edit: support as in commercial support with SLA etc

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#30

Post by jacquesm »

Because of one man: Donald Becker. At the beginning of the commodity super computer era Donald did an absolutely amazing job squeezing out every last bit of performance from commodity networking hardware for 'Beowulf' style clusters.

This gave Linux a head start and the self-reinforcing effects of such a head start did the rest, it made answering the question 'for which OS should we start writing drivers?' for specialty HPC hardware a no-brainer.

https://en.wikipedia.org/wiki/Donald_Becker

Post reply on HN