Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Why are supercomputers all running Linux and not BSD?

news.ycombinator.com

Post reply on HN
Print view View original

11–20 of 82 posts

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#11

Post by KaiserPro »

smilesnd wrote: post #6

A comment I found while looking at linux OS that are run on super computers. "Originally, the top 500 list was populated entirely by proprietary Unix systems from vendors like Cray research, SGI, etc. In June 1998, the first Linux system entered the top 500 list. By June 2003, Linux systems passed the 25% mark, accounting for 139 of the top 500. By November of 2003, Linux systems comprised over 56% of the top 500. By…

I think netflix use BSD because they wanted to use BSD. Sure some flavours of BSD have ZFS built in, but thats a pretty rare corner case.

Linux has two things that are extremely useful compared to BSD:

1) commercial backing (should one choose it) 2) first class support for inifinband, top end ethernet (should they use it) and storage controllers

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#12

Post by smilesnd »

KaiserPro wrote: post #11
smilesnd wrote: post #6

A comment I found while looking at linux OS that are run on super computers. "Originally, the top 500 list was populated entirely by proprietary Unix systems from vendors like Cray research, SGI, etc. In June 1998, the first Linux system entered the top 500 list. By June 2003, Linux systems passed the 25% mark, accounting for 139 of the top 500. By November of 2003, Linux systems comprised over 56% of the top 500. By…

I think netflix use BSD because they wanted to use BSD. Sure some flavours of BSD have ZFS built in, but thats a pretty rare corner case. Linux has two things that are extremely useful compared to BSD: 1) commercial backing (should one choose it) 2) first class support for inifinband, top end ethernet (should they use it) and storage controllers

Netflix uses bsd because it has great IO handling.

https://www.quora.com/Why-did-Netflix-choose-FreeBSD-over-Li...

But all the supercomputers use their own custom linux. So no commerical backing. Also these computers are not your standard data center. They cut networking and storage to a minimal because those are bottlenecks. These things are just massive ram/cpu/gpu boxes connected properly through pci.

Edit: I was looking at Sunway hardware specs the number one supercomputer they use a PCI-E 3.0 connection for all there nodes. Communication between the nodes is 12GB/second with a latency of 1 us. Their total ram is 1.31 PB

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#13

Post by frankharv »

snvzz wrote: post #10

Because BSDs don't scale in that direction. Dragonfly's design shows promise, but it's not anywhere near ready for supercomputers yet.

Yes I agree most with your comments. The reason is NUMA. FreeBSD performance on NUMA is poor. It was only implemented in 2015. It still needs tuning.

Dragonfly seems to be hard at work on the problems.

http://lists.dragonflybsd.org/pipermail/users/2017-February/...

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#14

Post by nl »

smilesnd wrote: post #6

A comment I found while looking at linux OS that are run on super computers. "Originally, the top 500 list was populated entirely by proprietary Unix systems from vendors like Cray research, SGI, etc. In June 1998, the first Linux system entered the top 500 list. By June 2003, Linux systems passed the 25% mark, accounting for 139 of the top 500. By November of 2003, Linux systems comprised over 56% of the top 500. By…

Time is rarely "sold" on supercomputers - or not in the sense it is in AWS.

Sometimes it is billed to different departments, but in most universities or research groups the "billing" is really quota allocation.

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#15

Post by profquail »

KaiserPro wrote: post #11
smilesnd wrote: post #6

A comment I found while looking at linux OS that are run on super computers. "Originally, the top 500 list was populated entirely by proprietary Unix systems from vendors like Cray research, SGI, etc. In June 1998, the first Linux system entered the top 500 list. By June 2003, Linux systems passed the 25% mark, accounting for 139 of the top 500. By November of 2003, Linux systems comprised over 56% of the top 500. By…

I think netflix use BSD because they wanted to use BSD. Sure some flavours of BSD have ZFS built in, but thats a pretty rare corner case. Linux has two things that are extremely useful compared to BSD: 1) commercial backing (should one choose it) 2) first class support for inifinband, top end ethernet (should they use it) and storage controllers

FreeBSD has commercial support from iXsystems.

FreeBSD (and perhaps some other BSDs) support “top-end” Ethernet as well. There was a great post on the Netflix blog a couple of months ago (discussed on HN) about how Netflix optimized their systems to serve video at 100Gbps.

FreeBSD does not support Infiniband, afaik.

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#16

Post by nl »

jpgvm wrote: post #9

HPC may look like COTS gear but it's not. BSD doesn't have drivers for Infiniband and other HPC interconnects. Nor does it have client drivers (let alone server implementation) for Lustre which is the distributed filesystem used by most super computers. I imagine MPI support on BSD is also likely non-existent. Then there is the matter of accelerator support, i.e NVidia GPUs and Intel Xeon Phi. It's not to say that so…

This is the correct answer (especially Infiniband - and Aries on Crays)

Also NUMA is very important on supercomputers, and it works well on Linux.

The other thing worth noting is the much better support IBM has for Linux on PowerPC (2 in the top 10). I think Sunway (most powerful in the world) is a Linux shop too.

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#17

Post by nerdponx »

ktpsns wrote: post #8

Scientific high energy physicist here with regional HPC center on the same floor. My observation is that administrators tend to enterprise distributions such as Scientific linux, Suse linux enterprise server (SLES), together with commercial MPI implementations such as IBM MPI and Intel MPI. On the other hand, people are used to Linux, in my environment literally everybody has Ubuntu on their notebook and workstation.…

Modules looks really interesting. Makes me wonder why Continuum is out there trying to reinvent the wheel with Anaconda. Glad to have something I can use at work to replace Conda environments. Now all it needs is Powershell/CMD support so I don't have to use it inside Cygwin...

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#18

Post by nl »

smilesnd wrote: post #12
KaiserPro wrote: post #11

Earlier quoted context omitted.

I think netflix use BSD because they wanted to use BSD. Sure some flavours of BSD have ZFS built in, but thats a pretty rare corner case. Linux has two things that are extremely useful compared to BSD: 1) commercial backing (should one choose it) 2) first class support for inifinband, top end ethernet (should they use it) and storage controllers

Netflix uses bsd because it has great IO handling. https://www.quora.com/Why-did-Netflix-choose-FreeBSD-over-Li... But all the supercomputers use their own custom linux. So no commerical backing. Also these computers are not your standard data center. They cut networking and storage to a minimal because those are bottlenecks. These things are just massive ram/cpu/gpu boxes connected properly through pci. Edit: I was…

But all the supercomputers use their own custom linux. So no commerical backing.

This is just wrong. Yes, they use custom Linux, but it is highly highly supported. You buy a Cray or a BlueGene and you get dedicated kernel engineers as well as on site support etc etc.

They cut networking and storage to a minimal because those are bottlenecks. These things are just massive ram/cpu/gpu boxes connected properly through pci.

This is just wrong. Networking is extremely important in supercomputers - but it isn't like setting up a LAN. They use custom networking, Infiniband, Aries, OmniPath etc. There isn't much information about the "PCIe Network" on the Sunway, but the fact it is PCIe isn't very interesting - everyone has fast optical networking. It's the topology and protocol which makes things interesting.

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#20

Post by Certhas »

nerdponx wrote: post #17
ktpsns wrote: post #8

Scientific high energy physicist here with regional HPC center on the same floor. My observation is that administrators tend to enterprise distributions such as Scientific linux, Suse linux enterprise server (SLES), together with commercial MPI implementations such as IBM MPI and Intel MPI. On the other hand, people are used to Linux, in my environment literally everybody has Ubuntu on their notebook and workstation.…

Modules looks really interesting. Makes me wonder why Continuum is out there trying to reinvent the wheel with Anaconda. Glad to have something I can use at work to replace Conda environments. Now all it needs is Powershell/CMD support so I don't have to use it inside Cygwin...

These solve completely different problems. On my HPC I load the conda module to run Python.

They are really different tools for different jobs.

Post reply on HN