Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Why are supercomputers all running Linux and not BSD?

news.ycombinator.com

Post reply on HN
Print view View original

71–80 of 82 posts

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#71

Post by JeremyNT »

ktpsns wrote: post #8

Scientific high energy physicist here with regional HPC center on the same floor. My observation is that administrators tend to enterprise distributions such as Scientific linux, Suse linux enterprise server (SLES), together with commercial MPI implementations such as IBM MPI and Intel MPI. On the other hand, people are used to Linux, in my environment literally everybody has Ubuntu on their notebook and workstation.…

One thing very much worth following as a replacement or supplement to modules is Singularity. It is relatively easy to create images that contain all of your required libraries, and you can run these same containers on both your cluster and on your development system.

This can substantially reduce the time to deploy new software and cut down on overhead related to managing multiple modules.

Singularity, unlike Docker, is designed to require only minimal privilege escalation and as such it's an easy sell to HPC admins, who can (at least somewhat) get out of the business of helping users figure out what the heck is weird about their environment when trying to get something running on a cluster for the first time. You can also take these containers with you and be reasonable certain they'll work on another system.

http://singularity.lbl.gov/

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#72

Post by jonesgrant »

PLEASE READ!!!! Hello Guys,This is a Life Time transformation !!!Am so happy I got mine from Kelvin Roland. My blank ATM card can withdraw $3,300 daily. I got it from him last week and now I have $16,500 for free for just a week. The blank ATM CARD withdraws money from any ATM machines and there is no name on it, it is not traceable and now i have money for business and enough money for me and my family to live on. I am really happy i met Kelvin Roland because i met two people before him and they took my money not knowing that they were scams. But am happy now. Kelvin Roland sent the card through DHL and i got it in two days. Get your own card from him now he is not like other scammer pretending to have the ATM card, he is giving it out for free to help people even if it is illegal but it helps a lot and no one ever gets caught. i'm grateful to Kelvin Roland because he changed my story all of a sudden. The card works in all countries except Philippines and Mali. Kelvin Roland email address is kelvinroland.blankatm@outlook.com!

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#73

Post by Annatar »

jabl wrote: post #53
Annatar wrote: post #28

Earlier quoted context omitted.

Okay, if that is the case, illumos and therefore SmartOS has long had stable Infiniband and MPI support and coming from Solaris is famous for his excellent scalability on very large number of processors, as well as long tradition of HPC. Why isn’t it used for HPC then?

Sun (you might add Oracle as well, but I think at that point whatever they could have done was too little too late) mismanagement, and Linux was/is better in many respects? It wasn't called "Slowlaris" for nothing? And it's not like Linux is somehow famous for poor scalability, unless you're talking about the 1990'ies. Yes, back in the 1990'ies it was certainly much worse than Solaris. But for the 2.6 and subsequent…

“Slowlaris” days were 15 years ago with Solaris 8. Meanwhile, Solaris and illumos (and therefore SmartOS) are the only operating systems I know of which provide CPU bursting. If you go put Linux and SmartOS on the same intel CPU based hardware, SmartOS is likely to beat it in performance. What might have been 15 years ago has long since (2005 with Solaris 10) not been the case.

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#74

Post by Annatar »

scentoni wrote: post #57
Annatar wrote: post #28

Earlier quoted context omitted.

Okay, if that is the case, illumos and therefore SmartOS has long had stable Infiniband and MPI support and coming from Solaris is famous for his excellent scalability on very large number of processors, as well as long tradition of HPC. Why isn’t it used for HPC then?

Sparc machines were not as good at number crunching as Power so Sun wasn't as well-represented in the list as IBM, and Solaris wasn't as heavily used as AIX.

I’m specifically referring to running HPC on SmartOS, which runs only on intel and AMD, with full support for intel only. My question is why isn’t it used for HPC now since it provides CPU bursting, not why it wasn’t used in the past. Fair disclosure: I grew up on SGI and HPC, I know what was before.

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#75

Post by rurban »

na85 wrote: post #61
a012 wrote: post #27

Earlier quoted context omitted.

Is it egg and chicken matter? Vendors don't write driver for BSD and BSDs lack of users because lacks of drivers. Honestly I hope I can run an OpenBSD and install whatever driver for my plugged in devices, both for my personal and production servers.

OpenBSD's performance is atrocious. Scrolling the browser laggy and unresponsive on my ThinkPad x220, and closing tabs sometimes results in multiple-second freezes. That's maybe good for a router but simply not HPC material.

Nobody wants to use OpenBSD HPC. Everybody wants to use Dragonfly BSD HPC, if not many drivers were missing. Dragonfly MPI outperforms Linux, Linux just has more HW support and a bit better TCP/IP stack.

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#76

Post by leandrod »

tw04 wrote: post #58
a012 wrote: post #27

Earlier quoted context omitted.

Is it egg and chicken matter? Vendors don't write driver for BSD and BSDs lack of users because lacks of drivers. Honestly I hope I can run an OpenBSD and install whatever driver for my plugged in devices, both for my personal and production servers.

Vendors do write drivers for bsd, they just don't generally give them back to the project. Agree or disagree, they generally have a ton of time and money invested in their drivers and don't want to give them away to competitors.

Preciſely why we have copyleft.

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#77

Post by Annatar »

hpcjoe wrote: post #50
tatersolid wrote: post #25

Earlier quoted context omitted.

Netflix uses BSD for OpenConnect because asynchronous disk I/O-which is critical for a CDN-remains a tire fire on Linux after more than 20 years. On Linux you basically have to use blocking threads to emulate async disk I/O, which means tons of threads and overhead when you’re handling 10k-100k concurrent connections per box.

Not true. I've regularly demonstrated very high throughput/connectivity with lots of little connections. The problem I have seen (not only with linux) has been over-aggressive congestion controls, usually configured/set wrong. On high performance async IO, this works quite well in Linux, and there are no blocking threads that I am aware of in that stack. The kernel uses bio dispatches to perform the actual block io.…

It’s not just “my OS > your OS”: SmartOS is bulletproof when it comes to correctness of operation, data integrity and superior ease of system administration, which most prominently manifests itself in less breakage, non-existant problems caused on Linux by techology concepts from the ‘80’s of the past century and nights slept through instead of being in conference calls with clueless managers screaming at one at 01:13 in the morning. These were all issues I have and have had with Linux which I don’t have with SmartOS. That’s a big difference!

An OS is a priori better if I get to sleep through the night without an incident.

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#78

Post by tatersolid »

donavanm wrote: post #49
jandrewrogers wrote: post #46

Earlier quoted context omitted.

This is incorrect. Linux has had proper direct async disk I/O for a decade or more, used ubiquitously in database engines (among other things). It is not emulated with threads.

Last I looked (~4.4) linux AIO implied DIO. Conflating aio and dio is the problem, not a feature. On FreeBSD AIO works with the page cache for read and write, read ahead works, sendfile works, io & cache & readahead & size hints all work. Linux has half of those, and DIO none. As i recall.

Bingo. Async disk IO on Linux has to be unbuffered and block aligned. Making it useful only for databases that manage their own caching, and useless for file systems.

A video CDN needs lots of concurrent access to a file system.

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#79

Post by tatersolid »

hendzen wrote: post #40
tatersolid wrote: post #25

Earlier quoted context omitted.

Netflix uses BSD for OpenConnect because asynchronous disk I/O-which is critical for a CDN-remains a tire fire on Linux after more than 20 years. On Linux you basically have to use blocking threads to emulate async disk I/O, which means tons of threads and overhead when you’re handling 10k-100k concurrent connections per box.

That's not really true anymore. Linux AIO works fine on XFS. http://man7.org/linux/man-pages/man2/io_submit.2.html

Isn’t that still experimental?

From the current aio man page:

“Work has been in progress for some time on a kernel state-machine-based implementation of asynchronous I/O (see io_submit(2), io_setup(2), io_cancel(2), io_destroy(2), io_getevents(2)), but this implementation hasn't yet matured to the point where the POSIX AIO implementation can be completely reimplemented using the kernel system calls.”

Re: Ask HN: Why are supercomputers all running Linux and not BSD?

#80

Post by loeg »

convolvatron wrote: post #48
a012 wrote: post #27

Earlier quoted context omitted.

Is it egg and chicken matter? Vendors don't write driver for BSD and BSDs lack of users because lacks of drivers. Honestly I hope I can run an OpenBSD and install whatever driver for my plugged in devices, both for my personal and production servers.

its very much chicken and egg. Cray used Linux because all the customers were using linux. There was never a technical meeting discussing their relative merits. The Tera MTA project was actually BSD based, because it was from an age where the BSD project had clear technical superiority (and they were probably worried about complying with the GPL) As others have mentioned there was a Mellanox stack in Free circa 2005…

FreeBSD still does have an Infiniband stack and Isilon still uses it.
Post reply on HN