Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Supercomputers: Obama orders world's fastest computer

bbc.co.uk

Post reply on HN
Print view View original

151–160 of 179 posts

Re: Supercomputers: Obama orders world's fastest computer

#151

Post by dekhn »

rwallace wrote: post #129
dekhn wrote: post #90

Earlier quoted context omitted.

I would say no. Here is my argument: when I worked for DOE, everybody told me I had to run my MD simulations on a super computer using all the processors, and I would judged on my parallel efficiency. This meant using a code that used MPI to communicate at every (or every N) timesteps. I asked, instead, "Why not just run N independent simulations, and pool the results?" In this case, you run an M-thread simulation on…

Interesting! Would it be accurate to say that as the amount of computing power and memory per CPU has increased over the years, so also has the percentage of scientific problems where a single simulation instance will fit on a single CPU? Certainly if you can do so, it's more efficient (in both machine and human resources) to partition by one job per CPU.

Yes, for example when I did my PhD work ~2001 with a T3E I could run a simulation of a duplex DNA in a box of water by running it in parallel. This was true both for memory and CPU reasons. It limited to me studying a single sequence at a time, or 2-3 which was the practical limit on the number of concurrent jobs. This used the well-balanced design of the T3E, which had a great MPI system.

Eventually it reached the point (~2007) where I could fit the whole simulation on a single 4-core Intel box with similar performance. Then, I ran one "task" per machine, and scaled to the number of available machines. This uses only inter-node communication, which goes over a hub or crossbar on the motherboard. Much faster.

Now, I can fit many copies of DNA on a single machine (one task per core). This is far and away the best, because each processor just accesses its own memory, greatly reducing motherboard traffic, so the problem is basically CPU-bound instead of communication bound (this also now applies to GPUs, such that single GPUs can run one large simulation within its own RAM and not have to spill data back and forth over the CPU/GPU communication path).

This moves the challenge to the IO subsystem- I generate so much simulation data that I need a fat MapReduce cluster to analyze the trajectories.

Re: Supercomputers: Obama orders world's fastest computer

#152

Post by dekhn »

pinewurst wrote: post #139
dekhn wrote: post #90

Earlier quoted context omitted.

I would say no. Here is my argument: when I worked for DOE, everybody told me I had to run my MD simulations on a super computer using all the processors, and I would judged on my parallel efficiency. This meant using a code that used MPI to communicate at every (or every N) timesteps. I asked, instead, "Why not just run N independent simulations, and pool the results?" In this case, you run an M-thread simulation on…

I know some people in the life sciences who were strongly encouraged to get Titan time. When they applied and presented ORNL with their embarrassingly parallel code, they were told to go away.

yes, precisely my point. If I wanted to run BLAST by partitioning it to run embarassingly parallel, they wanted to me use mpiBLAST- but mpiBLAST isn't actually any better for any real-world workfload.

This is because 50% of the cost of the machine was the interconnect, and if they let those codes run, it means they wasted budget and will get less next time.

Until I hear that the funders/builders are spending the same amount of budget on machines that let biologists run embarrassingly parallel codes as they spend on TOP500 machines, it's not going to change.

Re: Supercomputers: Obama orders world's fastest computer

#153

Post by dekhn »

lorenzhs wrote: post #141
dekhn wrote: post #92

Earlier quoted context omitted.

You are incorrect saying MapReduce isn't locality aware. Hadoop supports machine, rack, row, and cluster locality scheduling. Also, most modern Internet HPC systems dedicate a ton of design and equipment to having very high cross-sectional bandwidth, which enables the locality restrictions to be relaxed.

Well my point exactly. That "ton of design and equipment" doesn't scale particularly well, as its cost grows highly super-linearly with the computing power. You need to reduce communication volume to be cost effective at exascale.

This isn't true. You can build awesome high bandwidth clusters for extremely cheap. It takes an understanding of ethernet silicon and TCP implementations, but it can be done. Amazon for example recognized that superlinear cost scaling was killing their profits, and invested in building newer systems with better designs that solve these problems.

See also this paper http://research.google.com/pubs/pub36740.html

The main challenge is that because these are built with multistage routers, they have fairly high latency. So much of the effort in modern HPC systems used for Hadoopy workloads goes to latency hiding.

Re: Supercomputers: Obama orders world's fastest computer

#154

Post by ams6110 »

sevensor wrote: post #133
khm wrote: post #114

Earlier quoted context omitted.

Ah, a Cray. Buck up; there are better supercomputers.

It was, in fact. How did you infer that?

/scratch on a Cray is (last I used one) is a Lustre filesystem. It is the only file storage available to the compute nodes, which have no local storage of their own. No spinning disk, no SSD. So if the code is written such that it makes frequent small writes (e.g. it's peppered with print statements), the lustre nodes get hammered by all the compute nodes and become the bottleneck and they will eventually fall over.

Re: Supercomputers: Obama orders world's fastest computer

#155

Post by jen729w »

scottcha wrote: post #115
ypcx wrote: post #106

Earlier quoted context omitted.

Clearly designed to run AI for cyber-warfare / cyber-defense purposes? William Gibson's Black Ice coming to life?

More likely its for modeling nuclear weapons which I believe is the primary purpose of NSCI.

Serious question: what's left to model? They're already big enough to kill everyone, and we have heaps of them.

Re: Supercomputers: Obama orders world's fastest computer

#156

Post by robgibbons »

davegardner wrote: post #14

10 years ago the fastest supercomputer was BlueGene/L which was rated at 136.8 TFlop/s. The current fastest supercomputer is rated at 33,862.7 TFlop/s, or 247 times faster. It seems to me that the aim of taking 10 years to build a supercomputer that is only 20 times faster than the current one might fall a little short if it's aiming to take the top spot.

Computer hardware innovation is a textbook example of diminishing returns. With each improvement in processor performance, size, energy usage, and heat management, it becomes more expensive to push the tech further. We're currently witnessing this effect in action with the recent stagnation in consumer processor speeds. They are still getting better in size, energy and heat management, but average speeds have hovered around 2.5Ghz for years now.

Re: Supercomputers: Obama orders world's fastest computer

#157

Post by trsohmers »

tim333 wrote: post #147
trsohmers wrote: post #118

Earlier quoted context omitted.

The real problem is getting 1 exaflop (or around it) within a reasonable power budget. The DOE's power budget for all of their supercomputing resources is 20 Megawatts, so at a full system level we would need to be at 50 GFLOPs per watt, while the best system right now is at 5.

Nvidia's Pascale GPU, due next year, is supposed to do 28 terraflops with 1300 watts which I make 21 GFLOPs per watt.

That's single precision, which the DOE doesn't care about. The latest NVIDIA GPU's do 8 to 16 times better at single precision (32 bit) floating point compared to double precision (64 bit).

The best next generation DP GFLOPs/watt from one of the big players will most likely be the 2016 Xeon Phi, at ~10-12GFLOPs/watt... You are also forgetting that GPUs also have a ~100W+ CPU sitting next to it, which brings down total efficiency significantly.

Shameless self promotion: My startup (http://rexcomputing.com) is aiming for 64 double precision GFLOPs/watt, and 128 GFLOPs/watts single precision for its first chip next year.

Re: Supercomputers: Obama orders world's fastest computer

#158

Post by markhahn »

white-flame wrote: post #149
tim333 wrote: post #147

Earlier quoted context omitted.

Nvidia's Pascale GPU, due next year, is supposed to do 28 terraflops with 1300 watts which I make 21 GFLOPs per watt.

20+ GFLOPS/W for a single part isn't something new: http://streamcomputing.eu/blog/2012-08-27/processors-that-ca... (2012) A huge amount of overall system power is spent in data transport. Plus, double everything for cooling. That brings the total system efficiency way down from what the actual computational components spend.

not 20 real (64b) gflops/watt. and no, not much power is used on interconnect, and certainly not a PUE of 2.0.

Re: Supercomputers: Obama orders world's fastest computer

#159

Post by markhahn »

jen729w wrote: post #155
scottcha wrote: post #115

Earlier quoted context omitted.

More likely its for modeling nuclear weapons which I believe is the primary purpose of NSCI.

Serious question: what's left to model? They're already big enough to kill everyone, and we have heaps of them.

Often they model how the warhead ages, rather than how it explodes.

Re: Supercomputers: Obama orders world's fastest computer

#160

Post by markhahn »

dekhn wrote: post #90
rwallace wrote: post #87

Earlier quoted context omitted.

That's true for many purposes, but is it true for physical simulations? Don't they need communication on every time step?

I would say no. Here is my argument: when I worked for DOE, everybody told me I had to run my MD simulations on a super computer using all the processors, and I would judged on my parallel efficiency. This meant using a code that used MPI to communicate at every (or every N) timesteps. I asked, instead, "Why not just run N independent simulations, and pool the results?" In this case, you run an M-thread simulation on…

montecarlo-type simulations are fine in their niche. it's just asinine to claim you can do all science that way.
Post reply on HN