Live data from Hacker News

Supercomputers: Obama orders world's fastest computer

bbc.co.uk

171–179 of 179 posts

Re: Supercomputers: Obama orders world's fastest computer

#171

Earlier quoted context omitted.

Oh, you're talking about a policy limitation, not a technological one. (And if you're talking about the DOE or NSF/Teragrid/XSEDE clusters, then you're probably right. Haven't touched those in years -- and even when I did, I wasn't doing anything crazy.)

It is a technical limitation. Oftentimes the platform is so specialized that only a tiny handful of compilers are ported to it. Say, just gcc, g++, and gfortran, and xlc, xlC, and xlf. And just one version at that. Java would require porting the JVM to the cut-down, weird Linux on the compute nodes. Some $$$ machines don't even support dynamic linking! The number of these machines is so small that extensive compiler…

There is no problem running a JVM on cut-down linux nodes. A JVM is just a process.

Anyway, the issue with JVMs is that they don't have predictable performance, not that the compilers can't be ported.

Re: Supercomputers: Obama orders world's fastest computer

#172
post #153

Earlier quoted context omitted.

Well my point exactly. That "ton of design and equipment" doesn't scale particularly well, as its cost grows highly super-linearly with the computing power. You need to reduce communication volume to be cost effective at exascale.

This isn't true. You can build awesome high bandwidth clusters for extremely cheap. It takes an understanding of ethernet silicon and TCP implementations, but it can be done. Amazon for example recognized that superlinear cost scaling was killing their profits, and invested in building newer systems with better designs that solve these problems. See also this paper http://research.google.com/pubs/pub36740.html The ma…

You say "awesome high bandwidth" but at 1 Gbit/s per node you're still a long way from an InfiniBand 4X FDR Interconnect (54 Gbit/s and sub-microsecond latency, significantly lower than your network ). As you write, these are built with multistage routers, which add even more latency. So in effect they have reduced (but still high) communication capabilities to keep costs manageable, just as I said.

Re: Supercomputers: Obama orders world's fastest computer

#173
post #171

Earlier quoted context omitted.

It is a technical limitation. Oftentimes the platform is so specialized that only a tiny handful of compilers are ported to it. Say, just gcc, g++, and gfortran, and xlc, xlC, and xlf. And just one version at that. Java would require porting the JVM to the cut-down, weird Linux on the compute nodes. Some $$$ machines don't even support dynamic linking! The number of these machines is so small that extensive compiler…

There is no problem running a JVM on cut-down linux nodes. A JVM is just a process. Anyway, the issue with JVMs is that they don't have predictable performance, not that the compilers can't be ported.

The JVM probably calls fork() and system(), no? Not allowed. Dynamic thread creation? Not allowed. And 50% of your flops go away unless your program uses the BG/Q-specific "double hummer" floating point instructions. These are primitive machines, in terms of development environment and typically require significant rewriting to get even "standard" system software working.

Re: Supercomputers: Obama orders world's fastest computer

#174
post #153

Earlier quoted context omitted.

This isn't true. You can build awesome high bandwidth clusters for extremely cheap. It takes an understanding of ethernet silicon and TCP implementations, but it can be done. Amazon for example recognized that superlinear cost scaling was killing their profits, and invested in building newer systems with better designs that solve these problems. See also this paper http://research.google.com/pubs/pub36740.html The ma…

You say "awesome high bandwidth" but at 1 Gbit/s per node you're still a long way from an InfiniBand 4X FDR Interconnect (54 Gbit/s and sub-microsecond latency, significantly lower than your network ). As you write, these are built with multistage routers, which add even more latency. So in effect they have reduced (but still high) communication capabilities to keep costs manageable, just as I said.

1 Gbit/sec, if you look at the Jupiter paper, was the host speed in 2004. The Jupiter system works with 10G and 40G interfaces on the host.

What's important to recognize is you simply cannot buy Infiniband switches that let you contact a lot (10K+) of hosts together. The vendors won't sell you this, they won't do the R&D to make it, and it would cost infinite anyway.

This is a deliberate choice: for most Internet work, it's better to have really fat bisection bandwidth and non-blocking fabrics, and latency is ignored due to the high cost of building a crossbar that supports that with high radix.

Only if you have an algorithm that absolutely requires, and simply cannot be fixed, low latency, you are almost always better off building a cheaper, fatter fabric, and hiring engineers who know how to write applications that are latency tolerant.

Re: Supercomputers: Obama orders world's fastest computer

#175

Earlier quoted context omitted.

CG isn't truly asynchronous due to its reductions. It can be pipelined in various ways (we have several implementations in PETSc), but performance requires a quality implementation of asynchronous reduction (e.g., MPI_Iallreduce) which the vendors have been slow about developing (I've been working with some on fixing this and Cray has made recent progress). With respect to deep learning and other applications using C…

Is it possible to calculate the theoretical performance of a cluster under HPGMG and then do a practical run and come with an efficiency number like in HPL ? One of the biggest reasons for use of HPL is that many sizing considerations can be based off of the theoretical calculations. But anyway this is very interesting. I definitely need to check this out.

HPL has an abundance of flops at all scales (N^{1.5} flops on N data), so one can expect a decent fraction of peak flop/s on any architecture with enough memory and adequate cache performance. This is a problem because architectural tricks like doubling the vector registers without commensurate improvements in bandwidth, cache sizes, load/store/gather/scatter produce huge (nearly 2x) benefit for HPL and little or no benefit to a large fraction of real applications.

HPGMG is representative of most structure-exploiting algorithms in that it does not have this abundance of flops, thus theoretical performance is actively constrained by both memory bandwidth and flop/s. We see many active constraints in practice; e.g., improving any of peak flop/s, memory bandwidth, network latency, or network bandwidth produces a tangible improvement in HPGMG performance. Depending on the fidelity of the performance model, these dimensions can be a fairly accurate predictor of performance, but ILP, compiler quality, on-node synchronization latency, cache sizes, and similar factors also matter (more for HPGMG-FE than HPGMG-FV).

I think it is actually quite undesirable for benchmark performance to be trivially computed from one parameter in machine provisioning. No computing center has a mission statement asking for a place on a benchmark ranking list (like Top500). Instead, they have a scientific or engineering mandate. Press releases tend to overemphasize the ranking and I think it is harmful to the science any time the benchmark takes precedence over the expected scientific workload. HPGMG is intended to be representative in the sense that if you build an "HPGMG Machine", you'll get a balanced, versatile machine that scientists and engineers in most disciplines will be happy with. I'd still rather the centers focus on their workload instead of HPGMG.

Re: Supercomputers: Obama orders world's fastest computer

#176
post #134

Earlier quoted context omitted.

Interesting! I'm running this one: Intel® Core™ i7-4790K at 4.x Ghz (currently 4.2 for long term big problem stability) What does your setup look like? Cooling, RAM, etc...? And is it good for long term, like say crunch on it for a week type problems?

Water cooling from Antec, it fits into regular desktop case. Under load it makes sound like hoover. Some cheap RAM, some motherboard for $200, 500 Watt PSU. I use it for IDE (Scala compiler is slow). No problem with stability, it ran a few times over weekend at full load. Maximal temperature about 90C. I have put quite high voltage. Over summer it goes down to 4.8GHz.

Nice! I'll need to pick up a new machine soon. Might give this a go to compare with my existing 4.3Ghz machine.

Re: Supercomputers: Obama orders world's fastest computer

#177

Earlier quoted context omitted.

/scratch on a Cray is (last I used one) is a Lustre filesystem. It is the only file storage available to the compute nodes, which have no local storage of their own. No spinning disk, no SSD. So if the code is written such that it makes frequent small writes (e.g. it's peppered with print statements), the lustre nodes get hammered by all the compute nodes and become the bottleneck and they will eventually fall over.

Interesting! This was exactly the case on the system I used. I didn't realize Cray was the only vendor who went the no-local-storage route.

They're not. They're just the only ones who insist on doing it with a filesystem that can't handle the load.

The other problem is that their interconnect is relatively fragile. It's comparatively easy to crash the entire network, at which time your filesystem goes away and processing stops.

But thanks to Lustre, even when it's good, it's bad.

Re: Supercomputers: Obama orders world's fastest computer

#178
post #48
post #20

I suppose that "supercomputers" are all multi-processor these days, so the colossal FLOP numbers are counted as an aggregation over many processors and one has to coordinate these processors in any application that takes advantage of the FLOP specs. Now I am curious what is the fastest single processor?

On a per-core basis, probably Power 8: http://www.anandtech.com/show/9193/the-xeon-e78800-v3-review...

If your memory bandwidth limited then, yes. Otherwise your better of with a decently clocked intel. Anything that spends a portion of its time running out of L2 or better on the Xeon will be significantly faster.

Re: Supercomputers: Obama orders world's fastest computer

#179

Earlier quoted context omitted.

Sure, I'm asking sincerely.

One of the issues faced in super computing isn't just raw horsepower or more cpu's, it's latency. It's not enough to just connect a ton of machines via ethernet, you need specialized hardware to provide high-throughput sharing of data.

Thanks for the insight!
Post reply on HN