Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Berkeley Lab Debuts Perlmutter, World’s Fastest AI Supercomputer

hpcwire.com

Post reply on HN
Print view View original

31–40 of 61 posts

Re: Berkeley Lab Debuts Perlmutter, World’s Fastest AI Supercomputer

#31

Post by freeone3000 »

more_corn wrote: post #30
dpzmick wrote: post #26

Earlier quoted context omitted.

I believe there's a hint towards the end of the article: > Note: Perlmutter’s “AI performance” is based on Nvidia’s half-precision numerical format (FP16 Tensor Core) with Nvidia’s sparsity feature enabled. FP16 is a 16 bit floating point format. FLOPS for top 500 are measured with LINPACK HPL, which says it is over 64 bit floating point values (I think): > HPL is a software package that solves a (random) dense linea…

"These FP16 operations are very useful for some kinds of calculations." ML for example. If you use 16bit precision you can fit the model in half the memory and your lookups are twice as fast. Newer GPU models offer "mixed precision mode" It takes some doing to get that working in your tooling though.

Less than it used to! Apex now offers automatic mixed precision on cublas operations, so you get a good bit of result even without a significant code change.

Re: Berkeley Lab Debuts Perlmutter, World’s Fastest AI Supercomputer

#32

Post by freeone3000 »

jeffbee wrote: post #12

How do we know this is the largest ML facility in the field? Can anybody rule out the idea that I could go to GCP right now and launch 1500 nodes with 4 GPUs each?

1500 compute nodes on GPC don't make a supercomputer the same way that all the computers on campus don't. Table stakes are a high speed interconnect, shared data storage, and a job scheduler. I know of a few devices built out of DGX machines with InfiniBand interconnects that are about half the size, and they were massive projects as well. You can't get there from generic cloud compute.

Re: Berkeley Lab Debuts Perlmutter, World’s Fastest AI Supercomputer

#33

Post by jagger27 »

antonzabirko wrote: post #27

Doesnt Cerebras chip just out perform this?

I don’t think they are very comparable. This is still very much a conventional supercomputer despite the marketing.

Here’s a fun comparison of total silicon wafer space used.

A Cerebras die is 46,255 mm^2.

1,500 Milan 64-core CPUs * 8 compute chiplets each is 1,004,832 mm^2. (Not including the I/O chiplet).

6,159 NVidia A100 dies is 5,087,334 mm^2.

These are all made on TSMC 7nm, funnily enough.

Re: Berkeley Lab Debuts Perlmutter, World’s Fastest AI Supercomputer

#35

Post by paxys »

supernova87a wrote: post #34

Maybe this is off topic and a matter of taste, but wouldn't you hesitate to have a massive computer cluster named after you? While you're alive? Either out of modesty, or in the worst case, if something goes terribly wrong with it?

Modesty is a very personal preference. I imagine a scientist with a Nobel Prize and world renown is comfortable with something like this. And not much can go terribly wrong with a computing cluster. It may have bugs I guess? Or the whole building could catch fire? Not really reasons to weigh too heavily.

Re: Berkeley Lab Debuts Perlmutter, World’s Fastest AI Supercomputer

#37

Post by crawdog »

Doing back of the napkin math you could accomplish this with 770 cloud instances in Google Cloud all running 8 attached GPUs. Now running lustre in the cloud is a pita so you would have to find out how to take advantage of cloud native storage. Running Rapids should be possible using Dataproc. Anyone have the budget to kick off the largest cloud native HPC workload?

Re: Berkeley Lab Debuts Perlmutter, World’s Fastest AI Supercomputer

#38

Post by stainforth »

ArtWomb wrote: post #17
mooneater wrote: post #6

Not just for perl! > namesake of Saul Perlmutter, an astrophysicist at Berkeley Lab who shared the 2011 Nobel Prize in Physics for his contributions to research showing that the expansion of the universe is accelerating

Aptly named, as the first simulations will be detecting dark matter. What's incredible is that optical simulation in high energy physics can be reduced to ray tracing in computer graphics. Photon generation and propagation directly mapping to NVidia's OptiX framework ;) Opticks : GPU Optical Photon Simulation for Particle Physics using NVIDIAR OptiX https://iopscience.iop.org/article/10.1088/1742-6596/898/4/0... Proj…

Are there hard objectives that get reached or achieved with the simulations or whatever's being run? Or is it some amount of data that may or may not show something significant but up to interpretation?

Re: Berkeley Lab Debuts Perlmutter, World’s Fastest AI Supercomputer

#39

Post by boulos »

Disclosure: I used to work on Google Cloud.

Perlmutter seems like an awesome system. But, I think the “ai exaflops” is a “X GPUS times the NVIDIA peak rate”. The new sparsity features on A100 are promising, but haven’t been demonstrated to be nearly as awesome in practice (yet).

It also all comes down to workloads: large-scale distributed training is a funny workload! It’s not like LINPACK. If you make your model compute intensive enough, then the networking need mostly becomes bandwidth (for which multi-hundred Gbps worth of NICs is handy) but even without it there are lots of ways to max out your compute.

Similarly, storage is a serious need for say giant video corpora, but not for things like text! GPT-2 had like a 40 GiB corpus.

For those asking about largest-scale cloud GPU runs, there are basically three examples (chronological)

- the work OpenAI did on Five (thousands of V100s on GCP)

- the IceCube science work [1] on many clouds (51000 GPUs at peak!)

- OpenAI’s 10000-V100 cluster they used for GPT-3

The A100s used here are recently released and another step-change in perf per part (and memory). All major providers now offer them, though with different density and networking configurations (GCP went with 16 in a single box, most folks went with 8, some have lots of networking, etc.).

What everyone should be asking is: what awesome stuff is NERSC / LBL going to do with Perlmutter? You can’t just rent one for a few hours on GCP or any other provider :). (But, fwiw, most usage will be small slices: this is the sad fate of giant supercomputers!)

[1] https://insidehpc.com/2019/11/sdsc-conducts-50000-gpu-cloudb...

Re: Berkeley Lab Debuts Perlmutter, World’s Fastest AI Supercomputer

#40

Post by ganzuul »

jagger27 wrote: post #33
antonzabirko wrote: post #27

Doesnt Cerebras chip just out perform this?

I don’t think they are very comparable. This is still very much a conventional supercomputer despite the marketing. Here’s a fun comparison of total silicon wafer space used. A Cerebras die is 46,255 mm^2. 1,500 Milan 64-core CPUs * 8 compute chiplets each is 1,004,832 mm^2. (Not including the I/O chiplet). 6,159 NVidia A100 dies is 5,087,334 mm^2. These are all made on TSMC 7nm, funnily enough.

AI needs bandwidth. Those 1500 chips might as well be orbiting the Earth.
Post reply on HN