Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nvidia will build 700-petaflop supercomputer for University of Florida

venturebeat.com

Post reply on HN
Print view View original

81–90 of 112 posts

Re: Nvidia will build 700-petaflop supercomputer for University of Florida

#81

Post by QuotedForTruth »

tgb wrote: post #68
Blammar wrote: post #9

Earlier quoted context omitted.

It's $50 million. That's all. See https://blogs.nvidia.com/blog/2020/07/21/university-of-flori... for the numbers. It's probable the hardware is being offered at a deep discount. The building and infrastructure will cost an additional $20m, covered by the University.

It's "anchored" by $50 million (of which half is support). But what does "anchored" mean here? The article also lists an additional $20 million from UF. I wouldn't be surprised if there was even more.

Its a $70M effort "anchored" by $25M cash donation and $25M in hardware. I've heard anchored used to indicate an initial "getting us started" donation at the beginning of a fund raising drive. Its easier to raise the final portion once you've got momentum and people think the project is likely to be successfully funded.

Re: Nvidia will build 700-petaflop supercomputer for University of Florida

#82

Post by QuotedForTruth »

fma wrote: post #70
xvilka wrote: post #67

Earlier quoted context omitted.

Precisely explains why NVIDIA did that - to hook up all these 50k students to CUDA. Much more profitable in the long run.

All 50k Students aren't engineers...

As a UF CS grad, I bet less than 100 students will get hands on this computer each year. Maybe way less than 100.

Although I wasn't a graduate student so maybe way more of them have access than I realize.

Re: Nvidia will build 700-petaflop supercomputer for University of Florida

#83

Post by easton »

QuotedForTruth wrote: post #82
fma wrote: post #70

Earlier quoted context omitted.

All 50k Students aren't engineers...

As a UF CS grad, I bet less than 100 students will get hands on this computer each year. Maybe way less than 100. Although I wasn't a graduate student so maybe way more of them have access than I realize.

Do you know how the CS program is doing these days? I heard it was on the ropes a few years ago but this looks like they are investing in it again?

Re: Nvidia will build 700-petaflop supercomputer for University of Florida

#84

Post by rrss »

boulos wrote: post #13

Sigh, I wish people wouldn’t say “peta flop ” for these. https://www.nvidia.com/en-us/data-center/a100/ is the most official reference. If you scroll to the bottom, you’ll see that an A100 part can do ~20 Teraflops (either FP32 or FP64 in little-matrix aka tensor mode). When they say “each A100 can do 5 petaflops”, they mean each DGX which has 8 such cards and thus they mean 600-ish something-ops per card. The genero…

You're the "disclaimer: I work for google cloud" person, right? Maybe you can help solve this problem (and it's a real one, it massively sucks that everyone counts whatever "flops" they want).

Start here: https://cloud.google.com/tpu

> Cloud TPU v3 Pod

> 100+ petaflops

Is the Cloud TPU v3 even functionally capable of computing in fp64 at any performance?

Re: Nvidia will build 700-petaflop supercomputer for University of Florida

#85

Post by rrss »

Blammar wrote: post #9
fluffything wrote: post #2

I'm a bit baffled. How high are the tuitions at the university of florida? This supercomputer is more powerful than many at the national labs and must cost a fortune (multiple 100 millions of dollars).

It's $50 million. That's all. See https://blogs.nvidia.com/blog/2020/07/21/university-of-flori... for the numbers. It's probable the hardware is being offered at a deep discount. The building and infrastructure will cost an additional $20m, covered by the University.

I dunno if there's a deep discount.

This is 140 DGX A100 systems. The advertised price for each DGX A100 is $199,000, so 140 of them is ~$28 million. It's plausible the networking, storage, cooling, power distribution, etc (i.e. everything but the compute nodes) could cost $22 million.

Re: Nvidia will build 700-petaflop supercomputer for University of Florida

#86

Post by rrss »

xvilka wrote: post #67
auslegung wrote: post #40

Earlier quoted context omitted.

In addition to what everyone else has said, UF has over 50,000 students, making it one of the five largest universities in the USA, iirc.

Precisely explains why NVIDIA did that - to hook up all these 50k students to CUDA. Much more profitable in the long run.

There's a lot of ways to program these GPU supercomputers, and you are maybe overestimating how much of the HPC code is written in CUDA. Some is, for sure, but there's also a lot using OpenACC or OpenMP (C, C++, or Fortran with annotations), or frameworks like Kokkos.

Recently, it also became possible to use C++17 stdpar and get GPU acceleration (https://docs.nvidia.com/hpc-sdk/compilers/c++-parallel-algor...).

Re: Nvidia will build 700-petaflop supercomputer for University of Florida

#87

Post by rrss »

sillysaurusx wrote: post #44
Cthulhu_ wrote: post #33

Earlier quoted context omitted.

Risk mitigation, maintenance, resell value (?), support, reliability. The custom AI accelerators you mentioned, how long have those been in business for? How many units have they moved? How many generations of hardware have they produced? Will they still be around to replace or upgrade units in 5-10 years? How flexible are they in their workload? That's a lot of factors to keep in mind when you're spending millions o…

TPUs are perfectly positioned to capture this market. There are an endless number of reasons why, but to keep it short: Wanna see a magic trick? https://twitter.com/theshawwn/status/1286426454171975680 GPT-2 117M training at 1 million tokens/sec. Now, I don't have experience with DGX clusters, so I'm not going to make a firm statement. What I will say is that I, as an outsider, managed to achieve a performance level…

If "this market" means supercomputers (as 'Cthulhu_ was talking about), then TPUs are certainly not perfectly positioned to capture this market, since they are only capable of doing bfloat16 arithmetic with high performance, and AFAIK incapable of fp64 at all. That means a HPL score of zero.

TPUs will probably dominate AI training, but not so much supercomputing.

Re: Nvidia will build 700-petaflop supercomputer for University of Florida

#88

Post by fma »

QuotedForTruth wrote: post #82
fma wrote: post #70

Earlier quoted context omitted.

All 50k Students aren't engineers...

As a UF CS grad, I bet less than 100 students will get hands on this computer each year. Maybe way less than 100. Although I wasn't a graduate student so maybe way more of them have access than I realize.

According to the Hipergator Website: https://www.rc.ufl.edu/access/account-request/

To be eligible for an account on HiPerGator, you must be on an education or research project that has as one of its Principal or Co-Principal investigators a faculty or staff member at the University of Florida.

I'm also a UF alumni (undergrad & grad EE), and IIRC back in my day high performance computing was a grad class. I never took it, but it wouldn't surprise me if for the grad class there would be labs on the super computer where students can get limited access.

Re: Nvidia will build 700-petaflop supercomputer for University of Florida

#90

Post by rubatuga »

lorenzhs wrote: post #47
abainbridge wrote: post #41

Earlier quoted context omitted.

Seems like it has about 700/5 = 140 GPUs. And they are about 0.4kW each. Let's say it is 50% utilized for a year. That is 140 * 0.4 * 0.5 * 365 * 24 = 245280 kWh. Let's say electricity generation creates 0.5 kg CO2e per kWh. So 122640 kg CO2e per year. For comparison, driving a cars creates about 0.2kg CO2 per km. Or 0.32 kg per mile. So about the same as driving 122640 / 0.32 = 383250 miles per year. Call it 38 cars…

HPC system utilization is typically upwards of 90%, these things are not idle. There's almost always a queue of jobs waiting to run, at least in my experience.

Definitely, also it makes you hate those who hog up the resources. A good scheduler will take into account the past usage however.
Post reply on HN