I'm a bit baffled. How high are the tuitions at the university of florida? This supercomputer is more powerful than many at the national labs and must cost a fortune (multiple 100 millions of dollars).
[deleted]
Nvidia will build 700-petaflop supercomputer for University of Florida
11–20 of 112 posts
Re: Nvidia will build 700-petaflop supercomputer for University of Florida
#12I'm a bit baffled. How high are the tuitions at the university of florida? This supercomputer is more powerful than many at the national labs and must cost a fortune (multiple 100 millions of dollars).
No, those national lab supercomputers typically quote double-precision flops, whereas this appears to be about float16 tensor ops for inner product (they call it "AI flops").
The A100 is 9.7 general purpose Tflops of float64, versus 624 Tflops of float16 / bfloat16 (perhaps also just tensor cores not general purpose)?
So quoting it the same way, this Florida machine would only be a (700 * (9.7 / 624)) = 10.9 Pflop supercomputer, not counting host CPUs.
Granted though, the hot new HPC approach is to approximate certain expensive high-precision calculations with ML inference instead, but that only works for certain algorithms.
Re: Nvidia will build 700-petaflop supercomputer for University of Florida
#13Sigh, I wish people wouldn’t say “petaflop” for these.
https://www.nvidia.com/en-us/data-center/a100/ is the most official reference. If you scroll to the bottom, you’ll see that an A100 part can do ~20 Teraflops (either FP32 or FP64 in little-matrix aka tensor mode). When they say “each A100 can do 5 petaflops”, they mean each DGX which has 8 such cards and thus they mean 600-ish something-ops per card. The generous assumption is that’s FP16 or bfloat16 for sparse ops, and therefore they are “flops”.
The reality is that if someone says “supercomputer” in the general sense, they mean scientific computing and so mean a double-precision LINPACK benchmark. The 1120 A100 parts (8x140) doing 20 “real” teraflops each has an absolute peak of about 22 Petaflops (and older code without tensor mode would be half that on FP64).
tl;dr: ML isn’t scientific computing, and those are different flops, but “10 petaflops” just doesn’t sound as impressive.
Re: Nvidia will build 700-petaflop supercomputer for University of Florida
#14I'm a bit baffled. How high are the tuitions at the university of florida? This supercomputer is more powerful than many at the national labs and must cost a fortune (multiple 100 millions of dollars).
No, those national lab supercomputers typically quote double-precision flops, whereas this appears to be about float16 tensor ops for inner product (they call it "AI flops"). The A100 is 9.7 general purpose Tflops of float64, versus 624 Tflops of float16 / bfloat16 (perhaps also just tensor cores not general purpose)? So quoting it the same way, this Florida machine would only be a (700 * (9.7 / 624)) = 10.9 Pflop su…
Jinx :). I really am surprised the marketing folks didn’t claim this was an ~3 exa-op supercomputer based on int4 w/ sparsity.
Re: Nvidia will build 700-petaflop supercomputer for University of Florida
#15It seems like supercomputers are still generally many times more powerful than what AI researchers at top universities or orgs are using. Has any well known AI research been done on supercomputers? It seems like the case that literally just throwing money at the problem is a solid idea nowadays.
GPT-3 was another buckets worth of evidence in favor of the scaling hypothesis. Performance kept improving (and cost to train kept increasing) as more parameters were added. Even with 175 billion parameters, the performance had not yet plateaued. One take-away is that throwing a lot of compute at the problem helps tremendously :).
You can read more about GPT-3 here: https://lambdalabs.com/blog/gpt-3/
Re: Nvidia will build 700-petaflop supercomputer for University of Florida
#16I'm a bit baffled. How high are the tuitions at the university of florida? This supercomputer is more powerful than many at the national labs and must cost a fortune (multiple 100 millions of dollars).
The article mentions that the money/hardware was donated by Nvidia co-founders.
One of the founders, Chris Malachowsky, is an alum:
Re: Nvidia will build 700-petaflop supercomputer for University of Florida
#17I'm a bit baffled. How high are the tuitions at the university of florida? This supercomputer is more powerful than many at the national labs and must cost a fortune (multiple 100 millions of dollars).
No, those national lab supercomputers typically quote double-precision flops, whereas this appears to be about float16 tensor ops for inner product (they call it "AI flops"). The A100 is 9.7 general purpose Tflops of float64, versus 624 Tflops of float16 / bfloat16 (perhaps also just tensor cores not general purpose)? So quoting it the same way, this Florida machine would only be a (700 * (9.7 / 624)) = 10.9 Pflop su…
The FLOPs quoted by national lab supercomputers are usually measured double-precision FLOPs from HPL (High-performance Linpack - at least on the Top500 reports).
IIUC, HPL can use double-precision tensor operations. It might also be able to use mixed-precision for some computations.
Are the performance numbers for this HPC system at University of Florida also from HPL ?
Re: Nvidia will build 700-petaflop supercomputer for University of Florida
#18Earlier quoted context omitted.
No, those national lab supercomputers typically quote double-precision flops, whereas this appears to be about float16 tensor ops for inner product (they call it "AI flops"). The A100 is 9.7 general purpose Tflops of float64, versus 624 Tflops of float16 / bfloat16 (perhaps also just tensor cores not general purpose)? So quoting it the same way, this Florida machine would only be a (700 * (9.7 / 624)) = 10.9 Pflop su…
The FLOPs quoted by national lab supercomputers are usually measured double-precision FLOPs from HPL (High-performance Linpack - at least on the Top500 reports). IIUC, HPL can use double-precision tensor operations. It might also be able to use mixed-precision for some computations. Are the performance numbers for this HPC system at University of Florida also from HPL ?
> IIUC, HPL can use double-precision tensor operations.
Lets put it this way: As of the A100 GPU generation, the tensor cores are able to do double precision calculations too. By using the tensor core the peak FP64 flops double to 19.5 Tflops. And yes, the CUDA BLAS library (that the HPL benchmark uses) uses the tensor cores on the A100.
> It might also be able to use mixed-precision for some computations.
There is currently a lot of interest in mixed precision algorithms, but I'm not sure that the algorithm used for HPL is amenable to that approach, nor whether it's allowed per the benchmark rules.
Re: Nvidia will build 700-petaflop supercomputer for University of Florida
#19I'm a bit baffled. How high are the tuitions at the university of florida? This supercomputer is more powerful than many at the national labs and must cost a fortune (multiple 100 millions of dollars).
[deleted]
Re: Nvidia will build 700-petaflop supercomputer for University of Florida
#20I'm a bit baffled. How high are the tuitions at the university of florida? This supercomputer is more powerful than many at the national labs and must cost a fortune (multiple 100 millions of dollars).
> How high are the tuitions at the university of florida?
In-state tuition at the University of Florida is inexpensive compared to most major US universities and is generally considered one of the better values among universities. ~$6,400 per year in-state. Florida State is also comparable in cost.