Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nvidia CEO Introduces Nvidia Ampere Architecture, Nvidia A100 GPU

blogs.nvidia.com

Post reply on HN
Print view View original

111–120 of 347 posts

Re: Nvidia CEO Introduces Nvidia Ampere Architecture, Nvidia A100 GPU

#111

Post by codemk8 »

solidasparagus wrote: post #106
wegs wrote: post #31

Earlier quoted context omitted.

It's irrelevant to researchers. Research operates on rapid cycles: prototype, publish, move on. It does impact businesses. It doesn't prevent adoption for e.g. deep learning, but I haven't seen e.g. GPU-based databases reach broad adoption, or many other places where MIMD/SIMD would reduce costs or improve performance. Using classical hardware is clearly cheaper than the business risk and engineering time of relying…

> I haven't seen e.g. GPU-based databases reach broad adoption Because it's very questionable whether GPU-based databases are generally valuable. GPUs accelerate compute, not all of the other things that databases do and often GPUs are not cost effective. > I dabble in GPU, but never beyond prototypes, for those reasons And I bet if you dabbled a little further you still wouldn't use it because it isn't cost-effectiv…

Agree. The reason GPUs are not widely adopted in certain areas is not because of open sourced or not, but because it is not cost effective. GPUs are optimized for throughput, not latency.

Re: Nvidia CEO Introduces Nvidia Ampere Architecture, Nvidia A100 GPU

#112

Post by dadarepublic »

dijit wrote: post #18
drewg123 wrote: post #10

I'm confused. Is there any relationship between the recent Ampere Arm64 servers ( https://news.ycombinator.com/item?id=22475036 ) and Nvidia's "Ampere Architecture", or is it just a case of them using the same name?

I don't like that people downvoted you for asking a question. If someone thinks the question is stupid or not doesn't mean that a downvote is warranted. (nor an upvote, answer the question and move on.) To answer though; it's just a coincidence, as you might already know Nvidia uses famous scientists (especially in the field of electricity) as the names of their microarchitectures. * Volta (Alessandro Volta, inventor…

Thank you for calling out the downvote issue. I've refrained from asking questions for this exact reason.

People should be encouraged to ask questions, even if from a position of lesser knowledge of the matter at hand. Those that do answer are probably not only helping the person asking the question, but those who may not ask the question even if it is one in their mind.

How many others refrain from enriching the dialogue for the same reasons?

Re: Nvidia CEO Introduces Nvidia Ampere Architecture, Nvidia A100 GPU

#113

Post by wegs »

solidasparagus wrote: post #106
wegs wrote: post #31

Earlier quoted context omitted.

It's irrelevant to researchers. Research operates on rapid cycles: prototype, publish, move on. It does impact businesses. It doesn't prevent adoption for e.g. deep learning, but I haven't seen e.g. GPU-based databases reach broad adoption, or many other places where MIMD/SIMD would reduce costs or improve performance. Using classical hardware is clearly cheaper than the business risk and engineering time of relying…

> I haven't seen e.g. GPU-based databases reach broad adoption Because it's very questionable whether GPU-based databases are generally valuable. GPUs accelerate compute, not all of the other things that databases do and often GPUs are not cost effective. > I dabble in GPU, but never beyond prototypes, for those reasons And I bet if you dabbled a little further you still wouldn't use it because it isn't cost-effectiv…

You're making a lot of lousy assumptions. As a few points of reference:

* My list comprehensions run over gigabytes of data (but sometimes 3 orders of magnitude bigger or smaller). Stream processing of big data. It's not deep learning, but it's slow and potentially deeply parallel. It would move to MIMD trivially, and SIMD with just a little bit of work.

* There are programming languages which support models almost exactly like this for data processing. Sun Labs Fortress comes to mind as an early example. This would generalize to a lot of contexts -- much smaller than you're giving credit for.

* Most of the issues, like startup times, are implementation-specific, rather than fundamental, and could be mitigated for much smaller data too. You do need to wrap your head around changes in programming paradigms to make that work. There is some overhead for latency (you'll probably never do well moving a list with 10 items to a GPU), but most of those aren't where programs are performance-bound.

* Many database operations map very well to MIMD.

* You're making deep assumptions that you're talking to an idiot. That doesn't make you look smart or right, or lead to a constructive discussion.

Re: Nvidia CEO Introduces Nvidia Ampere Architecture, Nvidia A100 GPU

#114

Post by jabl »

barbegal wrote: post #30
krapht wrote: post #13

Earlier quoted context omitted.

Physics simulations. There's a rule of thumb that to get an n-bit accurate result after a long chain of calculations, intermediate results should be stored with 2n bits. Often using the full dynamic range of a float is necessary because the magnitude of different physical phenomena varies so wildly. I guess people do store intermediate results in floats in order to take advantage of GPU acceleration. However, once yo…

I've always argued that if you are getting close to having to worry about underflow, overflow etc. then you have an ill-conditioned problem and just increasing the size of your intermediate results won't help you a huge amount because you need more precision from your inputs. There are very few fields where you need more than the 7 decimal digits afforded by floats. Maybe the only exceptions are in astrophysics.

Double precision has been the mainstay of scientific computing for decades, and no, it's not because all those scientists are dumb.

Re: Nvidia CEO Introduces Nvidia Ampere Architecture, Nvidia A100 GPU

#115

Post by riffraff »

dijit wrote: post #18
drewg123 wrote: post #10

I'm confused. Is there any relationship between the recent Ampere Arm64 servers ( https://news.ycombinator.com/item?id=22475036 ) and Nvidia's "Ampere Architecture", or is it just a case of them using the same name?

I don't like that people downvoted you for asking a question. If someone thinks the question is stupid or not doesn't mean that a downvote is warranted. (nor an upvote, answer the question and move on.) To answer though; it's just a coincidence, as you might already know Nvidia uses famous scientists (especially in the field of electricity) as the names of their microarchitectures. * Volta (Alessandro Volta, inventor…

Pascal also invented a few mechanical calculators[0] which might considered in a way precursor to modern computers.

[0] https://en.wikipedia.org/wiki/Pascal%27s_calculator

Re: Nvidia CEO Introduces Nvidia Ampere Architecture, Nvidia A100 GPU

#116

Post by slaymaker1907 »

ethbro wrote: post #47
wegs wrote: post #31

Earlier quoted context omitted.

It's irrelevant to researchers. Research operates on rapid cycles: prototype, publish, move on. It does impact businesses. It doesn't prevent adoption for e.g. deep learning, but I haven't seen e.g. GPU-based databases reach broad adoption, or many other places where MIMD/SIMD would reduce costs or improve performance. Using classical hardware is clearly cheaper than the business risk and engineering time of relying…

Hypothetically, from an ISA perspective, why couldn't Intel and AMD extend x86-64 more fully with SIMD / MIMD instructions? (as in, way more fully than MMX / SSE / AVX) Naive question, because I literally don't know the link between CPU instruction stream and GPGPU instruction stream. But it seems like there would be an opportunity to seize the higher (open) ground at the ISA level, and then force Nvidia to implement…

GPUs have gone far beyond just SIMD these days. To effectively program a GPU, you need to program it like a GPU, not a CPU. In particular, while most people are aware that GPUs don't like branching at a high level, branching can actually be fine as long as each block (small group of processors in the GPU) take the same branch. Block 1 taking the branch while block 2 not taking the branch will have little impact on performance. Additionally, the memory hierarchy is completely different for GPUs with blocks sharing cache and a huge number of registers per core while having very little memory for a typical stack.

Sure, treating a GPU as a SIMD blackbox may work for many problems as a suitable abstraction, but in doing so you also overlook many of its capabilities. x86-64 can emulate many of the SIMD aspects without too much trouble, but the aspects like huge number of processors with many registers is not something that is able to be reproduced without a large number of tradeoffs.

The only way that I see it as being possible to have a true CPU/GPU hybrid that is effective would be to basically have two separate chips for the GPU and the CPU, maybe multiple chips. I think the reason why such a product has not really taken off is because at that point there really isn't a point over using it versus a separate GPU and CPU. Maybe if hardware designers figured out how to greatly improve CPU to GPU communication in such a setup over having the motherboard in-between it might be worth it. CPU to GPU communication is a bottleneck for many applications.

Re: Nvidia CEO Introduces Nvidia Ampere Architecture, Nvidia A100 GPU

#117

Post by jzer0cool »

Nvidia is one of the major GPU suppliers (as well as AMD and others) as there are more compute needs in autonomous vehicles, robotics, etc. For future demand, will there be more increases in demand for CPU's (or decrease here in favor of GPU's), GPU's or some development other favorable processing unit?

Google had TPU's mentioned a few years back as one example. So curious to know what market segments would likely increase, which would require such processors to fill the increased demand for those units.

Re: Nvidia CEO Introduces Nvidia Ampere Architecture, Nvidia A100 GPU

#118

Post by fluffything »

macksd wrote: post #71

The numbers for their SATURNV supercomputer are either untrue or absolutely staggering. 4.6 exaflops? #1 on the Top 500 list of supercomputers just barely passed 200 petaflops at peak performance. If you add up the entire list you only get 1.65 exaflops. And LINPACK isn't usually network-bound. How can this possibly be true?

Yeah, next generation national super-computers are "pre-exascale" supercomputers. Theoretical 4.6 exaflops today is an unbelievable jump. It would be interesting to know what's the HPL perf that it can sustain.

Re: Nvidia CEO Introduces Nvidia Ampere Architecture, Nvidia A100 GPU

#120

Post by wegs »

aseipp wrote: post #55
wegs wrote: post #31

Earlier quoted context omitted.

It's irrelevant to researchers. Research operates on rapid cycles: prototype, publish, move on. It does impact businesses. It doesn't prevent adoption for e.g. deep learning, but I haven't seen e.g. GPU-based databases reach broad adoption, or many other places where MIMD/SIMD would reduce costs or improve performance. Using classical hardware is clearly cheaper than the business risk and engineering time of relying…

GPU based databases haven't reached broad adoption because sending things over the PCIe link is a huge waste of time if you can avoid it. Working around this with custom design like NVLink/NVSwitch do is ridiculously expensive (and why a DGX costs a gajillion dollars), and there is simply not enough volume to subsidize it. They are largely analytics focused, because the parallel hardware can obviously map onto primit…

I don't underestimate the complexity. But I do claim that the complexity can and should be hidden behind programming language constructs. I've worked both on the design of MIMD hardware, back when I was a graduate student, and on programming languages. These aren't easy problems, but they are solvable.

The reason for openness isn't abstract. I don't think NVidia will solve these problems alone. NVidia can make really good tools for a few specific domains, but generalizing to how we apply this to JavaScript, databases, or Python interpreters requires an open community approach. It requires a lot of people experimenting and dabbling.

It's kind of like Nokia and friends thinking they could solve the problem of building phone apps alone. When Apple launched the iPhone, and there was a community pushing things forward, we were in a whole new world of progress.

I would argue NVidia underestimates both the potential and the complexity if they think they can go it (relatively) alone, come up with the right programming constructs, and provide the right set of tools for programmers to consume.

Post reply on HN