Nvidia CEO Introduces Nvidia Ampere Architecture, Nvidia A100 GPU

I know this isn't a traditional supercomputer and these aren't LINPACK benchmarks, but I am still blown away by the 5 petaflops figure, considering where we were just 10 or 15 years ago.

Yup- they're cheaper than renting from AWS.

I'm not making a claim about the necessity of experimentation. I spent years (and working a paid job) doing programming language work, and also design hardware these days in my spare time, so I'm not against that. I'm specifically addressing the claim that "GPU databases haven't taken off because of lack of open source CUDA" or whatnot. Database tech is one of the most R&D heavy engineering subfields, almost all major innovations come from it. The points I made up are not coming from thin air, they're the result of people (engineers) doing a lot of experimentation and coming to similar conclusions for many years. You don't need open source designs to prove this, by the way, you only need to do basic napkin math about the characteristics of the system, and how data moves around, to come to similar conclusions. You need a correct (and I hate this word) synergy of hardware, software, and programming model to do it. A programming language, a new model, does not change the theoretical bandwidth of PCIe 3.0, or the fact you have a memory hierarchy to optimize for best performance. Just having one and none of the others, or having lopsided characteristics, isn't sufficient, and innovations across the stack are one of the major things people are reaching for, in order to differentiate themselves.

That said, I agree and would love to see less crappy programming models here. As a PL geek, I have numerous reasons why I think that's necessary. It really needs to be easier to compose sets of small languages, and design them -- one for designing streaming systems, one for latency sensitive ones. They need to model the memory hierarchy available to us (a huge thing most do not do, and vital to system performance.) I'd love this. But it doesn't undermine anything I said earlier about why things are the way they are, today. No amount of fancy programming languages is going to change the fact a $10,000 Supermicro server is more cost effective than $70,000 worth of V100s for 90% of OLAP workloads you'd want a database for. Engineers design accordingly.

There is also the problem of needing huge amounts of capital, where most of this work can only be done by exceedingly well funded groups with deep ties to hardware divisions in question. The future of hardware innovation comes from billion dollar companies, because only they can sustain it, not plucky engineers. Sure, for us, CUDA being open source would be awesome. But you don't really need open source drivers when you're working directly with the vendor on your requirements and you pay them millions for support and you just use Linux for everything. You just let them solve it and move on. The engineering world is designed this way (both by engineers, and by capitalists), because it is how we make money from it in a capitalist society!

> I would argue NVidia underestimates both the potential and the complexity if they think they can go it (relatively) alone, come up with the right programming constructs, and provide the right set of tools for programmers to consume.

Nope. Nvidia understands that they alone may not hit a global optimum or whatever in all these fields. I suspect given that they have entire divisions of highly skilled engineers dedicated to programming tools -- they understand it better than either of us. But what they also understand is that their software stack is a differentiator for them, because it actually works (the competitors don't) and it makes them money to keep it that way. You're confusing a technical problem with one of politics and vision -- a categorical mistake about their priorities and where they lie. I don't want to sound crass, but people saying "I would argue that I, the sole, lone gun engineer, understand their business and future and everything way better than they do" is typical of engineers, and it is almost always a categorical mistake to think so.

Nvidia fully understands that maybe some nebulous benefit might come to them by open sourcing things, maybe years down the line. They understand plucky researchers can do amazing things, sometimes. But they understand much better that keeping it closed makes them money and keeps them distinct from their competitors in the short term. If you think this is a contradiction, or seemingly short sighted: don't worry, because you are correct, it is. What is more "surprising" is recognizing that all of capitalist society is built on these sorts of contradictions. I'm afraid we're all going to have to get used to waiting for FOSS nvidia drivers/CUDA.

EDIT: I'll also say that if this changes from their "major open source announcement" they were going to do at GTC, I'll eat my hat. I'm not expecting much from Nvidia in terms of open source, but I'd happily be proven wrong. But broadly I think my general point stands, which is that thinking about it from the POV of "open source drivers are the limitation" isn't really the right way to think about it.

People who are into RISC-V and other side projects/open stacks obviously have not worked on mission critical problems.

When you have a Jet engine hoisted up for a test rig, and something fails in your DSP library, you don't hesitate to call Matlab engineering support to help on within next 30 mins. Try that with some python library. People give a lot of flak to Matlab for being closed source but there is a reason they exist. Not for building a stupid toy project, but for real things where big $$$ is on the line. Python is also used in production everywhere, but if your application is a niche one and using PyVISA library to connect to some DSP hardware that you git cloned is not very "production" ready. You need solid deps.

Don't get me wrong - open source software runs in prod all the time - PostgreSQL/Linux, etc. The smaller the application domain (specific DSP libraries or analysis stacks for wind turbines and such), the lower the availability of high quality open source software (and support).

My point is that reality hits you hard when it is anything where a lot of $$$ or people's time depend on it. Don't blame their engineers for using closed source tools.

In general computing, FPGAs will likely see a rise in popularity (latest mac pro offers an FPGA card as an option). In task specific computing, GPUs and ASICs will likely lead.

Tesla replaced their Nvidia GPUs over a year ago with a custom chip (I believe they hired a chip designer away from Apple to develop their own silicon).