Tesla Dojo Custom AI Supercomputer at HC34

Dude just straight up lies. He may not realize he's lying, but he lies constantly.

Edit, because I'm getting downvoted. Here's some examples: self driving, battery swaps, robotic snake chargers, cybertruck windows, Bitcoin won't be converted to fiat, starlink speeds will improve, he will sell his home, first mars mission 2024, Tesla solar roofs, power packs at every supercharger, brake pads on Tesla cars will never need to be replaced, the gigafactory will be 100% renewable powered by 2020, fixing the Flint water crisis, making bricks from boring company waste, founding a media credibility organization, and probably a TON more.

Oh man, I forgot he took money to fly tourists to the moon.

There are so many lies he's told. So many.

I think they said that during the first AI Day. Here's a 19 minute supercut of AI Day: https://www.youtube.com/watch?v=keWEE9FwS9o

An ASIC can strip out features they don't need and save some space. But a good chunk of modern GPUs are memory-controllers, registers, and SIMD-cores. And modern GPUs (both AMD's MI250x and NVidia's A100) have 16-bit matrix multiplication units (aka: Tensor cores). Once we factor in the process disadvantage, I'm not sure if the D1 will be as competitive as they hope.

Tesla's hope is that their D1 chip has more 16-bit matrix multiplication cores than the NVidia / AMD designs. But A100 is quite solid, and NVidia Hopper has been announced at HC34 (aka: NVidia's next generation).

https://www.nvidia.com/en-us/technologies/hopper-architectur...

-------

Most of this presentation on the Tesla Dojo is about the interconnect system. Alas, NVidia's on like the 4th (or was it 5th?) generation of NVlink, available from their DGX servers (and I'm sure a Hopper version will come out soon).

AMD's not far behind, also with a lot of good presentations this year from HC34 that points out how AMD's "Frontier" Supercomputer has huge bandwidth. In particular, each MI250X GPU is a twin-chiplet design (two GPUs per... GPU), with 5 high-speed links to connect to other GPUs in a high-speed fashion. There's a reason why Frontier is the #1 supercomputer of the world right now, in both absolute Double-precision FLOPs, and in Green Double-precision FLOPS-per-watt.

NVidia's Hopper will be hitting 4nm. AMD's MI250x is 5nm. That means the D1 chip has less than 1/2 the transistors at the same area compared to NVidia.

> but I’d imagine ASIC based supercomputers (like Google TPU) are the way to go forward.

Only if you keep up with the process shrinks. 7nm is getting long in the tooth now. All eyes are looking forward to 5nm, 4nm, and even 3nm designs (now that Apple is the customer of TSMC's 3nm node).

-----------

That being said, if the 7nm node is cheaper, maybe this exercise was still cost-effective for Tesla. As the newer nodes obsolete the older nodes, the older nodes become more cost-effective.

Cost-efficiency is less popular / less cool, but still an effective business plan.

> but I’d imagine ASIC based supercomputers (like Google TPU) are the way to go forward.

The issue is that it probably costs hundreds-of-millions of dollars to design something like the D1. Sure, the mass production of the chip afterwards will be incredible, but chips have stupidly high startup costs (masking, engineering, research, etc. etc.)

GPUs on the other hand, are more general purpose and are applicable to more situations. So you can sell the GPU to more customers and spread out the R&D costs. In particular, GPUs capture the attention of the video game crowd, who will fund high-end GPU research just to play video games.

Much like how Intel's laptops allow servers to share the R&D effort, so too does NVidia's consumer GPUs share research/development costs with their high end A100 cards.