Made where? Isn’t foundry capacity the limiting factor on chips for AI right now?
TSMC manufactures the MAIA 100:
https://azure.microsoft.com/en-us/blog/azure-maia-for-the-er...
Microsoft CTO says he wants to swap most AMD and Nvidia GPUs for homemade chips
81–90 of 137 posts
Re: Microsoft CTO says he wants to swap most AMD and Nvidia GPUs for homemade chips
#82The most important note is: > The software titan is rather late to the custom silicon party. While Amazon and Google have been building custom CPUs and AI accelerators for years, Microsoft only revealed its Maia AI accelerators in late 2023. They are too late for now, they realistically hardware takes a couple generations to become a serious contender and by the time Microsoft has a chance to learn from their hardwar…
> hardware takes a couple generations to become a serious contender Not really and for the same reason Chinese players like Biren are leapfrogging - much of the workload profile in AI/ML is "embarrassingly parallel", thus reducing the need for individual ASICs to be bleeding edge performant. If you are able to negotiate competitive fabrication and energy supply deals, you can mass produce your way into providing "goo…
As I understood it the main bottleneck is interconnects, anyhow. It's more difficult to keep the ALUs fed than it is to make them fast enough, especially once your model can't fit in one die/PCB. And that's in principle a much trickier part of the design, so I don't really know how that shakes out (is there a good enough design that you can just buy as a block?)
Re: Microsoft CTO says he wants to swap most AMD and Nvidia GPUs for homemade chips
#83Earlier quoted context omitted.
Microsoft lacks the credibility and track record for this to be anything but talk. Hardware doesn’t simply go from zero to gigawatts of infrastructure on talk. Even Apple is better positioned for such a thing.
Microsoft has plenty of home grown hardware on Azure, some of which even has firmware written in Rust nowadays.
Yep, tons of decent homegrown stuff in Azure. And for a long time - see the Catapult FPGA cards.
Re: Microsoft CTO says he wants to swap most AMD and Nvidia GPUs for homemade chips
#84Earlier quoted context omitted.
I'm 99.999% sure that the claim of "all deep learning for Gemini on their own silicon" is not true. Maybe if you restrict it similarly to the Deepseek paper to "Gemini uses TPU for the final successful training run and for scaled inference" you might be correct, but there's no way that GPUs aren't involved for at minimum comparability and more rapid iteration reasons during the extremely buggy and error prone point o…
You seem to think GPUs are better than TPUs for rapid iteration. Why is that? There's no inherent reason why one is more suited to rapid iteration than another; it's entirely a matter of developer tooling and infrastructure. And Google famously has excellent tooling. And furthermore, the tooling Google exposes to the outside world is usually poorer than the tooling used internally by Googlers.
It was a few years ago since I last played with Googles hw, but iirc TPUs were inflexible and very fast. Worked well for linear and convolutional layers but could not accelerate certain LSTM configurations. For such networks GPUs were faster. It wouldn't surprise me the least if TPU hardware support lagged behind what the latest and greatest LLMs require for training.
Re: Microsoft CTO says he wants to swap most AMD and Nvidia GPUs for homemade chips
#85It always falls back on the software. AMD is behind, not because the hardware is bad, but because their software historically has played second fiddle to their hardware. The CUDA moat is real. So, unless they also solve that issue with their own hardware, then it will be like the TPU, which is limited to usage primarily at Google, or within very specific use cases. There are only so many super talented software engin…
The CUDA moat is real for general purpose computing and for researchers that want a swiss army knife, but when it comes to well known deployments, for either training or inference, the amount of stuff that you need from a chip is quite limited.
You do not need most of CUDA, or most of the GPU functionality, so dedicated chips make sense. It was great to see this theory put to the test in the original llama.cpp stack which showed just what you needed, the tiny llama.c that really shows how little was actually needed and more recently how a small team of engineers at Apple put together MLX.
Re: Microsoft CTO says he wants to swap most AMD and Nvidia GPUs for homemade chips
#86Earlier quoted context omitted.
Not sure what you mean. Google is a big company. Their TPUs have many users internally.
Very few developers outside of Google have ever written code for a TPU. In a similar way, far fewer have written code for AMD, compared to NVIDIA. If you're going to design a custom chip and deploy it in your data centers, you're also committing to hiring and training developers to build for it. That's a kind of moat, but with private chips. While you solve one problem (getting the compute you want), you create anoth…
I mean it's all true to some extent. But that doesn't mean implementing the few primitives to get transformers running requires CUDA, or that it's an impossible task. Remember, we're talking about >$1B companies here who can easily assemble teams of 10s-100s of developers.
You can compare CUDA to the first PC OS, DOS 1.0. Sure, DOS was viewed as a moat at the time, but it didn't keep others from kicking its ass.
Re: Microsoft CTO says he wants to swap most AMD and Nvidia GPUs for homemade chips
#87Earlier quoted context omitted.
> hardware takes a couple generations to become a serious contender Not really and for the same reason Chinese players like Biren are leapfrogging - much of the workload profile in AI/ML is "embarrassingly parallel", thus reducing the need for individual ASICs to be bleeding edge performant. If you are able to negotiate competitive fabrication and energy supply deals, you can mass produce your way into providing "goo…
As I understood it the main bottleneck is interconnects, anyhow. It's more difficult to keep the ALUs fed than it is to make them fast enough, especially once your model can't fit in one die/PCB. And that's in principle a much trickier part of the design, so I don't really know how that shakes out (is there a good enough design that you can just buy as a block?)
[deleted]
Re: Microsoft CTO says he wants to swap most AMD and Nvidia GPUs for homemade chips
#88It always falls back on the software. AMD is behind, not because the hardware is bad, but because their software historically has played second fiddle to their hardware. The CUDA moat is real. So, unless they also solve that issue with their own hardware, then it will be like the TPU, which is limited to usage primarily at Google, or within very specific use cases. There are only so many super talented software engin…
The CUDA moat is real for general purpose computing and for researchers that want a swiss army knife, but when it comes to well known deployments, for either training or inference, the amount of stuff that you need from a chip is quite limited. You do not need most of CUDA, or most of the GPU functionality, so dedicated chips make sense. It was great to see this theory put to the test in the original llama.cpp stack…
Absolutely agreed on the need for just specific parts of the chip and tailoring to that. My point is bigger than that. Even if you build a specific chip, you still need engineers who understand the full picture.
Re: Microsoft CTO says he wants to swap most AMD and Nvidia GPUs for homemade chips
#89Earlier quoted context omitted.
> hardware takes a couple generations to become a serious contender Not really and for the same reason Chinese players like Biren are leapfrogging - much of the workload profile in AI/ML is "embarrassingly parallel", thus reducing the need for individual ASICs to be bleeding edge performant. If you are able to negotiate competitive fabrication and energy supply deals, you can mass produce your way into providing "goo…
As I understood it the main bottleneck is interconnects, anyhow. It's more difficult to keep the ALUs fed than it is to make them fast enough, especially once your model can't fit in one die/PCB. And that's in principle a much trickier part of the design, so I don't really know how that shakes out (is there a good enough design that you can just buy as a block?)
The bet I'm seeing is to try and invest in custom ASICs to become integrated as part of an SoC to solve that interconnect bottleneck.
It's largely a solved problem based on Google/Broadcom's TPU work - almost everyone is working with Broadcom to design their own custom ASIC and SoC.
Re: Microsoft CTO says he wants to swap most AMD and Nvidia GPUs for homemade chips
#90On a slightly different tangent, is anyone working on analog machine learning ASICs? Sub-threshold CMOS or something? I mean even at the research level? Using a handful of transistor for an analog multiplier. And get all of the crazy fascinating translinear stuff of Barrie Gilbert fame. https://www.electronicdesign.com/technologies/analog/article... https://www.analog.com/en/resources/analog-dialogue/articles... http…
For large models, the bottlenecks are memory bandwidth, network, and power consumption by the DAC/ADC arrays It’s never come even close to penciling out in practice. For small models there are people working on this implemented in flash memory eg Mythic.
[dead]