Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The first two custom silicon chips designed by Microsoft for its cloud

theverge.com

Post reply on HN
Print view View original

91–100 of 282 posts

Re: The first two custom silicon chips designed by Microsoft for its cloud

#91

Post by scarface_74 »

kyboren wrote: post #63
andy_xor_andrew wrote: post #12

Damn, Microsoft is playing all sides of this, huh. Within a ten minute window: - Satya announced GPT-4 runs (at least partly) on a new AMD offering - Satya announced an in-house chip for ML acceleration - Satya brings NVidia CEO Jensen Huang on stage they've got every horse in the race, huh (disclaimer, I work for MS but all the stuff talked about here far is waaaay above my paygrade haha, and all brand new info to m…

> - Satya brings NVidia CEO Jensen Huang on stage This should be ringing alarm bells at FTC and DoJ. You know what's even better than trust busting and breaking up cartels? Preventing the formation of cartels and trusts in the first place.

Raising alarm bells because Microsoft is actually dealing with multiple companies?

Would it be better for competition if Microsoft only used one supplier?

Re: The first two custom silicon chips designed by Microsoft for its cloud

#92

Post by jedberg »

rapsey wrote: post #23
jedberg wrote: post #3

It was only a matter of time. Google announced theirs years ago, Amazon announced theirs last year. Right now NVIDIA has the lead because they have the better software, but they can't make the chips fast enough. Will be interesting to see if their better software continues to keep them in the lead or if people are more interested in getting the capacity in any form.

nVidia still has the monopoly on training. Everyone else is just making chips for inference.

Amazon announced Trainum last year and recently announced their partnership with Anthropic, where Claude will be trained on Trainium.

https://press.aboutamazon.com/2023/9/amazon-and-anthropic-an...

Re: The first two custom silicon chips designed by Microsoft for its cloud

#93

Post by sgillen »

adrian_b wrote: post #70
rapsey wrote: post #23

Earlier quoted context omitted.

nVidia still has the monopoly on training. Everyone else is just making chips for inference.

While NVIDIA provides the best absolute performance for training, Intel (i.e. Gaudi) already provides much better training performance per dollar. The funny thing is that this fact has been shown inadvertently by NVIDIA: https://www.servethehome.com/nvidia-shows-intel-gaudi2-is-4x...

That article shows that it takes about 50x as long to train gpt-3 with intel's offering vs Nvidia. At least in the current environment, if you are training llms I think almost no amount of cost savings can justify that.

Re: The first two custom silicon chips designed by Microsoft for its cloud

#94

Post by donatzsky »

scarface_74 wrote: post #84
jedberg wrote: post #3

It was only a matter of time. Google announced theirs years ago, Amazon announced theirs last year. Right now NVIDIA has the lead because they have the better software, but they can't make the chips fast enough. Will be interesting to see if their better software continues to keep them in the lead or if people are more interested in getting the capacity in any form.

If they “can’t make the chips fast enough” being TSMC’s second highest volume customer behind Apple and probably second in priority, what chance does Microsoft have getting enough of TSMCs capacity?

Microsoft does have the benefit of not having any customers other than themselves, so volumes are smaller.

Re: The first two custom silicon chips designed by Microsoft for its cloud

#96

Post by ShamelessC »

belval wrote: post #48
ShamelessC wrote: post #36

Earlier quoted context omitted.

Yikes those names are horrible.

Why? Inferentia => inference, trainium => training. Given the usually naming of AWS product, having one where the name roughly matches what it does is pretty good? TPU is pretty good but is associated with Google. MTIA is an acronym but still maps to what the chip does. ~~"Cobalt" is worse as it does not mean anything~~ . Cobalt is the CPU chip, MAIA is the accelerator so this matches Meta's naming.

> Why? Inferentia => inference, trainium => training.

Funny, that's precisely why I think the names are bad. It's like if Google had chosen "Search-ola" as their name. Way too on the nose and/or lazy. Having said that, I don't really care all that much and I imagine that may have been the spirit of those who chose the names.

Re: The first two custom silicon chips designed by Microsoft for its cloud

#97

Post by adrian_b »

sgillen wrote: post #93
adrian_b wrote: post #70

Earlier quoted context omitted.

While NVIDIA provides the best absolute performance for training, Intel (i.e. Gaudi) already provides much better training performance per dollar. The funny thing is that this fact has been shown inadvertently by NVIDIA: https://www.servethehome.com/nvidia-shows-intel-gaudi2-is-4x...

That article shows that it takes about 50x as long to train gpt-3 with intel's offering vs Nvidia. At least in the current environment, if you are training llms I think almost no amount of cost savings can justify that.

That 50X is only if you can afford one thousand NVIDIA H100.

There cannot be more than a handful of companies in the entire world that could afford such a huge price (tens of millions of $).

In comparison with a still extremely expensive cluster of 64 NVIDIA H100, the difference in speed would reduce to only two to three times, and paying several times less for the entire training becomes very attractive.

Re: The first two custom silicon chips designed by Microsoft for its cloud

#98

Post by solardev »

brucethemoose2 wrote: post #86
aleph_minus_one wrote: post #40

Earlier quoted context omitted.

> The masses will be left with the NVidia monopoly, while large companies will be able to free themselves from that. My bet: if it really becomes clear what capabilities an AI accelerator chip needs and lots of people want to run (or even train) AIs on their own computers, AI accelerators will appear at the market. This is how capitalism typically works. My further bet: these AI accelerators will initially come from…

The capital costs are enormous, not even counting the CUDA moat. It takes years to start producing a big AI processor. Yet many startups and existing designers anticipated this demand correctly, years in advance, and they are all still kinda struggling. Nvidia is massively supply constrained. AI customers would be buying up MI250s, CS-2s, IPUs, Tenstorrent accelerators, Gaudi 2s and so on en masse if they wanted to..…

Is there not a distributed computing potential here like there was for crypto mining? Some sort of seti@home/boinc like setup where home users can donate or sell compute time?

Re: The first two custom silicon chips designed by Microsoft for its cloud

#99

Post by kmod »

I thought an interesting point was the liquid cooling -- unclear how important this is to them, but I'm guessing it means that they designed it with a TDP that requires liquid cooling.

This (wanting higher density) is the opposite of the trade-off that I was expecting. In my (limited and out of date) experience, power was the limiting factor before space, and I believe AI racks have very high power draws already.

I would have guessed this would be because larger nodes would be better for AIs tight communication patterns, but they specifically call out datacenter space as the constraint. Curious if anyone knows more about this

Re: The first two custom silicon chips designed by Microsoft for its cloud

#100

Post by tomcam »

aleph_minus_one wrote: post #40
ronsor wrote: post #22

Earlier quoted context omitted.

Google does the same thing with their TPUs. The masses will be left with the NVidia monopoly, while large companies will be able to free themselves from that.

> The masses will be left with the NVidia monopoly, while large companies will be able to free themselves from that. My bet: if it really becomes clear what capabilities an AI accelerator chip needs and lots of people want to run (or even train) AIs on their own computers, AI accelerators will appear at the market. This is how capitalism typically works. My further bet: these AI accelerators will initially come from…

> My bet: if it really becomes clear what capabilities an AI accelerator chip needs and lots of people want to run (or even train) AIs on their own computers, AI accelerators will appear at the market.

My bet: in 6 months jart will have models running on local or server, with support for all platforms and using only 88K of ram ;)

Post reply on HN