Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building Meta's GenAI infrastructure

engineering.fb.com

Post reply on HN
Print view View original

101–110 of 314 posts

Re: Building Meta's GenAI infrastructure

#101

Post by andy99 »

danielhanchen wrote: post #27

float8 got a mention! x2 more FLOPs! Also xformers has 2:4 sparsity support now so another x2? Is Llama3 gonna use like float8 + 2:4 sparsity for the MLP, so 4x H100 float16 FLOPs? Pytorch has fp8 experimental support, whilst attention is still complex to do in float8 due to precision issues, so maybe attention is in float16, and RoPE / layernorms in float16 / float32, whilst everything else is float8?

Is there float8 support in any common CPU intrinsics? It sounds interesting but curious what will be the impact if any on CPU inference.

Re: Building Meta's GenAI infrastructure

#102

Post by theptip »

ZiiS wrote: post #41
fuddle wrote: post #35

How much are they paying for H100's? If they are paying $10k: 350,000 NVIDIA H100 x $10k = $3.5b

They may have to pay a premium to secure ~¼ of the output; certainly unlikely to be that steep a discount.

Semi analysis posted recently noting that Meta locked in these purchases a while ago; something like a year or more. So they probably didn’t pay today’s spot rate.

Re: Building Meta's GenAI infrastructure

#103

Post by nemothekid »

nova22033 wrote: post #75
crakenzak wrote: post #43

Earlier quoted context omitted.

The angle is that by releasing cutting edge AI research to the public openly, the relative difference between open source models/tech and closed source tech shrinks. Whether or not you think the "value" of AI products is proportional to their performance gap vs the next closest thing or not is up to you. Very interesting PG essay I read recently talks about the opposite of this (Superlinear returns) where if you're h…

New Linux versions don't "salt the earth" for Windows.

Linux is not competitive as a desktop platform for regular users, but linux did "salt the earth" for the Server market.

Re: Building Meta's GenAI infrastructure

#105

Post by verticalscaler »

fuddle wrote: post #100
trsohmers wrote: post #77

Earlier quoted context omitted.

Significantly more than that; MFN pricing for NVIDIA DGX H100 (which has been getting priority supply allocation, so many have been suckered into buying them in order to get fast delivery) is ~$309k, while a basically equivalent HGX H100 system is ~$250k, coming to a price per GPU at the full server level being ~$31.5k. With Meta’s custom OCP systems integrating the SXM baseboards from NVIDIA, my guess is that their…

350,000 NVIDIA H100 x $23k = $8b :0

Wait till you find out how much they spent on VR.

It is a real loophole in the economy. If you're a trillion dollar company the market will insist you set such sums on fire just to be in the race for $current-hype. If they do it drives their market cap higher still and if they don't they risk being considered un-innovative and therefore doomed to irrelevancy and the market cap will spiral downwards.

Sort of reminds me of The Producers.

Re: Building Meta's GenAI infrastructure

#106

Post by munificent »

echelon wrote: post #66
__loam wrote: post #55

Earlier quoted context omitted.

"My paycheck depends on this technology destroying every field producing cultural artifacts"

Said the butter churner, cotton ginner, and petrol pumper. I work in film. I've shot dozens of them the old fashioned way. I've always hated how labor, time, and cost intensive they are to make. Despite instructions from the luminaries to "just pick up a camera", the entire process is stone age. The field is extremely inequitable, full of nepotism and "who you know". Almost every starry-eyed film student winds up doi…

Call me crazy, but I don't think churning butter and writing a novel are in the same category of human endeavor at all.

Re: Building Meta's GenAI infrastructure

#107

Post by verticalscaler »

Dr_Birdbrain wrote: post #82
NBJack wrote: post #58

Earlier quoted context omitted.

What moat exactly? Much of the user data they have access to is drying up due to new regulations, some of which prohibit IIRC direct use on models as well. I'm not even sure they can use historical data. Meta certainly has an edge in engineer count, undoubtedly. But I'd say they really, really want the metaverse to succeed more to have their on walled garden (i.e. equivalent power of Apple and Google stores, etc.). T…

I think the raw text inside Facebook groups is at least as valuable as Reddit data. Even if demographics data is restricted under European law, the raw text of people interacting is quite valuable.

Indeed, my deranged auntie posting on FB is approximately as valuable as my ADHD/PTSD quaranteeny nephew redditing.

Re: Building Meta's GenAI infrastructure

#108

Post by oblio »

verticalscaler wrote: post #105
fuddle wrote: post #100

Earlier quoted context omitted.

350,000 NVIDIA H100 x $23k = $8b :0

Wait till you find out how much they spent on VR. It is a real loophole in the economy. If you're a trillion dollar company the market will insist you set such sums on fire just to be in the race for $current-hype. If they do it drives their market cap higher still and if they don't they risk being considered un-innovative and therefore doomed to irrelevancy and the market cap will spiral downwards. Sort of reminds m…

The thing is, this could be considered basic research, right? Basic research IS setting money on fire until (and if) that basic research turns into TCP/IP, Ethernet and the Internet.

Re: Building Meta's GenAI infrastructure

#109

Post by oblio »

rthnbgrredf wrote: post #62
DEDLINE wrote: post #16

I wonder if Meta would ever try to compete with AWS / MSFT / GOOG for AI workloads

Meta could build their own cloud offering. But it would take years to match the current existing offerings of AWS, Azure and GCP in terms of scale and wide range of cloud solutions.

The real question is: why aren't they? They had the infrastructure needed to seed a cloud offering 10 years ago. Heck, if Oracle managed to be in 5th (6th? 7th?) place, Facebook for sure could have been a top 5 contender, at least.
Post reply on HN