How Meta trains large language models at scale
engineering.fb.com
How Meta trains large language models at scale
1–10 of 213 posts
Re: How Meta trains large language models at scale
#2Random q, I wonder if gloo is used in these systems? https://github.com/facebookincubator/gloo
RDMA and GPUDirect capable. Coordinates over MPI or (hi)redia.
Re: How Meta trains large language models at scale
#3interesting that their domain is still engineering.fb.com
Re: How Meta trains large language models at scale
#4Posts like this underscore why the smart money is betting on Google as the long term AI winner. Meta, Microsoft, OpenAI, etc. are trying to address problems with consumer video cards and spending billions to try and out bid each other to win Nvidia's favor - while Google is on their 6th generation of custom silicon.
Literally the only thing that can stop Google now is the fact they keep bringing Microsoft and Oracle flunkies into leadership positions.
Re: How Meta trains large language models at scale
#5Posts like this underscore why the smart money is betting on Google as the long term AI winner. Meta, Microsoft, OpenAI, etc. are trying to address problems with consumer video cards and spending billions to try and out bid each other to win Nvidia's favor - while Google is on their 6th generation of custom silicon. Literally the only thing that can stop Google now is the fact they keep bringing Microsoft and Oracle…
How do TPUs perform compared to GPUs on LLMs and image generation?
Re: How Meta trains large language models at scale
#6Posts like this underscore why the smart money is betting on Google as the long term AI winner. Meta, Microsoft, OpenAI, etc. are trying to address problems with consumer video cards and spending billions to try and out bid each other to win Nvidia's favor - while Google is on their 6th generation of custom silicon. Literally the only thing that can stop Google now is the fact they keep bringing Microsoft and Oracle…
H100s are far from consumer video cards
Re: How Meta trains large language models at scale
#7[flagged]
Re: How Meta trains large language models at scale
#8Posts like this underscore why the smart money is betting on Google as the long term AI winner. Meta, Microsoft, OpenAI, etc. are trying to address problems with consumer video cards and spending billions to try and out bid each other to win Nvidia's favor - while Google is on their 6th generation of custom silicon. Literally the only thing that can stop Google now is the fact they keep bringing Microsoft and Oracle…
Google has been working on TPUs and Tensorflow for a decade with pretty mixed success; I don't think it's clear that they're going to win.
Re: How Meta trains large language models at scale
#9[flagged]
[flagged]
Re: How Meta trains large language models at scale
#10Posts like this underscore why the smart money is betting on Google as the long term AI winner. Meta, Microsoft, OpenAI, etc. are trying to address problems with consumer video cards and spending billions to try and out bid each other to win Nvidia's favor - while Google is on their 6th generation of custom silicon. Literally the only thing that can stop Google now is the fact they keep bringing Microsoft and Oracle…
Do you really think Google’s hardware expertise is better than Nvidia’s?
If needed these other companies have the $$$ to buy the best chips money can buy from Nvidia. Better chips than Google could ever produce.
If anything, this is why IMO Google will fail.