161–170 of 314 posts
> we have successfully used both RoCE and InfiniBand clusters for large, GenAI workloads (including our ongoing training of Llama 3 on our RoCE cluster) without any network bottlenecks. Interesting dig on IB. RoCE is the right solution since it is open standards and more importantly, available without a 52+ week lead time.
Yeah, and RoCE isn't single vendor. I'm not sure IB scales to the relevant cluster sizes, either.
Earlier quoted context omitted.
It will be ironic if Meta sinks all this money into the new trend and finds out later that it has been a huge boondoggle, just as publishers followed Facebook's "guidance" on video being the future, subsequently gutting the talent pool and investing into video production and staff - only to find out it was all a total waste.
It already paid off. When the world moved from determinisic to probablistic ad modeling. That's why their numbers are so good right now compared to every other advertiser
Earlier quoted context omitted.
350,000 NVIDIA H100 x $23k = $8b :0
Wait till you find out how much they spent on VR. It is a real loophole in the economy. If you're a trillion dollar company the market will insist you set such sums on fire just to be in the race for $current-hype. If they do it drives their market cap higher still and if they don't they risk being considered un-innovative and therefore doomed to irrelevancy and the market cap will spiral downwards. Sort of reminds m…
You don’t think earning increasing amounts of tens of billions of dollars in net income per year at some of the highest profit margins in the world at that size for 10+ years has anything to do with market cap?
Earlier quoted context omitted.
FB does not have the flywheel of running data centres - all three of those mentioned run hyper scale datacentres that they can then juice by “investing” billions in AI companies who then turn around and put those billions as revenue in the investors OpenAI takes money from MSFT and buys Azure services Anthropic takes Amazon money and buys AWS services (as do many robotics etc) I am fairly sure it’s not illegal but it…
Sounds like it's free equity at the very least
Earlier quoted context omitted.
It looks more like an infra problem than ML. "Software architect"s mixed with devops/infra/sre people
Well since I'm not a ML engineer of any kind - that's good!
The link mentions "our internal job scheduler" and how they had to optimize it for this work -- does anyone know what this job scheduler is called, or how it works?
it might be twine: https://www.usenix.org/system/files/osdi20-tang.pdf but I suspect its not that, because Twine is optimised for services rather than batch processing, and doesn't really have the concept of priorities.
I wonder if Meta would ever try to compete with AWS / MSFT / GOOG for AI workloads
FB does not have the flywheel of running data centres - all three of those mentioned run hyper scale datacentres that they can then juice by “investing” billions in AI companies who then turn around and put those billions as revenue in the investors OpenAI takes money from MSFT and buys Azure services Anthropic takes Amazon money and buys AWS services (as do many robotics etc) I am fairly sure it’s not illegal but it…
Here more on the deals (2003):
https://www.cnet.com/tech/services-and-software/aol-saga-ope...
Popular names included AOL, Cisco, Yahoo, etc.
Not sure if Amazon’s term sheets driving high valuation are nothing but AWS credits (Amazon’s own license to print money).
float8 got a mention! x2 more FLOPs! Also xformers has 2:4 sparsity support now so another x2? Is Llama3 gonna use like float8 + 2:4 sparsity for the MLP, so 4x H100 float16 FLOPs? Pytorch has fp8 experimental support, whilst attention is still complex to do in float8 due to precision issues, so maybe attention is in float16, and RoPE / layernorms in float16 / float32, whilst everything else is float8?
Is there float8 support in any common CPU intrinsics? It sounds interesting but curious what will be the impact if any on CPU inference.
Earlier quoted context omitted.
Sounds like it's free equity at the very least
How is it free equity? Spending money to invest it somewhere involves risks. You might recover some of it if the investment is valued by others, but there is no guarantee.
It might cost tomorrow though, when the company starts to use your services. However depending the deal structure they might not use all the credit, go belly up before credit is used or bought up by someone with real cash.
Earlier quoted context omitted.
I still, for the life of me, can't understand why Google doesn't just start selling their TPUs to everyone. Nvidia wouldn't be anywhere near their size if they only made H100s available through their DGX cloud, which is what Google is doing only making TPUs available through Google Cloud. Good hardware, good software support, and market is starving for performant competitors to the H100s (and soon B100s). Would sell…
Do you mean, sell TPU hardware to other companies that would run it in their data centers? I can't imagine that would ever really work. The only reason TPUs work at Google is because they have huge teams across many different areas to keep them running (SRE, hardware repair, SWE, hardware infra) and it's coupled to the design of the data centers. To vend and externalize the software would require google to setup simi…
We had a GSA for intranet search and other than the paint this was a standard Dell server. I remember not being impressed by what the GSA could do.
We also had Google Urchin for web analytics, it wasn't a hardware appliance but the product wasn't very impressive either. They then killed that and tried to get you onto Google Analytics.
They just didn't commit to these on premise enterprise products.