This is great news for Nvidia and their stock, but are they sure the LLMs and image models will scale indefinitely? nature and biology has a preference for sigmoids. What if we find out that AGI requries different kinds of cpu capabilities
Building Meta's GenAI infrastructure
111–120 of 314 posts
Re: Building Meta's GenAI infrastructure
#112Earlier quoted context omitted.
Proven technology, maybe, but proven product-market fit for the kinds of things Facebook is using it for? Their linked blog about AI features gives examples "AI stickers" and image editing... cool, but are these potential multi-billion dollar lifts to their existing business? I guess I'm skeptical it's worthwhile unless they're able to unseat ChatGPT with a market-leading general purpose assistant.
I have a few group chats just that devolve into hours of sending stickers or image generation back and forth, lately we've been "writing a book together" with @Meta AI as the ghost writer, and while it utterly sucks, its been a hilarious shared experience. I don't think anyone else has gotten that group chat with AI thing so nailed.
On the podcast TrashFuture, November Kelly recently described AI systems as “garbage dispensers” which is both a funny image (why would anyone make a garbage dispenser??) and an apt description. Certainly these tools have some utility, but there are a load of startups claiming to “democratize creativity” by allowing anyone to publish AI generated slop to major platforms. On the podcast this phrase was used during discussion of a website which lets you create AI generated music and push it to Spotify, a move which Spotify originally pushed back on but has now embraced. Garbage dispenser indeed.
Re: Building Meta's GenAI infrastructure
#113Earlier quoted context omitted.
Wait till you find out how much they spent on VR. It is a real loophole in the economy. If you're a trillion dollar company the market will insist you set such sums on fire just to be in the race for $current-hype. If they do it drives their market cap higher still and if they don't they risk being considered un-innovative and therefore doomed to irrelevancy and the market cap will spiral downwards. Sort of reminds m…
The thing is, this could be considered basic research, right? Basic research IS setting money on fire until (and if) that basic research turns into TCP/IP, Ethernet and the Internet.
I wish.
Funnily enough Arpanet and all that Xerox stuff were like Where as I think this more appropriately can be considered the meta PR budget. They simply can't not spend it, would look bad for Wall Street. Have to keep up with the herd.
Re: Building Meta's GenAI infrastructure
#114Earlier quoted context omitted.
> And if you think any Jack or Jill can just come in and text prompt a whole movie, you're crazy. It's still hard work and a metric ton of good taste. If you want anything good , yes. If you just want something … I reckon it'd take a week to assemble an incomprehensible-nonsense-film pipeline, after which it's just a matter of feeding the computer electricity. Short-term, this is going to funnel resources away from t…
> If you want anything good, yes. If you just want something ... You don't even need AI for that. https://en.wikipedia.org/wiki/YouTube_poop https://en.wikipedia.org/wiki/Skibidi_Toilet The idea that AI isn't going to be used as a creative tool too and that it won't lead to more and better art is a defeatist, Luddite attitude. Similarly shaped people thought that digital cameras would ruin cinema and photography. > S…
> Similarly shaped people thought that digital cameras would ruin cinema and photography.
Obviously, but you seem to be arguing that AI is just another evolution of productivity tools. You still need to have a photographer's eye while using this technology.
If you couldn't make a good composition on film, a digicam will not save you, and it definitely did not replace photographers. Perhaps lowered the barrier of entry for prosumers.
https://www.nytimes.com/2023/12/26/opinion/ai-future-photogr...
Re: Building Meta's GenAI infrastructure
#115I wonder if Meta would ever try to compete with AWS / MSFT / GOOG for AI workloads
Meta could build their own cloud offering. But it would take years to match the current existing offerings of AWS, Azure and GCP in terms of scale and wide range of cloud solutions.
And then there's sales. All of those three - and more you haven't considered, like the Chinese mega-IT companies - spend huge amounts on training, partnerships, consultancy, etc to get companies to use their services instead of their competitors. My current employer seems all-in on Azure, previous one was AWS.
There was one manager who worked at two large Dutch companies and sold AWS to them, as in, moving their entire IT, workloads and servers over to AWS. I wouldn't be surprised if there was a deal made there somewhere.
Re: Building Meta's GenAI infrastructure
#116Earlier quoted context omitted.
A company moving away from Nvidia/CUDA while the field is developing so rapidly would result in that company falling behind. When (if) the rate of progress in the AI space slows, then perhaps the big players will have the breathing room to consider rethinking foundational components of their infrastructure. But even at that point, their massive investment in Nvidia will likely render this impractical. Nvidia decisive…
People said the same thing when tensorflow was all the rage and pytorch was a side project. Granted, HW is much harder than SW, but I would not discount Meta's ability to displace NVIDIA entirely.
I don't think they could; nvidia has tons of talent, Meta would have to steal that. Meta doesn't do anything in either consumer or datacenter hardware that isn't for themselves either.
Meta is a services company, their hardware is secondary and for their own usage.
Re: Building Meta's GenAI infrastructure
#117Earlier quoted context omitted.
Meta could build their own cloud offering. But it would take years to match the current existing offerings of AWS, Azure and GCP in terms of scale and wide range of cloud solutions.
The real question is: why aren't they? They had the infrastructure needed to seed a cloud offering 10 years ago. Heck, if Oracle managed to be in 5th (6th? 7th?) place, Facebook for sure could have been a top 5 contender, at least.
because meta sucks at software, documentation and making sure end user products work in a supported way.
Offering reliable IaaS is super hard and capital intensive. Its also not profitable if you are perceived as shit.
Re: Building Meta's GenAI infrastructure
#118float8 got a mention! x2 more FLOPs! Also xformers has 2:4 sparsity support now so another x2? Is Llama3 gonna use like float8 + 2:4 sparsity for the MLP, so 4x H100 float16 FLOPs? Pytorch has fp8 experimental support, whilst attention is still complex to do in float8 due to precision issues, so maybe attention is in float16, and RoPE / layernorms in float16 / float32, whilst everything else is float8?
You're still bounded by memory bandwidth, so adding multiples to FLOPs is not going to give you a good representation of overall speedup.
Re: Building Meta's GenAI infrastructure
#119Earlier quoted context omitted.
> $3.5b Which is a fourth of what they spent in VR/AR in a year. And Gen AI is something they could easily get more revenue as it has now become proven technology, and Meta could possibly leapfrog others because of the data moat.
What moat exactly? Much of the user data they have access to is drying up due to new regulations, some of which prohibit IIRC direct use on models as well. I'm not even sure they can use historical data. Meta certainly has an edge in engineer count, undoubtedly. But I'd say they really, really want the metaverse to succeed more to have their on walled garden (i.e. equivalent power of Apple and Google stores, etc.). T…
> Much of the user data they have access to is drying up due to new regulations, some of which prohibit IIRC direct use on models as well.
Source would be appreciated, because this is opposite of obvious. Regulations against using public first party would be a big news and I haven't heard of anything like that. They use my data for recommending feed so why not for answering my question?
Re: Building Meta's GenAI infrastructure
#120float8 got a mention! x2 more FLOPs! Also xformers has 2:4 sparsity support now so another x2? Is Llama3 gonna use like float8 + 2:4 sparsity for the MLP, so 4x H100 float16 FLOPs? Pytorch has fp8 experimental support, whilst attention is still complex to do in float8 due to precision issues, so maybe attention is in float16, and RoPE / layernorms in float16 / float32, whilst everything else is float8?
You're still bounded by memory bandwidth, so adding multiples to FLOPs is not going to give you a good representation of overall speedup.
Well, those smaller floats require less BW to transfer back and forth as well. Perhaps not a reduction linear in the size of the float, as maybe smaller floats require more iterations and/or more nodes in the model graph to get an equivalent result.
But rest assured there's an improvement, it's not like people would be doing it if there wasn't any benefit!