Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta Unveils New AI Supercomputer

wsj.com

Post reply on HN
Print view View original

181–190 of 199 posts

Re: Meta Unveils New AI Supercomputer

#181

Post by dekhn »

sailingparrot wrote: post #129
karmasimida wrote: post #43

> Meta’s AI supercomputer houses 6,080 Nvidia graphics-processing units ..... By mid-summer, when the AI Research SuperCluster is fully built, it will house some 16,000 GPUs Honestly ... this is lot of GPUs ... but is it the biggest...? > Model training is done with mixed precision on the NVIDIA DGX SuperPOD-based Selene supercomputer powered by 560 DGX A100 servers networked with HDR InfiniBand in a full fat tree co…

At 16k it will definitely be the biggest. As for today, Nvidia has this a very slightly smaller cluster that you outlined at ~5k, Microsoft as a few of them roughly of that size, and Microsoft also built a 10k GPU cluster for OpenAI 2 years ago, but those are V100 GPUs. So, is 6k A100 "bigger" than 10k V100? Depends exactly how you use them, in a perfect usage scenario yes, slightly. In real life maybe not.

Systems like this are designed to reach nearly peak performance (IE # of flops per processing element * # of processing elements), explicitly by making a network that won't block or increase latency for the common expensive operations (allreduce, allvall) at the expensive of greatly increased cost.

The point of making this machine is to have a lot of A100s going at the same time, and that will unblock some small set of researchers who are working on time-sensitive competitive research projects by giving them a slightly throughput and latency advantage on the largest problems. The vast majority of users would be better served by a small number of cheaper, slower GPUs that they had exclusive access to for the longest time period they could afford to wait.

Re: Meta Unveils New AI Supercomputer

#182

Post by 52-6F-62 »

cm2012 wrote: post #35
bearjaws wrote: post #5

“The experiences we’re building for the metaverse require enormous compute power…and RSC will enable new AI models that can learn from trillions of examples, understand hundreds of languages, and more,” Meta CEO Mark Zuckerberg I don't really understand how AI processing is going to make the 'experiences' any better? This seems to me like investor fluff, saying they have some insane capability that other 'VR provider…

Zuck has vision, especially for what people will want to use. I am looking forward to what FB comes up with here.

But his vision, in summary, is to dictate a world view. I am not looking forward to what they come up with.

Re: Meta Unveils New AI Supercomputer

#183

Post by mindcrime »

space_fountain wrote: post #173
mindcrime wrote: post #78

Earlier quoted context omitted.

I've always imagined AGI (perhaps naively) as being achieved by clever usage of ML, plus some utilization of classical/symbolic AI from pre-AI winter days, plus probably some unknown elements. For what it's worth, this is my view as well. And I don't think it's particularly naive. Plenty of people have researched and/or are researching aspects of how to do this. But how to combine something like a neural network, wit…

It seems quite clear to me that human brains are not actually doing much symbolic logic. What symbolic logic we do do has been bolted on using other faculties. I think the problem is that reasoning about our own minds is incredible tough. We want there to be some sort of magic sauce to what makes us, us and so we reject things like ANN's that seem somehow too simple. I think it probably is right that we won't just be…

It seems quite clear to me that human brains are not actually doing much symbolic logic. What symbolic logic we do do has been bolted on using other faculties.

I agree. But my interest is in engineering something that works, not necessarily in creating an exact replica of the human brain. That's why my interest falls into the domain of symbolic / sub-symbolic integration - because it strikes me as a faster path to more usable computer intelligence.

I have no problem believing that a sufficiently large ANN, with the right training and inference algorithms, could achieve AGI. My problem is that A. right now achieving that seems very out of reach to me (but I could be wrong) and B. it seems unnecessary to me to remain wedded to the idea of 100% (or even 90% or 80% etc.) fidelity with our biological brains. After all, if we want something just like a human brain, we just need a man, a woman, and 9 months of time.

Anyway, I think it's OK to think of engineering in "short cuts" by using things we know computers are good at, and things we already know how to do, and trying to combine them with ANN's in such a way as to make something useful. Will it ever yield AGI? I have no way of knowing. And even if it does, would that approach actually be faster than a pure ANN approach? Again, I don't know. But for now, I spend my time on symbolic/sub-symbolic integration nonetheless.

I think the problem is that reasoning about our own minds is incredible tough.

Yes, definitely.

Re: Meta Unveils New AI Supercomputer

#184

Post by dekhn »

chundicus wrote: post #62

I can't shake the feelings that a trillion or a quadrillion parameters won't solve the fundamental shortcomings of ML models not being models of artificial intelligence. I guess there's no way of knowing until we reach AGI, but I've never heard a compelling argument for why pure ML would get us there. GPT3 seems more like an argument against that hypothesis (in my view) than for it. Even the best, most expensive mode…

it would be pretty embarrassing (or relieving?) if it eventually turned out there was nothing special about human intelligence, just that we crossed some threshold of neurons and other brain bits to ("a few quadrillion parameters") to convincingly fool ourselves that we are self aware, have agency, and do anything "intelligent" (other than some fancy stuff that looks like the physics/biology equivalent of state of the art ML).

I am a proponent of using a working theory that intelligence is an emergent property and we can in principle create new intelligences in a lab (or ML warehouse) if we provide the proper conditions, but that finding and maintaining those conditions is extremely hard. Some state of the art research today aims to integrate recognition capbilities (image recognititon and object detection/tracking on video, voice extraction from audio, text) with advanced generative models for language and behavior, as well as realtime rendering systems that can create realistic humans.

if we combine those we can make a bot that appears fully interactive, passes all turing tests, convinces typical person it's another person... and still has nothing inside researchers would call "artificial intelligence". It might even solve science problems that we can't without having any spark of creativity or agency. Or maybe when we make a bot with all those properties, some uncanny valley is crossed and out pops something that has objective AGI?

As the wise robot once said, "if you can't tell the difference, does it really matter?". We should forge ahead with building datacenter-scale brains and feed them with data and algorithms, while also maintaining a cadre of research scientists who are attuned to the ethical challenges of doing so, an ops team trained to recognize the early signs of sentience, and an exec team with humanity.

Re: Meta Unveils New AI Supercomputer

#185

Post by sailingparrot »

dekhn wrote: post #181
sailingparrot wrote: post #129

Earlier quoted context omitted.

At 16k it will definitely be the biggest. As for today, Nvidia has this a very slightly smaller cluster that you outlined at ~5k, Microsoft as a few of them roughly of that size, and Microsoft also built a 10k GPU cluster for OpenAI 2 years ago, but those are V100 GPUs. So, is 6k A100 "bigger" than 10k V100? Depends exactly how you use them, in a perfect usage scenario yes, slightly. In real life maybe not.

Systems like this are designed to reach nearly peak performance (IE # of flops per processing element * # of processing elements), explicitly by making a network that won't block or increase latency for the common expensive operations (allreduce, allvall) at the expensive of greatly increased cost. The point of making this machine is to have a lot of A100s going at the same time, and that will unblock some small set…

> Systems like this are designed to reach nearly peak performance

The system certainly is. The code running on that system generally isn't. Pulling 100% of the FLOPS the GPUs are able to provide is quite hard.

And my point was it also depends on the specific models you are training. Are you training a transformer model in FP32 precision? Then yes, 6K A100 will blow 10K V100. Are you training a ConvNet in FP16? Then no, 10K V100 will perform better.

The GPUs have different architecture, you have to use the architecture best suited for the A100 to achieve the speedup marketed by NVidia, which is presumably the number FB is using to claim that their 6k GPU cluster is bigger than OpenAI's 10K one.

Re: Meta Unveils New AI Supercomputer

#186

Post by space_fountain »

mindcrime wrote: post #183
space_fountain wrote: post #173

Earlier quoted context omitted.

It seems quite clear to me that human brains are not actually doing much symbolic logic. What symbolic logic we do do has been bolted on using other faculties. I think the problem is that reasoning about our own minds is incredible tough. We want there to be some sort of magic sauce to what makes us, us and so we reject things like ANN's that seem somehow too simple. I think it probably is right that we won't just be…

It seems quite clear to me that human brains are not actually doing much symbolic logic. What symbolic logic we do do has been bolted on using other faculties. I agree. But my interest is in engineering something that works, not necessarily in creating an exact replica of the human brain. That's why my interest falls into the domain of symbolic / sub-symbolic integration - because it strikes me as a faster path to mo…

Very fair takes. I could certainly imagine elements being pulled in. For example things like alpha zero are to my understanding already coupling things like tree searches to neural nets. I sort of expect that any general solution would include some of that, but symbolic approaches seem to consistently do worse despite lots of people thinking they won't and plenty of money to be made. I think part of the problem is that what we want with AI is to interface with humans, and humans are using something fuzzy to understand the world so trying to model that rigidly will be hard

Re: Meta Unveils New AI Supercomputer

#187

Post by sbierwagen »

d1sxeyes wrote: post #141
forgotmyoldacc wrote: post #32

Earlier quoted context omitted.

Example 1: GPT-3 is a decent chatbot. Training a similar model so you can have a conversation with AIs in the "metaverse" (god, that word is terrible) could be fun / useful. Example 2: Using AI upscaling (like Nvidia) to improve visual fidelity in games. Example 3: Hand/body tracking for avatars. The more AI compute, the more experimentation researchers can do.

One particularly interesting piece of tech I've seen is Nvidia's AI Video 'Compression'. In summary, rather than actually streaming video to the person you're chatting with, you send a keyframe, and then 'compressed' video is sent over the wire, and 'decompressed' at the receiver end. I'm putting 'compression' in quotations because to me I'm not sure I'm comfortable calling it compression. Basically, you're remotely…

A Fire Upon The Deep (1992):

>Fleet Central refused the full video link coming from the Out of Band … Kjet had to settle for a combat link: The screen showed a color image with high resolution. Looking at it carefully, one realized the thing was a poor evocation…. Kjet recognized Owner Limmende and Jan Skrits, her chief of staff, but they looked several years out of style: old video matched with the transmitted animation cues. The actual communication channel was less than four thousand bits per second

>The picture was crisp and clear, but when the figures moved it was with cartoonlike awkwardness. And some of the faces belonged to people Kjet knew had been transferred before the fall of Sjandra Kei. The processors here on the Ølvira were taking the narrowband signal from Fleet Central, fleshing it out with detailed (and out of date) background and evoking the image shown. No more evocations after this, Svensndot promised himself, at least while we’re down here.

Re: Meta Unveils New AI Supercomputer

#188

Post by Atlas667 »

ctoth wrote: post #11

Anybody else get the sense that we're just totally frickin doomed? Even if Yudkowsky is off about AGI (which is a big maybe!) in what possible world will this technology be used to make our individual lives better (assuming you're not a FAIR researcher?)

Yep, this will be used for precision tracking and context categorization. And of course, to more deeply understand human motivators and make platforms even more addictive to us. This will have minimal benefits to the working class and all the benefit to Meta and their clients.

I fear the ammount of human information this AI is going to be free to analyze from Facebook and what it will deduce about us and then how Meta will use it to generate capital.

Re: Meta Unveils New AI Supercomputer

#189

Post by lemax »

It feels eerie that these trillions of parameters and exabyte sized training sets will come from harvested user data. 17 years of user activity all culminating into some.. supercomputer? I wonder how comments I wrote when I, and all the other people from my generation, were like 14 and using FB will feed into this and sort of be immortalized in this strange way.

Re: Meta Unveils New AI Supercomputer

#190

Post by mchusma »

cm2012 wrote: post #172
vl wrote: post #94

Earlier quoted context omitted.

Honestly, this single GPU-based install is child's play compared to Google's multiple TPU exoflop supercomputers with hyper-cube optical interconnects. Google's ML setups allow synchronous weight update on thousand+ TPUs...

For what its worth, for attention based advertising (youtube and display, not search), FB targeting blows Google out of the water. Not sure why but its consistent across brands.

I have seen this myself, by I'm unsure if it's just a "ad quality" thing. For example, I can target exact placements on YouTube for my exact niche, and broad Facebook matching will outperform. I have tried YouTube and display for months with nothing within an order of magnitude as effective as Facebook.
Post reply on HN