Meta Unveils New AI Supercomputer • Hacker News Board

Re: Meta Unveils New AI Supercomputer

#121

Post by paxys » Mon, Jan 24, 2022, 9:08 PM UTC

penjelly wrote: post #38

it feels like the future of companies is to increasingly give tasks to AI. So eventually we'll have massive corps that have a couple execs and a super ai making them all obscenely rich? I want to hate this idea, but it would be the same as hating machines replacing manual labor over the last 100 years. im not sure what to think, nor how to prepare myself for the next 20 years.

20 years is optimistic. This future isn't something we'll have to worry about in our lifetime, if ever. People wildly overestimate the state of AI as it exists today.

Re: Meta Unveils New AI Supercomputer

#122

Post by rileyphone » Mon, Jan 24, 2022, 9:11 PM UTC

ddalex wrote: post #92

chundicus wrote: post #62

I can't shake the feelings that a trillion or a quadrillion parameters won't solve the fundamental shortcomings of ML models not being models of artificial intelligence. I guess there's no way of knowing until we reach AGI, but I've never heard a compelling argument for why pure ML would get us there. GPT3 seems more like an argument against that hypothesis (in my view) than for it. Even the best, most expensive mode…

I feel that for there are three requirements for a NN-based AGI, inspired by biology: a). an internal feedback loop that evaluates a possible output without actuating it, and self-modifies the parameters if the possible output is not what it's needed b). the capability (based on a) to model own behaviours without acting on them, and to model other agents behaviours and incorporate that model into the feedback c). the…

https://www.creativemachineslab.com/uploads/6/9/3/4/69340277...

"A robot modeled itself without prior knowledge of physics or its shape and used the self-model to perform tasks and detect self-damage."

Re: Meta Unveils New AI Supercomputer

#123

Post by sxv » Mon, Jan 24, 2022, 9:11 PM UTC

michelb wrote: post #16

All this to better predict behavior and present ads. What a waste.

A waste is when you throw something into a landfill. This is weaponization by an enemy of the people.

Re: Meta Unveils New AI Supercomputer

#124

Post by Nasrudith » Mon, Jan 24, 2022, 9:12 PM UTC

ddalex wrote: post #92

chundicus wrote: post #62

I can't shake the feelings that a trillion or a quadrillion parameters won't solve the fundamental shortcomings of ML models not being models of artificial intelligence. I guess there's no way of knowing until we reach AGI, but I've never heard a compelling argument for why pure ML would get us there. GPT3 seems more like an argument against that hypothesis (in my view) than for it. Even the best, most expensive mode…

I feel that for there are three requirements for a NN-based AGI, inspired by biology: a). an internal feedback loop that evaluates a possible output without actuating it, and self-modifies the parameters if the possible output is not what it's needed b). the capability (based on a) to model own behaviours without acting on them, and to model other agents behaviours and incorporate that model into the feedback c). the…

Personally I think that biology may be a flawed approach for most applications. Although the others arr worthy ends in themselves just for its role in understanding ourselves in a forensic archaeologist try to replicate sort of way, let alone any potential insights to biological brains.

Biology is glacially slow in comparison and one of the advantages from computing is being fast.

I believe that not modeling it is partially by design as a result of responsibility and blame frameworks. If you depend upon possible actions taken by others to be safe you are reckless. Extrapolating from current motions is more reliable than trying to profile everything. "They are moving towards the street at 3mph and 20 ft away, their vector will intersect with car, brake to avoid collision or accelerate enough to leave intersection zone before they can even reach us" seems a more reliable approach. It isn't like a kid will suddenly teleport into the road.

Re: Meta Unveils New AI Supercomputer

#125

Post by ravi-delia » Mon, Jan 24, 2022, 9:14 PM UTC

chundicus wrote: post #62

I can't shake the feelings that a trillion or a quadrillion parameters won't solve the fundamental shortcomings of ML models not being models of artificial intelligence. I guess there's no way of knowing until we reach AGI, but I've never heard a compelling argument for why pure ML would get us there. GPT3 seems more like an argument against that hypothesis (in my view) than for it. Even the best, most expensive mode…

I'm most of the way towards agreeing with you, but I think you underestimate how far you could get without any major changes. Most of the brain consists of feed-forward processing, and what closed loops exist are probably replacements for backprop rather than essential to cognition. That's all the low level processing, from visual to motor. Now obviously we have higher level processing too, and it might be super weird! But no model we've made comes close to the size of even specialized brain regions, and study after study has demonstrated the power of the subconscious mind. Once we have big enough models, we might find out that all we need to take it to that final step is a while loop.

Re: Meta Unveils New AI Supercomputer

#126

Post by curiousllama » Mon, Jan 24, 2022, 9:15 PM UTC

Jasper_ wrote: post #105

ddalex wrote: post #92

Earlier quoted context omitted.

I feel that for there are three requirements for a NN-based AGI, inspired by biology: a). an internal feedback loop that evaluates a possible output without actuating it, and self-modifies the parameters if the possible output is not what it's needed b). the capability (based on a) to model own behaviours without acting on them, and to model other agents behaviours and incorporate that model into the feedback c). the…

That's what gets me about self-driving cars. The road is a very social space, and follows social rules. Pretty much all of the communication and norms happening on the road are social ones. The thing that would convince me AGI is ready would be to play a convincing game of poker. Or join in on a conversation mid-way through, listen to it, and engage with it actively. Show that machines are able to pick up on social c…

NNS can win at poker - one recently beat a bunch of pros. Games are great challenges but bad tests.

The structure both makes them tractable and not as generalizable as we'd like. To your point, social interactions aren't nearly so structured.

https://www.nature.com/articles/d41586-019-02156-9

Re: Meta Unveils New AI Supercomputer

#127

Post by terafo » Mon, Jan 24, 2022, 9:16 PM UTC

bno1 wrote: post #6

I wonder if things like this are the real reason behind the GPU shortage. How many other AI super computers are being built right now?

This is definitely not the case. A100, which is used for most "AI supercomputers" is manufactured on TSMC fabs, while Nvidia's gaming cards are produced on Samsung fabs. AMD produces their gaming GPUs on TSMC, but they are somewhere around 10% of the market since they are unwilling to divert their capacity from CPUs, which are more profitable, and consoles, really not sure why.

Re: Meta Unveils New AI Supercomputer

#128

Post by endymi0n » Mon, Jan 24, 2022, 9:17 PM UTC

Jasper_ wrote: post #105

ddalex wrote: post #92

Earlier quoted context omitted.

I feel that for there are three requirements for a NN-based AGI, inspired by biology: a). an internal feedback loop that evaluates a possible output without actuating it, and self-modifies the parameters if the possible output is not what it's needed b). the capability (based on a) to model own behaviours without acting on them, and to model other agents behaviours and incorporate that model into the feedback c). the…

That's what gets me about self-driving cars. The road is a very social space, and follows social rules. Pretty much all of the communication and norms happening on the road are social ones. The thing that would convince me AGI is ready would be to play a convincing game of poker. Or join in on a conversation mid-way through, listen to it, and engage with it actively. Show that machines are able to pick up on social c…

https://www.nature.com/articles/d41586-019-02156-9/

Re: Meta Unveils New AI Supercomputer

#129

Post by sailingparrot » Mon, Jan 24, 2022, 9:18 PM UTC

karmasimida wrote: post #43

> Meta’s AI supercomputer houses 6,080 Nvidia graphics-processing units ..... By mid-summer, when the AI Research SuperCluster is fully built, it will house some 16,000 GPUs Honestly ... this is lot of GPUs ... but is it the biggest...? > Model training is done with mixed precision on the NVIDIA DGX SuperPOD-based Selene supercomputer powered by 560 DGX A100 servers networked with HDR InfiniBand in a full fat tree co…

At 16k it will definitely be the biggest.

As for today, Nvidia has this a very slightly smaller cluster that you outlined at ~5k, Microsoft as a few of them roughly of that size, and Microsoft also built a 10k GPU cluster for OpenAI 2 years ago, but those are V100 GPUs.

So, is 6k A100 "bigger" than 10k V100? Depends exactly how you use them, in a perfect usage scenario yes, slightly. In real life maybe not.

Re: Meta Unveils New AI Supercomputer

#130

Post by Nasrudith » Mon, Jan 24, 2022, 9:20 PM UTC

rococode wrote: post #70

kleiba wrote: post #40

I used to work at a university where my professor had been in automatic speech recognition for a long time, but basically gave up on that line of research about 10 years ago because he figured that universities simply cannot compete budget wise with the big industry players. I suppose the same will soon be true for most ML-related areas of research sooner or later, at least as far as applied ML is concerned. Already,…

I think the academic side will start shifting towards research on efficiency and speed while companies will continue to push the cutting edge. In the NLP space there's been a lot of work recently around reducing model sizes, since they've started to reach the point where model weights sometimes don't fit in the memory of most GPUs. There's also projects like MarianNMT which completely abandon Python and write heavily…

It would be a bit ironic for universities to compete on efficency and speed given those are two things companies optimize on. Not impossible of course, theory and encouragement to a bit more abstract could lead to providing that.

As for writing low level code, I thought that was something usually handled by the compiler or where even the advanced high performance for high price mostly tweaked the compiler after analyzing the output. Not my direct space so I speak with no authority.