Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The post-exponential era of AI and Moore’s Law

techcrunch.com

Post reply on HN
Print view View original

31–40 of 150 posts

Re: The post-exponential era of AI and Moore’s Law

#31

Post by the8472 »

npo9 wrote: post #19
_bxg1 wrote: post #6

Honestly, this is a bit of a relief. All of the AI nightmare scenarios (be they the Terminator kind, or the more realistic hyper-empowerment-of-a-few-elites kind) rely on that exponential growth continuing unimpeded. If there's no exponential growth, there's no runaway AI that swiftly outpaces human understanding.

Let me fuel your paranoia a little. It’s likely that cloud computing will continue to see decreases in costs, which means the availability of computing will continue to raise after Mores Law has stopped giving. Quantum computers might play a role in increasing computational power in the next decade. Right now a lot of AI research is low hanging fruit recently made available with advances in CPU and GPU. After the adv…

> Quantum computers might play a role in increasing computational power in the next decade.

It won't increase your FLOP/s in any way. It will mostly[0] help solving narrow problem classes, those outside P but in BQP[1]. To my understanding none of the problems encountered in NN training falls into that.

[0] ignoring polynomial speedups such as grover's algorithm [1] https://en.wikipedia.org/wiki/BQP

Re: The post-exponential era of AI and Moore’s Law

#32

Post by npo9 »

martinpw wrote: post #26
npo9 wrote: post #19

Earlier quoted context omitted.

Let me fuel your paranoia a little. It’s likely that cloud computing will continue to see decreases in costs, which means the availability of computing will continue to raise after Mores Law has stopped giving. Quantum computers might play a role in increasing computational power in the next decade. Right now a lot of AI research is low hanging fruit recently made available with advances in CPU and GPU. After the adv…

> It’s likely that cloud computing will continue to see decreases in costs, which means the availability of computing will continue to raise after Mores Law has stopped giving. Why would cloud computing continue to see significant decreases in costs once Moore's law stops? Clearly there will be some ongoing decline in infrastructure costs, but if your main requirement is for lots of compute (for AI) then it seems har…

I was speaking about linear decreases. Cheaper power, more refined manufacturing processes, and economy of scale benefits are where I expect some of the linear benefits to come from. We can also expect CPU manufactures to compete for awhile, getting increasingly marginal improvements.

Linear improvements will get us there in the end.

Re: The post-exponential era of AI and Moore’s Law

#33

Post by pmoriarty »

ChuckMcM wrote: post #2

The article discusses one of the impacts of compute performance growth rates slowing while compute demand (training AI models) is growing exponentially. The 'end of Moore's law' (as a measure of performance, not density) is probably the most significant "Tech" story of the next decade. Why? Because it is going to demand engineers who can write fast code over engineers who write code fast. A lot of frameworks and abst…

It's not just training AI models that's demanding exponentially increasing compute power, but also streaming, storing, and processing the enormous amounts of scientific data coming from (for example) high-end physics experiments at world-class laboratories like CERN or the LHC, and there's also the tremendous amounts of surveillance data and data processing that's sprung up in recent decades, with an insatiable appetite of various agencies and companies to track, record, and analyze everyone and everything.

Re: The post-exponential era of AI and Moore’s Law

#34

Post by jchw »

codesushi42 wrote: post #23
_bxg1 wrote: post #14

Earlier quoted context omitted.

It could be that the slowing of Moore's Law results in a more diverse array of specialized computing hardware. After all, the resurgence of ML didn't happen because x86 got fast enough, but because video games - of all things - funded the maturation of a whole new category of massively-parallel chips.

the resurgence of ML ... because video games - of all things - funded the maturation of a whole new category of massively-parallel chips Fake news. It has far more to do with the rise of distributed computing than the existence of GPUs.

I don’t really see this; All of the currently popular machine learning frameworks either support GPU or are oriented around GPU based execution. For most developers working on AI it is still the defacto standard.

Re: The post-exponential era of AI and Moore’s Law

#35

Post by sgt101 »

jchw wrote: post #34
codesushi42 wrote: post #23

Earlier quoted context omitted.

the resurgence of ML ... because video games - of all things - funded the maturation of a whole new category of massively-parallel chips Fake news. It has far more to do with the rise of distributed computing than the existence of GPUs.

I don’t really see this; All of the currently popular machine learning frameworks either support GPU or are oriented around GPU based execution. For most developers working on AI it is still the defacto standard.

I think.. both. Distributed computing (Hadoop) was why a lot of data got collected and made available(ish).

GPU's are the engines that made CNN's (in particular) tractable, and opened up a bunch of applications for many companies, and opened up a reasonable route to results for a generation of researchers.

Re: The post-exponential era of AI and Moore’s Law

#36

Post by WizardAustralis »

ChuckMcM wrote: post #2

The article discusses one of the impacts of compute performance growth rates slowing while compute demand (training AI models) is growing exponentially. The 'end of Moore's law' (as a measure of performance, not density) is probably the most significant "Tech" story of the next decade. Why? Because it is going to demand engineers who can write fast code over engineers who write code fast. A lot of frameworks and abst…

We will probably start to see this a lot in the video game consoles field. On the hardware front, looking at the early specs of the PS5 for instance show them emphasis faster storage access speed more so than added CPU/GPU/RAM.

Consoles were always more likely to see speed optimizations, but this had slid aside a bit in the last two decades.

Re: The post-exponential era of AI and Moore’s Law

#37

Post by sgt101 »

npo9 wrote: post #19
_bxg1 wrote: post #6

Honestly, this is a bit of a relief. All of the AI nightmare scenarios (be they the Terminator kind, or the more realistic hyper-empowerment-of-a-few-elites kind) rely on that exponential growth continuing unimpeded. If there's no exponential growth, there's no runaway AI that swiftly outpaces human understanding.

Let me fuel your paranoia a little. It’s likely that cloud computing will continue to see decreases in costs, which means the availability of computing will continue to raise after Mores Law has stopped giving. Quantum computers might play a role in increasing computational power in the next decade. Right now a lot of AI research is low hanging fruit recently made available with advances in CPU and GPU. After the adv…

>cloud computing will continue to see decreases in costs

cloud computing costs much more than on prem; it's available on demand, which is why it's attractive to some companies. For everyone else it's an outsourcing play that unlocks capex for revenue generation.

>Quantum computers might play a role in increasing computational power in the next decade.

Nope. No chance. Best case is that QC will provide improved molecular simulations in 10-20 years. But that will be a very specialised result.

>Right now a lot of AI research is low hanging fruit recently made available with advances in CPU and GPU. After the advances dry up maybe more fundamental research will produce much better results with our current technology.

100%, but the key word is maybe. One problem I anticipate is that the current skill set of grad students and the current management mind set in academia and commercial research is not well fitted to the kind of fundamental research that includes maybe and risk. I think that a more likely turn is that there will be a focus on how to better exploit the tech for data science and ML that we have. Currently we're pretty hit and miss and also the results are rather unpredictable and brittle. When did you last find an ML project on Github with unit tests?

Re: The post-exponential era of AI and Moore’s Law

#38

Post by coldtea »

ChuckMcM wrote: post #2

The article discusses one of the impacts of compute performance growth rates slowing while compute demand (training AI models) is growing exponentially. The 'end of Moore's law' (as a measure of performance, not density) is probably the most significant "Tech" story of the next decade. Why? Because it is going to demand engineers who can write fast code over engineers who write code fast. A lot of frameworks and abst…

>Why? Because it is going to demand engineers who can write fast code over engineers who write code fast.

Or just making slower things more orthogonally parallelized and throwing more hardware units at the problem (as opposed to more powerful hardware or faster code).

Most code by engineers "who write code fast" is not in performance critical domains...

Re: The post-exponential era of AI and Moore’s Law

#39

Post by martinpw »

npo9 wrote: post #32
martinpw wrote: post #26

Earlier quoted context omitted.

> It’s likely that cloud computing will continue to see decreases in costs, which means the availability of computing will continue to raise after Mores Law has stopped giving. Why would cloud computing continue to see significant decreases in costs once Moore's law stops? Clearly there will be some ongoing decline in infrastructure costs, but if your main requirement is for lots of compute (for AI) then it seems har…

I was speaking about linear decreases. Cheaper power, more refined manufacturing processes, and economy of scale benefits are where I expect some of the linear benefits to come from. We can also expect CPU manufactures to compete for awhile, getting increasingly marginal improvements. Linear improvements will get us there in the end.

> Linear improvements will get us there in the end.

No, they won't. That's the point. To quote directly from the article:

we’re talking about exponential rates of growth here, linear expense adjustments won’t move the needle.

Post reply on HN