Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The post-exponential era of AI and Moore’s Law

techcrunch.com

Post reply on HN
Print view View original

111–120 of 150 posts

Re: The post-exponential era of AI and Moore’s Law

#111

Post by dragontamer »

streetcat1 wrote: post #95
skohan wrote: post #63

I predict that the slowing of Moore's law is going to make HPC and optimization in general a much more valuable skillset in the next few decades. In the past 15 years or so, we've been more or less happy to treat CPU cycles as a limitless resource, and as a result modern software stacks have a lot of fat in them. At the end of the era of free speed increases, trimming the fat is going to be a lot more important.

CPU cycles are limitless. Most of the time the CPU is waiting for the memory system.

We're talking about 16-bit floats, or even INT8 or INT4, as the basis of neural nets these days, pushing the problem back to CPU-cycles.

CNNs in particular recycle the same set of weights over-and-over again, fitting inside of the tiny caches (or shared-memory in GPUs), allowing for the compute-portion of the hardware to really work the data.

> CPU cycles are limitless. Most of the time the CPU is waiting for the memory system.

CPUs go out-of-order, deeply pipelines, and speculative so that they have work to do even while waiting for the memory system.

The typical CPU has over 200+ instructions in flight in parallel these days. (200+ sized reorder buffers and "shadow registers" to support this hidden parallelism), and that's split between two threads for better efficiency ("Hyperthreading").

GPUs can have 8x warps / wavefronts per SM (NVidia) or CU (AMD) waiting for memory. If one warp/wavefront (a group of 32 or 64 threads) is waiting for memory, the GPU will switch to another "ready to run" warp/wavefront.

It takes some programmer effort to understand this process and write high-performance code. But its doable with some practice.

Re: The post-exponential era of AI and Moore’s Law

#112

Post by clarry »

fouc wrote: post #8
cft wrote: post #3

Earlier quoted context omitted.

Rust will fare well. Ruby won't.

Ruby isn't in the critical path, so that's a bad example. In general, the real issue is the layers of abstractions between the hardware and the end user. The over complicated architectures. The constant re-invention of databases/operating systems/virtual machines at each layer.

> The constant re-invention of databases/operating systems/virtual machines

I wish that were the case but operating systems and systems software in general seems to be the most stagnant field. Everyone just buys the same FLOSS stack for $0 and compatibility is king, so there's very little research going in this field and much less of it ends up in any product you're ever going to see in use.

I would love to see a vibrant scene of competing operating systems and databases with fresh new ideas and research.

Re: The post-exponential era of AI and Moore’s Law

#113

Post by AnimalMuppet »

TomMarius wrote: post #80
TheOtherHobbes wrote: post #72

Earlier quoted context omitted.

Which is dead wrong, because not only is there no usable definition of "improve itself", but there isn't even any understanding of the kinds of skills required to create a usable definition. It's the difference between a computer that is taught how to compose okay-ish music, and a computer that learns spontaneously how to compose really really great music and do all of the social, cultural, and financial things requi…

> Which is dead wrong, because not only is there no usable definition of "improve itself", but there isn't even any understanding of the kinds of skills required to create a usable definition. I disagree. A program improves itself when it reacts to a problem and implements a solution. Obviously that is very general, but enough. A human of IQ 100 certainly can develop software; a program of IQ 100 should be able to do…

A human with IQ 100 can develop software. Can they develop it well enough to improve AI software? Or can they just adequately develop general software?

Can a human with IQ 100 write general AI software that a human with IQ 100 can debug?

Re: The post-exponential era of AI and Moore’s Law

#114

Post by clarry »

hyperpallium wrote: post #86

Why can't we just have bigger dies? Same density, more transistors. If yields are too low, connect smaller dies somehow.

If you stop scaling node size down and start scaling die size up, the result will be higher prices and power consumption. If you don't mind, go buy a Threadripper or EPYC. Might consider getting multiple sockets while at it. Caveat: won't fit in your pocket.

I'm sure there'll be additional interconnect issues to worry about as the physical cluster of dies gets larger and larger.

Re: The post-exponential era of AI and Moore’s Law

#115

Post by AnimalMuppet »

TomMarius wrote: post #53
mikorym wrote: post #44

Earlier quoted context omitted.

Mathematician here, I don't think the singularity as popularised through people like Ray Kurzweil would happen. In particular, their metaphor of an "infinity" point seems to me like the incorrect application of mathematical ideas to social contexts. I think the question of artificial general intelligence is a different question than a singularity, much like how the Chinese room addresses a different question than the…

Why is it a different question? AFAIK the assumption is that if the computer can improve itself (which it can if it has IQ 100), there is nothing holding the singularity back.

See my comment below about whether IQ 100 is enough. But also, there still can be plenty holding the singularity back.

Remember Zeno's paradox. "Improving" could mean "incrementally approaching a limit". It doesn't necessarily mean "marching off to infinity".

Then, there's the matter of hardware. An infinitely smart AI running on finite hardware seems likely to be a contradiction. Is the AI going to be able to not just improve its code, but improve its hardware? Without depending on slow, ignorant humans, who might even be having second thoughts about the wisdom of giving this AI even more hardware? That takes considerably more than IQ 100; that takes robots carrying out the AI's wishes, which some factory needs to make. And an IQ of 100 probably isn't going to cut it to hack all the factory control systems, to take them over and have them start producing what the AI wants.

Re: The post-exponential era of AI and Moore’s Law

#116

Post by [deleted] »

Shorel wrote: post #82
sprash wrote: post #56

I predict there will be much more assembly programming required in the future to squeeze out as much as performance as possible because the end of Moore's Law is already very apparent for several applications that can not be easily parallelized. It is a complete myth that you "can't beat the C-compiler" as it is claimed so often. The compiler can't know many things you know about the problem at hand. So far I have be…

"Right now I'm looking for a C/Shader-like language for x86_64/Linux" You seem like the ideal candidate for writing such a language. And the book about it.

[deleted]

Re: The post-exponential era of AI and Moore’s Law

#117

Post by brudgers »

coldtea wrote: post #93
brudgers wrote: post #65

Earlier quoted context omitted.

I agree mostly. But how long can engineers throw more and more and exponentially more hardware at the phone that fits in my pocket?

> But how long can engineers throw more and more and exponentially more hardware at the phone that fits in my pocket? Why would they need to? They can sell improved cameras and other additions instead of improved CPUs...

I agree there's no need. What phone camera improvements do you envision that don't involve more transistors on the CMOS sensor, GPU, CPU etc? Microfilm?

Re: The post-exponential era of AI and Moore’s Law

#118

Post by streetcat1 »

dragontamer wrote: post #111
streetcat1 wrote: post #95

Earlier quoted context omitted.

CPU cycles are limitless. Most of the time the CPU is waiting for the memory system.

We're talking about 16-bit floats, or even INT8 or INT4, as the basis of neural nets these days, pushing the problem back to CPU-cycles. CNNs in particular recycle the same set of weights over-and-over again, fitting inside of the tiny caches (or shared-memory in GPUs), allowing for the compute-portion of the hardware to really work the data. > CPU cycles are limitless. Most of the time the CPU is waiting for the mem…

So I assume that we are talking about Moores law for cpu/gpu and not for memory.

The bottleneck for GPU today is the amount of memory on board and not compute. Especially with large size models.

In addition, the problem with deep learning, in general, is the seq nature of the alg. I.e it is parallel within the layer, but not between layers. And for multi gpu setup, again it is the communication link between the GPUs.

So I think that the nature of the current state of the art optimization alg are what matter.

Re: The post-exponential era of AI and Moore’s Law

#119

Post by ativzzz »

DennisP wrote: post #68
ivalm wrote: post #9

Earlier quoted context omitted.

I don't have a hat in the game but why won't Ruby do well or not well based on this? It's not like basic webapps need to be much more efficient, and no one was writing high performance compute in ruby anyways.

Looking over the techempower benchmarks, the fastest web frameworks are over fifty times faster than ruby/rails. If you can replace a hundred web servers with two, that matters.

The vast majority of web apps are CRUD apps, which can achieve sub 100ms responses on any language with caching, background workers and minor optimizations.

There will always be problems where you need faster responses, and the solutions to those problems are already not using Ruby. You don't use a hammer when a screwdriver is appropriate, but nails work pretty well in most places and screws are unnecessary.

Re: The post-exponential era of AI and Moore’s Law

#120

Post by djokkataja »

YeGoblynQueenne wrote: post #59
m3at wrote: post #10

The "Bitter Lesson" post from Rich Sutton from earlier this year [1] seems a very good complement to this article: he explain how all of the big improvements in the field came from new methods that leveraged the much larger compute available from Moore's law, instead of progressive buildup over existing methods. A great quote from McCarthy also regularly referenced by Sutton is "Intelligence is the computational part…

Great read. To follow up also read a comment on that piece titled "A better lesson" by Rodney Brooks: https://rodneybrooks.com/a-better-lesson/

> One of the most celebrated successes of Deep Learning is image labeling, using CNNs, Convolutional Neural Networks, but the very essence of CNNs is that the front end of the network is designed by humans to manage translational invariance, the idea that objects can appear anywhere in the frame. To have a Deep Learning network also have to learn that seems pedantic to the extreme, and will drive up the computational costs of the learning by many orders of magnitude.

Thanks for the link, this is a pretty strong rebuttal.

That said...

> Massive data sets are not at all what humans need to learn things so something is missing. Today’s data sets can have billions of examples, where a human may only require a handful to learn the same thing.

I've come across this idea before, the notion that we should be able to create an AI that can learn something from just a handful of examples because humans can do that. This assumes some rough level of equality in lack of related knowledge / capability between a tabula rasa AI system and a human adult. But a human adult has been awake and continuously learning & testing conscious and unconscious predictions about everything they interact with for at least 18 years (1 year = 31536000 seconds * 2/3 (accounting for sleep) = 21024000 seconds awake per year, * 18 years = 378 432 000 seconds). In other words, a human adult is already a highly trained system. And even very young human children have been awake and learning for quite a long time compared to anything we might imagine to be a tabula rasa system.

So I don't think that the human capacity for learning supports the idea that a tabula rasa AI system should be able to learn any particular thing with just a handful of examples.

Post reply on HN