Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The post-exponential era of AI and Moore’s Law

techcrunch.com

Post reply on HN
Print view View original

91–100 of 150 posts

Re: The post-exponential era of AI and Moore’s Law

#91

Post by oblio »

csomar wrote: post #58
cft wrote: post #3

Earlier quoted context omitted.

Rust will fare well. Ruby won't.

That has a lot to do with why I took on learning Rust. My last year Macbook Pro has a 6-core CPU. At this rate, we should have 32-64-core CPUs for consumers in a few years. Not using all the processors means your program only makes use of 3-1.5% of the available power. My prediction is that, in a few years, lots of programming languages and software developers will simply become obsolete because what the other guys c…

I hope you're right.

However Intel Core Duo was released in January 2006 and it had 2 cores. So in a bit more than 13 years we only have 3x as many cores. Core 2 Quad was released in January 2007, so if you count from that one, a bit more than 12 years for 1.5x as many cores.

I wouldn't bet on widespread 64 core CPUs for consumers in the next 10 years or so.

Most commercial software just doesn't use it efficiently and consumers don't need it for existing software.

We need a software breakthrough, a killer app, something that people really want and that really needs a ton of cores.

For now, I don't see it. But then again, that's how killer apps are, nobody sees them until they're there.

Re: The post-exponential era of AI and Moore’s Law

#92

Post by oblio »

fouc wrote: post #8
cft wrote: post #3

Earlier quoted context omitted.

Rust will fare well. Ruby won't.

Ruby isn't in the critical path, so that's a bad example. In general, the real issue is the layers of abstractions between the hardware and the end user. The over complicated architectures. The constant re-invention of databases/operating systems/virtual machines at each layer.

> The constant re-invention of databases/operating systems/virtual machines at each layer.

Those are because of the human struggle, the political and economic fights. They're not because of purely technical problems.

Each layer is usually backed by a party with a very clear interest, usually control, and each layer on top tries to escape that control. It's been that way since the times of p-code.

Re: The post-exponential era of AI and Moore’s Law

#93

Post by coldtea »

brudgers wrote: post #65
coldtea wrote: post #38

Earlier quoted context omitted.

> Why? Because it is going to demand engineers who can write fast code over engineers who write code fast. Or just making slower things more orthogonally parallelized and throwing more hardware units at the problem (as opposed to more powerful hardware or faster code). Most code by engineers "who write code fast" is not in performance critical domains...

I agree mostly. But how long can engineers throw more and more and exponentially more hardware at the phone that fits in my pocket?

>But how long can engineers throw more and more and exponentially more hardware at the phone that fits in my pocket?

Why would they need to? They can sell improved cameras and other additions instead of improved CPUs...

Re: The post-exponential era of AI and Moore’s Law

#94

Post by streetcat1 »

ChuckMcM wrote: post #2

The article discusses one of the impacts of compute performance growth rates slowing while compute demand (training AI models) is growing exponentially. The 'end of Moore's law' (as a measure of performance, not density) is probably the most significant "Tech" story of the next decade. Why? Because it is going to demand engineers who can write fast code over engineers who write code fast. A lot of frameworks and abst…

Correct. As long as your bottleneck is the CPU.

Re: The post-exponential era of AI and Moore’s Law

#95

Post by streetcat1 »

skohan wrote: post #63

I predict that the slowing of Moore's law is going to make HPC and optimization in general a much more valuable skillset in the next few decades. In the past 15 years or so, we've been more or less happy to treat CPU cycles as a limitless resource, and as a result modern software stacks have a lot of fat in them. At the end of the era of free speed increases, trimming the fat is going to be a lot more important.

CPU cycles are limitless. Most of the time the CPU is waiting for the memory system.

Re: The post-exponential era of AI and Moore’s Law

#96

Post by thfuran »

TomMarius wrote: post #80
TheOtherHobbes wrote: post #72

Earlier quoted context omitted.

Which is dead wrong, because not only is there no usable definition of "improve itself", but there isn't even any understanding of the kinds of skills required to create a usable definition. It's the difference between a computer that is taught how to compose okay-ish music, and a computer that learns spontaneously how to compose really really great music and do all of the social, cultural, and financial things requi…

> Which is dead wrong, because not only is there no usable definition of "improve itself", but there isn't even any understanding of the kinds of skills required to create a usable definition. I disagree. A program improves itself when it reacts to a problem and implements a solution. Obviously that is very general, but enough. A human of IQ 100 certainly can develop software; a program of IQ 100 should be able to do…

Have you taken an IQ test? They only test a few classes of problem. Performance on these for a human is deemed a workable poxy for intelligence but, for something that approaches the problems very differently, it may not be at all indicative of general intelligence. I think we probably already have reached or are near the point where we could train systems to achieve human-level performance on each of those basic tasks. We are, however, not seemingly near a humanlike AGI.

Re: The post-exponential era of AI and Moore’s Law

#97

Post by codesushi42 »

jchw wrote: post #34
codesushi42 wrote: post #23

Earlier quoted context omitted.

the resurgence of ML ... because video games - of all things - funded the maturation of a whole new category of massively-parallel chips Fake news. It has far more to do with the rise of distributed computing than the existence of GPUs.

I don’t really see this; All of the currently popular machine learning frameworks either support GPU or are oriented around GPU based execution. For most developers working on AI it is still the defacto standard.

Except that's "currently". NNs can and were trained on CPUs originally.

Re: The post-exponential era of AI and Moore’s Law

#98

Post by zozbot234 »

streetcat1 wrote: post #95
skohan wrote: post #63

I predict that the slowing of Moore's law is going to make HPC and optimization in general a much more valuable skillset in the next few decades. In the past 15 years or so, we've been more or less happy to treat CPU cycles as a limitless resource, and as a result modern software stacks have a lot of fat in them. At the end of the era of free speed increases, trimming the fat is going to be a lot more important.

CPU cycles are limitless. Most of the time the CPU is waiting for the memory system.

True enough, but modern software stacks also have a lot of wasteful memory use.

Re: The post-exponential era of AI and Moore’s Law

#99

Post by fspeech »

unkulunkulu wrote: post #69
sgt101 wrote: post #35

Earlier quoted context omitted.

I think.. both. Distributed computing (Hadoop) was why a lot of data got collected and made available(ish). GPU's are the engines that made CNN's (in particular) tractable, and opened up a bunch of applications for many companies, and opened up a reasonable route to results for a generation of researchers.

Can anybody elaborate on why this is downvoted? This would be my guess as well, simd parallelism of GPUs solves only part of the challenges, you still need a general purpose data crunching machine to prepare and handle learning data.

For one GPU speedup over CPU isn’t that dramatic for small to medium sized problems, e.g. MNIST or CIFAR that one would try algorithm ideas on. So I think it’s a stretch to see GPU as essential to the new algorithms. On the other hand for large problems like the original Alpha Go you need to figure out the distributed computing to really scale.

This isn’t to say that GPUs aren’t nice. They do save time or for the same amount of time let you produce more polished results, which means in a competitive environment everyone would use them.

Re: The post-exponential era of AI and Moore’s Law

#100

Post by streetcat1 »

zozbot234 wrote: post #98
streetcat1 wrote: post #95

Earlier quoted context omitted.

CPU cycles are limitless. Most of the time the CPU is waiting for the memory system.

True enough, but modern software stacks also have a lot of wasteful memory use.

Right. Hence Moore's law is irrelevant. I.e. the CPU is NOT the bottleneck.

For example, who cares if you have 64 core , 5Ghz cpu, if the still doing I/O from an HDD.

Post reply on HN