Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The post-exponential era of AI and Moore’s Law

techcrunch.com

Post reply on HN
Print view View original

61–70 of 150 posts

Re: The post-exponential era of AI and Moore’s Law

#61

Post by brazzy »

carlmr wrote: post #60
brazzy wrote: post #57

Earlier quoted context omitted.

Premature optimization will never become the right thing to do. Most code has absolutely no reason to be particularly fast because it doesn't get executed very frequently.

Nobody said that. Just that you should think about which algorithms you use, profile, and not waste so much. The wasted power at this point is significant.

That is sensible, but I've also heard people basically call any code that is not micro-optimized wasteful and harp on about how back in the day it was second nature for programmers to care about bytes and cycles in every line of code they wrote, and modern day programmers are just lazy and incompetent.

Re: The post-exponential era of AI and Moore’s Law

#62

Post by jononor »

ilaksh wrote: post #54

General purpose AI is not waiting for more compute power. Its waiting for algorithms that can do it. I'm not 100% sure deep learning can get all the way there. We might need a new approach. But the deep learning leaders are taking it very seriously and making some progress. For example, Yann LeCun is talking about self-supervised, learning models of the world. MILA (Bengio's group) is talking about "state representat…

Yeah, many ML applications are bottlenecked on availability of labels, not by compute. Especially once outside very well-defined and established task. I think self-supervised has large potential, at least in expanding ML applications towards "more general".

Re: The post-exponential era of AI and Moore’s Law

#63

Post by skohan »

I predict that the slowing of Moore's law is going to make HPC and optimization in general a much more valuable skillset in the next few decades. In the past 15 years or so, we've been more or less happy to treat CPU cycles as a limitless resource, and as a result modern software stacks have a lot of fat in them. At the end of the era of free speed increases, trimming the fat is going to be a lot more important.

Re: The post-exponential era of AI and Moore’s Law

#64

Post by skohan »

foota wrote: post #11
ChuckMcM wrote: post #2

The article discusses one of the impacts of compute performance growth rates slowing while compute demand (training AI models) is growing exponentially. The 'end of Moore's law' (as a measure of performance, not density) is probably the most significant "Tech" story of the next decade. Why? Because it is going to demand engineers who can write fast code over engineers who write code fast. A lot of frameworks and abst…

I'm not sure that's a given, who's to say we won't be happy with the current performance we have?

There are still problems, like protein folding, or training larger and larger deep learning models, which have nowhere near the performance you'd actually want in a perfect world. There's always going to be a bigger target in some of these domains, and it would be a shame to imagine we'll reach a ceiling in some areas of scientific inquiry because of hardware limitations.

Re: The post-exponential era of AI and Moore’s Law

#65

Post by brudgers »

coldtea wrote: post #38
ChuckMcM wrote: post #2

The article discusses one of the impacts of compute performance growth rates slowing while compute demand (training AI models) is growing exponentially. The 'end of Moore's law' (as a measure of performance, not density) is probably the most significant "Tech" story of the next decade. Why? Because it is going to demand engineers who can write fast code over engineers who write code fast. A lot of frameworks and abst…

> Why? Because it is going to demand engineers who can write fast code over engineers who write code fast. Or just making slower things more orthogonally parallelized and throwing more hardware units at the problem (as opposed to more powerful hardware or faster code). Most code by engineers "who write code fast" is not in performance critical domains...

I agree mostly. But how long can engineers throw more and more and exponentially more hardware at the phone that fits in my pocket?

Re: The post-exponential era of AI and Moore’s Law

#66

Post by fyp »

Since AI is heavily parallelizable, it only matters that cost(as in dollars) will keep exponentially decreasing.

It doesn't matter if you can't double the transistor density of a single cpu if you can just double the number of machines. At the end of the day you still managed to double performance for the same price.

See https://en.wikipedia.org/wiki/FLOPS#Hardware_costs (note: in another thread someone noted that this wiki is outdated/inaccurate. If anyone have the relevant expertise they should help edit it)

Re: The post-exponential era of AI and Moore’s Law

#67

Post by 7thaccount »

fyp wrote: post #66

Since AI is heavily parallelizable, it only matters that cost(as in dollars) will keep exponentially decreasing. It doesn't matter if you can't double the transistor density of a single cpu if you can just double the number of machines. At the end of the day you still managed to double performance for the same price. See https://en.wikipedia.org/wiki/FLOPS#Hardware_costs (note: in another thread someone noted that th…

Some problems don't parallelize well (some mixed integer programming problems) and some matrix operations. We look at the end of Moore's law with horror.

Re: The post-exponential era of AI and Moore’s Law

#68

Post by DennisP »

ivalm wrote: post #9
cft wrote: post #3

Earlier quoted context omitted.

Rust will fare well. Ruby won't.

I don't have a hat in the game but why won't Ruby do well or not well based on this? It's not like basic webapps need to be much more efficient, and no one was writing high performance compute in ruby anyways.

Looking over the techempower benchmarks, the fastest web frameworks are over fifty times faster than ruby/rails. If you can replace a hundred web servers with two, that matters.

Re: The post-exponential era of AI and Moore’s Law

#69

Post by unkulunkulu »

sgt101 wrote: post #35
jchw wrote: post #34

Earlier quoted context omitted.

I don’t really see this; All of the currently popular machine learning frameworks either support GPU or are oriented around GPU based execution. For most developers working on AI it is still the defacto standard.

I think.. both. Distributed computing (Hadoop) was why a lot of data got collected and made available(ish). GPU's are the engines that made CNN's (in particular) tractable, and opened up a bunch of applications for many companies, and opened up a reasonable route to results for a generation of researchers.

Can anybody elaborate on why this is downvoted? This would be my guess as well, simd parallelism of GPUs solves only part of the challenges, you still need a general purpose data crunching machine to prepare and handle learning data.

Re: The post-exponential era of AI and Moore’s Law

#70

Post by pjmlp »

skohan wrote: post #63

I predict that the slowing of Moore's law is going to make HPC and optimization in general a much more valuable skillset in the next few decades. In the past 15 years or so, we've been more or less happy to treat CPU cycles as a limitless resource, and as a result modern software stacks have a lot of fat in them. At the end of the era of free speed increases, trimming the fat is going to be a lot more important.

You already see that ongoing with Java and .NET, refocusing on having AOT as part of the toolchain instead of some third party toolchain, more language features for lowlevel control and GPGPU access as well.

Or the pendulum going back to compiled languages after the scripting craziness of the last decade.

Post reply on HN