Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The post-exponential era of AI and Moore’s Law

techcrunch.com

Post reply on HN
Print view View original

131–140 of 150 posts

Re: The post-exponential era of AI and Moore’s Law

#131

Post by codesushi42 »

solidasparagus wrote: post #128
codesushi42 wrote: post #23

Earlier quoted context omitted.

the resurgence of ML ... because video games - of all things - funded the maturation of a whole new category of massively-parallel chips Fake news. It has far more to do with the rise of distributed computing than the existence of GPUs.

I don't think this is very true. You can trace the deep learning revolution back to VGG and a fundamental driver in the success of the first multi-level networks was the ability to train in semi-reasonable amounts of time using GPUs. Even today distributed training is relatively uncommon while pretty much everyone uses NVIDIA GPUs.

VGG was a long time after the first deep learning revolution.

Re: The post-exponential era of AI and Moore’s Law

#132

Post by AnimalMuppet »

Andrew_nenakhov wrote: post #130
AnimalMuppet wrote: post #121

Earlier quoted context omitted.

> Why couldn't it just change batteries as needed, or just be wired up to power for that matter? Because the human it's playing can't. In a sport where endurance is a significant component, wiring one competitor up to power, while the other has to use only the fuel that he/she has stored, doesn't seem like an equitable competition.

Are you suggesting to ban human players from drinking water during the match?

No, I wasn't. I don't think that a drink of water is the same as a battery swap.

Re: The post-exponential era of AI and Moore’s Law

#133

Post by DennisP »

ativzzz wrote: post #119
DennisP wrote: post #68

Earlier quoted context omitted.

Looking over the techempower benchmarks, the fastest web frameworks are over fifty times faster than ruby/rails. If you can replace a hundred web servers with two, that matters.

The vast majority of web apps are CRUD apps, which can achieve sub 100ms responses on any language with caching, background workers and minor optimizations. There will always be problems where you need faster responses, and the solutions to those problems are already not using Ruby. You don't use a hammer when a screwdriver is appropriate, but nails work pretty well in most places and screws are unnecessary.

100ms with how many responses per second? Throughput matters too. And techempower does some pretty realistic-looking benchmarks. For example:

> In this test, the framework's ORM is used to fetch all rows from a database table containing an unknown number of Unix fortune cookie messages (the table has 12 rows, but the code cannot have foreknowledge of the table's size). An additional fortune cookie message is inserted into the list at runtime and then the list is sorted by the message text. Finally, the list is delivered to the client using a server-side HTML template. The message text must be considered untrusted and properly escaped and the UTF-8 fortune messages must be rendered properly.

https://www.techempower.com/benchmarks/

Re: The post-exponential era of AI and Moore’s Law

#134

Post by pmoriarty »

AnimalMuppet wrote: post #121
pmoriarty wrote: post #110

Earlier quoted context omitted.

Why couldn't it just change batteries as needed, or just be wired up to power for that matter? I'd expect something like wrestling would be much more challenging than non-contact sports, assuming, of course, that the robot's strength and torque was limited to human levels. Wrestling is actually a pretty interesting case, now that I think about it for a bit, as robots wouldn't need to be limited to human-like bodies.…

> Why couldn't it just change batteries as needed, or just be wired up to power for that matter? Because the human it's playing can't. In a sport where endurance is a significant component, wiring one competitor up to power, while the other has to use only the fuel that he/she has stored, doesn't seem like an equitable competition.

Humans make the rules, and it's doubtless that some human's sense of fairness will determine what rules they'd compete under, rather than simply letting the robot compete using its full capability, which we can already see will be superior in many ways.

Re: The post-exponential era of AI and Moore’s Law

#135

Post by vagab0nd »

> A couple of years ago I was talking to the CEO of an AI company who argued that AI progress was basically an S-curve, and we had already reached its top for sound processing, were nearing it for image and video, but were only halfway up the curve for text. No prize for guessing which one his company specialized in — but he seems to have been entirely correct.

This feels a little off. To me, it feels like image has made the most progress, then text, then sound, then video.

Anyone knows which company they were referring to?

Re: The post-exponential era of AI and Moore’s Law

#136

Post by YeGoblynQueenne »

djokkataja wrote: post #120
YeGoblynQueenne wrote: post #59

Earlier quoted context omitted.

Great read. To follow up also read a comment on that piece titled "A better lesson" by Rodney Brooks: https://rodneybrooks.com/a-better-lesson/

> One of the most celebrated successes of Deep Learning is image labeling, using CNNs, Convolutional Neural Networks, but the very essence of CNNs is that the front end of the network is designed by humans to manage translational invariance, the idea that objects can appear anywhere in the frame. To have a Deep Learning network also have to learn that seems pedantic to the extreme, and will drive up the computational…

>> So I don't think that the human capacity for learning supports the idea that a tabula rasa AI system should be able to learn any particular thing with just a handful of examples.

I don't know if Brooks is talking about learning everything from scratch here. I for one don't think that humans come into the world as tabulae rasae. I think instead that we come endowed with a rich knowledge of the world already, as well as strong inductive biases that allow us to learn new concepts from a combination of existing knowledge and new observations, with our characteristically high sample efficiency that remains unmatched by neural nets.

For instance, one theory about our ability to learn human language as infants is that we are born with a language endowment, Chomsky's "universal grammar", that predisposes us to pick out and learn human language, without having to figure it out from scratch.

I wonder then if what Brooks is saying is missing from neural nets is the ability to use background knowledge and strong inductive biases as effectively and efficiently as we do, so that they don't need hundreds of thousands of examples before they can learn a new concept.

Re: The post-exponential era of AI and Moore’s Law

#137

Post by AnimalMuppet »

pmoriarty wrote: post #134
AnimalMuppet wrote: post #121

Earlier quoted context omitted.

> Why couldn't it just change batteries as needed, or just be wired up to power for that matter? Because the human it's playing can't. In a sport where endurance is a significant component, wiring one competitor up to power, while the other has to use only the fuel that he/she has stored, doesn't seem like an equitable competition.

Humans make the rules, and it's doubtless that some human's sense of fairness will determine what rules they'd compete under, rather than simply letting the robot compete using its full capability, which we can already see will be superior in many ways.

Well, a human with a finite energy capacity against a robot with an infinite energy capacity (because it has a feed) is clearly an unequal match on that basis.

Similarly, a human player against a robot that spans the entire width of the court, and that has 48 rackets, is clearly not an equal match on that basis.

In the same way, we don't race Formula 1 cars against NASCAR.

Re: The post-exponential era of AI and Moore’s Law

#138

Post by fouc »

clarry wrote: post #112
fouc wrote: post #8

Earlier quoted context omitted.

Ruby isn't in the critical path, so that's a bad example. In general, the real issue is the layers of abstractions between the hardware and the end user. The over complicated architectures. The constant re-invention of databases/operating systems/virtual machines at each layer.

> The constant re-invention of databases/operating systems/virtual machines I wish that were the case but operating systems and systems software in general seems to be the most stagnant field. Everyone just buys the same FLOSS stack for $0 and compatibility is king, so there's very little research going in this field and much less of it ends up in any product you're ever going to see in use. I would love to see a vib…

I meant that it is re-implemented at every layer.. Like a browser is practically an OS on its own.

Re: The post-exponential era of AI and Moore’s Law

#139

Post by codesushi42 »

fspeech wrote: post #99
unkulunkulu wrote: post #69

Earlier quoted context omitted.

Can anybody elaborate on why this is downvoted? This would be my guess as well, simd parallelism of GPUs solves only part of the challenges, you still need a general purpose data crunching machine to prepare and handle learning data.

For one GPU speedup over CPU isn’t that dramatic for small to medium sized problems, e.g. MNIST or CIFAR that one would try algorithm ideas on. So I think it’s a stretch to see GPU as essential to the new algorithms. On the other hand for large problems like the original Alpha Go you need to figure out the distributed computing to really scale. This isn’t to say that GPUs aren’t nice. They do save time or for the sam…

Exactly. GPUs are necessary now, but did not originally herald the deep learning revolution.

Re: The post-exponential era of AI and Moore’s Law

#140

Post by carlmr »

brazzy wrote: post #61
carlmr wrote: post #60

Earlier quoted context omitted.

Nobody said that. Just that you should think about which algorithms you use, profile, and not waste so much. The wasted power at this point is significant.

That is sensible, but I've also heard people basically call any code that is not micro-optimized wasteful and harp on about how back in the day it was second nature for programmers to care about bytes and cycles in every line of code they wrote, and modern day programmers are just lazy and incompetent.

Yeah, I get where you're coming from. There's as always a sweet spot between the two extremes. But needing a new computer for running Word or Excel smoothly just seems like there's a lot of low hanging fruit left.
Post reply on HN