Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AlphaCode as a dog speaking mediocre English

scottaaronson.blog

Post reply on HN
Print view View original

21–30 of 263 posts

Re: AlphaCode as a dog speaking mediocre English

#21

Post by ianbutler »

stephc_int13 wrote: post #10
asdfasgasdgasdg wrote: post #5

Earlier quoted context omitted.

It's rarely productive to take internet criticism into account, but it feels like AI is an especially strong instance of this. It seems like a lot of folks just want to pooh pooh any possible outcome. I'm not sure why this is. Possibly because of animosity toward big tech, given big tech is driving a lot of the research and practical implementation in this area?

AI has been over-hyped, that's all. The machine learning techniques that were developed and enhanced during the last decade are not magical, like any other machines/software. But people have different and irrational expectations about AI.

Has it been over hyped? Some ML created in the last 8 years is in most major products now. It has been transformative even if you don’t see it, is informing most things you use. We’re not close to AGI but I’ve never heard an actual researcher make that claim or the orgs they work for. They just consistently show for the tasks they pick they beat most baselines and in a lot of cases humans. The models just don’t generalize but with transformers we’re able to get them to perform above baselines for multiple problems and that’s the excitement. I’m not sure who has overhyped it for you but it’s delivering in line with my expectations ever since the first breakthrough in 2013/2014 that let neural nets actually work.

Re: AlphaCode as a dog speaking mediocre English

#22

Post by qualudeheart »

turminal wrote: post #9

It would be interesting to know how much of this improvement in the last 25 since he was a student comes from Moore's law, other hardware improvements, various new technologies not related to AI, amount of money being thrown at the problem... and how much of it are advancements in our understanding of AI.

It’s mostly Moores law and scaling. Language model progress will be the next Moores law until diminishing returns are reached.

Re: AlphaCode as a dog speaking mediocre English

#23

Post by omnicognate »

ogogmad wrote: post #19
omnicognate wrote: post #18

As someone who is skeptical, but open minded, about the impact these technologies will have on practical programming I think I'm one of the "people" in "people are complaining..." The article makes some assumptions about what such people think that certainly aren't true for me: 1. That we are unimpressed. I'm gobsmacked. 2. That we don't think these are significant advances. They're obviously huge advances. 3. That w…

I think using these tools might become a science or an art form in its own right. You'll have to give these tools the input they need to produce the most useful answers to you. In the short term at least, this is not going to take away your need to think. But it might change how you think, and it might make you more productive when your problem aligns well with these tools.

Potentially, but that doesn't address the problem of replacing writing code with the (harder) process of reading, verifying and fixing it. However well I engineer my inputs I'll still have to review, verify and fix the outputs.

Re: AlphaCode as a dog speaking mediocre English

#24

Post by drooby »

I think I agree with this take. We barely even know how our own brains learn and solve problems. We don't know exactly what the inner workings of AGI will look like... maybe this kind of problem solving is the emergence of that, perhaps in a different way to our own, perhaps not.

Re: AlphaCode as a dog speaking mediocre English

#25

Post by sanxiyn »

stephc_int13 wrote: post #10
asdfasgasdgasdg wrote: post #5

Earlier quoted context omitted.

It's rarely productive to take internet criticism into account, but it feels like AI is an especially strong instance of this. It seems like a lot of folks just want to pooh pooh any possible outcome. I'm not sure why this is. Possibly because of animosity toward big tech, given big tech is driving a lot of the research and practical implementation in this area?

AI has been over-hyped, that's all. The machine learning techniques that were developed and enhanced during the last decade are not magical, like any other machines/software. But people have different and irrational expectations about AI.

I mean, it is magical, in a sense that we are not sure how and why it works.

Re: AlphaCode as a dog speaking mediocre English

#26

Post by amusedcyclist »

evrydayhustling wrote: post #3

I love this take. Most AI results provoke a torrent of articles listing pratfalls that prove it's not AGI. Of course it's not AGI! But it is as unexpected as a talking dog. Take a second to be amazed, at least amused. Then read how they did it and think about how to do better.

I mean I'm not so impressed, because it seems like someones figured out the ventriloquist trick and and is just spamming it to make anything talk. Its fun enough, but unclear what this is achieving

Re: AlphaCode as a dog speaking mediocre English

#27

Post by Isinlor »

turminal wrote: post #9

It would be interesting to know how much of this improvement in the last 25 since he was a student comes from Moore's law, other hardware improvements, various new technologies not related to AI, amount of money being thrown at the problem... and how much of it are advancements in our understanding of AI.

Roughly half/half, skewed more towards improvements in algorithms.

Measuring the Algorithmic Efficiency of Neural Networks: https://arxiv.org/abs/2005.04305

In this work, we argue that algorithmic progress has an aspect that is both straightforward to measure and interesting: reductions over time in the compute needed to reach past capabilities. We show that the number of floating-point operations required to train a classifier to AlexNet-level performance on ImageNet has decreased by a factor of 44x between 2012 and 2019. This corresponds to algorithmic efficiency doubling every 16 months over a period of 7 years. By contrast, Moore's Law would only have yielded an 11x cost improvement. We observe that hardware and algorithmic efficiency gains multiply and can be on a similar scale over meaningful horizons, which suggests that a good model of AI progress should integrate measures from both.

Mastering Atari Games with Limited Data: https://arxiv.org/abs/2111.00210

Reinforcement learning has achieved great success in many applications. However, sample efficiency remains a key challenge, with prominent methods requiring millions (or even billions) of environment steps to train. (...) This is the first time an algorithm achieves super-human performance on Atari games with such little data. EfficientZero's performance is also close to DQN's performance at 200 million frames while we consume 500 times less data. EfficientZero's low sample complexity and high performance can bring RL closer to real-world applicability.

500x improvement over ~10 years since DQN that roughly 2x improvement in sample complexity every year.

A Time Leap Challenge for SAT Solving: https://arxiv.org/abs/2008.02215

We compare the impact of hardware advancement and algorithm advancement for SAT solving over the last two decades. In particular, we compare 20-year-old SAT-solvers on new computer hardware with modern SAT-solvers on 20-year-old hardware. Our findings show that the progress on the algorithmic side has at least as much impact as the progress on the hardware side.

AI research has also tiny budgets compared to the biggest scientific projects:

GPT-3 - $0.01B

LIGO - $1B

LHC - $7.5B

JWST - $10B

ITER - $20B

ISS - $150B

Re: AlphaCode as a dog speaking mediocre English

#28

Post by udbhavs »

stephc_int13 wrote: post #10
asdfasgasdgasdg wrote: post #5

Earlier quoted context omitted.

It's rarely productive to take internet criticism into account, but it feels like AI is an especially strong instance of this. It seems like a lot of folks just want to pooh pooh any possible outcome. I'm not sure why this is. Possibly because of animosity toward big tech, given big tech is driving a lot of the research and practical implementation in this area?

AI has been over-hyped, that's all. The machine learning techniques that were developed and enhanced during the last decade are not magical, like any other machines/software. But people have different and irrational expectations about AI.

Most of the stuff featured on https://youtube.com/c/K%C3%A1rolyZsolnai looks pretty magical to me

Re: AlphaCode as a dog speaking mediocre English

#29

Post by derekja »

omnicognate wrote: post #23
ogogmad wrote: post #19

Earlier quoted context omitted.

I think using these tools might become a science or an art form in its own right. You'll have to give these tools the input they need to produce the most useful answers to you. In the short term at least, this is not going to take away your need to think. But it might change how you think, and it might make you more productive when your problem aligns well with these tools.

Potentially, but that doesn't address the problem of replacing writing code with the (harder) process of reading, verifying and fixing it. However well I engineer my inputs I'll still have to review, verify and fix the outputs.

This problem has been (albeit imperfectly) addressed in speech recognition. When error corrections are made through the the UI the engine can learn from those corrections. Presumably over time the corrections needed in alphacode will become more semantic than syntactical. But you're right, correcting subtly flawed code or text is way harder than writing from scratch.

Re: AlphaCode as a dog speaking mediocre English

#30

Post by qualudeheart »

Which rate of progress are we ITT expecting from alphacode-like models?

In another thread I predicted we’d see competitive programming “solved” in ten years or less.

I didn’t rigorously explain what I meant by that in that thread, so I’ll clarify what I meant.

I expect AI to beat humans at competitive programming at the same rate as AlphaGo beats human Go players.

There could be diminshing returns soon but I don’t see what would cause them.

Post reply on HN