AlphaCode as a dog speaking mediocre English • Hacker News Board

Re: AlphaCode as a dog speaking mediocre English

#101

Post by Jack000 » Sun, Feb 06, 2022, 4:54 PM UTC

I think this type of model will have a massive impact on the software industry. 99% of programming tasks in the wild don't involve any kind of algorithmic design, but are more like making a CRUD pattern, writing SQL queries etc. This kind of work is easier to automate but more difficult to source the training data. If and when these models are applied to more mundane problems, I'd expect immediately better performance and utility.

We're also in the very very early days of code generation models. Even I can see some ways to improve AlphaCode:

- the generate->cluster->test process feels like a form of manual feature engineering. This meta layer should be learned as well, possibly with RL

- programming is inherently compositional. Ideally it should perform the generate->cluster->test step for each function and hierarchically build up the whole program, instead of in a single step as it does now

- source code is really meant for humans to read. The canonical form of software is more like the object code produced by the compiler. You can probably just produce this directly

Re: AlphaCode as a dog speaking mediocre English

#102

Post by CTmystery » Sun, Feb 06, 2022, 4:56 PM UTC

stephc_int13 wrote: post #12

zarzavat wrote: post #7

Earlier quoted context omitted.

> The susceptibility of these systems to completely failing on weird inputs shows that it's a kind of massive memorization going on here, rather than true understanding. Isn’t that a truism? “Understanding” is equivalent to AGI. Nobody would argue that we have AGI yet, but the intelligence-sans-understanding is somewhat similar to animal intelligence, which was the precursor to human intelligence. What should scare u…

>What should scare us is that we know that animal to human was not a difficult step evolutionarily. Do we know that? If that is not difficult, why is it so rare?

Perhaps most of us in this forum have only the most basic exposure to biology? I say this because with any serious exposure you can't help but be dumbfounded by the complexity and "creativity" of nature. Everything is amazing! The fact that the pieces fit together is just wild and scary and awesome. To think that gradient descent and backprop will give us general intelligence, one of the great mysteries of nature, is incredibly hubristic, IMO. It's statistical inference at scale, with some heuristics up/downstream. It's a cool and useful tool for sure!

Re: AlphaCode as a dog speaking mediocre English

#103

Post by rakejake » Sun, Feb 06, 2022, 4:57 PM UTC

kgeist wrote: post #82

omnicognate wrote: post #18

As someone who is skeptical, but open minded, about the impact these technologies will have on practical programming I think I'm one of the "people" in "people are complaining..." The article makes some assumptions about what such people think that certainly aren't true for me: 1. That we are unimpressed. I'm gobsmacked. 2. That we don't think these are significant advances. They're obviously huge advances. 3. That w…

>The problem is that doing so is, I expect, harder for the human than writing the code in the first place. Programming is mostly about writing boilerplate code using well-known architectural patterns and technologies, nothing extraordinary but which takes time (at least in my experience). If I can describe a project in a few abstract words, and the AI generates the rest, it can considerably improve my productivity, a…

If a junior dev writes truly head-scratching code, you could ping that person and ask why they wrote this line a certain way, as opposed to a more straight-forward way. Correct me if I'm wrong but you can't ask an ML model to do that (yet).

Re: AlphaCode as a dog speaking mediocre English

#104

Post by sdenton4 » Sun, Feb 06, 2022, 5:03 PM UTC

stephc_int13 wrote: post #94

mrshadowgoose wrote: post #51

Earlier quoted context omitted.

> The machine learning techniques that were developed and enhanced during the last decade are not magical, like any other machines/software. You might be using a different definition of "magical" than what others are using in this context. Of course, when you break down ML techniques, it's all just math running on FETs. So no, it's not extra-dimensional hocus pocus, but absolutely nobody is using that particular defi…

I strongly disagree that we've seen anything unexpected so far. AlphaGo is nothing else than brute force. And brute force can go a long way, it should not be underestimated. But so far, this approach has not let to emergent behaviors, the ML blackbox is not giving back more than what was fed.

AlphaGo is decidedly not brute force, under any meaningful definition of the term. It's monte carlo tree search, augmented by a neutral network to give stronger priors on which branches are worth exploring. There is an explore/exploit trade-off to manage, which takes it out of the realm of brute force. The previous best go programs used Monte Carlo tree search alone, or with worse heuristics for the priors. Alpha Go improves drastically on the priors, which is arguably exactly the part of the problem that one would attribute to understanding the game: Of the available moves, which ones look the best?

They used a fantastic amount of compute for their solution, but, as has uniformly been the case for neutral networks, the compute required for both training and inference has dropped rapidly after the initial research result.

Re: AlphaCode as a dog speaking mediocre English

#105

Post by keewee7 » Sun, Feb 06, 2022, 5:04 PM UTC

Jack000 wrote: post #101

I think this type of model will have a massive impact on the software industry. 99% of programming tasks in the wild don't involve any kind of algorithmic design, but are more like making a CRUD pattern, writing SQL queries etc. This kind of work is easier to automate but more difficult to source the training data. If and when these models are applied to more mundane problems, I'd expect immediately better performanc…

It's interesting that AI is being aggressively applied to areas where AI practitioners are domain experts. Think programming, data analysis etc.

We programmers and data scientists might find ourselves among the first half of knowledge workers to be replaced and not among the last as we previously thought.

Re: AlphaCode as a dog speaking mediocre English

#106

Post by vain_cain » Sun, Feb 06, 2022, 5:04 PM UTC

mark_l_watson wrote: post #38

Right on, I couldn’t agree more. We are living during a period of exponential progress. I like AlphaCode’s approach of using language models with search. In the last year I have experimented with mixing language models for NLP with semantic web/linked data tasks, so much simpler than what AlphaCode does, but I have been having fun. I have added examples for this in new additions to two of my books, but if you want a…

I mostly work with data mining on my personal projects(which is a couple of hours every day), and I'm pretty sure I didn't have to write a single regex since I've started using Copilot. It's hard for me to even imagine how I used to do it before, and how much time I've wasted on stupid mistakes and typos. Now I just write a comment of what I want and an example string. It does the job without me having to modify anything 99% of the time, even for complex stuff. Sure, sometimes it gives an overly complicated solution but it almost always works. Exciting times.

Re: AlphaCode as a dog speaking mediocre English

#107

Post by axg11 » Sun, Feb 06, 2022, 5:05 PM UTC

stephc_int13 wrote: post #98

mark_l_watson wrote: post #38

Right on, I couldn’t agree more. We are living during a period of exponential progress. I like AlphaCode’s approach of using language models with search. In the last year I have experimented with mixing language models for NLP with semantic web/linked data tasks, so much simpler than what AlphaCode does, but I have been having fun. I have added examples for this in new additions to two of my books, but if you want a…

We're not seeing exponential rates of progress, in my opinion. We've been on the steep part of an S-curve.

When you're on that curve, it's indistinguishable until you hit the plateau. We're in an era where AI is continuing to improve and has already surpassed a level that many people doubted was achievable. Nobody knows when that progress will plateau. It's entirely possible that we plateau _after_ surpassing human-level intelligence.

Re: AlphaCode as a dog speaking mediocre English

#108

Post by hrgiger » Sun, Feb 06, 2022, 5:07 PM UTC

gambiting wrote: post #43

ianbutler wrote: post #21

Earlier quoted context omitted.

Has it been over hyped? Some ML created in the last 8 years is in most major products now. It has been transformative even if you don’t see it, is informing most things you use. We’re not close to AGI but I’ve never heard an actual researcher make that claim or the orgs they work for. They just consistently show for the tasks they pick they beat most baselines and in a lot of cases humans. The models just don’t gener…

It's just that the day to day instances of "AI" that you might run into are nowhere near the level of hype they initially got. For instance all kinds of voice assistants are just DUMB. Like, so so bad they actively put people off using them, with countless examples of them failing at even the most basic queries. And the instances where they feel smart it looks like it's only because you actually hit a magic passphras…

I agree about the assistants that they are not as much as I would expect but also there are self driving cars heavily using a.i. even at the current state I am personally impressed or indirectly we get the help during pandemic for protein folding/ mRNA vaccine development [1] , I also remember a completed competition for the speeding up the delivery of cold storage mRNA vaccines to quickly figure out which ones could fail

[1] https://ai.plainenglish.io/how-ai-actually-helped-in-the-dev...

Re: AlphaCode as a dog speaking mediocre English

#109

Post by axg11 » Sun, Feb 06, 2022, 5:07 PM UTC

keewee7 wrote: post #105

Jack000 wrote: post #101

I think this type of model will have a massive impact on the software industry. 99% of programming tasks in the wild don't involve any kind of algorithmic design, but are more like making a CRUD pattern, writing SQL queries etc. This kind of work is easier to automate but more difficult to source the training data. If and when these models are applied to more mundane problems, I'd expect immediately better performanc…

It's interesting that AI is being aggressively applied to areas where AI practitioners are domain experts. Think programming, data analysis etc. We programmers and data scientists might find ourselves among the first half of knowledge workers to be replaced and not among the last as we previously thought.

Compilers didn't replace any jobs, they created more. Similarly, this type of AI-assisted programming will allow more people to program and make existing programmers more productive.

Re: AlphaCode as a dog speaking mediocre English

#110

Post by notahacker » Sun, Feb 06, 2022, 5:09 PM UTC

mrshadowgoose wrote: post #51

stephc_int13 wrote: post #10

Earlier quoted context omitted.

AI has been over-hyped, that's all. The machine learning techniques that were developed and enhanced during the last decade are not magical, like any other machines/software. But people have different and irrational expectations about AI.

> The machine learning techniques that were developed and enhanced during the last decade are not magical, like any other machines/software. You might be using a different definition of "magical" than what others are using in this context. Of course, when you break down ML techniques, it's all just math running on FETs. So no, it's not extra-dimensional hocus pocus, but absolutely nobody is using that particular defi…

Was Go ever "safe from ML" as opposed to "[then] state of the art can't even play Go without a handicap"? Seems like exactly the sort of thing ML should be good at; approximating Nash equilibrium responses in a perfect information game with a big search space (and humans setting a low bar as we're nowhere near finding an algorithmic or brute force solution). Is it really magical that computers running enough simulations exposes limitations to human Go theory (arguably one interesting lesson was that humans were so bad at playing that AlphaGoZero was better off not having its dataset biased by curated human play)? Yes, it's a clear step forward compared with only being able to beat humans at games which can be fully brute forced, or a pocket calculator being much faster and reliable than the average humans at arithmetic due to a simple, tractable architecture, but also one of the least magical-seeming applications given we already had the calculators and chess engines (especially compared with something like playing Jeopardy) unless you had unjustifiably strong priors about how special human Go theory was.

I think people are completely wrong to pooh pooh the utility of computers being better at search and calculations in an ever wider range of applied fields, but linking computers surpassing humans at more examples of those problems to certainty we'll synthetically replicate brain functionality we barely understand is the sort of stretch which is exactly why AGI-sceptics feel the need to point out that this is just a tool iterating through existing programs and sticking lines of code together until the program outputs the desired output, not evidence of reasoning in a more human-like way.