Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Competitive Programming with AlphaCode

deepmind.com

Post reply on HN
Print view View original

261–270 of 415 posts

Re: Competitive Programming with AlphaCode

#261

Post by YeGoblynQueenne »

qualudeheart wrote: post #243
YeGoblynQueenne wrote: post #221

Earlier quoted context omitted.

>> This is technology that simply didn't exist in any form 2 years ago. A few examples of neural program synthesis from at least 2 years ago: https://sunblaze-ucb.github.io/program-synthesis/index.html Another example from June 2020: DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning https://arxiv.org/abs/2006.08381 RobustFill, from 2017: RobustFill: Neural Program Le…

Has someone tried classical program synthesis techniques on competitive programming problems? I wonder what would have been possible with tech from more than 2 years ago.

I don't know if anyone has tried it, but it's not a very objective evaluation. We have no good measure of the coding ability of the "median level competitor" so doing better or worse than that, doesn't really tell us anything useful about the coding capability of an automated system.

So my hunch is that it probably hasn't been done, or hasn't been done often, because the program synthesis community would recognise it's pointless.

What you really want to look at is formal program synthesis benchmarks and how systems like AlphaCode do on them (hint: not so good).

Re: Competitive Programming with AlphaCode

#262

Post by pedrosorio »

ahgamut wrote: post #250
pedrosorio wrote: post #245

Earlier quoted context omitted.

> Suppose the models were trained only on greedy algorithms and then I provided a dynamic programming problem in the test set, (how) would the model solve it? How many human beings do you personally know who were able to solve a dynamic programming problem at first sight without ever having seen anything but greedy algorithms? Deepmind is not claiming they have a machine capable of performing original research here.…

> How many human beings do you personally know who were able to solve a dynamic programming problem at first sight without ever having seen anything but greedy algorithms? Zero, which is why if a trained network could do it, that would be "impressive" to me, given my personal biases. >. If you could get a machine that takes in all of Github and can solve "any" DP problem you describe in natural language with a couple…

I understand wanting to look at different metrics to gauge progress, but what is the issue with this?

> not elo rating because that depends on a human reference

Re: Competitive Programming with AlphaCode

#263

Post by cjbprime »

FiberBundle wrote: post #242
bricemo wrote: post #241

Earlier quoted context omitted.

What do you think then is the difference between going from 50th to 99.9th percentile in their other domains? Is there something materially different between ago, protein folding, or coding? (I don’t know the answer, just curious if anyone else does)

Well with respect to Go the fundamental difference afaict is that you can apply self-supervised learning, which is an incredibly powerful approach (But note e.g. that even this approach wasn't successful in "solving" Starcraft). Unfortunately it's extremely difficult to frame real-world problems in that setting. I don't know anything about protein-folding and don't know what Deepmind uses to try to solve that problem…

> this approach wasn't successful in "solving" Starcraft)

Why do you say that? As I understand it, AlphaStar beat pros consistently, including a not widely reported showmatch against Serral when he was BlizzCon champ.

Re: Competitive Programming with AlphaCode

#264

Post by alasdair_ »

throw10920 wrote: post #91
alasdair_ wrote: post #80

Earlier quoted context omitted.

>If you make developers 50x more efficient, won't you need 50x fewer developers? Developers today are 50X more efficient than when they had to input machine code on punched tape, yet the number of developers needed today is far larger than it was in those times.

There's no reason to believe that we'll need another 50x more developers, though.

Even if not, the original claim was that we may see a 50X decrease and I personally don't think that is likely, pre-Singularity anyway :)

Re: Competitive Programming with AlphaCode

#265

Post by sheikheddy »

pkaye wrote: post #228
falcor84 wrote: post #187

Earlier quoted context omitted.

How long before the typical human coder can do so?

Are you saying you cannot write code from scratch?

Not the parent comment, but I cannot code from scratch (outside of very simple and small applications). Competitive Programming is at about the limit of what I can do without looking things up, and only because I've had practice specifically for that kind of artificial environment.

Re: Competitive Programming with AlphaCode

#266

Post by zwaps »

cjbprime wrote: post #263
FiberBundle wrote: post #242

Earlier quoted context omitted.

Well with respect to Go the fundamental difference afaict is that you can apply self-supervised learning, which is an incredibly powerful approach (But note e.g. that even this approach wasn't successful in "solving" Starcraft). Unfortunately it's extremely difficult to frame real-world problems in that setting. I don't know anything about protein-folding and don't know what Deepmind uses to try to solve that problem…

> this approach wasn't successful in "solving" Starcraft) Why do you say that? As I understand it, AlphaStar beat pros consistently, including a not widely reported showmatch against Serral when he was BlizzCon champ.

Not once humans adapted to it afaik. AlphaStar got to top grandmaster level and then that was it, as people found ways to beat it. Now, it may be that the team considered the project complete and stopped training it. But technically - as it stands - Starcraft is still the one game where humans beat AI.

Re: Competitive Programming with AlphaCode

#267

Post by doctor_eval »

I sometimes read these and wonder if I need to retrain. At my age, I’ll struggle to get a job at a similar level in a new industry.

And then I remember that the thing I bring to the table is the ability to turn domain knowledge into code.

Being able to do competitive coding challenges is impressive, but a very large segment of software engineering is about eliciting what the squishy humans in management actually want, putting it into code, and discovering as quickly as possible that it’s not what they really wanted after all.

It’s going to take a sufficiently long time for AI to take over management that I don’t think oldies like me need to worry too much.

Re: Competitive Programming with AlphaCode

#268

Post by jahewson »

bricemo wrote: post #241
FiberBundle wrote: post #217

It never ceases to amaze me what you can do with these transformer models. They created millions of potential solutions for each problem, used the provided examples for the problems to filter out 99% of incorrect solutions and then applied some more heuristics and the 10 available submissions to try to find a solution. All these approaches just seem like brute-force approaches: Let's just throw our transformer on thi…

What do you think then is the difference between going from 50th to 99.9th percentile in their other domains? Is there something materially different between ago, protein folding, or coding? (I don’t know the answer, just curious if anyone else does)

That’s a big question but I’m tempted to answer it with a yes. A protein sequence contains a complete description of the structure of a protein but a coding question contains unknowns and the answers contain subjective variability.

Re: Competitive Programming with AlphaCode

#269

Post by falcor84 »

pkaye wrote: post #228
falcor84 wrote: post #187

Earlier quoted context omitted.

How long before the typical human coder can do so?

Are you saying you cannot write code from scratch?

I can write some code from scratch, but my ability to write code is improved by an order of magnitude when I can refer to online resources, including example code.

Re: Competitive Programming with AlphaCode

#270

Post by visarga »

hackinthebochs wrote: post #59
visarga wrote: post #35

Earlier quoted context omitted.

> The "problem" is that as you move up the levels of abstraction, you need fewer people to do the same amount of work. This will lower the entry barrier to developing software so more people will go into the field. Before you needed to know a programming language, now you will just have a dialogue with a language model. > I've always felt that programmers would be the first class of knowledge workers to be put out of…

>This will lower the entry barrier to developing software so more people will go into the field. Indeed. The ideal future of programming is something out of star trek. I often noticed how everyone on the ship is a programmer of a sort, they whip up a simulation as the problem warrants regardless of their field. But in this future, the job of programmer basically doesn't exist. As a programmer, I should be allowed to…

Let your imagination fly. We always want more than it's possible, our wishes fill up any volume like an expanding gas. Humans are going to be crucial to orchestrate AI and extract the most utility out of it.
Post reply on HN