Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Competitive Programming with AlphaCode

deepmind.com

Post reply on HN
Print view View original

221–230 of 415 posts

Re: Competitive Programming with AlphaCode

#221

Post by YeGoblynQueenne »

ctoth wrote: post #148
e4e78a06 wrote: post #134

Earlier quoted context omitted.

I don't think it's quite as impressive as you make it out to be. Median performance in a Codeforces programming competition is solving the easiest 1-2 problems out of 5-6 problems. Like all things programming the top 1% is much, much better than the median. There's also the open problem of verifying correctness in solutions and providing some sort of flag when the model is not confident in its correctness. I give it…

This is technology that simply didn't exist in any form 2 years ago. For no amount of money could you buy a program that did what this one does. Having been watching the growth of Transformer-based models for a couple years now really has hammered home that just as soon as we figure out how an AI can do X, X is no longer AI, or at least no longer impressive. How this happens is with comments like yours, and I'd reall…

>> This is technology that simply didn't exist in any form 2 years ago.

A few examples of neural program synthesis from at least 2 years ago:

https://sunblaze-ucb.github.io/program-synthesis/index.html

Another example from June 2020:

DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning

https://arxiv.org/abs/2006.08381

RobustFill, from 2017:

RobustFill: Neural Program Learning under Noisy I/O

https://www.microsoft.com/en-us/research/wp-content/uploads/...

I could go on.

And those are only examples from neural program synthesis. Program synthesis, in general, is a field that goes way back. I'd suggest as usual not making big proclamations about its state of the art without being acquainted with the literature. Because if you don't know what others have done every announcement by DeepMind, OpenAI et al seems like a huge advance... when it really isn't.

Re: Competitive Programming with AlphaCode

#222

Post by qualudeheart »

jonas_kgomo wrote: post #98

Genuine question, what are the reasons to be a software engineer without much ML knowledge in 2022. Seems like a wake up call for developers

Find something that’s hard and interesting. Someone will probably have a business trying to solve it and will hire you.

Re: Competitive Programming with AlphaCode

#223

Post by Veedrac »

omnicognate wrote: post #195
Veedrac wrote: post #176

Earlier quoted context omitted.

The first example is almost correct, conditioned off a sentence description. The second example is the right idea, it just bit off more than it could chew when slicing it all together. Using string ops for binary manipulation in Python isn't even stupid; it can be faster in a lot of cases. This feels a lot like screaming at a child for imperfect grammar.

You're misunderstanding my point. Nobody's screaming at anything. Whether this thing is impressive isn't at issue. It's utterly astonishing. I'm trying to figure out whether copilot in its current form is a tool that will be useful to me in my job. (I'd be able to do this evaluation properly if they'd just let me on the damned beta.) Nearly right isn't good enough for this afaics. In fact, I expect there to be a slig…

Well, that also seems like the wrong question to ask. Whether it's currently useful to you for writing short algorithms, rather than as the non-programmer's API interface it's primarily marketed as, seems about the least interesting take-away for it. We'll get to smoothing over the cracks later, once it's not a capability we literally just discovered exists. Heck, Codex is already not SOTA for that, AlphaCode is.

Re: Competitive Programming with AlphaCode

#224

Post by solididiot »

jakey_bakey wrote: post #152

At the risk of sounding relentlessly skeptical - surely by training the code on GitHub data you're not actually creating an AI to solve problems, but creating an extremely obfuscated database of coding puzzle solutions?

Does it need to solve original problems? Most of the code we write is dealing with the same problems in a slightly different context each time.

As others say in commends it might be the case where we meet in the middle. Us writing some form of tests for AI-produced code to pass.

Re: Competitive Programming with AlphaCode

#225

Post by 37ef_ced3 »

deanmen wrote: post #215
37ef_ced3 wrote: post #5

The example problem (essentially, is T a subsequence of S with deletions of size N) is a classic problem with no doubt dozens of implementations in AlphaCode's training set. And yet, what a garbage solution it produces. To illustrate the difference between intelligence and regurgitation, someone tell me what CoPilot generates for this: // A Go function to swap the sixth bit and seventeenth bit of a 32-bit signed inte…

You can do it without a subtraction unsigned int swapbits(unsigned int a) { bool bit6 = a & (1

And, to be clear, this is a human solution.

Not as efficient as mine, but kudos.

Re: Competitive Programming with AlphaCode

#226

Post by solididiot »

EVa5I7bHFq9mnYK wrote: post #193
qualudeheart wrote: post #90

Calling it now: If current language models can solve competitive programming at an average human level, we’re only a decade or less off from competitive programming being as solved as Go or Chess. Deepmind or openAI will do it. If not them, it will be a Chinese research group on par with them. I’ll be considering a new career. It will still be in computer science but it won’t be writing a lot of code. There’ll be sev…

Don't worry, there are a lot of much simpler jobs, like drivers or cashiers that will surrender to AI before coder's job does. So UBI will be implemented long before that happens.

I wouldn't be so sure. Programmers (and drivers and cashiers) can "survive" in poverty like millions others already do. This transformation is coming in waves that keep the proverbial frog in the pan.

Re: Competitive Programming with AlphaCode

#227

Post by pedrosorio »

ahgamut wrote: post #206

I find almost every new advance in deep learning is accompanied by contrasting comments: it's either "AI will soon automate programming/ ", or "let me know when AI can actually do ". There are many views on this spectrum, but these two are sure to be present in every comment section. IIUC, AlphaCode was trained on Github code to solve competitive programming challenges on Codeforces, some of which are "difficult for…

> Suppose AlphaCode was trained on Github code that contains the entire set of solutions on Codeforces, is it actually doing anything "difficult"?

They tested it on problems from recent contests. The implication being: the statements and solutions to these problems were not available when the Github training set was collected.

From the paper [0]: "Our pre-training dataset is based on a snapshot of selected public GitHub repositories taken on 2021/07/14" and "Following our GitHub pre-training dataset snapshot date, all training data in CodeContests was publicly released on or before 2021/07/14. Validation problems appeared between 2021/07/15 and 2021/09/20, and the test set contains problems published after 2021/09/21. This temporal split means that only information humans could have seen is available for training the model."

At the very least, even if some of these problems had been solved exactly before, you still need to go from "all of the code in Github" + "natural language description of the problem" to "picking the correct code snippet that solves the problem". Doesn't seem trivial to me.

> I don't believe it would be difficult for a human to solve problems on Codeforces when given access to the entirety of Github (indexed and efficiently searchable).

And yet, many humans who participate in these contests are unable to do so (although I guess the issue here is that Github is not properly indexed and searchable for humans?).

[0] https://storage.googleapis.com/deepmind-media/AlphaCode/comp...

Re: Competitive Programming with AlphaCode

#229

Post by ctoth »

YeGoblynQueenne wrote: post #221
ctoth wrote: post #148

Earlier quoted context omitted.

This is technology that simply didn't exist in any form 2 years ago. For no amount of money could you buy a program that did what this one does. Having been watching the growth of Transformer-based models for a couple years now really has hammered home that just as soon as we figure out how an AI can do X, X is no longer AI, or at least no longer impressive. How this happens is with comments like yours, and I'd reall…

>> This is technology that simply didn't exist in any form 2 years ago. A few examples of neural program synthesis from at least 2 years ago: https://sunblaze-ucb.github.io/program-synthesis/index.html Another example from June 2020: DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning https://arxiv.org/abs/2006.08381 RobustFill, from 2017: RobustFill: Neural Program Le…

Of course program synthesis has been a thing for years, I remember some excellent papers out of MSR 10 years ago. But which of those could read a prompt and build the program from the prompt? Setting up a whole bunch of constraints and having your optimizer spit out a program that fulfills them is program synthesis and is super interesting, but not at all what I think of when I'm told we can make the computer program for us. For instance, RobustFill takes its optimization criteria from a bundle of pre-completed inputs and outputs of how people want the program to behave instead of having the problem described in natural language and creating the solution program.

Re: Competitive Programming with AlphaCode

#230

Post by omnicognate »

Veedrac wrote: post #223
omnicognate wrote: post #195

Earlier quoted context omitted.

You're misunderstanding my point. Nobody's screaming at anything. Whether this thing is impressive isn't at issue. It's utterly astonishing. I'm trying to figure out whether copilot in its current form is a tool that will be useful to me in my job. (I'd be able to do this evaluation properly if they'd just let me on the damned beta.) Nearly right isn't good enough for this afaics. In fact, I expect there to be a slig…

Well, that also seems like the wrong question to ask. Whether it's currently useful to you for writing short algorithms, rather than as the non-programmer's API interface it's primarily marketed as, seems about the least interesting take-away for it. We'll get to smoothing over the cracks later, once it's not a capability we literally just discovered exists. Heck, Codex is already not SOTA for that, AlphaCode is.

It may not be the question that interests you but who are you to say it's the "wrong question" for me to ask? I want to know if I'm on the verge of having access to a tool that is going to transform the way I do my job, as people keep claiming.
Post reply on HN