Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Competitive Programming with AlphaCode

deepmind.com

Post reply on HN
Print view View original

61–70 of 415 posts

Re: Competitive Programming with AlphaCode

#61

Post by hackinthebochs »

skulk wrote: post #48
37ef_ced3 wrote: post #14

Earlier quoted context omitted.

Ridiculous. It's a joke, right? Perhaps the most inefficient and naive solution ever? CoPilot is regurgitating some solution from its training set, the solution of an inept programmer who would manipulate bits via conversion to string... yikes.

The next iteration of code assistant needs to be able to parse responses like your comment and update the code accordingly. Once a human+computer pair can converge on a correct and admissible solution to _any_ tractable programming task through natural language dialogue, we should start worrying about our jobs going away. Until then, for each line of code generated by AI, there will be two jobs created to maintain th…

Which direction in feature space do you move in response to "you inept POS"?

Re: Competitive Programming with AlphaCode

#62

Post by chongli »

msoad wrote: post #19

This seems to have a narrower scope than GitHub Copilot. It generates more lines of code to a more holistic problem vs. GitHub Copilot that works as a "more advanced autocomplete" in code editors. Sure Copilot can synthesize full functions and classes but for me, it's the most useful when it suggests another test case's title or writes repetitive code like this.foo = foo; this.bar = bar etc... Having used Copilot I c…

repetitive code like this.foo = foo; this.bar = bar etc...

This sort of boilerplate code is best solved by the programming language. Either via better built-in syntax or macros. Using an advanced machine learning model to generate this code is both error-prone and a big source of noise and code bloat. This is not an issue that will go away with better tooling; it will only get worse.

Re: Competitive Programming with AlphaCode

#63

Post by electroly »

skulk wrote: post #48
37ef_ced3 wrote: post #14

Earlier quoted context omitted.

Ridiculous. It's a joke, right? Perhaps the most inefficient and naive solution ever? CoPilot is regurgitating some solution from its training set, the solution of an inept programmer who would manipulate bits via conversion to string... yikes.

The next iteration of code assistant needs to be able to parse responses like your comment and update the code accordingly. Once a human+computer pair can converge on a correct and admissible solution to _any_ tractable programming task through natural language dialogue, we should start worrying about our jobs going away. Until then, for each line of code generated by AI, there will be two jobs created to maintain th…

Copilot can do that, sorta. You undo the completion and add something like "... but don't convert it to a string" to the comment, then have it try completing again.

Re: Competitive Programming with AlphaCode

#64

Post by omnicognate »

quenix wrote: post #7
37ef_ced3 wrote: post #5

The example problem (essentially, is T a subsequence of S with deletions of size N) is a classic problem with no doubt dozens of implementations in AlphaCode's training set. And yet, what a garbage solution it produces. To illustrate the difference between intelligence and regurgitation, someone tell me what CoPilot generates for this: // A Go function to swap the sixth bit and seventeenth bit of a 32-bit signed inte…

Here is what CoPilot wrote in Python: # A function to swap the sixth bit and seventeenth bit of a 32-bit signed integer. def swap_bits(x): # Get the value of the sixth bit. bit6 = x & (1 > 5 bit17 = bit17 >> 16 # Combine the swapped values of the sixth and seventeenth bit. x = x ^ (bit6 It actually synthesised about 10 viable solutions. Some interesting ones: def swap_six_seventeen(x): # Get the binary representation…

The first one (swap_bits) sets both bits to the same value, which is the original two bits XORed together. Eg.

    bin(swap_bits(0b_1_0000000000_0_00000))
    '0b10000000000100000'

    bin(swap_bits(0b_0_0000000000_1_00000))
    '0b10000000000100000'

    bin(swap_bits(0b_1_0000000000_1_00000))
    '0b0'

    bin(swap_bits(0b_0_0000000000_0_00000))
    '0b0'
The second one converts the value to a string and uses string operations, which is wildly inefficient and a very common mistake made by inexperienced programmers unaware of bitwise operations (so presumably common in the training set). It also attempts to swap the 6th and 17th most significant bits rather than the 6th and 17th least significant bits, i.e. counts in the opposite direction to the first one (the comment doesn't specify but typically you count from the least significant bit in these situations).

Worse, though, it gets the string manipulation completely wrong. I think it's trying for `binary[:5] + binary[16] + binary[6:16] + binary[5] + binary[17:]`, i.e. characters 1-5, then character 17, then characters 7-16, then character 6, then characters 18-32. The manipulation it does just completely mangles the string.

I'm very keen to try Github Copilot if they ever admit me to the beta (I've been waiting forever) and will adopt it enthusiastically if it's useful. However, this is exactly what I've pessimistically expected. Analysing these truly awful implementations to identify the subtle and bizarre misbehaviours has taken me far, far longer than it would have taken me to just write and test a working implementation myself. And I'm supposed to evaluate 10 of these to see if one of them might possibly do the right thing?!?!

Re: Competitive Programming with AlphaCode

#65

Post by pretendscholar »

alphabetting wrote: post #58
pretendscholar wrote: post #50

Earlier quoted context omitted.

Yes it will help the already powerful players disproportionately.

They opensourced alphafold for anyone to use commercially despite big financial incentive to keep it private and use in their new drug discovery lab. No idea how this works or differs from alphafold but imagine they'll do the same here if possible

Only after another lab made their own open source one that was comparable.

Re: Competitive Programming with AlphaCode

#66

Post by xmprt »

chongli wrote: post #62
msoad wrote: post #19

This seems to have a narrower scope than GitHub Copilot. It generates more lines of code to a more holistic problem vs. GitHub Copilot that works as a "more advanced autocomplete" in code editors. Sure Copilot can synthesize full functions and classes but for me, it's the most useful when it suggests another test case's title or writes repetitive code like this.foo = foo; this.bar = bar etc... Having used Copilot I c…

repetitive code like this.foo = foo; this.bar = bar etc... This sort of boilerplate code is best solved by the programming language. Either via better built-in syntax or macros. Using an advanced machine learning model to generate this code is both error-prone and a big source of noise and code bloat. This is not an issue that will go away with better tooling; it will only get worse.

I don't think I agree. Most people spend more time reading than writing code so programming languages should be optimized to be easier to read whereas tooling should be made to simplify writing code. New syntax or macros sounds like it would make the language harder to read. I agree that an advanced machine learning model for generating boilerplate code isn't the right approach but I also don't think we should extend languages for this. Tooling like code generators and linters are a good middle ground.

Re: Competitive Programming with AlphaCode

#68

Post by AnIdiotOnTheNet »

streetcat1 wrote: post #30
hmate9 wrote: post #2

Between this and OpenAI's Github Copilot "programming" will slowly start dying probably. What I mean by that is that sure, you have to learn how to program, but our time will be spent much more on just the design part and writing detailed documentation/specs and then we just have one of these AIs generate the code. It's the next step. Binary code Historically its always been about abstracting and writing less code to…

First, If this is correct, if alpha code succeeded, this will bring to its own demise. I.e. as soon as it starts replacing humans, it will not have enough human generated training data, since all of programming will be done by models like himself. Second, alphacode was specifically trained for competitive programming : 1. short programs. 2. Each program has 100's of human generated solutions. However, commercial prog…

> as soon as it starts replacing humans, it will not have enough human generated training data, since all of programming will be done by models like himself.

As a natural born pessimist, I can't help but feel that by the time we get to that point we'll just keep blundering forward and adapting our world around the wild nonsense garbage code the model ends up producing in this scenario.

After all, that's basically what we've done with the entire web stack.

Re: Competitive Programming with AlphaCode

#69

Post by ialyos »

chroem- wrote: post #43
streetcat1 wrote: post #30

Earlier quoted context omitted.

First, If this is correct, if alpha code succeeded, this will bring to its own demise. I.e. as soon as it starts replacing humans, it will not have enough human generated training data, since all of programming will be done by models like himself. Second, alphacode was specifically trained for competitive programming : 1. short programs. 2. Each program has 100's of human generated solutions. However, commercial prog…

Reinforcement learning and adversarial training can render both of those concerns as non-issues in practice.

The phrase "in practice" doesn't really work when you're referring to highly finicky strategies like RL and adversarial training

Re: Competitive Programming with AlphaCode

#70

Post by valyagolev »

chongli wrote: post #62
msoad wrote: post #19

This seems to have a narrower scope than GitHub Copilot. It generates more lines of code to a more holistic problem vs. GitHub Copilot that works as a "more advanced autocomplete" in code editors. Sure Copilot can synthesize full functions and classes but for me, it's the most useful when it suggests another test case's title or writes repetitive code like this.foo = foo; this.bar = bar etc... Having used Copilot I c…

repetitive code like this.foo = foo; this.bar = bar etc... This sort of boilerplate code is best solved by the programming language. Either via better built-in syntax or macros. Using an advanced machine learning model to generate this code is both error-prone and a big source of noise and code bloat. This is not an issue that will go away with better tooling; it will only get worse.

it is a very similar argument to the one for powerful IDEs and underwhelming languages. to be fair, it’s not necessarily fruitless - e.g. with smalltalk. i fail to see the analoguous smalltalk-style empowerment of language using AI but perhaps something is there.

anyway. programming is automation; automation of programming is abstraction. using AI to write your code is just a bad abstraction - we are used to them

Post reply on HN