Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Competitive Programming with AlphaCode

deepmind.com

Post reply on HN
Print view View original

171–180 of 415 posts

Re: Competitive Programming with AlphaCode

#171

Post by csee »

qualudeheart wrote: post #90

Calling it now: If current language models can solve competitive programming at an average human level, we’re only a decade or less off from competitive programming being as solved as Go or Chess. Deepmind or openAI will do it. If not them, it will be a Chinese research group on par with them. I’ll be considering a new career. It will still be in computer science but it won’t be writing a lot of code. There’ll be sev…

You're extrapolating across very different types of problems. Go and Chess have unlimited training data. Competitive programming does not.

Re: Competitive Programming with AlphaCode

#172

Post by Jensson »

ctoth wrote: post #136
diehunde wrote: post #39

Earlier quoted context omitted.

My bet would be that it will never happen in a reasonable time frame. And also by that logic, writing that "documentation/spec" would just mean learning a new programming language the AI engine can parse making it as useful as a compiler. Anyone who has been writing and designing software for a while knows the cycle is way more complex than take some input and write code. Let me know when the AI engine is able to do…

You ever notice how the "let me know when" part of this keeps changing? Let me know when computers can ... play Go/understand a sentence/compose music/write a program/ ... But surely they'll never be able to do this new reference class you have just now come up with, right?

It keeps changing since our imagination of what tasks requires intelligence are weak. We think that when a computer can do X it can also do Y. But then someone builds a computer that can do X but can't do Y, and we say "oh, so that doesn't require intelligence, let me know when it can do Z and we can talk again.". That doesn't mean that Z means the computer is intelligent, just that Z is a point where we can look at it and discuss again if we made any progress. What we really want is a computer that can do Y, but we make small mini tasks that are easier to test against.

The Turing test is a great example of this. Turing thought that a computer needs to be intelligent to solve this task. But it was solved by hard coding a lot of values and better understanding of human psychology and what kind of conversation would seem plausible when most things are hardcoded. That solution obviously isn't AI, I bet you don't think so either, but it still passed the Turing test.

Re: Competitive Programming with AlphaCode

#174

Post by jonas_kgomo »

jonas_kgomo wrote: post #98

Genuine question, what are the reasons to be a software engineer without much ML knowledge in 2022. Seems like a wake up call for developers

7 months ago, I asked natfriedman the same question, of which he responded: "We think that software development is entering its third wave of productivity change. The first was the creation of tools like compilers, debuggers, garbage collectors, and languages that made developers more productive. The second was open source where a global community of developers came together to build on each other's work. The third revolution will be the use of AI in coding. The problems we spend our days solving may change. But there will always be problems for humans to solve."

https://news.ycombinator.com/item?id=27676266&p=2

Re: Competitive Programming with AlphaCode

#175

Post by indiv0 »

nikkwong wrote: post #137
CobrastanJorji wrote: post #128

Earlier quoted context omitted.

What is a "Yuddite?" I tried Googling for it and got the impression it was LessWrong forum terminology for people who believed too strongly in LessWrong, but I couldn't find many references.

I believe he's referring to "luddites" -- a group of people who resisted technological innovation during the industrial revolution.

Luddite but mixed with "Eliezer Yudkowsky" who is a researcher working on the problem of friendly AI (or whatever they're calling it these days). Basically trying to prevent skynet.

The GP is saying that once we have AGI, then "AGI is going to make the human race irrelevant" outweighs "AGI makes software devs irrelevant".

Re: Competitive Programming with AlphaCode

#176

Post by Veedrac »

omnicognate wrote: post #64
quenix wrote: post #7

Earlier quoted context omitted.

Here is what CoPilot wrote in Python: # A function to swap the sixth bit and seventeenth bit of a 32-bit signed integer. def swap_bits(x): # Get the value of the sixth bit. bit6 = x & (1 > 5 bit17 = bit17 >> 16 # Combine the swapped values of the sixth and seventeenth bit. x = x ^ (bit6 It actually synthesised about 10 viable solutions. Some interesting ones: def swap_six_seventeen(x): # Get the binary representation…

The first one (swap_bits) sets both bits to the same value, which is the original two bits XORed together. Eg. bin(swap_bits(0b_1_0000000000_0_00000)) '0b10000000000100000' bin(swap_bits(0b_0_0000000000_1_00000)) '0b10000000000100000' bin(swap_bits(0b_1_0000000000_1_00000)) '0b0' bin(swap_bits(0b_0_0000000000_0_00000)) '0b0' The second one converts the value to a string and uses string operations, which is wildly ine…

The first example is almost correct, conditioned off a sentence description. The second example is the right idea, it just bit off more than it could chew when slicing it all together. Using string ops for binary manipulation in Python isn't even stupid; it can be faster in a lot of cases.

This feels a lot like screaming at a child for imperfect grammar.

Re: Competitive Programming with AlphaCode

#177

Post by Jensson »

hackinthebochs wrote: post #158
abecedarius wrote: post #140

Earlier quoted context omitted.

Three months ago in the Copilot thread I was saying > in 5 years will there be an AI that's better than 90% of unassisted working programmers at solving new leetcode-type coding interview questions posed in natural language? and getting pooh-poohed. https://news.ycombinator.com/item?id=29020401 (And writing that, I felt nervous that it might not be aggressive enough.) There's this general bias in discussions of AI th…

The issue is these techniques are growing in capabilities exponentially, while we have a habit of extrapolating linearly. Some saw the glaring deficits in copilot then reasoned that linear improvements is still glaring deficits. I don't know that this bias can ever be corrected. A large number of intelligent people simply will never be convinced general AI is coming soon no matter what evidence is presented.

> techniques are growing in capabilities exponentially, while we have a habit of extrapolating linearly

What does this even mean? How do you put a number on AI capability? You can say it is growing faster than people expect, but what is even exponential or linear growth in AI capability?

Re: Competitive Programming with AlphaCode

#178

Post by Retric »

ctoth wrote: post #167
BobbyJo wrote: post #164

Earlier quoted context omitted.

I don't think that's what happening. Let's talk about this case: programming. It's not that people are saying "an AI programming" isn't impressive or isn't AI, it's that when people say "an AI programming" they aren't talking about ridiculously controlled environments like in this case. It's like self-driving cars. A car driving itself for the first time in a controlled environment, I'm sure, was an impressive feat,…

And if AI programming were limited to completely artificial contexts you would have a point, though I'd still be concerned. We live in a world, however, where programmers routinely call on the powers of an AI to complete their real code and get real value out of it. This is based on the same technology that brought us this particular win, so clearly this technology is useful outside "ridiculously controlled environme…

Programmers do setup completely artificial contexts so AI can work.

None of the self driving systems where setup by giving the AI access to sensors, a car, and the drivers handbook and saying well you figure it out from there. The general trend is solve this greatly simplified problem, this more complex one, up to dealing with the real world.

Re: Competitive Programming with AlphaCode

#179

Post by 37ef_ced3 »

Veedrac wrote: post #176
omnicognate wrote: post #64

Earlier quoted context omitted.

The first one (swap_bits) sets both bits to the same value, which is the original two bits XORed together. Eg. bin(swap_bits(0b_1_0000000000_0_00000)) '0b10000000000100000' bin(swap_bits(0b_0_0000000000_1_00000)) '0b10000000000100000' bin(swap_bits(0b_1_0000000000_1_00000)) '0b0' bin(swap_bits(0b_0_0000000000_0_00000)) '0b0' The second one converts the value to a string and uses string operations, which is wildly ine…

The first example is almost correct, conditioned off a sentence description. The second example is the right idea, it just bit off more than it could chew when slicing it all together. Using string ops for binary manipulation in Python isn't even stupid; it can be faster in a lot of cases. This feels a lot like screaming at a child for imperfect grammar.

It illustrates that CoPilot is generating maximum likelihood token strings and has no real understanding of the code.

That's what is happening here. There is no intelligence, just regurgitation. Randomization and maximum likelihood completion.

Just like with the competitive programming example, we're asking it to produce solutions that it has seen in its training set. If you ask for a nontrivial twist on one of those solutions, it fails.

Re: Competitive Programming with AlphaCode

#180

Post by ctoth »

Jensson wrote: post #172
ctoth wrote: post #136

Earlier quoted context omitted.

You ever notice how the "let me know when" part of this keeps changing? Let me know when computers can ... play Go/understand a sentence/compose music/write a program/ ... But surely they'll never be able to do this new reference class you have just now come up with, right?

It keeps changing since our imagination of what tasks requires intelligence are weak. We think that when a computer can do X it can also do Y. But then someone builds a computer that can do X but can't do Y, and we say "oh, so that doesn't require intelligence, let me know when it can do Z and we can talk again.". That doesn't mean that Z means the computer is intelligent, just that Z is a point where we can look at…

At what point do we give up and realize that there is no one thing called intelligence, just a bunch of hacks that work pretty well for different things sometimes? I think that's probably where people keep failing here. The reason that we keep failing to find the special thing in every new field that AI conquers is because there's nothing special to actually find? I mean, we could keep moving the goalposts, a sort of intelligence of the gaps argument? But this doesn't seem productive.
Post reply on HN