Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Competitive Programming with AlphaCode

deepmind.com

Post reply on HN
Print view View original

191–200 of 415 posts

Re: Competitive Programming with AlphaCode

#191

Post by qualudeheart »

indiv0 wrote: post #175
nikkwong wrote: post #137

Earlier quoted context omitted.

I believe he's referring to "luddites" -- a group of people who resisted technological innovation during the industrial revolution.

Luddite but mixed with "Eliezer Yudkowsky" who is a researcher working on the problem of friendly AI (or whatever they're calling it these days). Basically trying to prevent skynet. The GP is saying that once we have AGI, then "AGI is going to make the human race irrelevant" outweighs "AGI makes software devs irrelevant".

That’s the idea.

Re: Competitive Programming with AlphaCode

#192

Post by falcor84 »

d0mine wrote: post #168

It reminds me that median reputation on StackOverflow is 1. All AlphaSO would have to do is to register to receive median reputation on SO ;) (kidding aside AlphaCode sounds like magic) Inventing relational DBs hasn't replaced programmers, we just write custom DB engines less often. Inventing electronic spreadsheets hasn't deprecated programmers, it just means that we don't need programmers for corresponding tasks (w…

>AI won't replace programmers until it grows to replace the humanity as a whole.

Yes, but after seeing this progress in the former, my time estimate of time remaining until the latter had just significantly shortened.

Re: Competitive Programming with AlphaCode

#193

Post by EVa5I7bHFq9mnYK »

qualudeheart wrote: post #90

Calling it now: If current language models can solve competitive programming at an average human level, we’re only a decade or less off from competitive programming being as solved as Go or Chess. Deepmind or openAI will do it. If not them, it will be a Chinese research group on par with them. I’ll be considering a new career. It will still be in computer science but it won’t be writing a lot of code. There’ll be sev…

Don't worry, there are a lot of much simpler jobs, like drivers or cashiers that will surrender to AI before coder's job does. So UBI will be implemented long before that happens.

Re: Competitive Programming with AlphaCode

#194

Post by 37ef_ced3 »

Veedrac wrote: post #182
37ef_ced3 wrote: post #179

Earlier quoted context omitted.

It illustrates that CoPilot is generating maximum likelihood token strings and has no real understanding of the code. That's what is happening here. There is no intelligence, just regurgitation. Randomization and maximum likelihood completion. Just like with the competitive programming example, we're asking it to produce solutions that it has seen in its training set. If you ask for a nontrivial twist on one of those…

It got the value of the sixth and seventeenth bits, moved them into the right positions, and inserted them into the original value. Off a one-line description written in English ! I really cannot empathize with the idea that this is not a meaningful capability. If intelligence only means to you “equal in all capabilities to an experienced human”, you are never going to be able to see anything coming ever.

If you ask CoPilot to solve something it hasn't seen, it won't be able to solve it.

It's a transformer. Do you understand what that means? It's just matrix multiplication.

It generates maximum likelihood token strings, based on its training data.

It doesn't "understand" what those token string mean.

You are amazed because you're testing the transformer by asking the transformer to generate human-written code THAT IT WAS TRAINED ON. To make CoPilot fail, all you have to do is ask it to generate something unlikely, something it hasn't seen in training.

Maximum likelihood token strings. Period.

Re: Competitive Programming with AlphaCode

#195

Post by omnicognate »

Veedrac wrote: post #176
omnicognate wrote: post #64

Earlier quoted context omitted.

The first one (swap_bits) sets both bits to the same value, which is the original two bits XORed together. Eg. bin(swap_bits(0b_1_0000000000_0_00000)) '0b10000000000100000' bin(swap_bits(0b_0_0000000000_1_00000)) '0b10000000000100000' bin(swap_bits(0b_1_0000000000_1_00000)) '0b0' bin(swap_bits(0b_0_0000000000_0_00000)) '0b0' The second one converts the value to a string and uses string operations, which is wildly ine…

The first example is almost correct, conditioned off a sentence description. The second example is the right idea, it just bit off more than it could chew when slicing it all together. Using string ops for binary manipulation in Python isn't even stupid; it can be faster in a lot of cases. This feels a lot like screaming at a child for imperfect grammar.

You're misunderstanding my point. Nobody's screaming at anything. Whether this thing is impressive isn't at issue. It's utterly astonishing.

I'm trying to figure out whether copilot in its current form is a tool that will be useful to me in my job. (I'd be able to do this evaluation properly if they'd just let me on the damned beta.)

Nearly right isn't good enough for this afaics. In fact, I expect there to be a slightly paradoxical effect where nearly-right is worse than obviously-wrong. An analysis of a piece of code like I did above is time consuming and cognitively taxing. An obviously wrong solution I can just reject immediately. An almost-right (or at least vaguely plausible) one like these takes thought to reject. Much more thought, in this case (for me, at least) than just writing the thing myself in the first place.

Edit: BTW, I don't get what you're saying with

"The first example is almost correct, conditioned off a sentence description. The second example is the right idea, it just bit off more than it could chew when slicing it all together."

The first one is completely (if subtly) wrong. It's supposed to swap two bits but it sets them to the same value. There's no interpretation of the description in which that's correct.

The second one is definitely not "the right idea". It tries to do it with string manipulations, which (regardless of the fact that it does so incorrectly) is completely the wrong approach. This one is actually "better" than the other in the paradoxical sense I mentioned above, because I could reject it the moment I saw it convert the number to a string.

Re: Competitive Programming with AlphaCode

#196

Post by qualudeheart »

zerr wrote: post #131
qualudeheart wrote: post #90

Calling it now: If current language models can solve competitive programming at an average human level, we’re only a decade or less off from competitive programming being as solved as Go or Chess. Deepmind or openAI will do it. If not them, it will be a Chinese research group on par with them. I’ll be considering a new career. It will still be in computer science but it won’t be writing a lot of code. There’ll be sev…

The thing is, Competitive Programming (CP) is a completely different discipline/subject with its own trivia knowledge and tricks. CP uses Computer Science the same way as e.g. Biology uses Mathematics. It has very little in common with a real world software development.

I said as much in another comment.

Automating the software development profession proper is going to be much harder and will require autonomous agents with coherent world models, because that’s what you need to act in a business context.

Re: Competitive Programming with AlphaCode

#198

Post by throwaway5752 »

Most people here are programmers (or otherwise involved in the production of software). We shouldn't look at RPA and other job automation trends dispassionately. SaaS valuations aren't were they are (and accounting doesn't treat engineering salary as cost of goods sold) because investors believe that they will require armies of very well paid developers in perpetuity.

Re: Competitive Programming with AlphaCode

#199

Post by hackinthebochs »

Jensson wrote: post #177
hackinthebochs wrote: post #158

Earlier quoted context omitted.

The issue is these techniques are growing in capabilities exponentially, while we have a habit of extrapolating linearly. Some saw the glaring deficits in copilot then reasoned that linear improvements is still glaring deficits. I don't know that this bias can ever be corrected. A large number of intelligent people simply will never be convinced general AI is coming soon no matter what evidence is presented.

> techniques are growing in capabilities exponentially, while we have a habit of extrapolating linearly What does this even mean? How do you put a number on AI capability? You can say it is growing faster than people expect, but what is even exponential or linear growth in AI capability?

I take your point that the linear/exponential terminology is a bit dubious. But the simple way to make sense of it is just going by various benchmarks. E.g. the power-law relationship between the model accuracy and the model size: https://eliaszwang.com/paper-reviews/scaling-laws-neural-lm/

Re: Competitive Programming with AlphaCode

#200

Post by falcor84 »

pjmorris wrote: post #47
hmate9 wrote: post #2

Between this and OpenAI's Github Copilot "programming" will slowly start dying probably. What I mean by that is that sure, you have to learn how to program, but our time will be spent much more on just the design part and writing detailed documentation/specs and then we just have one of these AIs generate the code. It's the next step. Binary code Historically its always been about abstracting and writing less code to…

I'd note that assembly, C, and Python didn't replace 'programming' but were expected to do so. I'd wager that what you now call 'detailed documentation/specs' will still be called programming in 10 or even 20 years.

If you could change a sentence in the documentation and then run a ~1min compilation to see the resulting software, it would be a very different kind of programming. I suppose it'll give a new meaning to Readme-Driven-Development.
Post reply on HN