Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Competitive Programming with AlphaCode

deepmind.com

Post reply on HN
Print view View original

331–340 of 415 posts

Re: Competitive Programming with AlphaCode

#331

Post by OOPMan »

p1esk wrote: post #307
GuB-42 wrote: post #304

Earlier quoted context omitted.

I think programmers are relatively "safe" from AI for the simple reason they are the ones who talk to AI. The job of programmers is to have machines do stuff so that humans don't have to, and of course, they do it for themselves too. Scripts, libraries, compilers, they are just tools to avoid flipping bits by hand. If something like copilot is not embraced by all programmers, it is that it is often less than helpful,…

there will always be someone who tells the computer what to do Until the computer starts telling people what to do

Uh...that already happens and AI isn't event required

Re: Competitive Programming with AlphaCode

#332

Post by Veedrac »

YeGoblynQueenne wrote: post #318
Veedrac wrote: post #300

Earlier quoted context omitted.

> Btw, APPS is not much of a benchmark. It evaluates code generation according to how close it resembles code written by humans. No, the metric used in this paper was the percentage of questions it could solve against the hidden tests. > That points to a further limitation of the approach: it works for Codeforces problems but not for APPS problems (so it's very purpose-specific). This does not matter typically since…

>> No, the metric used in this paper was the percentage of questions it could solve against the hidden tests. Right, that's my mistake. The APPS dataset has natural language specifications and test cases for evaluation. It actually includes Codeforces problems. The excuse quoted in the second part of your comment is an excuse. If a large language model can complete a code generation task, that's because it's seen an…

Your comment is an excuse, not mine: “ignore how good the advertised model is, because a different, much smaller version without all the techniques merely does pretty well on an extremely tough problem set.”

> Any claims to the contrary need very strong evidence to support them and there's typically no such thing in papers like the AlphaCode one.

This is the opposite of how burden of proof works. You are the one making a claim with certainty based off of guesswork, not me. And the paper does actually have a section on this, and finds copying isn't pervasive outside of utility snippets and functions, which also occur in human solutions. It's a weak objection anyway; just the task of translating english prose into the general algorithm you want to apply is already an impressive feat.

Re: Competitive Programming with AlphaCode

#334

Post by eightails »

p1esk wrote: post #307
GuB-42 wrote: post #304

Earlier quoted context omitted.

I think programmers are relatively "safe" from AI for the simple reason they are the ones who talk to AI. The job of programmers is to have machines do stuff so that humans don't have to, and of course, they do it for themselves too. Scripts, libraries, compilers, they are just tools to avoid flipping bits by hand. If something like copilot is not embraced by all programmers, it is that it is often less than helpful,…

there will always be someone who tells the computer what to do Until the computer starts telling people what to do

> Until the computer starts telling people what to do

My phone has me well trained. All it has to do is play a short message tone and I'll come running...

Re: Competitive Programming with AlphaCode

#335

Post by briga »

plutonorm wrote: post #298
briga wrote: post #280

Earlier quoted context omitted.

Another way to frame it is that these models still perform very poorly at the task they're designed to do. Imagine if real programmer needed to write a solution a hundred times before they were able to achieve (average) performance. You'd probably wonder if it was just blind luck that got them to the solution. You'd also fire them. What these models are very good at doing is plagiarizing content, so part of me wonder…

How do you know the inner workings of the mind don't operate in a similar manner? How many different solutions to the problem are constructed within your mind before the correct one 'just arrives'?

I suspect there is some similarity between language models and the structure of language in the mind, but there's a whole lot more going on behind the scenes in the brain than simple runtime statistical model output. Intentionality, planning, narrativity, memory formation, object permanence... Language models are exciting and interesting because apparently they can do abstract symbolic manipulation and produce coherent text, but I wouldn't call AGI solved quite yet.

Re: Competitive Programming with AlphaCode

#336

Post by briga »

Vetch wrote: post #305
briga wrote: post #280

Earlier quoted context omitted.

Another way to frame it is that these models still perform very poorly at the task they're designed to do. Imagine if real programmer needed to write a solution a hundred times before they were able to achieve (average) performance. You'd probably wonder if it was just blind luck that got them to the solution. You'd also fire them. What these models are very good at doing is plagiarizing content, so part of me wonder…

> Imagine if real programmer needed to write a solution a hundred times To be fair, a lot of creative work requires plenty of trial and error. And since no problems are solved from scratch, all things considered, the most immediate contributors to your result and you might have iterated through tens of dozens of possibilities. My advantage as a human is I can often tell you why I am eliminating this branch of the sea…

>The limitations to GPT3 codex and AlphaCode seems to be they're relatively weak at selection

This really does seem like the key here--the knowledge apparently is all in the language model, we just haven't found the best ways to extract that knowledge in a consistent and coherent manner. Right now it's just: generate a bunch of examples and cherry pick the good ones.

Re: Competitive Programming with AlphaCode

#337

Post by [deleted] »

YeGoblynQueenne wrote: post #297
Veedrac wrote: post #294

Earlier quoted context omitted.

The APPS benchmark was for a “small 1B parameter model”, fine-tuned “without using clustering, tags, ratings, value conditioning, or prediction”. > Overall, that's a brute-force, almost random approach that is ignoring entire decades of program synthesis work. You don't get answers to these questions by random search. Not even close. I have looked at non-neural program synthesis papers. It is not remotely competitive…

Yes, apparently they couldn't use their full approach because "of missing information in the dataset". That points to a further limitation of the approach: it works for Codeforces problems but not for APPS problems (so it's very purpose-specific). Btw, APPS is not much of a benchmark. It evaluates code generation according to how close it resembles code written by humans. That's standard fare for text generation benc…

[deleted]

Re: Competitive Programming with AlphaCode

#338

Post by faizshah »

briga wrote: post #280
FiberBundle wrote: post #217

It never ceases to amaze me what you can do with these transformer models. They created millions of potential solutions for each problem, used the provided examples for the problems to filter out 99% of incorrect solutions and then applied some more heuristics and the 10 available submissions to try to find a solution. All these approaches just seem like brute-force approaches: Let's just throw our transformer on thi…

Another way to frame it is that these models still perform very poorly at the task they're designed to do. Imagine if real programmer needed to write a solution a hundred times before they were able to achieve (average) performance. You'd probably wonder if it was just blind luck that got them to the solution. You'd also fire them. What these models are very good at doing is plagiarizing content, so part of me wonder…

I was really impressed with a lot of the GPT3 stuff I had seen people showing so I gave it a spin myself. I was surprised by how repetitive it seemed to be, it would write new sentences but it would repeat the same concepts among similar prompts. I wish I saved the examples, it was like when a chat bot gets in a loop but GPT3 varied the sentence structure. I think that if you look closely at transformer models outputs you can expect the same sort of thing. Its like in high school when people would copy homework but use different wording.

I also think generally in ML and DL the overarching progress gets hyped but in the background there are murmurs about the limitations in the research community. Thats how we end up with people in 2012 saying FSD is a couple years away but in 2022 we know we aren't even close yet. We tend to oversell how capable these systems are.

Re: Competitive Programming with AlphaCode

#339

Post by [deleted] »

YeGoblynQueenne wrote: post #287
bricemo wrote: post #241

Earlier quoted context omitted.

What do you think then is the difference between going from 50th to 99.9th percentile in their other domains? Is there something materially different between ago, protein folding, or coding? (I don’t know the answer, just curious if anyone else does)

>> What do you think then is the difference between going from 50th to 99.9th percentile in their other domains? Is there something materially different between ago, protein folding, or coding? Yes, it's the size of the search space for each problem. The search space for arbitrary programs in a language with Universal Turing Machine expressivity is infinite. Even worse, for any programming problem there are an infini…

[deleted]

Re: Competitive Programming with AlphaCode

#340

Post by [deleted] »

FemmeAndroid wrote: post #103

This is extremely impressive, but I do think it’s worth noting that these two things were provided: - a very well defined problem. (One of the things I like about competitive programming and the like is just getting to implement a clearly articulated problem, not something I experience on most days.) - existing test data. This is definitely a great accomplishment, but I think those two features of competitive program…

[deleted]
Post reply on HN