Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Competitive Programming with AlphaCode

deepmind.com

Post reply on HN
Print view View original

181–190 of 415 posts

Re: Competitive Programming with AlphaCode

#181

Post by ctoth »

Retric wrote: post #178
ctoth wrote: post #167

Earlier quoted context omitted.

And if AI programming were limited to completely artificial contexts you would have a point, though I'd still be concerned. We live in a world, however, where programmers routinely call on the powers of an AI to complete their real code and get real value out of it. This is based on the same technology that brought us this particular win, so clearly this technology is useful outside "ridiculously controlled environme…

Programmers do setup completely artificial contexts so AI can work. None of the self driving systems where setup by giving the AI access to sensors, a car, and the drivers handbook and saying well you figure it out from there. The general trend is solve this greatly simplified problem, this more complex one, up to dealing with the real world.

By AI programming I mean the AI doing programming, not programming the AI. Though soon enough the first will be doing the second and that's where the loop really closes...

Re: Competitive Programming with AlphaCode

#182

Post by Veedrac »

37ef_ced3 wrote: post #179
Veedrac wrote: post #176

Earlier quoted context omitted.

The first example is almost correct, conditioned off a sentence description. The second example is the right idea, it just bit off more than it could chew when slicing it all together. Using string ops for binary manipulation in Python isn't even stupid; it can be faster in a lot of cases. This feels a lot like screaming at a child for imperfect grammar.

It illustrates that CoPilot is generating maximum likelihood token strings and has no real understanding of the code. That's what is happening here. There is no intelligence, just regurgitation. Randomization and maximum likelihood completion. Just like with the competitive programming example, we're asking it to produce solutions that it has seen in its training set. If you ask for a nontrivial twist on one of those…

It got the value of the sixth and seventeenth bits, moved them into the right positions, and inserted them into the original value. Off a one-line description written in English! I really cannot empathize with the idea that this is not a meaningful capability. If intelligence only means to you “equal in all capabilities to an experienced human”, you are never going to be able to see anything coming ever.

Re: Competitive Programming with AlphaCode

#183

Post by machiaweliczny »

machiaweliczny wrote: post #157
udev wrote: post #94

I am thinking whether this result can create a type of loop that can self-optimize. We have AI to generate reasonable code from text problem description. Now what if the problem description text is to generate such a system in the first place? Would it be possible to close the loop, so to speak, so that over many iterations: - text description is improved - output code is improved Would it be possible to create somet…

I am actually trying this. Basically by asking questions to AI and teaching it to generate code / google when it doesn't know something. The other process checks if code is valid and either ask it to get more context or executes code and feeds back to file :)

I think one can make problem "differentiable" via some heuristics and if you have NN trained to rate code quality and some understanding what should be used for type of problem, memory and speed and than can classify problem to group then rate solution it should be able to guide the process (in competitive programming).

Re: Competitive Programming with AlphaCode

#184

Post by indiv0 »

machiaweliczny wrote: post #157
udev wrote: post #94

I am thinking whether this result can create a type of loop that can self-optimize. We have AI to generate reasonable code from text problem description. Now what if the problem description text is to generate such a system in the first place? Would it be possible to close the loop, so to speak, so that over many iterations: - text description is improved - output code is improved Would it be possible to create somet…

I am actually trying this. Basically by asking questions to AI and teaching it to generate code / google when it doesn't know something. The other process checks if code is valid and either ask it to get more context or executes code and feeds back to file :)

Do you have a blog or a github or something? This sounds really neat.

Re: Competitive Programming with AlphaCode

#185

Post by softwaredoug »

I think CoPilot, etc will be revolutionary tools AND I think human coders are needed. Specifically I love CoPilot for the task of "well specified algorithm to solve problem with well-defined inputs and outputs". The kind of problem you could describe as a coding challenge.

BUT, our jobs have a lot more complexity

- Local constraints - We almost always work in a large, complex existing code base with specific constraints

- Correctness is hard - writing lots of code is usually not the hard part, it's proving it correct against amorphous requirements, communicated in a variety of human social contexts, and bookmarked.

- Precision is extremely important - Even if 99% of the time, CoPilot can spit out a correct solution, the 1% of the time it doesn't creates a bevy of problems

Are those insurmountable problems? We'll see I suppose, but we begin to verge on general AI if we can gather and understand half a dozen modalities of social context to build a correct solution.

Not to mention much of the skill needed in our jobs has much more to do with soft skills, and the bridge between the technical and the non technical, and less to do with hardcore heads-down coding.

Exciting times!

Re: Competitive Programming with AlphaCode

#186

Post by bmc7505 »

hmate9 wrote: post #2

Between this and OpenAI's Github Copilot "programming" will slowly start dying probably. What I mean by that is that sure, you have to learn how to program, but our time will be spent much more on just the design part and writing detailed documentation/specs and then we just have one of these AIs generate the code. It's the next step. Binary code Historically its always been about abstracting and writing less code to…

I disagree that programming is dying -- tools like Copilot will lead to a Renaissance in the art of computer programming by enabling a larger population to design programs and explore the implications of their design choices. I wrote a short essay [1] on the history automated programming and where I think it is heading in the future.

[1]: https://breandan.net/public/programming_with_intelligent_mac...

Re: Competitive Programming with AlphaCode

#187

Post by falcor84 »

pkaye wrote: post #129
qualudeheart wrote: post #90

Calling it now: If current language models can solve competitive programming at an average human level, we’re only a decade or less off from competitive programming being as solved as Go or Chess. Deepmind or openAI will do it. If not them, it will be a Chinese research group on par with them. I’ll be considering a new career. It will still be in computer science but it won’t be writing a lot of code. There’ll be sev…

How long before it can write the code without plagiarizing code from online?

How long before the typical human coder can do so?

Re: Competitive Programming with AlphaCode

#188

Post by qualudeheart »

CobrastanJorji wrote: post #128
qualudeheart wrote: post #115

Earlier quoted context omitted.

Fully automating software engineering won’t happen until AGI. As a good Yuddite I expect us to have bigger problems when that happens. You need an agent with a large and coherent world model, in order to understand how your programs relate to the real world, in order to solve business tasks. This isn’t something any program synthesis tech currently available can do, because none of it has a coherent world model. GPT-…

What is a "Yuddite?" I tried Googling for it and got the impression it was LessWrong forum terminology for people who believed too strongly in LessWrong, but I couldn't find many references.

I am a follower of Elizier Yudkowsky.

Re: Competitive Programming with AlphaCode

#189

Post by chinabot »

ohwellhere wrote: post #117
jakub_g wrote: post #109

Earlier quoted context omitted.

100% agree. Someone (who?) had to take time and write the detailed requirements. In real jobs you rarely get good tickets with well defined expectations; it's one of most important developer's jobs to transform fuzzy requirement into a good ticket. (Side note: I find that many people skip this step, and go straight from fuzzy-requirement-only-discussed-on-zoom-with-Bob to code; open a pull request without much contex…

Is the next step in the evolution of programming having the programmer become the specifier? Fuzzy business requirements -> programmer specifies and writes tests -> AI codes

If its anything like my job

while(1) { Fuzzy business requirements -> programmer specifies and writes tests -> AI codes }

Re: Competitive Programming with AlphaCode

#190

Post by Jensson »

rfoo wrote: post #147
gfd wrote: post #95

Earlier quoted context omitted.

You can find the rating distribution filtered for >5 contests here: https://codeforces.com/blog/entry/71260 I am rated at 2100+ so I do agree that 1300 rating is low. But at the same time it solved https://codeforces.com/contest/1553/problem/D which is rated at 1500 which was actually non-trivial for me already. I had one wrong submit before getting that problem correct and I do estimate that 50% of the regular compe…

1553D is a quite confusing case though. On the AlphaCode Attention Visualization website [1], the Accepted code shown for 1553D is a O(n^2) Python one, which is supposed to be TLE. It correctly implements a two-pointer solution, but failed to "realize" that list.pop(0) is O(n) in Python. I'm not sure how it passed. [1] https://alphacode.deepmind.com/#layer=30,problem=34,heads=11...

Likely the python runtime has a strange string implementation for cases like this, just like javascript strings.
Post reply on HN