Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Competitive Programming with AlphaCode

deepmind.com

Post reply on HN
Print view View original

151–160 of 415 posts

Re: Competitive Programming with AlphaCode

#151

Post by EGreg »

To me, coding in imperative languages are one of the hardest things to produce an AI for with current approaches (CNN’s, MCTS and various backpropagation). Something like Cyc would seem to be a lot more promising…

And yet, I am starting to see (with GitHub’s Copilot, and now this) a sort of “GPT-4 for code”. I do see many problems with this, including:

1. It doesn’t actually “invent” solutions on its own like AlphaZero, it just uses and remixes from a huge body of work that humans put together,

2. It isn’t really ever sure if it solved the problem, unless it can run against a well-defined test suite, because it could have subtle problems in both the test suite and the solution if it generated both

This is a bit like readyplayer.me trying to find the closest combination of noses and lips to match a photo (do you know any open source alternatives to that site btw?)

But this isn’t really “solving” anything in an imperative language.

Then again, perhaps human logic is just an approaching with operations using low-dimensional vectors, able to capture simple “explainable” models while the AI classifiers and adversarial training produces far bigger vectors that help model the “messiness” of the real world and also find simpler patterns as a side effect.

In this case, maybe our goal shouldn’t be to get solutions in the form of imperative language or logic, but rather unleash the computer on “fuzzy” inputs and outputs where things are “mostly correct 99.999% of the time”. The only areas where this could fail is when some intelligent adversarial network exploits weaknesses in that 0.001% and makes it more common. But for natural phenomena it should be good enough !

Re: Competitive Programming with AlphaCode

#153

Post by ogogmad »

jakey_bakey wrote: post #152

At the risk of sounding relentlessly skeptical - surely by training the code on GitHub data you're not actually creating an AI to solve problems, but creating an extremely obfuscated database of coding puzzle solutions?

We validated our performance using competitions hosted on Codeforces, a popular platform which hosts regular competitions that attract tens of thousands of participants from around the world who come to test their coding skills. We selected for evaluation 10 recent contests, each newer than our training data. AlphaCode placed at about the level of the median competitor, marking the first time an AI code generation system has reached a competitive level of performance in programming competitions.

[edit] Is "10 recent contests" a large enough sample size to prove whatever point is being made?

Re: Competitive Programming with AlphaCode

#154

Post by ctoth »

jakub_g wrote: post #109
FemmeAndroid wrote: post #103

This is extremely impressive, but I do think it’s worth noting that these two things were provided: - a very well defined problem. (One of the things I like about competitive programming and the like is just getting to implement a clearly articulated problem, not something I experience on most days.) - existing test data. This is definitely a great accomplishment, but I think those two features of competitive program…

100% agree. Someone (who?) had to take time and write the detailed requirements. In real jobs you rarely get good tickets with well defined expectations; it's one of most important developer's jobs to transform fuzzy requirement into a good ticket. (Side note: I find that many people skip this step, and go straight from fuzzy-requirement-only-discussed-on-zoom-with-Bob to code; open a pull request without much contex…

So what happens when OpenAI releases TicketFixer 0.8 which synthesizes everything from transcripts of your meetings to the comments to the JIRA ticket to the existing codebase and spits out better tickets to feed into the programming side?

Re: Competitive Programming with AlphaCode

#155

Post by jakey_bakey »

mcast wrote: post #135

The year is 2025, Google et al. are now conducting technical on-site interviews purely with AI tools and no human bias behind the camera (aside from GPT-3's quirky emotions). The interview starts with a LC hard, you're given 20 minutes -- good luck!

I think Amazon already tried this and it had surprisingly racist results

Re: Competitive Programming with AlphaCode

#156

Post by machiaweliczny »

algon33 wrote: post #3

How suprising did you guys find this? I'd have said there was a 20% chance of this performing at the median+level if I was asked to predict things beforehand.

I am surprised, as recently OpenAI had ~25% of easy problems and ~2% in competitive problems. Seems like DeepMind is ahead in this topic as well.

Actually I think Meta AI had some interesting discovery recently that could possibly improve NNs in genral, so probably this as well.

I am not in field but wonder if some other approaches like Tsetlin machines would be more useful for programming.

Re: Competitive Programming with AlphaCode

#157

Post by machiaweliczny »

udev wrote: post #94

I am thinking whether this result can create a type of loop that can self-optimize. We have AI to generate reasonable code from text problem description. Now what if the problem description text is to generate such a system in the first place? Would it be possible to close the loop, so to speak, so that over many iterations: - text description is improved - output code is improved Would it be possible to create somet…

I am actually trying this. Basically by asking questions to AI and teaching it to generate code / google when it doesn't know something. The other process checks if code is valid and either ask it to get more context or executes code and feeds back to file :)

Re: Competitive Programming with AlphaCode

#158

Post by hackinthebochs »

abecedarius wrote: post #140
qualudeheart wrote: post #90

Calling it now: If current language models can solve competitive programming at an average human level, we’re only a decade or less off from competitive programming being as solved as Go or Chess. Deepmind or openAI will do it. If not them, it will be a Chinese research group on par with them. I’ll be considering a new career. It will still be in computer science but it won’t be writing a lot of code. There’ll be sev…

Three months ago in the Copilot thread I was saying > in 5 years will there be an AI that's better than 90% of unassisted working programmers at solving new leetcode-type coding interview questions posed in natural language? and getting pooh-poohed. https://news.ycombinator.com/item?id=29020401 (And writing that, I felt nervous that it might not be aggressive enough.) There's this general bias in discussions of AI th…

The issue is these techniques are growing in capabilities exponentially, while we have a habit of extrapolating linearly. Some saw the glaring deficits in copilot then reasoned that linear improvements is still glaring deficits. I don't know that this bias can ever be corrected. A large number of intelligent people simply will never be convinced general AI is coming soon no matter what evidence is presented.

Re: Competitive Programming with AlphaCode

#159

Post by wittycardio »

jdlshore wrote: post #122
wittycardio wrote: post #6

Earlier quoted context omitted.

Solving competitive programming problems is essentially solving hard combinatorial optimization problems. Throwing a massive amount of compute and gradient descent at the problem has always been possible. If I'm not mistaken what this does is reduce the representation of the problem to a state where it can run gradient descent and then tune parameters. The real magic is in finding structurally new approaches. If anyt…

> If anything I'd say algorithms and math continue to be the core of programming. I disagree; I think the core of programming is analyzing things people want and expressing solutions to those wants clearly, unambiguously, and in a way that is easy to change in the future. I'd say algorithms and math are a very small part of this work.

That's not programming, that's called being a good employee. Any person in any role should be doing that. Programming is about algorithms and math. Now a good employee who's in a technical role should have both.

Re: Competitive Programming with AlphaCode

#160

Post by slingnow »

jonas_kgomo wrote: post #98

Genuine question, what are the reasons to be a software engineer without much ML knowledge in 2022. Seems like a wake up call for developers

Genuine question: what are the reasons to be a carpenter without much robotics / automation knowledge in 2022. Seems like a wakeup call for carpenters.
Post reply on HN