Competitive Programming with AlphaCode

TL;DR In 2020 community of 169 people and the best forecasters were assigning ~15% that it will happen by July 2021.

More specifically, on Dec 31, 2016 in partnership with Center for the Study of Existential Risk, Machine Intelligence Research Institute, and The Future of Life Institute they asked:

How long until a machine-learning system can take a simple text description and turn it into a program coded in C/Python?

https://www.metaculus.com/questions/405/when-will-programs-w...

First 19 forecasters in March 2017 were predicting mid-2021, the best forecasters were predicting late 2024. When the question closed in 2020 the community was predicting January 2027 and the best forecasters were predicting March 2030.

The question resolved on July 2021 when Codex was published.

Community and the best forecasters were assigning ~15% that it will happen by July 2021.

I'm currently 14th best forecaster there and I was predicting 33% before July 2021. It was my last prediction, and it was made on October 2018.

I'm also predicting 75% that we will have AGI by 2040 as defined in this question:

https://www.metaculus.com/questions/3479/when-will-the-first...

20% that it will happen before 2030.

There is also stronger operationalization:

https://www.metaculus.com/questions/5121/when-will-the-first...

My prediction here is 60% before 2040 and 5% before 2030.

I have also "canary in the coal mine" questions:

When will AI achieve competency on multi-choice questions across diverse fields of expertise? Community predicts 50% before 2030, I agree.

https://www.metaculus.com/questions/5276/ai-competence-in-di...

When will AI be able to learn to play Montezuma's Revenge in less than 30 min? Community predicts 50% before 2025, I think 50% before 2027.

https://www.metaculus.com/questions/5460/ai-rapidly-learning...

It is clear writing code will soon be something of the past; maybe it is a bad idea to train our children to code. Let's make sure we milk every penny before the party is over!

IIRC you could and Serral did set his own custom keybindings on the machine. The main difference was different keyboard and mouse.

Yes, it's the size of the search space for each problem. The search space for arbitrary programs in a language with Universal Turing Machine expressivity is infinite. Even worse, for any programming problem there are an infinite number of candidate programs that may or may not solve it and that differ in only minute ways from each other.

For Go and protein structure prediction from sequences the search space is finite, although obviously not small. So there is a huge difference in the complexity of the problems right there.

Btw, I note yet again that AlphaCode performs abysmally badly on the formal benchmark included in the arxiv preprint (see Section 5.4, and table 10). That makes sense because AlphaCode is a very dumb generate-and-test, brute-force search approach that doesn't even try to be smart and tries to make up for the lack of intelligence with an awesome amount of computational resources. Most work in program synthesis is also basically a search through the space of programs, but people in the field have come up with sophisticated techniques to avoid having to search an infinite number of programs- and to avoid having to generate millions of program candidates, like DeepMind actually brags about:

At evaluation time, we create a massive amount of C++ and Python programs for each problem, orders of magnitude larger than previous work.

They say that as if generating "orders of magnitude more" progams than previous work is a good thing, but it's not. It means their system is extremely bad at generating correct programs. It is orders of magnitude worse than earlier systems, in fact.

(The arxiv paper linked from the article quantifies this "massive" amount as "millions"; see Section 4.4).

I say maybe because so far the code that Copilot has generated for me has been impressive for what it is, but riddled with obvious and subtle bugs. It’s like outsourcing my function implementations to a C-student undergraduate intern. I definitely wouldn’t use any of its code without close scrutiny.

AI will make some software engineering tasks more efficient and more accessible but human programmers are not going anywhere any time this side of the Singularity.