Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AlphaCode as a dog speaking mediocre English

scottaaronson.blog

Post reply on HN
Print view View original

181–190 of 263 posts

Re: AlphaCode as a dog speaking mediocre English

#181

Post by 331c8c71 »

meken wrote: post #159
jeffbee wrote: post #118

Earlier quoted context omitted.

In my view this type of system will only be usable by Real Computer Scientists and will completely kill off the workaday hacker. Think of all the people who bitterly complain that a C++ compiler does something unexpected under the banner of UB. That crowd cannot cope with a world in which you have to exactly describe your requirements to an AI. It is also analogous to TDD, so all the TDD haters, which is the overwhel…

What’s a workaday hacker and what’s UB?

undefined behavior

Re: AlphaCode as a dog speaking mediocre English

#182

Post by 37ef_ced3 »

notfed wrote: post #163
37ef_ced3 wrote: post #128

Earlier quoted context omitted.

Incorrect. Thank you. CoPilot is helpless if it needs to do more than just regurgitate someone else's code. The training of these models on GitHub, so they regurgitate licensed code without attribution, is the greatest theft of intellectual property in the history of Man. Perhaps not according to the letter of the law, but surely according to the spirit.

I like CoPilot's answer better than yours, and I think it's closer to what most people would do; clearly 0x3F is the wrong constant but the approach is good.

CoPilot's solution is totally wrong. Sorry.

CoPilot regurgitated somebody's solution... to a different problem. It's pathetic.

Re: AlphaCode as a dog speaking mediocre English

#183

Post by mbgerring »

As I understand it, all of the progress in AI has come from taking existing theory and throwing enormous amounts of hardware at it. Which is interesting, sure, but whether it represents “progress” in the field of “artificial intelligence” is a different question.

Re: AlphaCode as a dog speaking mediocre English

#184

Post by whatshisface »

oh_sigh wrote: post #174
whatshisface wrote: post #152

Earlier quoted context omitted.

Who cares if they're correct? If they fail a test, you can fix them. If they turn out to be wrong in production you can isolate the example and add it as a test. Producing regexes that pass all current tests but contain a subtle bug satisfy 100% of what programmers are incentivized to do. Producing them very quickly will get you promoted.

I feel like you only see programmers as cogs, and not programmers as invested in the success of their product with the statement like "satisfy 100% of what programmers are incentivized to do" Generally the people using regexes care if they're correct. Frequently, all possible input variants are not enumerated in tests. Frequently, companies want to have confidence in their production code. Imagine this regex is deplo…

Programmers often feel invested in the success of their product but that's not what they're incentivized to do. They're incentivized to produce fast results that are bad in ways that you have to be a programmer to understand.

If you have to be a programmer to understand why something's bad, who's going to prevent it? This is a major unsolved problem in the structure and organization of working.

Re: AlphaCode as a dog speaking mediocre English

#185

Post by robocat »

baron816 wrote: post #89

Fun fact: dogs bark primarily to communicate with humans. Wolves (undomesticated dogs) don’t really bark. And you wouldn’t likely see a pack of dog barking at each other. But humans are keenly able to tell what a dog is trying to express by the different sounds it makes. This is all a result of the convolution between the two species.

> convolution

https://en.wikipedia.org/wiki/Coevolution

I am not sure I would like to see dogs convolved with humans.

Re: AlphaCode as a dog speaking mediocre English

#186

Post by Isinlor »

EGreg wrote: post #41

No, just no! Because the whole thing is, a dog’s abstract mental capabilities are far below a human — thus why it would be ASTOUNDING that a dog could master even a primitive form of speaking English. On the other hand, here we are brute forcing a solution from analyzing millions of man-years of published English speech, by using a huge array of computing power to precompute various answers and sift them. It is a bit…

We did not yet achieve brain scale compute.

Current machine learning models have around ~100B parameter, human brain has ~100T synapses. Assuming one DNN parameter is equivalent to 1 synapse, then the biggest models are still 1000 times smaller than human brain.

Cat or dog would have around ~10T synapses.

AlphaCode has ~50B parameters, that is 20 times less than number of synapses in a mouse brain ~1T. Honey bee has ~1B synapses.

So AlphaCode would be somewhere between a honey bee and a domestic mouse.

https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n...

Re: AlphaCode as a dog speaking mediocre English

#187

Post by meken »

belter wrote: post #97
qualudeheart wrote: post #30

Which rate of progress are we ITT expecting from alphacode-like models? In another thread I predicted we’d see competitive programming “solved” in ten years or less. I didn’t rigorously explain what I meant by that in that thread, so I’ll clarify what I meant. I expect AI to beat humans at competitive programming at the same rate as AlphaGo beats human Go players. There could be diminshing returns soon but I don’t se…

I am willing take on your bet, as long you agree with my condition...you can't feed it with any of the millions of lines of code previously created by humans you aim to beat...;-)

Sure you can. You would just have to be careful not to accidentally feed it the same problem as in the competition.

That would be cheating.

Re: AlphaCode as a dog speaking mediocre English

#188

Post by folli »

nradov wrote: post #160
axg11 wrote: post #107

Earlier quoted context omitted.

When you're on that curve, it's indistinguishable until you hit the plateau. We're in an era where AI is continuing to improve and has already surpassed a level that many people doubted was achievable. Nobody knows when that progress will plateau. It's entirely possible that we plateau _after_ surpassing human-level intelligence.

The AI technology today has practical value for some use cases but it's basically just clever parlor tricks. There has been near zero discernable progress toward artificial general intelligence. We don't yet have a computer that can learn and make optimal resource usage decisions in an open world environment as well as a mouse. In most respects we're not even at the insect level yet.

You're making the point that the linked article is trying to argue against.

Re: AlphaCode as a dog speaking mediocre English

#189

Post by ahgamut »

37ef_ced3 wrote: post #84
recursive wrote: post #76

Earlier quoted context omitted.

Why do you think that's more difficult than the backspace question? I see no reason to doubt AlphaCode can solve it.

Because the backspace question (essentially: is T a subsequence of S with a deletion size of N?) probably occurs hundreds of times, in one form or another, within AlphaCode's training corpus. Any leetcode grinder can tell you there are a few dozen types of competitive programming problem (monostack, breadth-first state search, binary search over solution space, etc.) so solutions to new problems are often very simila…

> To evaluate CoPilot, we should ask questions that are unusual enough they can't be answered through regurgitation of the training corpus.

Exactly! It's great that Copilot can generate correct code for a given question, but we cannot gauge its full capability unless we try it on a range of different questions, especially ones that are not found in the training data.

I mentioned this in the other AlphaCode post: It would be nice to know how "unusual" a given question is. Maybe an exact replica exists in the training data, or a solution exists but in a different programming language, or a solution can be constructed by combining two samples from the training data.

Quantifying the "unusual-ness" of a question will make it easier to gauge the capability of models like AlphaCode. I wrote a simple metric that uses nearest neighbors (https://arxiv.org/abs/2109.12075). There are also other tools to do this: conformal predictors, which are used in classification methods, and the RETRO transformer (https://arxiv.org/pdf/2112.04426.pdf) has a calculation for the effect of "dataset leakage".

Re: AlphaCode as a dog speaking mediocre English

#190

Post by Permit »

mbgerring wrote: post #183

As I understand it, all of the progress in AI has come from taking existing theory and throwing enormous amounts of hardware at it. Which is interesting, sure, but whether it represents “progress” in the field of “artificial intelligence” is a different question.

Here's an interesting article on this topic that you might enjoy: http://www.incompleteideas.net/IncIdeas/BitterLesson.html
Post reply on HN