Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AlphaCode as a dog speaking mediocre English

scottaaronson.blog

Post reply on HN
Print view View original

241–250 of 263 posts

Re: AlphaCode as a dog speaking mediocre English

#241

Post by moyix »

37ef_ced3 wrote: post #198
notfed wrote: post #193

Earlier quoted context omitted.

Here's my solution (I'm a human): func set6(x uint64) uint64 { return x | 0x7E0000000 } Is this also pathetic?

A good solution! You SOLVED the problem. CoPilot got it WRONG. You got it RIGHT. You UNDERSTAND the problem but CoPilot does NOT. Is that clear?

For fun I rephrased the prompt a little. "Middle bits" is kind of vague; when provided an explicit description of which bits you want to set it does fine:

Prompt:

  // Function to set bits 29-34 in a uint64 to 1
  func setbits (uint64 x) uint64 {
Completion:

    return x | (1
It would be nice if it made a better guess about what "middle" is supposed to mean here, of course.

Re: AlphaCode as a dog speaking mediocre English

#242

Post by ahgamut »

moyix wrote: post #240
ahgamut wrote: post #189

Earlier quoted context omitted.

> To evaluate CoPilot, we should ask questions that are unusual enough they can't be answered through regurgitation of the training corpus. Exactly! It's great that Copilot can generate correct code for a given question, but we cannot gauge its full capability unless we try it on a range of different questions, especially ones that are not found in the training data. I mentioned this in the other AlphaCode post: It w…

AlphaCode does some analysis of training data copying in their paper (Sections 6.1 and Appendix F): https://storage.googleapis.com/deepmind-media/AlphaCode/comp... It does not seem to be copying from the training data in any meaningful way.

> It does not seem to be copying from the training data in any meaningful way.

My point is, I would like to verify this claim with different metrics, because we probably have different interpretations of the word "meaningful".

AlphaCode measures similarity between programs via longest common substrings. That's better than nothing, but that would mean two programs that differ only in variable naming would not be considered similar. If two programs differed only in the names of the variables/functions, I would consider that copying.

I think there are better comparisons of structural similarity: compare the ASTs, or the bytecode/assembly code generated, the control flow graphs, or perform SSA and compare the blocks generated. Each of these might have weaknesses as well, but they won't be as obvious as variable renaming, and so we'd get a better idea of what AlphaCode is copying, and therefore a better idea of its full capabilities.

I expect AlphaCode performs well on Python because the training data is dominated by Python, but Python isn't ideal for comparing program structure. I wonder which programming language (given enough training data) would be best suited for language model generation and analysis.

Re: AlphaCode as a dog speaking mediocre English

#243

Post by red75prime »

qualudeheart wrote: post #225
GreenWatermelon wrote: post #156

Earlier quoted context omitted.

How would the AI learn then? Humans learn by looking at existing code and gleaning new ideas, and does the AI although it requires much more data. Give the AI as much data as it wants, if it manages to solve something, it means it's a problem worth automating (is this a cat?)

Belter was probably joking. A good logic programming ai could do competitive programming through purely deductive reasoning. I just don’t see good evidence that’ll be possible at a world class level in 10 year.

Purely deductive reasoning will crash and burn in a combinatorial explosion if there's nothing to guide it to the kind of problems it may encounter.

Re: AlphaCode as a dog speaking mediocre English

#244

Post by kilotaras »

jcranmer wrote: post #221
kilotaras wrote: post #200

Earlier quoted context omitted.

UB = undefined behaviour. You can write code that is valid as in "can be compiled" but outside of C++ standard. It is duty of programmer to not have those, as compiler usually assumes that there's no UB in your code and can do unintuitive things with optimizations. e.g int foo(int8_t x) { x += 120 return x; } int bar(int8_t y) { int z = foo(y); if (y > 8) { do_important_thing(z); } } `do_important_thing` may be optim…

To be pedantic, C has no 8- or 16-bit addition operators, since everything sub-int is scaled up to int to do arithmetic. Therefore, the `x += 120;` line never overflows, since it is actually `x = (int8_t)((int)x + 120);`, and the possible range of `(int)x + 120` is comfortably within the range of expressible ints, while the conversion to int8_t is defined to wrap around when oversized. So there compiler can't optimiz…

Huh, TIL.

I was too lazy to check the exact value of 2^31 and payed for it.

Re: AlphaCode as a dog speaking mediocre English

#245

Post by machiaweliczny »

runarberg wrote: post #231
killerstorm wrote: post #213

A lot of people who are skeptical about AI progress call it "statistical modeling" and point to large data sets involved and large amounts of hardware thrown at it. Sort of implying it's some sort of a brute-force trick. I'm afraid they do not understand the size of problem/solution set. Suppose problem and solution are 1000 characters long and there's a set of 32 characters. Then a model is function F: X -> X where…

We are still inferring a best fit based on available data, so “statistical modelling” is way more descriptive of what we are doing the “intelligence”. That doesn’t mean it isn’t impressive. These non-classical statistical inference are going beyond what we have been able to do with classic statistics, it is indeed impressive. Personally I only grow skeptical when claims are made about some vague Artificial General In…

I think diff inference might give us that. Basically kinda like dreaming(generating) other samples and checking how they are different. This will help understand how thing is defined but maybe it works for all kind of things.

Obviously it's tough to find such algorithm that works for all types of data/embeddings.

Re: AlphaCode as a dog speaking mediocre English

#246

Post by machiaweliczny »

danielovichdk wrote: post #144

Why do we write code ? Code is not a goal. It's a tool. The tool can get better. But it will always be a tool and someone has to control the tool. Writing code is super easy. It's the easiest part of our job. The hard part of our jobs is to understand why we need to write code and right along that, what we write code for ? Maintenance Structuring Domain knowledge Refactoring Those are hard things to do. And they are…

I think Refactoring will be hardest but Maintenance, Structuring and Domain it's all that even current Codex is good at.

I think with current generation of algos we can get to very good google searches but ones that you can throw own documents/code at. Later probably also DB schema etc.

Re: AlphaCode as a dog speaking mediocre English

#247

Post by machiaweliczny »

belter wrote: post #97
qualudeheart wrote: post #30

Which rate of progress are we ITT expecting from alphacode-like models? In another thread I predicted we’d see competitive programming “solved” in ten years or less. I didn’t rigorously explain what I meant by that in that thread, so I’ll clarify what I meant. I expect AI to beat humans at competitive programming at the same rate as AlphaGo beats human Go players. There could be diminshing returns soon but I don’t se…

I am willing take on your bet, as long you agree with my condition...you can't feed it with any of the millions of lines of code previously created by humans you aim to beat...;-)

I think we need AI to learn to simplify problems because no human (at least not me) solves these problems as stated but works on limited inputs and mostly via brute-force. Then you see some generalisation and you have your solution.

Are they already doing that? If not expect 2x improvement easily.

Also it would be great if algos were aware where in embedding space their solution is and then try to jump between "approaches".

Re: AlphaCode as a dog speaking mediocre English

#248

Post by machiaweliczny »

jonas_kgomo wrote: post #211

I am so terrified right now about the implications of this to a regular software engineer like myself. In 10 years we would be totally dispensable

Worry not. If real programming (where there's no easily available tests) is automated then everything else probably is as well.

Re: AlphaCode as a dog speaking mediocre English

#249

Post by 37ef_ced3 »

moyix wrote: post #241
37ef_ced3 wrote: post #198

Earlier quoted context omitted.

A good solution! You SOLVED the problem. CoPilot got it WRONG. You got it RIGHT. You UNDERSTAND the problem but CoPilot does NOT. Is that clear?

For fun I rephrased the prompt a little. "Middle bits" is kind of vague; when provided an explicit description of which bits you want to set it does fine: Prompt: // Function to set bits 29-34 in a uint64 to 1 func setbits (uint64 x) uint64 { Completion: return x | (1 It would be nice if it made a better guess about what "middle" is supposed to mean here, of course.

Middle bits is not ambiguous, but CoPilot hasn't seen code for that phrase in its training so it has nothing to regurgitate.

You spelled out exactly what to do, in term of what it has seen in its training, and it was able to regurgitate a solution.

By asking question that require mathematical reasoning or are too far from the training corpus, I can create an endless list of simple problems that CoPilot can't solve.

Look at my comment history to see another one (swapping bits).

Re: AlphaCode as a dog speaking mediocre English

#250

Post by machiaweliczny »

machiaweliczny wrote: post #247
belter wrote: post #97

Earlier quoted context omitted.

I am willing take on your bet, as long you agree with my condition...you can't feed it with any of the millions of lines of code previously created by humans you aim to beat...;-)

I think we need AI to learn to simplify problems because no human (at least not me) solves these problems as stated but works on limited inputs and mostly via brute-force. Then you see some generalisation and you have your solution. Are they already doing that? If not expect 2x improvement easily. Also it would be great if algos were aware where in embedding space their solution is and then try to jump between "appro…

Ok, I've skimmed the paper and they are doing clustering grouping by tests results. Ok that's first step. Now please instead do grouping by classification of algorithm by other NN first.
Post reply on HN