Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AlphaCode as a dog speaking mediocre English

scottaaronson.blog

Post reply on HN
Print view View original

191–200 of 263 posts

Re: AlphaCode as a dog speaking mediocre English

#191

Post by YeGoblynQueenne »

>> Update: A colleague of mine points out that one million, the number of candidate programs that AlphaCode needs to generate, could be seen as roughly exponential in the number of lines of the generated programs.

To clarify, "one million" is the number of programs generated by AlphaCode on the CodeContests dataset, not the CodeForces one (although the former is a superset of the latter). The results on CodeContests are reported in table 5 (page 15 of the pdf) of the DeepMind preprint [1]. The results on CodeForces are reported in table 4 of the preprint (page 14 of the pdf).

I can't find where the paper lists the number of samples drawn for the CodeForces results, but on Section 4.4 the preprint says:

Sampling from transformer models can be easily parallelized, which allowed us to scale to millions of samples per problem (...).

Note the plural. "Millions" can mean one million, or a hundred million, but, as far as I can tell, the preprint never makes it clear which it is. Note that the results on CodeForces (table 4) are averaged over 3 evaluations in each of which AlphaCode generated "millions" of samples and finally submitted 10. So it's 3 times "millions" of samples for the actual results in table 4 (CodeForces). I assume those were the top 3 of all evaluations.

The idea that the lines of code in a target program are related to the cardinality of the program space that must be searched before a target program can be found is not completely unfounded. For sure, the cardinality of the search space for programs is some function of the number of tokens that must be combined to form each program in that space. An _exponential_ function because we're talking about combinations of tokens (assuming only grammatical strings are generated it gets a bit better, but not by much). We can take lines-of-code as a rough proxy of number of tokens, and in any case it's clear that the cardinality of the set of one-line programs is less than the cardinality of the set of two-line programs, and so on.

I'm not sure if the "Update" bit above is claiming, that AlphaCode must generate the _entire_ program space of _all_ k-line programs before it can find a solution to that problem. To be honest, even I don't think that AlphaCode is _that_ bad. Having to go through the entire program space to find one program is the worst case. On the other hand, AlphaCode does perform very poorly so who knows?

Regarding talking dogs, I have never met any, but I have heard of a horse that can do arithmetic [2].

__________

[1] https://storage.googleapis.com/deepmind-media/AlphaCode/comp...

[2] https://en.wikipedia.org/wiki/Clever_Hans

Re: AlphaCode as a dog speaking mediocre English

#192

Post by shawnz »

stephc_int13 wrote: post #98
mark_l_watson wrote: post #38

Right on, I couldn’t agree more. We are living during a period of exponential progress. I like AlphaCode’s approach of using language models with search. In the last year I have experimented with mixing language models for NLP with semantic web/linked data tasks, so much simpler than what AlphaCode does, but I have been having fun. I have added examples for this in new additions to two of my books, but if you want a…

We're not seeing exponential rates of progress, in my opinion. We've been on the steep part of an S-curve.

The steep part of a sigmoid curve DOES grow almost exponentially though. What's the point of saying this? It's like saying, "sure we are making great progress, but eventually the universe will come to an end and we'll all be dead." Who cares? Why not worry about that when we're there?

Re: AlphaCode as a dog speaking mediocre English

#193

Post by notfed »

37ef_ced3 wrote: post #182
notfed wrote: post #163

Earlier quoted context omitted.

I like CoPilot's answer better than yours, and I think it's closer to what most people would do; clearly 0x3F is the wrong constant but the approach is good.

CoPilot's solution is totally wrong. Sorry. CoPilot regurgitated somebody's solution... to a different problem. It's pathetic.

Here's my solution (I'm a human):

    func set6(x uint64) uint64 {
       return x | 0x7E0000000
    }
Is this also pathetic?

Re: AlphaCode as a dog speaking mediocre English

#194

Post by qayxc »

notfed wrote: post #163
37ef_ced3 wrote: post #128

Earlier quoted context omitted.

Incorrect. Thank you. CoPilot is helpless if it needs to do more than just regurgitate someone else's code. The training of these models on GitHub, so they regurgitate licensed code without attribution, is the greatest theft of intellectual property in the history of Man. Perhaps not according to the letter of the law, but surely according to the spirit.

I like CoPilot's answer better than yours, and I think it's closer to what most people would do; clearly 0x3F is the wrong constant but the approach is good.

I like the manual solution better: it includes thought process without negative impact on readability or performance.

This makes it easier to match the code against the specification, something a random (and in this case even wrong) magic number fails to do.

But maybe I'm overthinking it.

Re: AlphaCode as a dog speaking mediocre English

#195

Post by yojo »

Jack000 wrote: post #101

I think this type of model will have a massive impact on the software industry. 99% of programming tasks in the wild don't involve any kind of algorithmic design, but are more like making a CRUD pattern, writing SQL queries etc. This kind of work is easier to automate but more difficult to source the training data. If and when these models are applied to more mundane problems, I'd expect immediately better performanc…

Reading this I’m reminded of the debates around ORMs. At a basic level they drastically simplify your CRUD app. Until they make trivial errors no self-respecting programmer would (think N+1 queries), and then you need someone who actually understands what’s going on to fix it.

That doesn’t mean you shouldn’t ever use ORMs, or that in simple cases they aren’t “good enough”. But at some level of complexity it breaks down.

AI-assisted programming is the new leaky abstraction.

Re: AlphaCode as a dog speaking mediocre English

#196

Post by robrorcroptrer »

folli wrote: post #188
nradov wrote: post #160

Earlier quoted context omitted.

The AI technology today has practical value for some use cases but it's basically just clever parlor tricks. There has been near zero discernable progress toward artificial general intelligence. We don't yet have a computer that can learn and make optimal resource usage decisions in an open world environment as well as a mouse. In most respects we're not even at the insect level yet.

You're making the point that the linked article is trying to argue against.

How good is this model at fighting other insects?

Re: AlphaCode as a dog speaking mediocre English

#197

Post by qayxc »

Isinlor wrote: post #27
turminal wrote: post #9

It would be interesting to know how much of this improvement in the last 25 since he was a student comes from Moore's law, other hardware improvements, various new technologies not related to AI, amount of money being thrown at the problem... and how much of it are advancements in our understanding of AI.

Roughly half/half, skewed more towards improvements in algorithms. Measuring the Algorithmic Efficiency of Neural Networks: https://arxiv.org/abs/2005.04305 In this work, we argue that algorithmic progress has an aspect that is both straightforward to measure and interesting: reductions over time in the compute needed to reach past capabilities. We show that the number of floating-point operations required to train a…

Where did you get the $10M figure for GPT-3? That sounds awfully cheap considering the cost of compute alone: one estimate was $4.6M for a single training run [0], while other sources [1] put it at $12M per run. I highly doubt that OpenAI nailed the training process right on the second or even first go respectively (according to your figure).

So even conservative estimates put the compute cost alone at least one order of magnitude higher than your figure of $10M.

[0] https://lambdalabs.com/blog/demystifying-gpt-3/#:~:text=But%....

[1] https://venturebeat.com/2020/06/01/ai-machine-learning-opena...

Re: AlphaCode as a dog speaking mediocre English

#198

Post by 37ef_ced3 »

notfed wrote: post #193
37ef_ced3 wrote: post #182

Earlier quoted context omitted.

CoPilot's solution is totally wrong. Sorry. CoPilot regurgitated somebody's solution... to a different problem. It's pathetic.

Here's my solution (I'm a human): func set6(x uint64) uint64 { return x | 0x7E0000000 } Is this also pathetic?

A good solution! You SOLVED the problem.

CoPilot got it WRONG. You got it RIGHT.

You UNDERSTAND the problem but CoPilot does NOT.

Is that clear?

Re: AlphaCode as a dog speaking mediocre English

#199

Post by oblvious-earth »

axg11 wrote: post #107
stephc_int13 wrote: post #98

Earlier quoted context omitted.

We're not seeing exponential rates of progress, in my opinion. We've been on the steep part of an S-curve.

When you're on that curve, it's indistinguishable until you hit the plateau. We're in an era where AI is continuing to improve and has already surpassed a level that many people doubted was achievable. Nobody knows when that progress will plateau. It's entirely possible that we plateau _after_ surpassing human-level intelligence.

If you have actual data you can take the derivative of the curve and see you're on the S curve with a lot of confidence by the time you hit the middle and long before the plateau: https://miro.medium.com/max/700/1*6A3A_rt4YmumHusvTvVTxw.png

Re: AlphaCode as a dog speaking mediocre English

#200

Post by kilotaras »

meken wrote: post #159
jeffbee wrote: post #118

Earlier quoted context omitted.

In my view this type of system will only be usable by Real Computer Scientists and will completely kill off the workaday hacker. Think of all the people who bitterly complain that a C++ compiler does something unexpected under the banner of UB. That crowd cannot cope with a world in which you have to exactly describe your requirements to an AI. It is also analogous to TDD, so all the TDD haters, which is the overwhel…

What’s a workaday hacker and what’s UB?

UB = undefined behaviour.

You can write code that is valid as in "can be compiled" but outside of C++ standard. It is duty of programmer to not have those, as compiler usually assumes that there's no UB in your code and can do unintuitive things with optimizations.

e.g

  int foo(int8_t x) {
     x += 120
     return x;
  }

  int bar(int8_t y) {
     int z = foo(y);
     if (y > 8) {
         do_important_thing(z);
     }
  }
`do_important_thing` may be optimized out because:

1. signed overflow is a UB. Compiler than assumes that everything passed to foo is less than 8;

2. We pass y to foo => y 3. if branch can then be optimized out

Post reply on HN