Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AlphaCode as a dog speaking mediocre English

scottaaronson.blog

Post reply on HN
Print view View original

201–210 of 263 posts

Re: AlphaCode as a dog speaking mediocre English

#201

Post by 37ef_ced3 »

qayxc wrote: post #194
notfed wrote: post #163

Earlier quoted context omitted.

I like CoPilot's answer better than yours, and I think it's closer to what most people would do; clearly 0x3F is the wrong constant but the approach is good.

I like the manual solution better: it includes thought process without negative impact on readability or performance. This makes it easier to match the code against the specification, something a random (and in this case even wrong) magic number fails to do. But maybe I'm overthinking it.

You're not overthinking it.

You're correct.

Re: AlphaCode as a dog speaking mediocre English

#202

Post by stephc_int13 »

shawnz wrote: post #192
stephc_int13 wrote: post #98

Earlier quoted context omitted.

We're not seeing exponential rates of progress, in my opinion. We've been on the steep part of an S-curve.

The steep part of a sigmoid curve DOES grow almost exponentially though. What's the point of saying this? It's like saying, "sure we are making great progress, but eventually the universe will come to an end and we'll all be dead." Who cares? Why not worry about that when we're there?

The point is that it is a completely different framework to think about the future.

The exponential model is something close than what is behind Ray Kurzweil reasoning about the Great Singularity and how the future will be completely different and we're all going to be gods or immortals or doomed or something dramatic in that vein.

The S-Curve is more boring, it means that the future of computing technology might not be that mind blowing after all, we might already have reaped most of the low-hanging fruits.

A bit like airplanes, or space tech you know, have you seen those improving by a 10x factor recently?

Re: AlphaCode as a dog speaking mediocre English

#203

Post by stephc_int13 »

notahacker wrote: post #149
stephc_int13 wrote: post #131

Earlier quoted context omitted.

The underlying rule here, in my opinion, is the law of diminishing returns. (log shaped curve) AlphaZero is already capable of optimizing itself in the limited problem space of Chess. Infinitely increasing the computing power of this system won't give it properties it does not already have, there is no singularity point to be found ahead. And I am not sure that there are any singularities lying ahead in any other dom…

And building on that, the real bottleneck in most domains isn't going to be computer power, it's going to be human understanding of how to curate data or tweak parameters. We've already seen in games with simple rules and win conditions that giving computers data on what we think are good human games can make them perform worse than not giving them data. Most problems aren't possible for humans to ebncapsulate perfec…

We've seen with ML and Chess (an go to some extent) that brute forcing the problem space is clever than using heuristics.

I think this is only true because, so far, our heuristics are not that clever.

Re: AlphaCode as a dog speaking mediocre English

#204

Post by vain_cain »

37ef_ced3 wrote: post #124
vain_cain wrote: post #106

Earlier quoted context omitted.

I mostly work with data mining on my personal projects(which is a couple of hours every day), and I'm pretty sure I didn't have to write a single regex since I've started using Copilot. It's hard for me to even imagine how I used to do it before, and how much time I've wasted on stupid mistakes and typos. Now I just write a comment of what I want and an example string. It does the job without me having to modify anyt…

How do you know the regular expressions are correct, without understanding them?

I never said anything about not understanding them. If it's something simple, like getting a date from a string, a quick glance will tell me if it'll work. If it's more complex, a quick glance or two will give me the general idea and then I can test it against what I think will be edge cases. If you don't know what kind of a string you'll have to process then you can't really know if any regex is correct, and if you do testing it in most cases is pretty easy and quick. You'd have to test even if you write it yourself. And in the cases where it's wrong it usually gives me a good starting point.

Re: AlphaCode as a dog speaking mediocre English

#205

Post by zacmps »

Jack000 wrote: post #101

I think this type of model will have a massive impact on the software industry. 99% of programming tasks in the wild don't involve any kind of algorithmic design, but are more like making a CRUD pattern, writing SQL queries etc. This kind of work is easier to automate but more difficult to source the training data. If and when these models are applied to more mundane problems, I'd expect immediately better performanc…

> source code is really meant for humans to read. The canonical form of software is more like the object code produced by the compiler. You can probably just produce this directly

They key advantage of producing source code is that you can usually tell what the produced program does.

Re: AlphaCode as a dog speaking mediocre English

#206

Post by solveit »

whatshisface wrote: post #184
oh_sigh wrote: post #174

Earlier quoted context omitted.

I feel like you only see programmers as cogs, and not programmers as invested in the success of their product with the statement like "satisfy 100% of what programmers are incentivized to do" Generally the people using regexes care if they're correct. Frequently, all possible input variants are not enumerated in tests. Frequently, companies want to have confidence in their production code. Imagine this regex is deplo…

Programmers often feel invested in the success of their product but that's not what they're incentivized to do. They're incentivized to produce fast results that are bad in ways that you have to be a programmer to understand. If you have to be a programmer to understand why something's bad, who's going to prevent it? This is a major unsolved problem in the structure and organization of working.

The CTO who used to be a dev. More generally anyone in management with a technical background. They may not exist in some companies, but that's not because it's "a major unsolved problem in the structure and organization of working", it's because the company sucks in that regard.

Re: AlphaCode as a dog speaking mediocre English

#207

Post by solveit »

oblvious-earth wrote: post #199
axg11 wrote: post #107

Earlier quoted context omitted.

When you're on that curve, it's indistinguishable until you hit the plateau. We're in an era where AI is continuing to improve and has already surpassed a level that many people doubted was achievable. Nobody knows when that progress will plateau. It's entirely possible that we plateau _after_ surpassing human-level intelligence.

If you have actual data you can take the derivative of the curve and see you're on the S curve with a lot of confidence by the time you hit the middle and long before the plateau: https://miro.medium.com/max/700/1*6A3A_rt4YmumHusvTvVTxw.png

Even small errors in measurement will greatly change the location of the projected plateau so this is not usually useful in practice.

Re: AlphaCode as a dog speaking mediocre English

#208

Post by Isinlor »

qayxc wrote: post #197
Isinlor wrote: post #27

Earlier quoted context omitted.

Roughly half/half, skewed more towards improvements in algorithms. Measuring the Algorithmic Efficiency of Neural Networks: https://arxiv.org/abs/2005.04305 In this work, we argue that algorithmic progress has an aspect that is both straightforward to measure and interesting: reductions over time in the compute needed to reach past capabilities. We show that the number of floating-point operations required to train a…

Where did you get the $10M figure for GPT-3? That sounds awfully cheap considering the cost of compute alone: one estimate was $4.6M for a single training run [0], while other sources [1] put it at $12M per run. I highly doubt that OpenAI nailed the training process right on the second or even first go respectively (according to your figure). So even conservative estimates put the compute cost alone at least one orde…

Yes, I was using the $12M estimate. Take it as within order of magnitude approximation.

Re: AlphaCode as a dog speaking mediocre English

#209

Post by p1esk »

nlitened wrote: post #164
whatshisface wrote: post #152

Earlier quoted context omitted.

Who cares if they're correct? If they fail a test, you can fix them. If they turn out to be wrong in production you can isolate the example and add it as a test. Producing regexes that pass all current tests but contain a subtle bug satisfy 100% of what programmers are incentivized to do. Producing them very quickly will get you promoted.

I think we will very soon start seeing a clear separation between programmers ("co-pilot operators") and software engineers (those who do the thinking and understanding when there's someone "who cares").

This separation has existed since the first computer was built.

Re: AlphaCode as a dog speaking mediocre English

#210

Post by jonas_kgomo »

vain_cain wrote: post #106
mark_l_watson wrote: post #38

Right on, I couldn’t agree more. We are living during a period of exponential progress. I like AlphaCode’s approach of using language models with search. In the last year I have experimented with mixing language models for NLP with semantic web/linked data tasks, so much simpler than what AlphaCode does, but I have been having fun. I have added examples for this in new additions to two of my books, but if you want a…

I mostly work with data mining on my personal projects(which is a couple of hours every day), and I'm pretty sure I didn't have to write a single regex since I've started using Copilot. It's hard for me to even imagine how I used to do it before, and how much time I've wasted on stupid mistakes and typos. Now I just write a comment of what I want and an example string. It does the job without me having to modify anyt…

would like to talk about the paintpoints you experienced while doing data preparation and mining
Post reply on HN