Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Research acceleration: The view inside OpenAI

openai.com

Post reply on HN
Print view View original

201–210 of 210 posts

Re: Research acceleration: The view inside OpenAI

#201

Post by ACCount37 »

sfn42 wrote: post #200
ACCount37 wrote: post #199

Earlier quoted context omitted.

Yes, I do. IPCC's reports are sensible. They're not unreliable just because they don't support the "doom and burning land" narratives. By the way, there is no "just realized we're missing 1.5C". That projection was always the very low end of possibilities - the "assume rapid, radical climate action on global level" scenario. Yes, that's a dumb thing to assume. We've never been on track for it. But the "assume extreme…

I'm pessimistic. We're still producing more CO2 each year than the last, and several feedback loops are kicking in that accelerate the warming further such as permafrost thawing, arctic and antarctic sea ice disappearing, glaciers are melting, the amazon is being demolished, etc. I don't know how much of this is included in the projections. I hope the optimists are right, it just doesn't look like it to me at all. It…

As a rule: feedback loops are overrated.

They are, in fact, included in the projections - we'd be on track to ~2C by 2100 instead of ~3C by 2100 if they weren't. They just aren't that big.

There is no "Make Earth Into Venus Feedback Loop Of Doom" that a lot of people seem to imagine when they hear "feedback loop". There is, however, a dozen of things that add about +5% each.

Re: Research acceleration: The view inside OpenAI

#202

Post by dextrous »

dextrous wrote: post #169
nozzlegear wrote: post #77

I want an all-powerful AI that's aligned with my values, but not necessarily yours. Is that so much to ask for?

See Amodei’s comments regarding Iain M Banks’s Culture, his goal is benevolent machine rule. I suspect many HN folks would agree; I, for one, was rooting for the Iridians.

typo: Idirans

Re: Research acceleration: The view inside OpenAI

#203

Post by Yoric »

21asdffdsa12 wrote: post #159
Yoric wrote: post #131

Earlier quoted context omitted.

Am I the only one who's a bit disappointed that we're spending trillions, destroying the ecosystem, drowning democracies and learning in slop, preparing a big financial crash, all of this to achieve an "average AI research intern"? A long time ago, I used to be a (AI-adjacent) research intern, and frankly, I wouldn't trust any non-trivial task to that younger me. Fortunately, by opposition to an already trained LLM o…

Is this not what you wanted? You created a culture that dissolves responsibility by making it the worst thing to strife for - so everyone dissolves it, in processes, mass decisions and AI. Its the system, society, god, the great spirit. This is what you strove for, how can you be unhappy with things you demanded yourself?

Who, me?

Re: Research acceleration: The view inside OpenAI

#204

Post by andai »

jaggederest wrote: post #178
andai wrote: post #174

Earlier quoted context omitted.

Thanks! > If I had that many tokens/dollars I would be running canaries and adversarial verification in prod based on e.g. traffic replay, live fuzzing, all kinds of things to build confidence without direct human line-by-line review. This part jumped out at me. There's something to watch out for here. I recently had a funny experience. I delegated a major feature to an agent. It turned out that it had implemented it…

Yes, you need some kind of other source of truth. I think the best way to get that is to do clean room development with a different agent, but ultimately if you give them the wrong idea they'll do the wrong thing. The other thing I do, not as much as I should, but it's very powerful, is to generate spikes and deliberately throw them away to understand how to prompt better. Like I generated a swift version of the reac…

> I think the best way to get that is to do clean room development with a different agent

What is this about? Could you give an example?

Re: Research acceleration: The view inside OpenAI

#205

Post by HarHarVeryFunny »

marcosdumay wrote: post #102
jazzyjackson wrote: post #16

Earlier quoted context omitted.

Yes the exponential self improvement folks have never heard of an eigenvalue I guess. You can loop forever using output as input but at some point the result will stop changing (depending on the function)

The name you are looking for is "fixed points", not "eingevalues".

The "You can loop forever using output as input but at some point the result will stop changing" idea seems more related to dynamics, feedback and attractor states.

If you have a feedback loop of "recursively improving" by generating synthetic data, to train on, to generate more synthetic data, etc, then this is indeed a matter of looping "using output as input", and without any other system inputs it would indeed "converge" to some attractor states.

Re: Research acceleration: The view inside OpenAI

#206

Post by anthonyrstevens »

achierius wrote: post #167
mrob wrote: post #145

Earlier quoted context omitted.

Calling a machine with no drives beyond maximizing a number a "slave" is far worse than saying it "thinks". The problem isn't the emotive language, it's that it implies human motivations such as self-preservation and desire for freedom that it doesn't have. Even on HN, people regularly claim it would be "irrational" for an ASI to do things like killing all biological life. That would be irrational for a slave, but no…

Have you read a single paper in ai safety?

Magic 8-Ball says:

VERY DOUBTFUL

Re: Research acceleration: The view inside OpenAI

#207

Post by anthonyrstevens »

shwaj wrote: post #89
hndc wrote: post #57

Earlier quoted context omitted.

Recursion reduces each step toward a base case: each step is defined in terms of previous/simpler steps, not more advanced ones. The "recursive" in "recursive self improvement" has things precisely backward. Iteration correctly describes a process where each step is the starting point of its successive step, so it should be "iterative self improvement" but I guess that didn't sound as cool.

I think you’re conflating the direction of definition with the direction of evaluation. Compare the similarity of: AI(n) = improve(AI(n-1)) With: Fib(n) = Fib(n-1) + Fib(n-2) The latter is a classic example of recursion. So why isn’t the former? Edit: formatting

I think you're confusing a recurrence relationship with recursive execution. Until AI has control over its inputs (including physical inputs - compute hardware, etc.) it's not quite RSI. We'll get there. They'll get there, rather.

Re: Research acceleration: The view inside OpenAI

#208

Post by jaggederest »

andai wrote: post #204
jaggederest wrote: post #178

Earlier quoted context omitted.

Yes, you need some kind of other source of truth. I think the best way to get that is to do clean room development with a different agent, but ultimately if you give them the wrong idea they'll do the wrong thing. The other thing I do, not as much as I should, but it's very powerful, is to generate spikes and deliberately throw them away to understand how to prompt better. Like I generated a swift version of the reac…

> I think the best way to get that is to do clean room development with a different agent What is this about? Could you give an example?

So, if I'm building, say an iOS app, I have a) the code itself (swift/react native), b) the internal test suite (unit tests), and c) the e2e test suite with something like XCUITest that drives the physical development phone, plus d) the backend (which has similar tests but is basically a mirror, so I won't elaborate)

So when you're creating a (and hopefully b), you use one model with one context, and then you might use some other model to implement c. I kind of round robin the models and present them with seperate context - so for example I don't build a, b, and c together, I build a plus some b, then later go for a pass over b and c together, then maybe I used c to drive improvements in a, and I vary between openai and anthropic models as I do.

Heres an example set of minimal prompts:

a-focus:

implement feature based on #ticket in github, be sure to reference the engineering standards documents and the swift and react native skills as needed

b-focus:

improve test coverage in the repo for to ensure that is covered completely, and fix any outstanding gaps or omissions in that feature as you go (standard references above)

c-focus (possibly in a separate repo):

You are creating a black box xcuitest to drive a physical phone for testing , here is the user specification and known issues, create failing tests for each known issue and an overall robust suite to ensure any user facing or ui issues are caught

Re: Research acceleration: The view inside OpenAI

#209

Post by ryan_n »

SamPatt wrote: post #192
ryan_n wrote: post #161

Earlier quoted context omitted.

I think people very much should care about who ends up owning these solutions. The person or entity that controls things like that just has more power, which isn’t necessarily a good thing

It is a good thing when it didn't exist before and it does exist now and wouldn't have existed without them. If they profit immensely from curing cancer, good.

And if they decide they are in charge of determining who gets this cancer cure and who doesn’t? Still good?

Maybe it is, idk, you may be right. But I think it’s something people should care about.

Re: Research acceleration: The view inside OpenAI

#210

Post by mjburgess »

ekidd wrote: post #90
mjburgess wrote: post #82

Earlier quoted context omitted.

The claim is that the RSI operation is just finding a fixed point of improvement, RSI(LLM ) = RSI(LLM ) -- for an optimal LLM* which is a fixed point of RSI As for eigenvalues/vectors, they're fixed points of (1/val)A or A*val

Eigenvectors represent fixed directions , not fixed magnitudes . From Wikipedia: > More precisely, an eigenvector v of a linear transformation T is scaled by a constant factor lambda when the linear transformation is applied to it: Tv = lambda v . In other words, repeated multiplication of an eigenvector by a matrix can still create exponential growth.

> of (1/val)A
Post reply on HN