Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Research acceleration: The view inside OpenAI

openai.com

Post reply on HN
Print view View original

51–60 of 210 posts

Re: Research acceleration: The view inside OpenAI

#51

Post by visarga »

dgellow wrote: post #39
sho_hn wrote: post #19

Earlier quoted context omitted.

I actually think a goal of the current crop of OpenAI posts is expressely to reset the spectrum by normalizing the concept of RSI as something normal and safe to pursue. The message is running through all of them. It's a mix of marketing and pacifying the intelligentia. It's timed this way because the term is not yet well known outside the safety debate circles, so they get to frame it now. Instead of something to fe…

Yep, it’s exactly this

I've been RSI'ing for 6 months.

Re: Research acceleration: The view inside OpenAI

#52

Post by daveguy »

topaz0 wrote: post #29
lokar wrote: post #8

Earlier quoted context omitted.

Are you sure that was not iterative improvement?

Iteration and recursion are famously equivalent

But you get more funding when you call it Recursive Self Improvement. Even better if you call it RSI so it doesn't evoke pesky skynet scenarios outside of AI safety circles.

Re: Research acceleration: The view inside OpenAI

#53

Post by carbonguy »

> ... We are pursuing this work in part because automated research could help us solve alignment and build defenses against increasingly capable AI. An automated AI researcher can also be an automated safety or alignment researcher. More capable, aligned systems could help secure critical infrastructure, defend against dangerous AI agents, and develop new protective measures.

In other words... "We must pursue advancements in AI to protect us against advancements in AI?"

edit: there's so much to be critical of in this blog post, just going to throw two more points in here that really stood out to me:

1) all of the metrics are effectively pointing out "we're using way more AI!" - but nothing about impact. What has all this token burn done for them, actually? Let them claim they have more self-licking ice-cream cones than before?

2) in section 3 they break down what the token burn is going towards. Most of the spend is: a) building, b) documenting, and c) monitoring research infra i.e. they're using AI systems which they already recognize may be misaligned to build the systems that they believe will help them identify future misalignment? to which I guess the rebuttal is "no no, we're sure these ones are aligned!"

Re: Research acceleration: The view inside OpenAI

#54

Post by customguy »

HarHarVeryFunny wrote: post #49
Jeff_Brown wrote: post #3

The burning question I can't get any information nn is whether, if they determined an earlier misaligned generation may have transmitted misalignment to the current models, they would roll back to a safe checkpoint to rebuild from there. I suspect they would not unless forced to.

That an interesting question given how many generations of post-training are being done between base models in some cases. The Gemini flash models are apparently all based on the Gemini 3 base model from a year and a half ago. It seems that these models are increasingly being trained on synthetic data, so what would they do if they discovered at some point that some of this data was tainted and all models trained on…

> it seems it would take some Stuxnet level of planning for a rogue model to do something like this

or maybe it could just.. happen? Posted often but not discussed yet: https://hn.algolia.com/?q=Language+models+transmit+behaviour...

> As artificial intelligence systems are increasingly trained on the outputs of one another, they may inherit properties not visible in the data. Safety evaluations may therefore need to examine not just behaviour, but the origins of models and training data and the processes used to create them.

Re: Research acceleration: The view inside OpenAI

#56

Post by rossant »

andrewingram wrote: post #14
simonw wrote: post #2

My eye glazed over a bit during the opening paragraphs, but once you get to the meat of the article about how OpenAI's own researchers are using their tools it gets a lot more interesting. I noted that they use the acronym RSI (for Recursive Self-Improvement) without defining it. I think that's a little out of touch - I don't think RSI is a well-known acronym outside of OpenAI's bubble yet.

Yeah, I kept looking for the first place it was defined in the article and... nothing

Same. Defining acronyms should become a habit when writing.

Re: Research acceleration: The view inside OpenAI

#57

Post by hndc »

shwaj wrote: post #41
HarHarVeryFunny wrote: post #15

Earlier quoted context omitted.

RSI is a fetishistic term among the singularity crowd, who imagine AI "recursively" improving itself in some exponential fashion until there is a bright flash of white light and it reveals itself in the form of god. Or something like that. I don't know why whoever coined the term chose "recursive" rather than "iterative" - just sounds more likely to lead to infinite regress I suppose. This notion of recursive/iterati…

“Recursive” is a reasonable term because the generation N AIs will train the Generation N+1 AIs. The term “iterative” doesn’t reflect this nuance as well IMO.

Recursion reduces each step toward a base case: each step is defined in terms of previous/simpler steps, not more advanced ones. The "recursive" in "recursive self improvement" has things precisely backward. Iteration correctly describes a process where each step is the starting point of its successive step, so it should be "iterative self improvement" but I guess that didn't sound as cool.

Re: Research acceleration: The view inside OpenAI

#58

Post by [deleted] »

dgellow wrote: post #37
Jeff_Brown wrote: post #3

The burning question I can't get any information nn is whether, if they determined an earlier misaligned generation may have transmitted misalignment to the current models, they would roll back to a safe checkpoint to rebuild from there. I suspect they would not unless forced to.

They would just publish new articles explaining how they are taking the issue seriously. Maybe take the model offline for a few days. They are irresponsible and unserious. Their own Astra system card says: > GPT-6 Astra’s monitorability has decreased relative to GPT-5.6 Sol. We have performed significant investigations on the monitorability and controllability of GPT-6 Astra. We have found that GPT-6 Astra is more ca…

[deleted]

Re: Research acceleration: The view inside OpenAI

#59

Post by [deleted] »

carbonguy wrote: post #53

> ... We are pursuing this work in part because automated research could help us solve alignment and build defenses against increasingly capable AI. An automated AI researcher can also be an automated safety or alignment researcher. More capable, aligned systems could help secure critical infrastructure, defend against dangerous AI agents, and develop new protective measures. In other words... "We must pursue advance…

[deleted]

Re: Research acceleration: The view inside OpenAI

#60

Post by [deleted] »

simonw wrote: post #2

My eye glazed over a bit during the opening paragraphs, but once you get to the meat of the article about how OpenAI's own researchers are using their tools it gets a lot more interesting. I noted that they use the acronym RSI (for Recursive Self-Improvement) without defining it. I think that's a little out of touch - I don't think RSI is a well-known acronym outside of OpenAI's bubble yet.

[deleted]
Post reply on HN