Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Research acceleration: The view inside OpenAI

openai.com

Post reply on HN
Print view View original

151–160 of 210 posts

Re: Research acceleration: The view inside OpenAI

#151

Post by HarHarVeryFunny »

red75prime wrote: post #118
HarHarVeryFunny wrote: post #32

Earlier quoted context omitted.

> How do you think why there's this fad of producing general purpose humanoid robots? For doing physical work? So a swarm of robots builds the shell of your fab overnight, and then what? Where is the EUV machine coming from? So far the most we're seen TeslaBot do is serve drinks via tele-operation, and I don't think it's exactly built for construction site work.

For example, TSMC uses behavioral cloning to scale up human-bottlenecked parts of the manufacturing process to meet the growing demand, while automated research laboratories do thousands experiments in parallel to find better manufacturing processes.

The production bottleneck in a fab isn't the human workers - the process is mostly automated. The bottleneck more derives from how many wafers per hour you can process, which comes down to the etching process and EUV throughput.

ASMLs EUV machines are literally the most complex machine that mankind has ever built, which is why no other country, including the US, has yet been able to duplicate it. It's not just the machine itself, but a global supply chain of irreplaceable components such as focusing mirrors made by Zeiss to an incomprehensible level of accuracy - differences in surface height no more than the size of a hydrogen atom (or if you scaled the mirror up to the size of the country of Germany, then surface differences in height of 0.1mm).

Robots are useful to automate things, but they are zero help when trying to build tech like this that you are incapable of building in the first place.

The US has fallen way behind in manufacturing expertise, and no swarm of robots is going to help.

Re: Research acceleration: The view inside OpenAI

#152

Post by red75prime »

HarHarVeryFunny wrote: post #151
red75prime wrote: post #118

Earlier quoted context omitted.

For example, TSMC uses behavioral cloning to scale up human-bottlenecked parts of the manufacturing process to meet the growing demand, while automated research laboratories do thousands experiments in parallel to find better manufacturing processes.

The production bottleneck in a fab isn't the human workers - the process is mostly automated. The bottleneck more derives from how many wafers per hour you can process, which comes down to the etching process and EUV throughput. ASMLs EUV machines are literally the most complex machine that mankind has ever built, which is why no other country, including the US, has yet been able to duplicate it. It's not just the ma…

You've missed a part where it's TSMC that does behavioral cloning (to build more EUV machines). The full vertical integration is a bit farther down the line.

Etching a model's weights on silicon is another way to utilize non-top-notch tech-processes, while maintaining or improving performance. (and it suits robotics well)

Re: Research acceleration: The view inside OpenAI

#153

Post by whateverboat »

> For AGI to benefit all of humanity, we believe it must be democratically governed. This can only happen through an informed public debate about the capabilities, risks and safeguards of highly capable AI systems. People everywhere need to understand the likely future trajectory of frontier AI, so they can have a meaningful voice in how it develops.

This first and foremost also means that means of generating intelligence should be democratically available to everyone.

Re: Research acceleration: The view inside OpenAI

#154

Post by HarHarVeryFunny »

jazzyjackson wrote: post #16
HarHarVeryFunny wrote: post #15

Earlier quoted context omitted.

RSI is a fetishistic term among the singularity crowd, who imagine AI "recursively" improving itself in some exponential fashion until there is a bright flash of white light and it reveals itself in the form of god. Or something like that. I don't know why whoever coined the term chose "recursive" rather than "iterative" - just sounds more likely to lead to infinite regress I suppose. This notion of recursive/iterati…

Yes the exponential self improvement folks have never heard of an eigenvalue I guess. You can loop forever using output as input but at some point the result will stop changing (depending on the function)

I think the limit of what can be achieved with RL and synthetic data generation is better simply described as a leveling off of gains as you extract all the intelligence and knowledge from the original human training data.

Of course things will change at some point in the future as we go beyond LLMs, to build creative intelligence not just imitative/predictive intelligence, but right now these companies are stuck in this loop of building synthetic data and RLVR training from that, which means they are essentially building the "generative closure" of the original human training data - trying to squeeze all the juice out of it.

To go beyond this they need to add creativity of some sort to generate data that is not ultimately based on the original human training data. They could try something like brute force search (cf agent swarms/graphs), but this is just a more thorough way of exploring the search space defined by the training data - it may find you the "move 37" or low-hanging mathematical proof, but as Demis Hassabis has said, the goal of AGI is not to find move 37 but rather to create something capable of inventing as compelling a game as Go in the first place.

Re: Research acceleration: The view inside OpenAI

#155

Post by 12eeie »

dsign wrote: post #133

It's a funny read if you pull together "AI 2027" and what we all know is going on. Essentially, open AI employee or model is writing "things are going exactly as bad as AI 2027 predicted, but my (golden/RL-) cuffs are too heavy and all I can do is publish this code-speak for 'send help'". It's not a pretty place to be.

Yup the doom and gloom posts are not only pathetic but demonstrate how little people can think for themselves.

Re: Research acceleration: The view inside OpenAI

#156

Post by hgoel »

nojs wrote: post #74
hedgehog wrote: post #13

This roughly lines up with my personal experience that in March a combination of stronger models and better tooling on my end let me start running jobs unattended 24/7 (using Anthropic sub and my own hardware). Their $8000/day per researcher spend is crazy though, I'm curious how they keep track of the work.

> let me start running jobs unattended 24/7 (using Anthropic sub and my own hardware) How are you running jobs unattended 24/7 without hitting your token limits?

It depends on the time the job itself takes. If you're having the LLM handle a training run for another model, the LLM is probably spending most of its time waiting for iterations rather than consuming tokens.

For a task I left a local model running on overnight, only ~100k tokens were used because most of the time was just waiting on tests to finish, then waking up, tweaking a few settings and trying again.

Re: Research acceleration: The view inside OpenAI

#157

Post by 12eeie »

piokoch wrote: post #142

One more marketing stunt. We are so good, AI is so powerful so we need to use AI to fight with it. The message is: if you don't buy from us, your competitor will purchase all of this amazing power... I understand that investors are buying this, after all they believed in all of other crap that led to the 2008 crisis, but please...

Yup it’s getting annoying

What they’re doing is strategic for both insiders and investors - they know china is coming so they need to pull theatrics to keep the valuations inflated.

I personally test all models all the time - chinese models are right up there and superior when you actually do the proper economic analysis.

Re: Research acceleration: The view inside OpenAI

#158

Post by ryan_n »

gatio wrote: post #125
bix6 wrote: post #103

Earlier quoted context omitted.

Personally I’d like to see them actually start benefiting humanity by doing all the things Sam has claimed they will like curing disease, cancer, global warming, etc. But I guess a computer intern so we can avoid paying / training the next generation is better.

> Personally I’d like to see them actually start benefiting humanity by doing all the things Sam has claimed they will like curing disease, cancer, global warming, etc. It makes more sense to leave curing disease & cancer to the experts, with tools (like AI) being developed by AI experts. Call me crazy, but I want separate organizations and experts for medical vs finance vs space vs climate vs AI research.

What the op was pointing out is that guys like Altman and Dario are repeatedly saying they’re going to cure xyz diseases and solve xyz huge global problems. Maybe their companies will eventually do these things, but haven’t yet.

I don’t have an opinion either way, I think it’s too soon to tell if llms will be able to cure cancer or whatever. But at the very least it will be a good tool to help researchers do their jobs.

Re: Research acceleration: The view inside OpenAI

#159

Post by 21asdffdsa12 »

Yoric wrote: post #131
p1esk wrote: post #98

Earlier quoted context omitted.

I've hired many AI research interns (and was one many years ago), and I agree with them - frontier models are currently at the level of an average AI research intern.

Am I the only one who's a bit disappointed that we're spending trillions, destroying the ecosystem, drowning democracies and learning in slop, preparing a big financial crash, all of this to achieve an "average AI research intern"? A long time ago, I used to be a (AI-adjacent) research intern, and frankly, I wouldn't trust any non-trivial task to that younger me. Fortunately, by opposition to an already trained LLM o…

Is this not what you wanted? You created a culture that dissolves responsibility by making it the worst thing to strife for - so everyone dissolves it, in processes, mass decisions and AI. Its the system, society, god, the great spirit. This is what you strove for, how can you be unhappy with things you demanded yourself?

Re: Research acceleration: The view inside OpenAI

#160

Post by andrethegiant »

piyh wrote: post #12
Jeff_Brown wrote: post #3

The burning question I can't get any information nn is whether, if they determined an earlier misaligned generation may have transmitted misalignment to the current models, they would roll back to a safe checkpoint to rebuild from there. I suspect they would not unless forced to.

Opus was trained based on it's internal CoT due to a bug for generations. Gemini's depression extended through models. OpenAI has killed people. We've already seen cross gen misalingment.

Source?
Post reply on HN