Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Neural Network Diffusion

arxiv.org

Post reply on HN
Print view View original

51–60 of 92 posts

Re: Neural Network Diffusion

#51

Post by astrange »

rralian wrote: post #46
astrange wrote: post #40

Earlier quoted context omitted.

I think people can read books (self improvement) and have children (recursive), but neither of those are both.

New generations build onto the scientific knowledge of previous generations. It may not be fast but that sounds like recursive improvement to me. It seems reasonable for AI to accelerate this process.

I think saying all of society is doing it is plausible, but not the same thing as a single human or AI doing it.

Though… still don't think it's true. Isn't "society is self improving" what they call Whig history?

Re: Neural Network Diffusion

#52

Post by astrange »

always2slow wrote: post #47
astrange wrote: post #34

Earlier quoted context omitted.

> Is there a law of thermodynamics which prevents AI from writing code which would train a better AI? You need to apply Wittgenstein here. This appears to be true because you haven't defined "better". If you define it, it'll become obvious that this is either false or true, but if it is true it'll be obvious in a way that doesn't make it sound interesting anymore. (For one thing our current "AI" don't come from "writ…

>> I guess people working there believe in magic. >Yes, OpenAI was literally founded by a computer worshipping religious cult. What cult is this?

HPMOR readers who live in group home polycules in Berkeley who think they need to invent a good computer god to stop the evil computer god.

Re: Neural Network Diffusion

#53

Post by astrange »

alanbernstein wrote: post #41
astrange wrote: post #34

Earlier quoted context omitted.

> Is there a law of thermodynamics which prevents AI from writing code which would train a better AI? You need to apply Wittgenstein here. This appears to be true because you haven't defined "better". If you define it, it'll become obvious that this is either false or true, but if it is true it'll be obvious in a way that doesn't make it sound interesting anymore. (For one thing our current "AI" don't come from "writ…

> they just come from training bigger models on the same data Are you arguing that all AI models are using the same network structure? This is only true in the most narrow sense, looking at models that are strictly improvements over previous generation models. It ignores the entire field of research that works by developing new models with new structures, or combining ideas from multiple previous works.

I sure am ignoring that, because the bitter lesson of AI is usually applicable and implies that all such research will be replaced by larger generic transformer networks as time goes on.

The exception is when you care about efficiency (in training or inference costs) but at the limit or if you care about "better" then you don't.

Re: Neural Network Diffusion

#54

Post by Vecr »

astrange wrote: post #52
always2slow wrote: post #47

Earlier quoted context omitted.

>> I guess people working there believe in magic. >Yes, OpenAI was literally founded by a computer worshipping religious cult. What cult is this?

HPMOR readers who live in group home polycules in Berkeley who think they need to invent a good computer god to stop the evil computer god.

I think they cleaned out some of the EAs around the time of the board situation, but I don't know what the non-EA overlap is with your description.

Re: Neural Network Diffusion

#55

Post by gwern »

This doesn't seem all that impressive when you compare it to earlier work like 'g.pt' https://arxiv.org/abs/2209.12892 Peebles et al 2022. They cite it in passing, but do no comparison or discussion, and to my eyes, g.pt is a lot more interesting (for example, you can prompt it for a variety of network properties like low vs high score, whereas this just generates unconditionally) and more thoroughly evaluated. The autoencoder here doesn't seem like it adds much.

Re: Neural Network Diffusion

#56

Post by SubiculumCode »

vessenes wrote: post #18

I wasn't sure if this paper was parody on reading the abstract. It's not parody. Two things stand out to me: first is the idea of distilling these networks down into a smaller latent space, and then mucking around with that. That's interesting, and cross-sections a bunch of interesting topics like interpretability, compression, training, over- and under-.. The second is that they show the diffusion models don't just…

Perhaps doing this to generate 10 similar but different versions of a model can then be fed into mixture of experts?

Re: Neural Network Diffusion

#57

Post by hoc »

Hm, so does this actually improve/condense the representation for certain applications or is this some more some kind of global expand and collect in network space?

Re: Neural Network Diffusion

#58

Post by Nevermark »

astrange wrote: post #34
killerstorm wrote: post #23

Earlier quoted context omitted.

> which is just a belief that magic is real Is there a law of thermodynamics which prevents AI from writing code which would train a better AI? Never learned that one in school. And FYI here's OpenAI plan to align superintelligence: "Our goal is to build a roughly human-level automated alignment researcher. We can then use vast amounts of compute to scale our efforts, and iteratively align superintelligence." I guess…

> Is there a law of thermodynamics which prevents AI from writing code which would train a better AI? You need to apply Wittgenstein here. This appears to be true because you haven't defined "better". If you define it, it'll become obvious that this is either false or true, but if it is true it'll be obvious in a way that doesn't make it sound interesting anymore. (For one thing our current "AI" don't come from "writ…

> Humans don't have a "recursive self-improvement" ability

They do.

Humans can learn from new information, but also by iteratively distilling existing information or continuously optimizing performance on an existing task.

Mathematics is a pure instance of this, in the sense that all the patterns for conjectures and proven theorems are available to any entity to explore, no connection to the world needed.

But any information being analyzed for underlying patterns, or task being optimized for better performance, creates a recursive learning driver.

Finally, any time two or more humans compete at anything, they drive each other to learn and perform better. Models can do that too.

Re: Neural Network Diffusion

#60

Post by [deleted] »

astrange wrote: post #51
rralian wrote: post #46

Earlier quoted context omitted.

New generations build onto the scientific knowledge of previous generations. It may not be fast but that sounds like recursive improvement to me. It seems reasonable for AI to accelerate this process.

I think saying all of society is doing it is plausible, but not the same thing as a single human or AI doing it. Though… still don't think it's true. Isn't "society is self improving" what they call Whig history?

[deleted]
Post reply on HN