All AI models might be the same
blog.jxmo.io
All AI models might be the same
1–10 of 157 posts
Re: All AI models might be the same
#2Especially if they are all me-too copies of a Transformer.
When we arrive at AGI, you can be certain it will not contain a Transformer.
Re: All AI models might be the same
#3Especially if they are all me-too copies of a Transformer. When we arrive at AGI, you can be certain it will not contain a Transformer.
I don't think architecture matters. It seems to be more a function of the data somehow.
I once saw a LessWrong post claiming that the Platonic Representation Hypothesis doesn't hold when you only embed random noise, as opposed to natural images: http://lesswrong.com/posts/Su2pg7iwBM55yjQdt/exploring-the-p...
Re: All AI models might be the same
#4This only works when different sources share similar feature distributions and semantic relationships.
The M or B game breaks down when you play with someone who knows obscure people you've never heard of. Either you can't recognize their references, or your sense of "semantic distance" differs from theirs. The solution is to match knowledge levels: experts play with experts, generalists with generalists.
The same applies to decoding ancient texts, if ancient civilizations focused on completely different concepts than we do today, our modern semantic models won't help us understand their writing.
Re: All AI models might be the same
#5Has there been research on using this to make models smaller? If models converge on similar representations, we should be able to build more efficient architectures around those core features.
Re: All AI models might be the same
#6[deleted]
Re: All AI models might be the same
#7Has there been research on using this to make models smaller? If models converge on similar representations, we should be able to build more efficient architectures around those core features.
It's more likely that such an architecture would be bigger rather than smaller. https://arxiv.org/abs/2412.20292 demonstrated that score-matching diffusion models approximate a process that combines patches from different training images. To build a model that makes use of this fact, all you need to do is look up the right patch in the training data. Of course a model the size of its training data would typically be rather unwieldy to use. If you want something smaller, we're back to approximations created by training the old-fashioned way.
Re: All AI models might be the same
#8Has there been research on using this to make models smaller? If models converge on similar representations, we should be able to build more efficient architectures around those core features.
I've been thinking about this a lot. I want to know what's the smallest a model needs to be, before letting it browse search engines, or files you host locally is actually an avenue an LLM can go through to give you more informed answers. Is it 2GB? 8GB? Would love to know.
Re: All AI models might be the same
#9This is kind of fascinating because I just tried to play mussolini or bread with chatgpt and it is absolutely _awful_ at it, even with reasoning models.
It just assumes that your answers are going to be reasonably bread-like or reasonably mussolini-like, and doesn't think laterally at all.
It just kept asking me about varieties of baked goods.
edit: It did much better after I added some extra explanation -- that it could be anything that it may be very unlike either choice, and not to try and narrow down too quickly
Re: All AI models might be the same
#10This only works when different sources share similar feature distributions and semantic relationships. The M or B game breaks down when you play with someone who knows obscure people you've never heard of. Either you can't recognize their references, or your sense of "semantic distance" differs from theirs. The solution is to match knowledge levels: experts play with experts, generalists with generalists. The same ap…
I've played this game with friends occasionally and -- when it's a person -- don't think I've ever completed a game.