Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

There are no new ideas in AI, only new datasets

blog.jxmo.io

Post reply on HN
Print view View original

91–100 of 307 posts

Re: There are no new ideas in AI, only new datasets

#91

Post by strangescript »

If you work with model architecture and read papers, how could not know there are a flood of new ideas? Only few yield interesting results though.

I kind of wonder if libraries like pytorch have hurt experimental development. So many basic concepts no one thinks about anymore because they just use the out of the box solutions. And maybe those solutions are great and those parts are "solved", but I am not sure. How many models are using someone else's tokenizer, or someone else's strapped on vision model just to check a box in the model card?

Re: There are no new ideas in AI, only new datasets

#92

Post by russellbeattie »

Paradigm shifts are often just a conglomeration of previous ideas with one little tweak that suddenly propels a technology ahead 10x which opens up a whole new era.

The iPhone is a perfect example. There were smartphones with cameras and web browsers before. But when the iPhone launched, it added a capacitive touch screen that was so responsive there was no need for a keyboard. The importance of that one technical innovation can't be overstated.

Then the "new new thing" is followed by a period of years where the innovation is refined, distributed, applied to different contexts, and incrementally improved.

The iPhone launched in 2007 is not really that much different than the one you have in your pocket today. The last 20 years has been about improvements. The web browser before that is also pretty much the same as the one you use today.

We've seen the same pattern happen with LLMs. The author of the article points out that many of AI's breakthroughs have been around since the 1990s. Sure! And the Internet was created in the 1970s and mobile phones were invented in the 1980s. That doesn't mean the web and smartphones weren't monumental technological events. And it doesn't mean LLMs and AI innovation is somehow not proceeding apace.

It's just how this stuff works.

Re: There are no new ideas in AI, only new datasets

#93

Post by sigmoid10 »

vladimirralev wrote: post #48
EternalFury wrote: post #30

What John Carmack is exploring is pretty revealing. Train models to play 2D video games to a superhuman level, then ask them to play a level they have not seen before or another 2D video game they have not seen before. The transfer function is negative. So, in my definition, no intelligence has been developed, only expertise in a narrow set of tasks. It’s apparently much easier to scare the masses with visions of ASI…

He is not using appropriate models for this conclusion and neither is he using state of the art models in this research and moreover he doesn't have an expensive foundational model to build upon for 2d games. It's just a fun project. A serious attempt at video/vision would involve some probabilistic latent space that can be noised in ways that make sense for games in general. I think veo3 proves that ai can generaliz…

Veo3's world model is still pretty limited. That becomes obvious very fast once you prompt out of distribution video content (i.e. stuff that you are unlikely to find on youtube). It's extremely good at creating photorealistic surfaces and lighting. It even has some reasonably solid understanding of fluid dynamics for simulating water. But for complex human behaviour (in particular certain motions) it simply lacks the training data. Although that's not really a fault of the model and I'm pretty sure there will be a way to overcome this as well. Maybe some kind of physics based simulation as supplement training data.

Re: There are no new ideas in AI, only new datasets

#94

Post by raincole »

justanotherjoe wrote: post #68
EternalFury wrote: post #30

What John Carmack is exploring is pretty revealing. Train models to play 2D video games to a superhuman level, then ask them to play a level they have not seen before or another 2D video game they have not seen before. The transfer function is negative. So, in my definition, no intelligence has been developed, only expertise in a narrow set of tasks. It’s apparently much easier to scare the masses with visions of ASI…

I don't get why people are so invested in framing it this way. I'm sure there are ways to do the stated objective. John Carmack isn't even an AI guy why is he suddenly the standard.

Because it "confirms" what they already believe in.

Re: There are no new ideas in AI, only new datasets

#95

Post by [deleted] »

tough wrote: post #39
ferguess_k wrote: post #38

Earlier quoted context omitted.

Can you please explain "the transfer function is negative"? I'm wondering whether one has tested with the same model but on two situations: 1) Bring it to superhuman level in game A and then present game B, which is similar to A, to it. 2) Present B to it without presenting A. If 1) is not significantly better than 2) then maybe it is not carrying much "knowledge", or maybe we simply did not program it correctly.

I think the problem is we train models to pattern match, not to learn or reason about world models

[deleted]

Re: There are no new ideas in AI, only new datasets

#96

Post by [deleted] »

Swizec wrote: post #85
nomel wrote: post #74

Earlier quoted context omitted.

There are many people who mentally developed while paralyzed that literally drive around their bodies via motorized wheelchair. I don't think there's any evidence that a brain couldn't exist or develop in a jar, given only the inputs modern AI now has (text, video, audio).

> any evidence that a brain couldn't exist or develop in a jar The brain could . Of course it could. It's just a signals processing machine. But would it be missing anything we consider core to the way humans think? Would it struggle with parts of cognition? For example: experiments were done with cats growing up in environments with vertical lines only. They were then put in a normal room and had a hard time underst…

[deleted]

Re: There are no new ideas in AI, only new datasets

#97

Post by fullshark »

EternalFury wrote: post #30

What John Carmack is exploring is pretty revealing. Train models to play 2D video games to a superhuman level, then ask them to play a level they have not seen before or another 2D video game they have not seen before. The transfer function is negative. So, in my definition, no intelligence has been developed, only expertise in a narrow set of tasks. It’s apparently much easier to scare the masses with visions of ASI…

Just sounds like an example of overfitting. This is all machine learning at its root.

Re: There are no new ideas in AI, only new datasets

#98

Post by EternalFury »

veqz wrote: post #79
ferguess_k wrote: post #66

Earlier quoted context omitted.

I kinda think I'm more or less the same...OK maybe we have different definitions of "pattern matching".

It's Plato's cave: We train the models on what are basically shadows, and they learn how to pattern match the shadows. But the shadows are only depictions of the real world, and the LLMs never learn about that.

100%

Re: There are no new ideas in AI, only new datasets

#99

Post by raincole »

goatlover wrote: post #87
EternalFury wrote: post #30

What John Carmack is exploring is pretty revealing. Train models to play 2D video games to a superhuman level, then ask them to play a level they have not seen before or another 2D video game they have not seen before. The transfer function is negative. So, in my definition, no intelligence has been developed, only expertise in a narrow set of tasks. It’s apparently much easier to scare the masses with visions of ASI…

I've wondered about the claim that the models played those Atari/2D video games at superhuman levels, because I clearly recall some humans achieving superhuman levels before models were capable of it. Must have been superhuman compared to average human player, not someone who spent an inordinate amount of time mastering the game.

I'm not sure why you think so. AI outperforms humans in many games already. Basically all the games we care to put money to train a model.

AI has beat the best human players in Chess, Go, Mahjong, Texas hold'em, Dota, Starcraft, etc. It would be really, really surprising that some Atari game is the holy grail of human performance that AI cannot beat.

Re: There are no new ideas in AI, only new datasets

#100

Post by lsy »

This seems simplistic, tech and infrastructure play a huge part here. A short and incomplete list of things that contributed:

- Moore's law petering out, steering hardware advancements towards parallelism

- Fast-enough internet creating shift to processing and storage in large server farms, enabling both high-cost training and remote storage of large models

- Social media + search both enlisting consumers as data producers, and necessitating the creation of armies of Mturkers for content moderation + evaluation, later becoming available for tagging and rlhf

- A long-term shift to a text-oriented society, beginning with print capitalism and continuing through the rise of "knowledge work" through to the migration of daily tasks (work, bill paying, shopping) online, that allows a program that only produces text to appear capable of doing many of the things a person does

We may have previously had the technical ideas in the 1990s but we certainly didn't have the ripened infrastructure to put them into practice. If we had the dataset to create an LLM in the 90s, it still would have been astronomically cost-prohibitive to train, both in CPU and human labor, and it wouldn't have as much of an effect on society because you wouldn't be able to hook it up to commerce or day-to-day activities (far fewer texts, emails, ecommerce).

Post reply on HN