Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

There are no new ideas in AI, only new datasets

blog.jxmo.io

Post reply on HN
Print view View original

51–60 of 307 posts

Re: There are no new ideas in AI, only new datasets

#51

Post by smokel »

EternalFury wrote: post #30

What John Carmack is exploring is pretty revealing. Train models to play 2D video games to a superhuman level, then ask them to play a level they have not seen before or another 2D video game they have not seen before. The transfer function is negative. So, in my definition, no intelligence has been developed, only expertise in a narrow set of tasks. It’s apparently much easier to scare the masses with visions of ASI…

The subject you are referring to is most likely Meta-Reinforcement Learning [1]. It is great that John Carmack is looking into this, but it is not a new field of research.

[1] https://instadeep.com/2021/10/a-simple-introduction-to-meta-...

Re: There are no new ideas in AI, only new datasets

#52

Post by chasd00 »

voxleone wrote: post #16

I'd say with confidence: we're living in the early days. AI has made jaw-dropping progress in two major domains: language and vision. With large language models (LLMs) like GPT-4 and Claude, and vision models like CLIP and DALL·E, we've seen machines that can generate poetry, write code, describe photos, and even hold eerily humanlike conversations. But as impressive as this is, it’s easy to lose sight of the bigger…

> Language and vision are just the beginning..

Based on the architectures we have they may also be the ending. There’s been a lot of news in the past couple years about LLMs but has there been any breakthroughs making headlines anywhere else in AI?

Re: There are no new ideas in AI, only new datasets

#53

Post by IshKebab »

fsmv wrote: post #45
NBJack wrote: post #40

Earlier quoted context omitted.

In other words, they learn the game, not how to play games .

They memorize the answers not the process to arrive at answers

This has been disproven so many times... They clearly do both. You can trivially prove this yourself.

Re: There are no new ideas in AI, only new datasets

#54

Post by antisthenes »

tough wrote: post #39
ferguess_k wrote: post #38

Earlier quoted context omitted.

Can you please explain "the transfer function is negative"? I'm wondering whether one has tested with the same model but on two situations: 1) Bring it to superhuman level in game A and then present game B, which is similar to A, to it. 2) Present B to it without presenting A. If 1) is not significantly better than 2) then maybe it is not carrying much "knowledge", or maybe we simply did not program it correctly.

I think the problem is we train models to pattern match, not to learn or reason about world models

Where do you draw the line between pattern matching and reasoning about world models?

A lot of intelligence is just pattern matching and being quick about it.

Re: There are no new ideas in AI, only new datasets

#55

Post by voxleone »

dinfinity wrote: post #36
voxleone wrote: post #16

I'd say with confidence: we're living in the early days. AI has made jaw-dropping progress in two major domains: language and vision. With large language models (LLMs) like GPT-4 and Claude, and vision models like CLIP and DALL·E, we've seen machines that can generate poetry, write code, describe photos, and even hold eerily humanlike conversations. But as impressive as this is, it’s easy to lose sight of the bigger…

> Language and vision are just the beginning — the parts we were able to digitize first - not necessarily the most central to intelligence. I respectfully disagree. Touch gives pretty cool skills, but language, video and audio are all that are needed for all online interactions . We use touch for typing and pointing, but that is only because we don't have a more efficient and effective interface. Now I'm not saying t…

> Language and vision are just the beginning — the parts we were able to digitize first - not necessarily the most central to intelligence.

I probably made a mistake when i asserted that -- should have thought it over. Vision is evolutionarily older and more “primitive”, while language is uniquely human [or maybe, more broadly, primate, cetacean, cephalopod, avian...] symbolic, and abstract — arguably a different order of cognition altogether. But i maintain that each and every sense is important as far as human cognition -- and its replication -- is concerned.

Re: There are no new ideas in AI, only new datasets

#56

Post by seydor »

There are new ideas, people are finding new ways to build vision models, which then are applied to language models and vice versa (like diffusion).

The original idea of connectionism is that neural networks can represent any function, which is the fundamental mathematical fact. So we should be optimistic, neural nets will be able to do anything. Which neural nets? So far people have stumbled on a few productive architectures, but it appears to be more alchemy than science. There is no reason why we should think there won't be both new ideas and new data. Biology did it, humans will do it too.

> we’re engaged in a decentralized globalized exercise of Science, where findings are shared openly

Maybe the findings are shared, if they make the Company look good. But the methods are not anymore

Re: There are no new ideas in AI, only new datasets

#57

Post by dragonwriter »

chasd00 wrote: post #52
voxleone wrote: post #16

I'd say with confidence: we're living in the early days. AI has made jaw-dropping progress in two major domains: language and vision. With large language models (LLMs) like GPT-4 and Claude, and vision models like CLIP and DALL·E, we've seen machines that can generate poetry, write code, describe photos, and even hold eerily humanlike conversations. But as impressive as this is, it’s easy to lose sight of the bigger…

> Language and vision are just the beginning.. Based on the architectures we have they may also be the ending. There’s been a lot of news in the past couple years about LLMs but has there been any breakthroughs making headlines anywhere else in AI?

> There’s been a lot of news in the past couple years about LLMs but has there been any breakthroughs making headlines anywhere else in AI?

Yeah, lots of stuff tied to robotics, for instance; this overlaps with vision, but the advances go beyond vision.

Audio has seen quite a bit. And I imagine there is stuff happening in niche areas that just aren't as publicly interesting as language, vision/imagery, audio, and robotics.

Re: There are no new ideas in AI, only new datasets

#58

Post by singron »

tough wrote: post #39
ferguess_k wrote: post #38

Earlier quoted context omitted.

Can you please explain "the transfer function is negative"? I'm wondering whether one has tested with the same model but on two situations: 1) Bring it to superhuman level in game A and then present game B, which is similar to A, to it. 2) Present B to it without presenting A. If 1) is not significantly better than 2) then maybe it is not carrying much "knowledge", or maybe we simply did not program it correctly.

I think the problem is we train models to pattern match, not to learn or reason about world models

I think this is clearly a case of over fitting and failure to generalize, which are really well understood concepts. We don't have to philosophize about what pattern matching really means.

Re: There are no new ideas in AI, only new datasets

#59

Post by genewitch »

FeepingCreature wrote: post #29
Y_Y wrote: post #13

Earlier quoted context omitted.

You raise a really interesting point. I'm sure it's just missed my notice, but I'm not familiar with any projects from antediluvian AI that have been resurrected to run on modern hardware and see where they'd really asymptote if they'd had the compute they deserved.

To be fair, usually those projects would need considerable work to be ported to modern multicore machines, let alone GPUs.

can you name a couple so i can see how much work is involved? markov chains compile fast and respond fast, sure, and neural nets train pretty quick too, so i'm wondering where the cutoff is; expert systems?

Re: There are no new ideas in AI, only new datasets

#60

Post by tantalor »

piinbinary wrote: post #28

AI training is currently a process of making the AI remember the dataset. It doesn't involve the AI thinking about the dataset and drawing (and remembering) conclusions. It can probably remember more facts about a topic than a PhD in that topic, but the PhD will be better at thinking about that topic.

Maybe that's why PhDs keep the textbooks they use at hand, so they don't have to remember everything.

Why should the model need to memorize facts we already have written down somewhere?

Post reply on HN