Live data from Hacker News

There are no new ideas in AI, only new datasets

blog.jxmo.io

171–180 of 307 posts

Re: There are no new ideas in AI, only new datasets

#171
post #64

I wrote about it around a year ago here: "There weren't really any advancements from around 2018. The majority of the 'advancements' were in the amount of parameters, training data, and its applications. What was the GPT-3 to ChatGPT transition? It involved fine-tuning, using specifically crafted training data. What changed from GPT-3 to GPT-4? It was the increase in the number of parameters, improved training data,…

And when winter does arrive, then what? The technology is slowing down while its popularity picks up. Can sparks fly out of snow?

the trillion dollar funding tap is turned off, the prices charged then will have to reflect the costs

shortly thereafter the entire ecosystem will collapse

Re: There are no new ideas in AI, only new datasets

#172

What John Carmack is exploring is pretty revealing. Train models to play 2D video games to a superhuman level, then ask them to play a level they have not seen before or another 2D video game they have not seen before. The transfer function is negative. So, in my definition, no intelligence has been developed, only expertise in a narrow set of tasks. It’s apparently much easier to scare the masses with visions of ASI…

I don't get why people are so invested in framing it this way. I'm sure there are ways to do the stated objective. John Carmack isn't even an AI guy why is he suddenly the standard.

Credentialism is bad, especially when used as a stick

Re: There are no new ideas in AI, only new datasets

#173

What John Carmack is exploring is pretty revealing. Train models to play 2D video games to a superhuman level, then ask them to play a level they have not seen before or another 2D video game they have not seen before. The transfer function is negative. So, in my definition, no intelligence has been developed, only expertise in a narrow set of tasks. It’s apparently much easier to scare the masses with visions of ASI…

He is not using appropriate models for this conclusion and neither is he using state of the art models in this research and moreover he doesn't have an expensive foundational model to build upon for 2d games. It's just a fun project. A serious attempt at video/vision would involve some probabilistic latent space that can be noised in ways that make sense for games in general. I think veo3 proves that ai can generaliz…

> I think veo3 proves that ai can generalize 2d and even 3d games, generating a video under prompt constraints is basically playing a game.

In the same way that keeping a dream journal is basically doing investigative journalism, or talking to yourself is equivalent to making new friends, maybe.

The difference is that while they may both produce similar, "plausible" output, one does so as a result of processes that exist in relation to an external reality.

Re: There are no new ideas in AI, only new datasets

#175

What John Carmack is exploring is pretty revealing. Train models to play 2D video games to a superhuman level, then ask them to play a level they have not seen before or another 2D video game they have not seen before. The transfer function is negative. So, in my definition, no intelligence has been developed, only expertise in a narrow set of tasks. It’s apparently much easier to scare the masses with visions of ASI…

Seeing comments here saying “this problem is already solved”, “he is just bad at this” etc. feels bad. He has given a long time to this problem by now. He is trying to solve this to advance the field. And needless to say, he is a legend in computer engineering or w/e you call it.

It should be required to point to the “solution” and maybe how it works to say “he just sucks” or “this was solved before”.

IMO the problem with current models is that they don’t learn categorically like: lions are animals, animals are alive. goats are animals, goats are alive too. So if lions have some property like breathing and goats also have it, it is likely that other similar things have the same property.

Or when playing a game, a human can come up with a strategy like: I’ll level this ability and lean on it for starting, then I’ll level this other ability that takes more time to ramp up while using the first one, then change to this play style after I have the new ability ready. This might be formulated completely based on theoretical ideas about the game, and modified as the player gets more experience.

With current AI models as far as I can understand, it will see the whole game as an optimization problem and try to find something at random that makes it win more. This is not as scalable as combining theory and experience in the way that humans do. For example a human is innately capable of understanding there is a concept of early game, and the gains made in early game can compound and generate a large lead. This is pattern matching as well but it is on a higher level .

Theory makes learning more scalable compared to just trying everything and seeing what works

Re: There are no new ideas in AI, only new datasets

#176
post #86

I will respectfully disagree. All "new" ideas come from old ideas. AI is a tool to access old ideas with speed and with new perspectives that hasn't been available up until now. Innovation is in the cracks: recognition of holes, intersections, tangents, etc. on old ideas. It has bent said that innovation is done on the shoulders of giants. So AI can be an express elevator up to an army of giant's shoulders? It all de…

Imagine a human had read every book/publication in every field of knowledge that mankind has ever produced AND couldn’t come up with anything entirely new. Hard to imagine.

My hypothesis of the mismatch is centered around "read" - I think that when you wrote it, and when others similarly think about that scenario, the surprise is because our version of "read" is the implied "read and internalized" or at bare minimum "read for comprehension" but as very best I can tell the LLM's version is "encoded tokens into vector space" and not "encoded into semantic graph"

I welcome the hair-splittery that is sure to follow about what it means to "understand" anything

Re: There are no new ideas in AI, only new datasets

#178
post #103
post #85

Earlier quoted context omitted.

> any evidence that a brain couldn't exist or develop in a jar The brain could . Of course it could. It's just a signals processing machine. But would it be missing anything we consider core to the way humans think? Would it struggle with parts of cognition? For example: experiments were done with cats growing up in environments with vertical lines only. They were then put in a normal room and had a hard time underst…

This isn't remotely a hypothetical, so I imagine there are some examples out there, especially from back when polio was a problem. Although, for practical reasons, they might have had limited exposure to novelty , which could have negative consequences.

I agree it’s not hypothetical and also as a layperson I don’t know how much impact on cognition has been studied. Would be cool if it has!

I do know of studies that showed blind people start using their visual cortex to process sounds. That is pretty cool imo

Re: There are no new ideas in AI, only new datasets

#179
post #86

I will respectfully disagree. All "new" ideas come from old ideas. AI is a tool to access old ideas with speed and with new perspectives that hasn't been available up until now. Innovation is in the cracks: recognition of holes, intersections, tangents, etc. on old ideas. It has bent said that innovation is done on the shoulders of giants. So AI can be an express elevator up to an army of giant's shoulders? It all de…

Imagine a human had read every book/publication in every field of knowledge that mankind has ever produced AND couldn’t come up with anything entirely new. Hard to imagine.

It is possible that such a human wouldn't come up with anything new, even if they could.

Re: There are no new ideas in AI, only new datasets

#180

Sometimes we get confused by the difference between technological and scientific progress. When science makes progress it unlocks new S-curves that progress at an incredible pace until you get into the diminishing returns region. People complain of slowing progress but it was always slow, you just didn’t notice that nothing new was happening during the exponential take off of the S-curve, just furious optimization.

The crypto mind cannot comprehend
Post reply on HN