There are no new ideas in AI, only new datasets • Hacker News Board

Re: There are no new ideas in AI, only new datasets

#261

Post by pjmorris » Tue, Jul 01, 2025, 2:22 PM UTC

dinfinity wrote: post #235

computably wrote: post #228

Earlier quoted context omitted.

Language is literally an abstraction of sensory inputs and cognitive processes. One can make similar arguments about image generation. These abstractions might characterize the higher cognitive abilities of humans, but it makes no sense to ignore "lower level" cognition. Embodiment is the foundation of our rich internal world models, in particular spacetime, causality, etc. Current generative models merely mimic the…

> Language is literally an abstraction of sensory inputs and cognitive processes. Language can exist entirely independently from senses and cognition. It is an encoding of patterns in the world where the only thing that matters is if anybody or anything wielding it can map the encodings to and from the patterns they encode for (which is more of a sociological/synchronisation challenge). Does C, or Java, 'make no sens…

> Does C, or Java, 'make no sense' because it 'ignores lower level cognition'?

It makes sense in context, but that context includes the machine on which the compiled code runs. Without the underlying machine, there's no real purpose for C or Java. I'm open to the idea that 'lower level cognition' may be as relevant to language as the machine is to C or Java.

Re: There are no new ideas in AI, only new datasets

#262

Post by surecoocoocoo » Tue, Jul 01, 2025, 2:22 PM UTC

justanotherjoe wrote: post #238

surecoocoocoo wrote: post #154

Earlier quoted context omitted.

Ah some No True Scotsman Not sure why justanotherjoe is a credible resource on who is and isn’t expert in some new dialectic and euphemism for machine state management. You’re that nobody to me :shrug: Yann LeCun is an AI guy and has simplified it as “not much more than physical statistics.” WWhole lot of AI is decades old info theory books applied to modern computer. Either a mem value is or isn’t what’s expected. E…

I'm not being gatekeeper here. John Carmack came into AI around 2021 iirc and came in Lex Friedman and said he's going to bring about AGI. It's okay for him to try so but he had no particular expertise in the field. He's a brilliant guy and I'm not gonna say he's not going to succeed, or that his opinion is worthless. But people seeemed to think that the whole field is a farce just waiting for an adult to come in and…

[dead]

Re: There are no new ideas in AI, only new datasets

#263

Post by ordu » Tue, Jul 01, 2025, 2:29 PM UTC

dinfinity wrote: post #36

voxleone wrote: post #16

I'd say with confidence: we're living in the early days. AI has made jaw-dropping progress in two major domains: language and vision. With large language models (LLMs) like GPT-4 and Claude, and vision models like CLIP and DALL·E, we've seen machines that can generate poetry, write code, describe photos, and even hold eerily humanlike conversations. But as impressive as this is, it’s easy to lose sight of the bigger…

> Language and vision are just the beginning — the parts we were able to digitize first - not necessarily the most central to intelligence. I respectfully disagree. Touch gives pretty cool skills, but language, video and audio are all that are needed for all online interactions . We use touch for typing and pointing, but that is only because we don't have a more efficient and effective interface. Now I'm not saying t…

> Touch gives pretty cool skills, but language, video and audio are all that are needed for all online interactions. We use touch for typing and pointing, but that is only because we don't have a more efficient and effective interface.

It may be, that we are not using touch for anything important as adults. But babies rely on touch to explore their surroundings. They stick anything into their mouth, why? Because a tongue is the most touch-sensitive organ. They are exploring things by touching them with their tongues.

I can only guess what people get from that, but my guess is they get understanding of geometry and of surface properties of objects, which you'll have problems to get by processing photos or texts.

> your comment was specifically about intelligence.

Talking about intelligence, I do not believe that LLMs can match humans without deep understanding or 3d-space and material science^W intuition. It needs touch and temperature sensitivity at least. Probably you can replace it with billions of words of texts describing these things, but I doubt it.

Re: There are no new ideas in AI, only new datasets

#264

Post by Night_Thastus » Tue, Jul 01, 2025, 2:57 PM UTC

Culonavirus wrote: post #207

Night_Thastus wrote: post #69

Man I can't wait for this '''''AI''''' stuff to blow over. The back and forth gets a bit exhausting.

Within the bounds of HN audience I would definitely describe myself as an A(G)I skeptic. But even I can see that this ""AI"" stuff is not going to blow over. That ship has sailed. Even if the current models get only marginal improvements, the momentum is unquestionably, inarguably there to make the adoption and productization 10x or even 100x wider than it is now. Robotics, automatization, self-driving, all kinds of…

The reason I think it's going to blow over is that even the best models are frequently quite terrible. The fundamental problems of how they work can't really be fixed because they're a feature - the way that LLMs work.

And no one has found a way to make any money with it. All the tech companies are burning money by the truckload so investors don't lose confidence, but none of them have actually shown it's a good financial investment.

At the end of the day, I don't think anyone is going to want to pay what it really costs to run these models, just for a result that is so unreliable. Once they start to stagnate everyone will lose interest.

The only reason it might stick around is because investors will get desperate to get returns and go full sunk-cost once it starts looking like they made a bad call. (Which they will blame the companies for, of course)

Re: There are no new ideas in AI, only new datasets

#265

Post by ahmedhawas123 » Tue, Jul 01, 2025, 3:01 PM UTC

I think this is reflective of current state, but does not mean this will be the future. I think there is a lot of innovation to come on revisiting some of the 1990s principles of back propagation and optimization. Imagine if you could train current models to optimal weights in 1 day or 1 hour instead of weeks/months?

Just a hypothesis of mine

Re: There are no new ideas in AI, only new datasets

#266

Post by 93po » Tue, Jul 01, 2025, 3:12 PM UTC

ozgrakkurt wrote: post #175

EternalFury wrote: post #30

What John Carmack is exploring is pretty revealing. Train models to play 2D video games to a superhuman level, then ask them to play a level they have not seen before or another 2D video game they have not seen before. The transfer function is negative. So, in my definition, no intelligence has been developed, only expertise in a narrow set of tasks. It’s apparently much easier to scare the masses with visions of ASI…

Seeing comments here saying “this problem is already solved”, “he is just bad at this” etc. feels bad. He has given a long time to this problem by now. He is trying to solve this to advance the field. And needless to say, he is a legend in computer engineering or w/e you call it. It should be required to point to the “solution” and maybe how it works to say “he just sucks” or “this was solved before”. IMO the problem…

I'm a huge fan of Carmack and read the book (Masters of Doom) multiple times and love it, too. But he's a legend for pioneering PC gaming graphics in a way that was feasible for a single (very talented) person to accomplish, and was also pioneering something that already existed on consoles. I think there's a big leap from very cleverly recreating existing very basic and simple 3d graphics for a new platform versus the massive task that is AGI/ASI, which I don't think is something a single person can meaningfully move forward at this point. Even the big jump we got from GPTs was due to many many people.

Re: There are no new ideas in AI, only new datasets

#267

Post by qaq » Tue, Jul 01, 2025, 3:22 PM UTC

justanotherjoe wrote: post #240

qaq wrote: post #77

Earlier quoted context omitted.

Keen includes researchers like Richard Sutton, Joseph Modayil etc. Also John has being doing it full time for almost 5 years now so given his background and aptitude for learning I would imaging by this time he is more of an AI guy then a fairly large percentage of AI PhDs.

Yeah and in another 5 years he'd probably be at nobel laureate level in AI. I don't think that's how it works. What do you mean? Even a phd program can take 5 years sometimes. Also the man started saying he'd bring about AGI right at the gate. He wasn't being exactly humble. God I hate sounding like this. I swear I'm not too good for John Carmack, as he's infinitely smarter than me. But I just find it a bit weird. I'…

He stated AGI is an interesting problem to work on could you provide a reference on him claiming "he'd bring about AGI right at the gate"?

Re: There are no new ideas in AI, only new datasets

#268

Post by motorest » Tue, Jul 01, 2025, 4:06 PM UTC

pjmorris wrote: post #261

dinfinity wrote: post #235

Earlier quoted context omitted.

> Language is literally an abstraction of sensory inputs and cognitive processes. Language can exist entirely independently from senses and cognition. It is an encoding of patterns in the world where the only thing that matters is if anybody or anything wielding it can map the encodings to and from the patterns they encode for (which is more of a sociological/synchronisation challenge). Does C, or Java, 'make no sens…

> Does C, or Java, 'make no sense' because it 'ignores lower level cognition'? It makes sense in context, but that context includes the machine on which the compiled code runs. Without the underlying machine, there's no real purpose for C or Java. I'm open to the idea that 'lower level cognition' may be as relevant to language as the machine is to C or Java.

> Without the underlying machine, there's no real purpose for C or Java.

They do express algorithms, don't they?

Re: There are no new ideas in AI, only new datasets

#269

Post by dinfinity » Tue, Jul 01, 2025, 4:26 PM UTC

ordu wrote: post #263

dinfinity wrote: post #36

Earlier quoted context omitted.

> Language and vision are just the beginning — the parts we were able to digitize first - not necessarily the most central to intelligence. I respectfully disagree. Touch gives pretty cool skills, but language, video and audio are all that are needed for all online interactions . We use touch for typing and pointing, but that is only because we don't have a more efficient and effective interface. Now I'm not saying t…

> Touch gives pretty cool skills, but language, video and audio are all that are needed for all online interactions. We use touch for typing and pointing, but that is only because we don't have a more efficient and effective interface. It may be, that we are not using touch for anything important as adults. But babies rely on touch to explore their surroundings. They stick anything into their mouth, why? Because a to…

It is trivial to train AI on 3D representations. In fact, that already happens in cases where robot algorithms are trained in simulations.

Another thing to remember is that the senses we have aren't the only ones in biology and far from the only ones possible. In fact, anything that gives you another type of information about the world (you're modeling) is a different sense. In that sense (ha), AI has access to an incredibly vast and varied array of senses that is inaccessible to humans. Lidar is a very simple example of that.

I don't think touch and temperature sensitivity are needed to achieve it, but I do agree that training with senses specifically for understanding 3D space is very important. At the very least binocular video.

Re: There are no new ideas in AI, only new datasets

#270

Post by dinfinity » Tue, Jul 01, 2025, 4:33 PM UTC

actionfromafar wrote: post #247

dinfinity wrote: post #188

Earlier quoted context omitted.

Vision is interesting in that it leverages the maximum speed with which it is easily possible to gather information about our surroundings in this universe. I believe that is what makes it special and very valuable. I also believe this aspect makes it a strong attractor for convergent evolution. Language allows encoding and compression of information about the world, which is of course incredibly powerful and increas…

Regarding visual communication, I think you downsell posturing, gesturing and facial expressions a little. They may not be as high bandwidth as talking but they are very low latency and pretty stealthy if necessary.

Well, there is sign language, so I guess you're right. It would be interesting to see how high bandwidth gesturing can be compared to speaking.

I thought about this some more and I think the prevalence of making sounds rather than gesturing etc. is due to sound being a broadcasting mechanism that works over long distances and without line of sight.

Visually indicating that you've claimed some territory is pretty hard.