Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

There are no new ideas in AI, only new datasets

blog.jxmo.io

Post reply on HN
Print view View original

161–170 of 307 posts

Re: There are no new ideas in AI, only new datasets

#161

Post by timewizard »

voxleone wrote: post #16

I'd say with confidence: we're living in the early days. AI has made jaw-dropping progress in two major domains: language and vision. With large language models (LLMs) like GPT-4 and Claude, and vision models like CLIP and DALL·E, we've seen machines that can generate poetry, write code, describe photos, and even hold eerily humanlike conversations. But as impressive as this is, it’s easy to lose sight of the bigger…

> has made jaw-dropping progress

They took 1970s dead tech and deployed it on machines 1 million times more powerful. I'm not sure I'd qualify this as progress. I'd also need an explanation as to what systemic improvements in models and computations that give an exponential growth in performance are planned.

I don't see anything.

Re: There are no new ideas in AI, only new datasets

#163

Post by timewizard »

tippytippytango wrote: post #10

Sometimes we get confused by the difference between technological and scientific progress. When science makes progress it unlocks new S-curves that progress at an incredible pace until you get into the diminishing returns region. People complain of slowing progress but it was always slow, you just didn’t notice that nothing new was happening during the exponential take off of the S-curve, just furious optimization.

You're being awfully generous to describe basic hype as "technological progress."

Re: There are no new ideas in AI, only new datasets

#164

Post by _giorgio_ »

strangescript wrote: post #91

If you work with model architecture and read papers, how could not know there are a flood of new ideas? Only few yield interesting results though. I kind of wonder if libraries like pytorch have hurt experimental development. So many basic concepts no one thinks about anymore because they just use the out of the box solutions. And maybe those solutions are great and those parts are "solved", but I am not sure. How ma…

It's the opposite.

Frameworks like pytorch are really flexible. You can implement any architecture, and if it's not enough, you can learn CUDA.

Keras it's the opposite, it's probably like you describe things.

Re: There are no new ideas in AI, only new datasets

#165

Post by hackinthebochs »

MichaelZuo wrote: post #158
hackinthebochs wrote: post #156

Earlier quoted context omitted.

You have probably seen examples of LLMs doing the "mirror test", i.e. identifying themselves in screenshots and referring to the screenshot from the first person. That is a genuinely novel question as an "LLM mirror test" wasn't a concept that existed before about a year ago.

Elephant mirror tests existed, so it doesn’t seem all that novel when the word “elephant” could just be substituted for the word “LLM”?

The question isn't about universal novelty, but whether the prompt/context is novel enough such that the LLM answering competently demonstrates understanding. The claim of parroting is that the dataset contains a near exact duplicate of any prompt and so the LLM demonstrating what appears to be competence is really just memorization. But if an LLM can generalize from an elephant mirror test to an LLM mirror test in an entirely new context (showing pictures and being asked to describe it), that demonstrates sufficient generalization to "understand" the concept of a mirror test.

Re: There are no new ideas in AI, only new datasets

#166

Post by gerdesj »

amelius wrote: post #153
GuB-42 wrote: post #150

Earlier quoted context omitted.

Who is an "AI guy"? The field as we know it is fairly new. Sure, neural nets are old hat, but a lot has happened in the last few years. John Carmack founded Keen technology in 2022 and has been working seriously on AI since 2019. From his experience in the video game industry, he knows a thing or two about linear algebra and GPUs, that is the underlying maths and the underlying hardware. So, for all intent and purpos…

But the logic seems flawed. He has built an AI system that fails to do X. That does not mean there isn't an AI system that can do X. Especially considering that a lot is happening in AI, as you say. Anyway, Carmack knows a lot about optimizing computations on modern hardware. In practice, that happens to be also necessary for AI. However, it is not __sufficient__ for AI.

"He has built an AI system that fails to do X."

Perhaps you have put your finger on the fatal flaw ...

Re: There are no new ideas in AI, only new datasets

#167

Post by pevansgreenwood »

tippytippytango wrote: post #10

Sometimes we get confused by the difference between technological and scientific progress. When science makes progress it unlocks new S-curves that progress at an incredible pace until you get into the diminishing returns region. People complain of slowing progress but it was always slow, you just didn’t notice that nothing new was happening during the exponential take off of the S-curve, just furious optimization.

That's putting the cart before the horse. Thermodynamics came after the steam engine was made practical. Flight came before aerodynamics. Metallurgy before materials science. Radio before electromagnetic theory took hold. Even LLMs are the result of a lot of tinkering rather than scientific insight. It’s the successful tinkering that creates the puzzle science later formalises.

Re: There are no new ideas in AI, only new datasets

#168

Post by mikewarot »

Hardware isn't even close to being out of steam. There are some breathtakingly obvious premature optimizations that we can undo to get at least 99% power reduction for the same amount of compute.

For example, FPGAs use a lot of area and power routing signals across the chip. Those long lines have a large capacitance, and thus cause a large amount of dynamic power loss. So does moving parameters around to/from RAM instead of just loading up a vast array of LUTs with the values once.

Re: There are no new ideas in AI, only new datasets

#169

Post by SamaraMichi »

This brings us to the problem AI companies are facing, the lack of data, they have already hoovered as much as they can from the internet and desperately need more data.

Which make sit blatantly obvious why we're beginning to see products being marketed under the guise of assistants/tools to aid you whose actual purpose is to gather real world picture and audio data, think meta glasses and what Ives and Altman are cooking up with their partnership.

Re: There are no new ideas in AI, only new datasets

#170

Post by magicalhippo »

tough wrote: post #39
ferguess_k wrote: post #38

Earlier quoted context omitted.

Can you please explain "the transfer function is negative"? I'm wondering whether one has tested with the same model but on two situations: 1) Bring it to superhuman level in game A and then present game B, which is similar to A, to it. 2) Present B to it without presenting A. If 1) is not significantly better than 2) then maybe it is not carrying much "knowledge", or maybe we simply did not program it correctly.

I think the problem is we train models to pattern match, not to learn or reason about world models

In the Physics of Language Models[1] they argue that you must augment your training data by changing sentences and such, in order for the model to be able to learn the knowledge. As I understand their argument, language models don't have a built-in way to detect what is important information and what is not, unlike us. Thus the training data must aid it by presenting important information in many different ways.

Doesn't seem unreasonable that the same holds in a gaming setting, that one should train on many variations of each level. Change the lengths of halls connecting rooms, change the appearance of each room, change power-up locations etc, and maybe even remove passages connecting rooms.

[1]: https://physics.allen-zhu.com/part-3-knowledge/part-3-1

Post reply on HN