Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

There are no new ideas in AI, only new datasets

blog.jxmo.io

Post reply on HN
Print view View original

231–240 of 307 posts

Re: There are no new ideas in AI, only new datasets

#231

Post by imiric »

IshKebab wrote: post #208
imiric wrote: post #123

Earlier quoted context omitted.

LLMs don't "memorize" concepts like humans do. They generate output based on token patterns in their training data. So instead of having to be trained on every possible problem, they can still generate output that solves it by referencing the most probable combination of tokens for the specified input tokens. To humans this seems like they're truly solving novel problems, but it's merely a trick of statistics. These…

> To humans this seems like they're truly solving novel problems Because they are . This is some crazy semantic denial. I should stop engaging with this nonsense. We have AI that is kind of close to passing the Turing test and people still say it's not intelligent...

> Because they _are_.

Not really. Most of those seemingly novel problems are permutations of existing ones, like the one you mentioned. A solution is simply a specific permutation of tokens in the training data which humans are not able to see.

This doesn't mean that the permutation is something that previously didn't exist, let alone that it's something that is actually correct, but those scenarios are much rarer.

None of this is to say that these tools can't be useful, but thinking that this is intelligence is delusional.

> We have AI that is kind of close to passing the Turing test and people still say it's not intelligent...

The Turing test was passed arguably decades ago. It's not a test of intelligence. It's an _imitation game_ where the only goal is to fool humans into thinking they're having a text conversation with another human. LLMs can do this very well.

Re: There are no new ideas in AI, only new datasets

#232

Post by timewizard »

ekunazanu wrote: post #182
timewizard wrote: post #161

Earlier quoted context omitted.

> has made jaw-dropping progress They took 1970s dead tech and deployed it on machines 1 million times more powerful. I'm not sure I'd qualify this as progress. I'd also need an explanation as to what systemic improvements in models and computations that give an exponential growth in performance are planned. I don't see anything.

Winning two Nobel prizes wasn't enough progress?

Is progress measured in nobel prizes? My understanding is those are put to a vote by institutional committee.

Putting that aside. The shared prize in 2024 was given for work done in the 1970s and 1980s. Was this meant to be a confirmation of my point? You've done so beautifully.

In 2022 they saw fit to award Ben Bernanke. Yep. That one. For, I kid you not, work on the impacts of financial crises. Ironically also work originally done in the 1970s and 80s.

Re: There are no new ideas in AI, only new datasets

#233

Post by imiric »

tolerance wrote: post #70
lossolo wrote: post #64

I wrote about it around a year ago here: "There weren't really any advancements from around 2018. The majority of the 'advancements' were in the amount of parameters, training data, and its applications. What was the GPT-3 to ChatGPT transition? It involved fine-tuning, using specifically crafted training data. What changed from GPT-3 to GPT-4? It was the increase in the number of parameters, improved training data,…

And when winter does arrive, then what? The technology is slowing down while its popularity picks up. Can sparks fly out of snow?

> And when winter does arrive, then what?

If the technology is useful, the Slope of Enlightenment, followed by the Plateau of Productivity.

Re: There are no new ideas in AI, only new datasets

#234

Post by timewizard »

petesergeant wrote: post #185
timewizard wrote: post #161

Earlier quoted context omitted.

> has made jaw-dropping progress They took 1970s dead tech and deployed it on machines 1 million times more powerful. I'm not sure I'd qualify this as progress. I'd also need an explanation as to what systemic improvements in models and computations that give an exponential growth in performance are planned. I don't see anything.

> They took 1970s dead tech and deployed it on machines 1 million times more powerful. I’m not sure I’d qualify this as progress If this isn’t meant to be sarcasm or irony, you’ve got some really exciting research and learning ahead of you! At the moment it reads very “computers are just addition and multiplication and we’ve had that for thousands of years!”

> you’ve got some really exciting research and learning ahead of you

I've done the research. Which is why I made the point I did. You're being dismissive and rude instead of putting forth any sort of argument. It's the paper hat of fake intellect. Yawn.

> At the moment it reads very “computers are just addition and multiplication and we’ve had that for thousands of years!”

Let's be specific then. The problem with the models is they require exponential cost growth for model generation giving only linear increases in output performance. This cost curve is currently a factor or two stronger than the curve of increasing hardware performance. Putting the technology, absent any actual fundamental algorithmic improvements, which do /not/ seem forthcoming despite billions in speculative funding, into a strict coffin corner. In short: AI winter 2.0.

Got any plans for that? Any specific research that deals with that? Any thoughts of your own on this matter?

Re: There are no new ideas in AI, only new datasets

#235

Post by dinfinity »

computably wrote: post #228
dinfinity wrote: post #36

Earlier quoted context omitted.

> Language and vision are just the beginning — the parts we were able to digitize first - not necessarily the most central to intelligence. I respectfully disagree. Touch gives pretty cool skills, but language, video and audio are all that are needed for all online interactions . We use touch for typing and pointing, but that is only because we don't have a more efficient and effective interface. Now I'm not saying t…

Language is literally an abstraction of sensory inputs and cognitive processes. One can make similar arguments about image generation. These abstractions might characterize the higher cognitive abilities of humans, but it makes no sense to ignore "lower level" cognition. Embodiment is the foundation of our rich internal world models, in particular spacetime, causality, etc. Current generative models merely mimic the…

> Language is literally an abstraction of sensory inputs and cognitive processes.

Language can exist entirely independently from senses and cognition. It is an encoding of patterns in the world where the only thing that matters is if anybody or anything wielding it can map the encodings to and from the patterns they encode for (which is more of a sociological/synchronisation challenge).

Does C, or Java, 'make no sense' because it 'ignores lower level cognition'?

There are many parts of non-programming languages that similarly have nothing to do with embodiment. Some of them are even about incredibly abstract things impossible in our universe. One could argue that for many fields genius lies in being able to mentally model what is so foreign to the intuition our embodiment has imbued us with or to be able to find a mapping to facilitate that intuition. Said otherwise: the experience our embodiment has given us might limit how well we can understand the world (Quantum Mechanics anyone?).

Again, embodiment is interesting and worth pursuing, but far from a requirement for far-reaching intelligence.

Re: There are no new ideas in AI, only new datasets

#236

Post by tsimionescu »

voxleone wrote: post #16

I'd say with confidence: we're living in the early days. AI has made jaw-dropping progress in two major domains: language and vision. With large language models (LLMs) like GPT-4 and Claude, and vision models like CLIP and DALL·E, we've seen machines that can generate poetry, write code, describe photos, and even hold eerily humanlike conversations. But as impressive as this is, it’s easy to lose sight of the bigger…

I think one counterpoint to this idea is the compute cost.

To a great extent, it's not AI research that is the primary driver behind the huge advances in AI, either in terms of techniques (transformers) or data sets. Instead, the biggest single factor responsible for this huge boost are advances in compute hardware and compute power in general. Even if we had known about the Transformer architecture 20 years earlier, and we had had the datasets that OpenAI and Google amassed 20 years earlier, we still would not have been able to get anywhere close to training an LLM on hardware from 20 years ago.

And given this, and given that LLMs have already pushed this compute power to the limit, it's very possible that we'll stagnate at more or less the current level unless and until a new 10x or even 100x boost in compute power happens. It's very unlikely that you could train a model on 100x as much data as you get today without that, which is what you would likely require to add multiple modalities and then combine them.

Re: There are no new ideas in AI, only new datasets

#237

Post by [deleted] »

varjag wrote: post #73
justanotherjoe wrote: post #68

Earlier quoted context omitted.

I don't get why people are so invested in framing it this way. I'm sure there are ways to do the stated objective. John Carmack isn't even an AI guy why is he suddenly the standard.

What in your opinion constitutes an AI guy?

[deleted]

Re: There are no new ideas in AI, only new datasets

#238

Post by justanotherjoe »

surecoocoocoo wrote: post #154
justanotherjoe wrote: post #68

Earlier quoted context omitted.

I don't get why people are so invested in framing it this way. I'm sure there are ways to do the stated objective. John Carmack isn't even an AI guy why is he suddenly the standard.

Ah some No True Scotsman Not sure why justanotherjoe is a credible resource on who is and isn’t expert in some new dialectic and euphemism for machine state management. You’re that nobody to me :shrug: Yann LeCun is an AI guy and has simplified it as “not much more than physical statistics.” WWhole lot of AI is decades old info theory books applied to modern computer. Either a mem value is or isn’t what’s expected. E…

I'm not being gatekeeper here. John Carmack came into AI around 2021 iirc and came in Lex Friedman and said he's going to bring about AGI. It's okay for him to try so but he had no particular expertise in the field. He's a brilliant guy and I'm not gonna say he's not going to succeed, or that his opinion is worthless. But people seeemed to think that the whole field is a farce just waiting for an adult to come in and fix it. I find that biased. By the way this is how people end up worshipping figures like Musk. There's a limit to transfer function of human expertise, ironically to the discussion at hand.

That's just what I think anyway.

Re: There are no new ideas in AI, only new datasets

#239

Post by petesergeant »

timewizard wrote: post #234
petesergeant wrote: post #185

Earlier quoted context omitted.

> They took 1970s dead tech and deployed it on machines 1 million times more powerful. I’m not sure I’d qualify this as progress If this isn’t meant to be sarcasm or irony, you’ve got some really exciting research and learning ahead of you! At the moment it reads very “computers are just addition and multiplication and we’ve had that for thousands of years!”

> you’ve got some really exciting research and learning ahead of you I've done the research. Which is why I made the point I did. You're being dismissive and rude instead of putting forth any sort of argument. It's the paper hat of fake intellect. Yawn. > At the moment it reads very “computers are just addition and multiplication and we’ve had that for thousands of years!” Let's be specific then. The problem with the…

> I've done the research

Great. What's the 1970s equivalent of word2vec or embeddings, that we've simply scaled up? Where are the papers about the transformer architecture or attention from the 1970s? Sure feels like you think LLMs are just big perceptrons.

> The problem with the models is they require exponential cost growth

Let's stick to the assertion I was disputing instead.

Re: There are no new ideas in AI, only new datasets

#240

Post by justanotherjoe »

qaq wrote: post #77
justanotherjoe wrote: post #68

Earlier quoted context omitted.

I don't get why people are so invested in framing it this way. I'm sure there are ways to do the stated objective. John Carmack isn't even an AI guy why is he suddenly the standard.

Keen includes researchers like Richard Sutton, Joseph Modayil etc. Also John has being doing it full time for almost 5 years now so given his background and aptitude for learning I would imaging by this time he is more of an AI guy then a fairly large percentage of AI PhDs.

Yeah and in another 5 years he'd probably be at nobel laureate level in AI. I don't think that's how it works. What do you mean? Even a phd program can take 5 years sometimes. Also the man started saying he'd bring about AGI right at the gate. He wasn't being exactly humble.

God I hate sounding like this. I swear I'm not too good for John Carmack, as he's infinitely smarter than me. But I just find it a bit weird.

I'm not against his discovery, just against the vibe and framing of the op.

Post reply on HN