Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Many in the AI field think the bigger-is-better approach is running out of road

economist.com

Post reply on HN
Print view View original

121–130 of 354 posts

Re: Many in the AI field think the bigger-is-better approach is running out of road

#121

Post by Legend2440 »

michaelteter wrote: post #63

Isn't the fundamental problem that LLM's don't actually understand anything (as greater concepts), but rather operate as complex probability machines? My 2 month active experience with ChatGPT-4 gave me the following takeaways: - when it's right, it's amazing; and when you, the operator, can recognize the niche use case where it performs really well, it can be a game-changer (although you could have programmed a tool…

It's a game-changer in that it shows computers being able to climb levels of abstraction on their own.

In the past, if you wanted programs to work with a high-level idea, you had to explicitly hand it to them. Humans had to do the work of turning data into concepts. These generative AI systems are different - they can learn abstractions from data, and manipulate them in complex ways.

Is ChatGPT a good and accurate chatbot? Maybe not. But it's a fundamental change in what computers are capable of.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#122

Post by Legend2440 »

anikan_vader wrote: post #117
thereare5lights wrote: post #114

Earlier quoted context omitted.

There's no evidence for nondeterminism in the human brain

There is plenty of evidence for non-determinism in matter, which the brain is notably made out of.

Not necessarily. Everything is deterministic above the quantum level, and it's possible that quantum non-determinism is the result of deterministic processes we can't see.

Lots of deterministic processes (like PRNGs) look random from the outside - that's what chaos theory is about. I think it's likely that everything in the universe is deterministic.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#123

Post by esafak »

willmadden wrote: post #102

Define “running out of road”. Road to control narratives or road to be useful?

There are no datasets that are an order of magnitude bigger, and the latest model itself may be a mixture of experts; i.e., more of the same.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#124

Post by geysersam »

usrbinbash wrote: post #88
nomel wrote: post #49

Earlier quoted context omitted.

> that don't hallucinate “Hallucination” is part of thought. Solving a new problem requires hallucinating new, non existing, possible outcomes and solutions, to find one that will work. It seems that eliminating the ability to interpolate and extrapolate (hallucinations) would make intelligence impossible. It would eliminate creativity, tying together new concepts, creation, etc. Is the goal AI, or a nice database fr…

The problem is, what we call "hallucinating" in LMs isn't a way of creative thinking and coming up with novel solutions. It also has nothing to do with "interpolate and extrapolate". It's simply when the predicted probable sequence isn't grounded in reality. When I ask an LLM to summarize the great water wars of 1999, and how the Trade Union was ultimately defeated by the Antarctic Coalitions hovercraft-fleet under V…

It's a great illusionist. But ultimately it cannot separate relevant information from simple word correlations.

> What is heavier, a small floating passenger ferry or a two metric ton heavy rock that sinks to the bottom of the ocean.

> A two metric ton heavy rock would be heavier than a small floating passenger ferry. The weight of the rock is two metric tons, which is equivalent to 2,000 kilograms or 4,409 pounds. The weight of the passenger ferry would depend on its specific design and construction materials, but it is unlikely to be heavier than two metric tons. Therefore, the heavy rock would have a greater weight than the small floating passenger ferry.

It completely relies on surface information such as "small floating" and ignores the deeper "correlation" that all ferries are heavy.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#125

Post by visarga »

michaelteter wrote: post #63

Isn't the fundamental problem that LLM's don't actually understand anything (as greater concepts), but rather operate as complex probability machines? My 2 month active experience with ChatGPT-4 gave me the following takeaways: - when it's right, it's amazing; and when you, the operator, can recognize the niche use case where it performs really well, it can be a game-changer (although you could have programmed a tool…

One field where it works rather well is in semantic data extraction. Do you remember the dream of semantic web? Having access to structured information from unstructured sources, auto-matching API functions and arguments to make interoperation easier. We can do that now.

Here is a sample where I used my own post (just copy pasted the raw text from the browser) and got this schema

    comment:
      meta:
        points:
        author:
        time:
        action:
          parent:
          next:
          edit:
          delete:
        post:
      content:
      action:
        reply:
It also guessed it was a Hacker News comment by the formatting of the meta section.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#126

Post by JimtheCoder »

goldenshale wrote: post #120

As if anyone is good at predicting the future. Please can we stop acting like expertise equates to fortune telling capabilities?! Nobody has any clue what a 1000x sized GPT model could do, and anybody who makes strong claims is a charlatan. In this age of paranoid AI risk cultists we need to cultivate humility and calm, a willingness to follow data rather than beliefs and predictions.

Sir, this is a discussion forum...

Re: Many in the AI field think the bigger-is-better approach is running out of road

#127

Post by ruleryak »

dwaltrip wrote: post #103

The last paragraph in the article: > That such big performance increases can be extracted from relatively simple changes like rounding numbers or switching programming languages might seem surprising. But it reflects the breakneck speed with which llms have been developed. For many years they were research projects, and simply getting them to work well was more important than making them elegant. Only recently have t…

It's an incredibly important point -

We have so many a-ha moments ahead of us in this field. Seemingly minor changes yielding task speed multipliers, fresh eyes on foundational codebases saying now why the heck did they do it that way when xyz exists and works better, etc. A recent graphics driver update took my local SD performance from almost 4 seconds per iteration to 2.7it/s because someone somewhere had an a-ha moment. We're practically in the Commodore 64 era of this technology and there are only going to be more and more people putting their eyes and minds on these problems every day.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#128

Post by wzdd »

bemmu wrote: post #111
michaelteter wrote: post #63

Isn't the fundamental problem that LLM's don't actually understand anything (as greater concepts), but rather operate as complex probability machines? My 2 month active experience with ChatGPT-4 gave me the following takeaways: - when it's right, it's amazing; and when you, the operator, can recognize the niche use case where it performs really well, it can be a game-changer (although you could have programmed a tool…

I’m not convinced the language part of my brain isn’t just a complex probability machine, just with different trade-offs.

To expand on this, healthy human brains seem to be made of many interacting systems which function in different ways. It may be that in the future we view the search for one simple formula for "intelligence" / "understanding" / "consciousness" (pick your poorly-defined poison) the same way biologists view the concept of elan vital.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#129

Post by acover »

usrbinbash wrote: post #87
Animats wrote: post #11

We need a way to make tight little specialist models that don't hallucinate and reliably report when they don't know. Trying to cram all of the web into a LLM is a dead end.

> and reliably report when they don't know. Then we need a new system, because LMs, no matter if they are large or not, cannot do that, for a very simple reason: A LM doesn't understand "truthfulness". It has no concept of a sequence being true or not, only of a sequence being probable. And that probability cannot work as a standin for truthfulness, because the LM doesn't produce improbable sequences to begin with...…

An llm can detect when an llm has outputted an inconsistent world.

It can reason. To an extent.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#130

Post by Legend2440 »

andsoitis wrote: post #104
tbrownaw wrote: post #75

Earlier quoted context omitted.

> don't actually understand anything (as greater concepts), but rather operate as complex probability machines? But, things are defined by how they interact with the world around them. A concept is its relations to other concepts. Which does seem to be the general sort of thing that these models are trying to get at, even if they don't seem to do a great job of it.

> A concept is its relations to other concepts. where does a concept begin? where does a concept end? what is a concept?

A concept is an abstraction.

Abstractions are a way to characterize the behavior of complex systems. It's not feasible to directly compute their behavior, but they still have predictable properties. Concepts let you handle emergence; you can manipulate an object as its own thing rather than a collection of atoms.

(And yes, I did just read Stephen Wolfram's book and this idea is largely based on it. I think he has some delusions of grandeur but is also onto something.)

Post reply on HN