Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Many in the AI field think the bigger-is-better approach is running out of road

economist.com

Post reply on HN
Print view View original

201–210 of 354 posts

Re: Many in the AI field think the bigger-is-better approach is running out of road

#201

Post by regularfry »

usrbinbash wrote: post #87
Animats wrote: post #11

We need a way to make tight little specialist models that don't hallucinate and reliably report when they don't know. Trying to cram all of the web into a LLM is a dead end.

> and reliably report when they don't know. Then we need a new system, because LMs, no matter if they are large or not, cannot do that, for a very simple reason: A LM doesn't understand "truthfulness". It has no concept of a sequence being true or not, only of a sequence being probable. And that probability cannot work as a standin for truthfulness, because the LM doesn't produce improbable sequences to begin with...…

What's needed, ideally, is a checker. Something that takes the LLM's output, can go back to the training material, and verify the output for consistency with it.

I don't think those steps are out of the bounds of possibility, really.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#202

Post by attemptone »

sheepscreek wrote: post #56

Sam Altman has been saying this for months. Nothing noteworthy here for someone following the industry closely. A16z’s latest summary of the landscape was way more useful and relevant than this.

Following research instead of the industry would this make a basic fact known for a while.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#203

Post by simion314 »

TX81Z wrote: post #171
simion314 wrote: post #45

Earlier quoted context omitted.

>GPT4 is really good at code for popular languages, though for JS it most of the time outputs obsolete syntax and code. Today i tried to do a bit of scripting with my son in Garrys Mod, it uses Expression 2 for a Wiremod module. GPT hallucinated a lot of functions and the worst part it switched almost each time from e2 to lua. It is good at solving homeworks for students, or solving popular problems in popular langua…

I’ve been writing JavaScript for 25 years and have always felt it has utterly trash syntax. I’m not sure that’s 100% on GPT. (And I don’t mean to be rude - I just really hate that syntax!)

What I mean it will generate code using var instead of let or const like instead of a nice array.forEach it will create for(var i

i mean use at least "let"

Re: Many in the AI field think the bigger-is-better approach is running out of road

#204

Post by MrScruff »

chaosjevil wrote: post #172
bemmu wrote: post #111

Earlier quoted context omitted.

I’m not convinced the language part of my brain isn’t just a complex probability machine, just with different trade-offs.

Nah. Human utterances convey purpose on a discursive level; including your comment or mine. We say stuff because we want to do something, like showing [dis]agreement or inform another speaker or change the actions of the other speaker. This is not just probabilistic - it's a way to handle the world. In the meantime those large language models simply predict the next word based on the preceding words.

We're able to do something analogous to reinforcement learning (take on new example data to update our 'weights').

Why do I spend time debating these ideas on Hacker News? Probably the underlying motivation is improving the reliability of my model of the world, which over my lifetime and the lifetimes of creatures before me has led to (somewhat indirectly) positive outcomes in survival and reproduction.

Is my model of the world that different to that of an LLM? I'm sure it is in many ways, but I expect their are similarities as well. An LLMs model encodes in a form a bunch of higher order relationships between concepts as defined by the word embedding. I think my brain encodes something similar, although the relationships are probably orders of magnitude more complex than the relationships encoded with GPT-4.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#205

Post by asddubs »

bemmu wrote: post #111
michaelteter wrote: post #63

Isn't the fundamental problem that LLM's don't actually understand anything (as greater concepts), but rather operate as complex probability machines? My 2 month active experience with ChatGPT-4 gave me the following takeaways: - when it's right, it's amazing; and when you, the operator, can recognize the niche use case where it performs really well, it can be a game-changer (although you could have programmed a tool…

I’m not convinced the language part of my brain isn’t just a complex probability machine, just with different trade-offs.

maybe, but the language part of your brain isn't the only part of your brain

Re: Many in the AI field think the bigger-is-better approach is running out of road

#206

Post by mo_42 »

zappchance wrote: post #163
kedean wrote: post #133

Earlier quoted context omitted.

If it's not actually possible to simulate a system within the confines of physics, does it being deterministic actually matter outside of thought experiments? I feel like a random system and a deterministic system which cannot be simulated are effectively the same thing.

It only matters in matters of free will and ethics. One actual scenario where it's relavant would be the discussion around the criminal justice system. If the universe is deterministic, how can punitive justice be justified?

> If the universe is deterministic, how can punitive justice be justified?

Determinism doesn't necessarily mean that organisms always act in the same way. They act in the same way given the exact configuration of them and the world.

Obviously, justice changes the configuration of an organism (fines, prison, ...). To me it boils down to the question whether justice decreases the likelihood to commit crimes again. Given that our systems of justice have evolved over a long time, I'd give them the benefit of the doubt.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#207

Post by MrScruff »

tchaffee wrote: post #179
Legend2440 wrote: post #170

Earlier quoted context omitted.

The point of punishment is to discourage you (and other people) from doing that action in the future. You don't need free will for that.

If you don't have free will then how would something discourage you from doing something that is already determined you will do? That doesn't make any sense.

Not the commenter you're replying to, and I'm sure you already realise this, but crime/punishment in a fully deterministic universe could be seen as a 'self correction' mechanism of the whole system.

One could imagine an impossibly vast cellular automata system which develops individual cellular 'agents'. Over time the agents develop some means of reproduction/death/reward system and become more and more complex. Then the agents that evolve with cooperative behaviours start to dominate. One might imagine that the system as a whole would also evolve these kind of 'self corrective' behaviours for anti-cooperative behaviours of the individual agents.

This is ignoring the whole philosophical ethics discussion and consciousness of course.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#208

Post by layer8 »

tchaffee wrote: post #179
Legend2440 wrote: post #170

Earlier quoted context omitted.

The point of punishment is to discourage you (and other people) from doing that action in the future. You don't need free will for that.

If you don't have free will then how would something discourage you from doing something that is already determined you will do? That doesn't make any sense.

Because the punishment is part of the input that determines what you will do.

In fact, if free will were absolute, punishment wouldn't make any sense because it wouldn't have any effect on your will.

Re: Many in the AI field think the bigger-is-better approach is running out of road

#209

Post by layer8 »

Legend2440 wrote: post #162
goatlover wrote: post #140

Earlier quoted context omitted.

Or the Many Worlds Interpretation is the correct understanding of quantum mechanics. The MWI people will say that indeterminism comes from the Copenhagen idea of there being a random collapse. But since measuring devices and human brains are also quantum systems, there's no reason to propose a collapse. Decoherence would be the reason we only see one result.

Maybe, but there's no data to support one interpretation over another. I believe that all quantum interpretations are incomplete and therefore wrong. Quantum mechanics is an abstraction over a deeper level of physics we can't measure yet.

Objective-collapse theories can actually be experimentally tested: https://en.wikipedia.org/wiki/Objective-collapse_theory#Test...

Re: Many in the AI field think the bigger-is-better approach is running out of road

#210

Post by murderfs »

tchaffee wrote: post #176
brutusborn wrote: post #166

Earlier quoted context omitted.

It can be justified in a deterministic universe because it makes criminals less likely to commit crime in future. As a recipient of punitive justice myself, being punished had a tangible effect on how I thought about crime and thus how I behaved post-punishment. Whether you believe that was deterministic or due to my own free will doesn’t change the outcome.

Punishment is not effective for most people in reducing crime. One thing that is effective is increasing the perceived risk of getting caught. People rarely commit crimes when they are sure they will be caught.

For the sake of argument, let's assume that punishment has absolutely no effect on whether someone will commit a crime. Even so, there's still a benefit to imprisonment: it separates criminals from society for a period of time, during which they won't be committing crimes against it.
Post reply on HN