Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You are the scariest monster in the woods

jamie.ideasasylum.com

Post reply on HN
Print view View original

241–250 of 405 posts

Re: You are the scariest monster in the woods

#241

Post by yjftsjthsd-h »

myrmidon wrote: post #30
alansaber wrote: post #26

Earlier quoted context omitted.

Perhaps the AGI people thinking we can catch up millions of years of evolution in a handful of years

If you make the same argument for flight it looks really weak. Cognition is (to me) not even the most impressive and out-of-reach achievement: That would be how (our) and animals bodies are self-assembling, self-repairing and self-replicating, with an impressive array of sensors and actors in a highly integrated package. I honestly believe our current technology is much closer to emulating a human brain than it is to…

As sibling comment points out, flight is physically pretty simple. Also, it took us centuries to figure it out. I'd say comparing to flight makes it look pretty strong.

Re: You are the scariest monster in the woods

#242

Post by fullstackchris »

myrmidon wrote: post #13

I struggle to understand how people can just dismiss the possibility of artificial intelligence. Human cognition was basically bruteforced by evolution-- why would it be impossible to achieve the exact same result in silicon, especially after we already demonstrated some parts of those results (e.g. use of language) that critically set us apart from other animals? I'm not buying the whole "AI has no agency" line eith…

Of course it's possible. It's just DEFINITELY not possible using a large neural net, or basically a markov chain on steroids. C'mon, this should be very obvious by now in the world of agents / LLMs.

When is silicon valley gonna learn that token input and output =/= AGI?

Re: You are the scariest monster in the woods

#243

Post by fullstackchris »

reactordev wrote: post #217
Aerroon wrote: post #176

Earlier quoted context omitted.

> I'm not buying the whole "AI has no agency" line either; this might be true for now, but this is already being circumvented with current LLMs (by giving them web access etc). They don't have agency, because they don't have persistent state. They're like a function that you can query and get an answer. During that answer the LLM has state, but once it's done the state is gone. Humans (and other "agents") have persis…

The trick here is never turning it off so the ICL keeps growing and learning to the point where it’s aware.

but even as humans we still don't know what "aware" even means!

Re: You are the scariest monster in the woods

#244

Post by the8472 »

BoxOfRain wrote: post #181
progbits wrote: post #156

Earlier quoted context omitted.

Computer can simulate human brain on subatomic level (in theory). Do you agree this would be "sentient and intelligent" and not just symbol manipulating? If yes, everything else is just optimization.

Say we do have a 1:1 representation of the human brain in software. How could we know if we're talking to a conscious simulation of a human being, versus some kind of philosophical zombie which appears conscious but isn't? Without a solid way to differentiate 'conscious' from 'not conscious' any discussion of machine sentience is unfalsifiable in my opinion.

How do you tell the difference in other humans? Do you just believe them because they claim to be conscious instead of pointing a calibrated and certified consciousness-meter at them?

Re: You are the scariest monster in the woods

#245

Post by zamadatix »

Aerroon wrote: post #176
myrmidon wrote: post #13

I struggle to understand how people can just dismiss the possibility of artificial intelligence. Human cognition was basically bruteforced by evolution-- why would it be impossible to achieve the exact same result in silicon, especially after we already demonstrated some parts of those results (e.g. use of language) that critically set us apart from other animals? I'm not buying the whole "AI has no agency" line eith…

> I'm not buying the whole "AI has no agency" line either; this might be true for now, but this is already being circumvented with current LLMs (by giving them web access etc). They don't have agency, because they don't have persistent state. They're like a function that you can query and get an answer. During that answer the LLM has state, but once it's done the state is gone. Humans (and other "agents") have persis…

> During that answer the LLM has state, but once it's done the state is gone.

This is an operational choice. LLMs have state, and you never have to clear it. The problems come from the amount of state being extremely limited (in comparison to the other axes) and the degradation of quality as the state scales. Because of these reasons, people tend to clear the state of LLMs. That is not the same thing as not having state, even if the result looks similar.

Re: You are the scariest monster in the woods

#246

Post by KronisLV »

throwuxiytayq wrote: post #209
Aerroon wrote: post #176

Earlier quoted context omitted.

> I'm not buying the whole "AI has no agency" line either; this might be true for now, but this is already being circumvented with current LLMs (by giving them web access etc). They don't have agency, because they don't have persistent state. They're like a function that you can query and get an answer. During that answer the LLM has state, but once it's done the state is gone. Humans (and other "agents") have persis…

I have no words to express how profoundly disappointed I am to keep reading these boring, shallow, shorttermist, unimaginative takes that are invalidated by a model/arch upgrade next week, or - in this case - more like years ago, since pretty much all big LLM platforms are already augmented by RAG and memory systems. Do you seriously think you’re discussing a serious long term limitation here?

> pretty much all big LLM platforms are already augmented by RAG and memory systems

I think they're more focusing on the fact that training and inference are two fundamentally different processes, which is problematic on some level. Adding RAG and various memory addons on top of the already trained model is trying to work around that, but is not really the same to how humans or most other animals think and learn.

That's not to say that it'd be impossible to build something like that out of silicon, just that it'd take a different architecture and approach to the problem, something to avoid catastrophic forgetting and continuously train the network during its operation. Of course, that'd be harder to control and deploy for commercial applications, where you probably do want a more predictable model.

Re: You are the scariest monster in the woods

#247

Post by mediaman »

santadays wrote: post #238
mediaman wrote: post #216

Earlier quoted context omitted.

In Context Learning (ICL) is already a rapidly advancing area. You do not need to modify their weights for LLMs to persist state. The human brain is not that different. Our long-term memories are stored separately from our executive function (prefrontal cortex), and specialist brain functions such as the hippocampus serve to route, store, and retrieve those long term memories to support executive function. Much of th…

It seems like there is a bunch of research/working implementations that allow efficient fine tuning of models. Additionally there are ways to tune the model to outcomes vs training examples. Right now the state of the world with LLMs is that they try to predict a script in which they are a happy assistant as guided by their alignment phase. I'm not sure what happens when they start getting trained in simulations to b…

That's already happening. It started happening when they incorporated reinforcement learning into the training process.

It's been some time since LLMs were purely stochastic average-token predictors; their later RL fine tuning stages make them quite goal-directed, and this is what has given some big leaps in verifiable domains like math and programming. It doesn't work that well with nonverifiable domains, though, since verifiability is what gives us the reward function.

Re: You are the scariest monster in the woods

#248

Post by Stevvo »

woeirua wrote: post #44

There's this weird disconnect in tech circles, where everyone is deathly afraid of AGI, but totally asleep on the very real possibility of thermonuclear war breaking out in Europe or Asia over the next 10 years. There's already credible evidence that we came perilously close to the use of tactical nuclear weapons in Ukraine which likely would've spiraled out of control. AGI might happen, but the threat of nuclear war…

There is no evidence that use of tactical nuclear weapons in Ukraine would spiral out of control. I like to think that the US/UK/France would stay out of it, if only because the leaders value their own lives if not those of others.

Re: You are the scariest monster in the woods

#249

Post by btilly »

myrmidon wrote: post #13

I struggle to understand how people can just dismiss the possibility of artificial intelligence. Human cognition was basically bruteforced by evolution-- why would it be impossible to achieve the exact same result in silicon, especially after we already demonstrated some parts of those results (e.g. use of language) that critically set us apart from other animals? I'm not buying the whole "AI has no agency" line eith…

I agree that we should not dismiss the possibility of artificial intelligence.

But the central argument of the article can be made without that point. Because the truth is that right now, LLMs are good enough to be a force multiplier for those who know how to use them. Which eventually becomes synonymous with "those who have power". This means that the power of AI will naturally get used to further the ends of corporations.

The potential problem there is that corporations are natural paperclip maximizers. They operate on a model of the world where "more of this results in more of that, which gets of more of the next thing, ..." And, somewhere down the chain, we wind up with money and resources that feed back into the start to create a self-sustaining, exponentially growing loop. (The underlying exponential nature of these loops has become a truism that people rely on in places as different as finance, and technology improvement curves.)

This naturally leads to exponential growth in resource consumption, waste, economic growth, wealth, and so on. In the USA this growth has averaged about 3-3.5% per year. With growth varying by area. Famously, growth rates tend to be much higher in tech. Likewise growth rates are higher in some areas than others. (The best known example is the technology curve described by Moore's law. Which has had a tremendous impact on our world.)

The problem is that we are undergoing exponential growth in a world with ultimately limited resources. Which means that the most innocuous things will eventually have a tremendous impact. The result isn't simply converting everything into a mountain of paperclips. We have mountains of many different things that we have produced, and multiple parallel environmental catastrophes from the associated waste.

Even with no agency, AI serves as a force multiplier for this underlying dynamic. But since AI is being inserted as a crucial step at so many places, AI is on a particularly steep growth curve. Estimates for total global electricity spent on AI are in the range 0.2-0.4%. That seems modest, but annual growth rates are projected as being in the range of 10-30%. (The estimates are far apart because a lot of the data is not public, and so has to be estimated.) This is a Moore's law level growth. We are likely to see the electricity consumption of AI grow past all other uses within our lifetimes. And that will happen even without the kind of sudden leaps in capability that machine learning regularly delivers.

I hope we humans like those paperclips. Humans, armed with AI, are going to make a lot of them. And they're not actually free.

Post reply on HN