Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The changing goalposts of AGI and timelines

mlumiste.com

Post reply on HN
Print view View original

321–330 of 411 posts

Re: The changing goalposts of AGI and timelines

#321

Post by godelski »

daxfohl wrote: post #314
godelski wrote: post #301

Earlier quoted context omitted.

> there's no reason we'd need to have humans working jobs that only involve typing stuff into a computer and going to meetings all day I'm not sure I understand, and want to check. That really applies to a lot of jobs. That's all admins, accountants, programmers, probably includes lawyers, and probably includes all C-suite execs. It's harder for me to think of jobs that don't fit under this umbrella. I can think of s…

Actually it occurs to me that even if we did have AGI, or even if ASI, heck if ASI even moreso, we'd still need desk jobs to maintain the guardrails. Intelligence is one thing, being able to figure out how get a task done (say). But understanding that no, I don't want you to exploit a backdoor or blackmail my teammate or launch a warhead even though that might expedite the task. Or why some task is more important tha…

  > we'd still need desk jobs to maintain the guardrails.
Agreed. I don't get why people think it is a good idea not to. I'd wager even the AGI would agree. The reason is quite simple: different perspectives help. Really for mission critical things it makes sense to have multiple entities verifying one another. For nuclear launches there's a chain of responsibility and famously those launching have two distinct keys that must be activated simultaneously. Though what people don't realize is that there's a chain of people who act and act independently during this process. It isn't just the president deciding to nuke a location and everyone else carrying out the commands mindlessly. But in far lower stakes settings... we have code review. Or a common saying in physical engineering as well among many tradesmen "measure twice, cut once".

It would be absolutely bonkers to just hand over absolute control of any system to a machine before substantial verification. These vetting processes are in place for a reason. They can be annoying because they slow things down, but they're there because they speed things up in the long run. Because their existence tends to make things less sloppy, so they are less needed. But their existence also catches mistakes that were they made slow down processes far more than all the QA annoyances and slowdowns could ever cause combined.

  > And why not? If you can get AI to do the work of the scientist for a tenth of the price
And what are the assumptions being made here? Equal quality work? To my question, this is part of the implication. Price is an incredibly naive metric. We use it because we need something, but a grave mistake is to interpret some metric as more meaningful than it actually is. Goodhart's Law? Or just look at any bureaucracy. I think we need to be more refined than "price". It's going to be god awfully hard to even define what "equal quality" means. But it seems like you're recognizing that given your other statements.

Re: The changing goalposts of AGI and timelines

#322

Post by dragonwriter »

marcus_holmes wrote: post #320
shepherdjerred wrote: post #207

Earlier quoted context omitted.

They define AGI in their charter > artificial general intelligence (AGI)—by which we mean highly autonomous systems that outperform humans at most economically valuable work

y'see, I would not define a system as "highly autonomous" if it only responds to requests. And I get that there are workarounds; effectively a cron job every second prompting "do the next thing". But in my personal definition of "highly autonomous" it would not need prompting at all. It would be thinking all the time, independently of requests.

The model is not the system. The model is a component of the system. The "cron job" (or other means by which a continuous action loop is implemented) and the necessary prompting for it to gather input (including subsequent user input or other external data) and to pursue a set of objectives which evolves based on input are all also parts of the system.

Re: The changing goalposts of AGI and timelines

#323

Post by daxfohl »

godelski wrote: post #321
daxfohl wrote: post #314

Earlier quoted context omitted.

Actually it occurs to me that even if we did have AGI, or even if ASI, heck if ASI even moreso, we'd still need desk jobs to maintain the guardrails. Intelligence is one thing, being able to figure out how get a task done (say). But understanding that no, I don't want you to exploit a backdoor or blackmail my teammate or launch a warhead even though that might expedite the task. Or why some task is more important tha…

> we'd still need desk jobs to maintain the guardrails. Agreed. I don't get why people think it is a good idea not to. I'd wager even the AGI would agree. The reason is quite simple: different perspectives help. Really for mission critical things it makes sense to have multiple entities verifying one another. For nuclear launches there's a chain of responsibility and famously those launching have two distinct keys th…

And "maintaining guardrails" may be far more grandiose than it sounds. It's like if we have this energy source that could destroy the planet, but the closer you get to it without going past some threshold, the energy you get from it is proportional to the inverse of how close you are to it. There's some wiggle room and you can poke and prod and recover if it starts to go ballistic, but your goal is to extract as much energy (or wealth or whatever) out of it as possible. Every company in the world, every engineer on the planet would be pushing to extract just a little bit more without going beyond the limit.

AI could go the same way. It's a creation engine like nothing that's ever been seen before, but it can also become a destruction engine in ways that we could never understand or hope to counter, and left unchecked, the odds of that soar to near certainty. So the first job is to place dummy guardrails around it. That's where we are now. But soon that becomes too restrictive. What can we loosen? How do we know? How can we recover if we're wrong? We're not quite there yet, but we're not not there either.

Of course eventually somebody is going to trigger it and it's going to go ballistic. Our only hope is that it happens at exactly the right time where AGI can cause enough damage for people to notice, but not enough to be irrecoverable. Maybe we should rename this whole AGI thing to Project Icarus.

Re: The changing goalposts of AGI and timelines

#324

Post by [deleted] »

wongarsu wrote: post #216
Jensson wrote: post #185

Earlier quoted context omitted.

It is very easy to tell if we still need humans in the loop. We still do so its not AGI.

For certain types of "human in the loop". If it can't write working code without a human in the loop then it's not AGI. But a human-level coder also has lots of humans in the loop: a more senior developer doing code review, several layers of management, a product owner that interfaces the project with outside reality, sales people, etc. Now I already hear you typing "but those roles should also be handles by AI if it…

[deleted]

Re: The changing goalposts of AGI and timelines

#325

Post by sebastos »

ufmace wrote: post #169
sebastos wrote: post #156

Earlier quoted context omitted.

Firstly, the models that pass the Math Olympiad aren’t the same models as the ones you’re saying “pass the Turing test”. Secondly, nothing actually passes the Turing test. They pass a vibes check of “hey that’s pretty good!” but if your life depended on it, you could easily find ways to sniff out an LLM agent. Thirdly, none of these models learn in real time, which is an obviously essential feature. We’ll know AGI wh…

> nothing actually passes the Turing test Says who? I had already found this study, published almost a year ago, saying that they do: https://arxiv.org/abs/2503.23674 There doesn't seem to be a super-rigorous definition of the Turing Test, but I don't think it's reasonable to require it to fool an expert whose life depends on the correct choice. It already seems to be decently able to fool a person of average intelli…

The real answer is that once LLMs passed a "casual" application of the Turing test, it just made us realize that the "casual Turing test" is not particularly interesting. It turns out to be too easy to ape human behavior over short time frames for it to be a good indicator of human-like intelligence.

Now, you could argue that this right here is the aforementioned moving of the goalposts. After all, we're deciding that the casual Turing test wasn't interesting precisely after having seen that LLMs could pass it.

However, in my view, the Turing test _always_ implied the "rigorous" Turing test, and it's only now that we're actually flirting with passing it that it had to be clarified what counts as a true Turing test. As I see it, the Turing test can still be salvaged as a criteria for genera intelligence, but only if you allow it to be a no-holds-barred, life-depends-on-it test to exhaustion. This would involve allowing arbitrarily long questioning periods, for instance. I think this is more in the spirit of the original formulation, because the whole idea is to pit a machine against all of human intelligence, proving it has a similar arsenal of adaptability at its disposal. If it only has to passingly fool a human for brief periods, well... I'm afraid that just doesn't prove much. All sorts of stuff briefly fools humans. What requires intelligence is to consistently anticipate and adapt to all lines of questioning in a sustained manner until the human runs out of ideas for how to differentiate.

Re: The changing goalposts of AGI and timelines

#326

Post by hintymad »

Spivak wrote: post #311
hintymad wrote: post #295

Earlier quoted context omitted.

> It has no real world model, no ability to learn in any but superficial ways I also think so, and in the meantime I have to admit a lot of people don't learn deeply either. Take math for example, how many STEM students from elite universities truly understood the definition of limit, let alone calculus beyond simple calculation? Or how many data scientists can really intuitively understand Bayesian statistics? Yet m…

Well part of that is because STE folks aren't typically required to take any kind of theoretical maths. It's $Math for Engineers and it eschews theoretical underpinnings for application. I don't think it's any kind of failing, it's just different. My statistics class was a dense treatise in measure theory. Anyone who took the regular stats class is almost surely way better than me at designing an experiment, but I ca…

I was not talking about theoretical foundations like Analysis or measure theory, but just basics in college-level math class. There can be other examples. The point is that many people didn’t have intuitive understanding of what they use everyday — in a way they are like AI, only slower and know less than AI

Re: The changing goalposts of AGI and timelines

#327

Post by remarkEon »

btown wrote: post #106
remarkEon wrote: post #66

Earlier quoted context omitted.

Agree, and I think the labeling of them (Anthropic) a supply chain risk was handled poorly and will likely be reverted over time. That being said, I would be nervous if I was in the Pentagon and depended on Anthropic tooling for something, even if that something was unrelated to kinetic operations. How do they audit that Anthropic can't alter model outputs for contexts they (the ethics board or whatever it's called,…

A counter-argument here: if a private company knows that its technology may be used for human-not-in-loop targeting/surveillance, and knows that its technology is not yet ready to fulfill that use case without meaningful unintended casualties... does that company have an ethical obligation to contractually delineate its inability to offer that service? In a version of a trolley problem where you're on a track that wi…

It's an interesting question, but it's mostly irrelevant.

This problem is really difficult to discuss because we are all wrapping the capabilities of these tools into our response framing. These are tools, or weapons. Your hypothetical could just as easily be applied to GBU-39s, a smaller laser guided bomb that's meant to take out, say, a single vehicle in a convoy versus the entire set of vehicles. If you're not confident in what the product is supposed to do, and you've already sold it to the government, you have lied and they are going to come back to you asking some direct questions.

Re: The changing goalposts of AGI and timelines

#328

Post by gosub100 »

runarberg wrote: post #201
tokai wrote: post #150

Earlier quoted context omitted.

Turing test is generally misunderstood, much like Schrodinger's cat, it has devolved in to a pop cultural meme. The test is to evaluate if a machine can think . Not if it is intelligent, not if it is human-like. Its dismissed as a useful by most experts in philosophy of mind, AI, language, etc.. Thinking cool and all but not that extraordinary. Even plants does it.

I like the analogy with Schrödinger’s cat. Like Schrödinger’s cat it is actually not a good thought experiment. Both have been debunked. Schrödinger’s cat is applying quantum behavior (of a single interaction) to a macro system (with trillions of interactions). While the Turing test can be explained away with Searle’s Chinese room thought experiment. I would argue that Schrödinger’s cat has done more damage to the ge…

What do you mean by Schrodinger's cat experiment being "debunked"? The only way I can think to debunk it is to say there are ways to determine if the cat is alive such as heartbeat or temperature, which are impossible to isolate at a quantum level. I don't think anyone claimed the animal was in a superposition.

Re: The changing goalposts of AGI and timelines

#329

Post by d0able »

choult wrote: post #6

The writing was on the wall as soon as it went all-in on commercializing the tech. This will never happen, LLMs are already being used very unsafely, and if this HN headline stays where it is OpenAI will quietly remove their charter from their website.

It's just lip service at this point.

Re: The changing goalposts of AGI and timelines

#330

Post by wise_blood »

djoldman wrote: post #116

Anytime I see "Artificial General Intelligence," "AGI," "ASI," etc., I mentally replace it with "something no one has defined meaningfully." Or the long version: "something about which no conclusions can be drawn because the proposed definitions lack sufficient precision and completeness." Or the short versions: "Skippetyboop," "plipnikop," and "zingybang."

the ARC definition is the one I like the best, something like:

"it is AGI when we can no longer come up with tasks easy for humans to solve but hard for computers"

Post reply on HN