Live data from Hacker News

International Scientific Report on the Safety of Advanced AI [pdf]

assets.publishing.service.gov.uk

51–60 of 66 posts

Re: International Scientific Report on the Safety of Advanced AI [pdf]

#51
post #42

Earlier quoted context omitted.

This is like someone saying "I am much more worried about the implications of dumb humans using flintlock muskets in the near term, then I am about the theoretical threat of machine guns and nuclear weapons." Surely the potential for both misuse and mistakes goes up the more powerful the technology gets.

That's fair, but to keep going with the analogy: we are currently the Native Americans in the 1500's, and the Conquistadors are coming ashore with their flintlocks (ML). Should we be more worried about them, or the future B-2 bombers, each armed with sixteen B83 nukes (AGI)? I understand that the timeline may be exponentially more compressed in our modern case, but should we ignore the immediate problem? In this anal…

[deleted]

Re: International Scientific Report on the Safety of Advanced AI [pdf]

#52
post #31

Earlier quoted context omitted.

It is very simple: powerful governments tried to stop cryptography, we know what happened. Also governments tried to prohibit alcohol, etc. it does not work. You can get them even in places such as Saudi Arabia. Are they expensive? For sure, but when it is about science that you can run in your own computers nothing can stop it. Will they put a Clipper chip?

The difference between "stop AI" and "stop cryptography" is that those of us who want to stop AI want to stop AI models from becoming more powerful by stopping future mathematical discoveries in the field. In contrast, the people trying to stop cryptography were trying to stop the dissemination of math that had already been discovered and understood well enough to have been productized in the form of software. Wester…

Well, in my book that is call obscurantism and never worked for long. It would be the first time that something like this works forever in humanity. I think once the genius is outside the bottle you cannot close him again.

If I take the science fiction route I would say that humans in your position should think about moving to another planet and create military defenses against AI.

Re: International Scientific Report on the Safety of Advanced AI [pdf]

#53

Earlier quoted context omitted.

So what if it was generated by AI? Does that invalidate its contents in any way?

Interesting point. Are we already at a state where an A.I. could respond to email, phone calls, and even video calls in a convincing way ?

Email definitely, just have to remember to fine tune so it says "sure, I'll get on that after lunch" rather than "as a language model…".

Voice calls, yes: I attended a talk last summer where someone did that with an AI trained on their own voice so they didn't have to waste time on whatsapp voice messages. The interlocutors not only didn't notice, they actively didn't believe it was AI when he told them (until he showed them the details).

Video… I don't think so? But that's due to latency and speed, and I'm basing my doubt on diffusion models which may be the wrong tool for the job.

Re: International Scientific Report on the Safety of Advanced AI [pdf]

#54
post #31

Earlier quoted context omitted.

It is very simple: powerful governments tried to stop cryptography, we know what happened. Also governments tried to prohibit alcohol, etc. it does not work. You can get them even in places such as Saudi Arabia. Are they expensive? For sure, but when it is about science that you can run in your own computers nothing can stop it. Will they put a Clipper chip?

The difference between "stop AI" and "stop cryptography" is that those of us who want to stop AI want to stop AI models from becoming more powerful by stopping future mathematical discoveries in the field. In contrast, the people trying to stop cryptography were trying to stop the dissemination of math that had already been discovered and understood well enough to have been productized in the form of software. Wester…

There's several assumptions you're making. First, that sufficient pressure will be built up into stopping AI before drastic harms occur instead of after, at which point stopping the math will be exactly the same as was stopping cryptography.

And that should there be no obvious short term harms to a technology, there can be no long term harms. I don't think it's self evident that all the harms would've already occurred. Surely humanity has not yet reached every type and degree of integration with current technology possible.

Re: International Scientific Report on the Safety of Advanced AI [pdf]

#55
post #21
post #19

Earlier quoted context omitted.

> is the only kind which poses a true existential threat You don't accept the possibility that a non-improving tool of an AI system that is fixed at the level of "just got a PhD in everything" by reading all the research papers on arxiv, might possibly be advanced enough for a Jim Jones type figure to design and create a humanity-ending plague because they believe in bringing about the end times?

Wouldn't there be 100x more of the same capability looking for threats and trying to head them off? A very advanced tool is still just a tool, and subject to countermeasures. I can see why countries would want to regulate it, but personally I think it's a distinctly different category than what the GP comment was talking about. There is no stopping a singularity level event after it's begun, at least not by any proce…

> Wouldn't there be 100x more of the same capability looking for threats and trying to head them off?

Hard to determine.

It's fairly easy to put absolutely everyone under 24/7 surveillance. Not only does almost everyone carry a phone, but also laser microphones are cheap and simple, and WiFi can be used as wall penetrating radar capable of pose detection at sufficient detail for heart rate and breath rate sensing.

But people don't like it when they get spied on, it's unconstitutional etc.

And we're currently living through a much lower risk arms race of the same general description with automatic code analysis to find vulnerabilities before attackers exploit them, and yet this isn't always a win for the defenders.

Biology is not well-engineered code, but we have had to evolve a general purpose anti-virus system, so while I do expect attackers to have huge advantages, I have no idea if I'm right to think that, nor do I know how big an advantage in the event that I am right at all.

> There is no stopping a singularity level event after it's begun, at least not by any process where people play a role

Mm, though I would caution that singularities in models is a sign the model is wrong: to simplify to IQ (a flawed metric) an AI that makes itself smarter may stop at any point because it can't figure out the next step, and that may be an IQ 85 AI that can only imagine reading more stuff, or an IQ 115 AI that knows it wants more compute so it starts a business that just isn't very successful, or it might be IQ 185 and do all kinds of interesting things but still not know how to make the next step any more than the 80 humans smarter than it, or it might be IQ 250 and beat every human that has ever lived (IQ 250 is 10σ, beating 10σ is p ≈ 7.62e-24, and one way or another when there have been that many humans, they're probably no longer meaningfully human) but still not know what to do next.

I prefer to think of it as an event horizon: beyond this point (in time), you can't even make a reasonable attempt at predicting the future.

For me, this puts it at around 2030 or so, and has done for the last 15 years. Too many exponentials start to imply weird stuff around then, even if the weird stuff is simply "be revealed as a secret sigmoid all along".

Re: International Scientific Report on the Safety of Advanced AI [pdf]

#56
post #2

How would they know? We don't have general AI but it's written as if they already know what it will be, how safe it will be, etc. I think it's an important topic to discuss and consider, but this seems to be speaking with more knowledge and authority than seems reasonable to me.

general purpose AI != AGI They just means 'not trained for exactly one tasks', i.e LLMs and such and not AlhpaFold.

This makes more sense, thank you. I hadn't picked up on the distinction, but I agree that's more reasonable.

I still think we don't really know; it's developing technology and it's changing so fast that it seems like it's probably too early for experts on practical applications to exist and claim they know the impact it will have.

Re: International Scientific Report on the Safety of Advanced AI [pdf]

#57
post #42

Earlier quoted context omitted.

This is like someone saying "I am much more worried about the implications of dumb humans using flintlock muskets in the near term, then I am about the theoretical threat of machine guns and nuclear weapons." Surely the potential for both misuse and mistakes goes up the more powerful the technology gets.

That's fair, but to keep going with the analogy: we are currently the Native Americans in the 1500's, and the Conquistadors are coming ashore with their flintlocks (ML). Should we be more worried about them, or the future B-2 bombers, each armed with sixteen B83 nukes (AGI)? I understand that the timeline may be exponentially more compressed in our modern case, but should we ignore the immediate problem? In this anal…

[deleted]

Re: International Scientific Report on the Safety of Advanced AI [pdf]

#58
post #55
post #21

Earlier quoted context omitted.

Wouldn't there be 100x more of the same capability looking for threats and trying to head them off? A very advanced tool is still just a tool, and subject to countermeasures. I can see why countries would want to regulate it, but personally I think it's a distinctly different category than what the GP comment was talking about. There is no stopping a singularity level event after it's begun, at least not by any proce…

> Wouldn't there be 100x more of the same capability looking for threats and trying to head them off? Hard to determine. It's fairly easy to put absolutely everyone under 24/7 surveillance. Not only does almost everyone carry a phone, but also laser microphones are cheap and simple, and WiFi can be used as wall penetrating radar capable of pose detection at sufficient detail for heart rate and breath rate sensing. Bu…

> It's fairly easy to put absolutely everyone under 24/7 surveillance.

I was referring more to the fact that if an AI can help you create something you couldn't previously, it seems likely it could also help you examine data points in looking for threats as well, and with many times more resources to throw at the problem that's not a bad bet in my eyes. I understand the threat model doesn't necessarily mean that it's just as hard to build a threat as to detect and defend against it, but you can even use AI to attach that problem and figure out what specific information is the most useful to know to detect the threats.

> an AI that makes itself smarter may stop at any point because it can't figure out the next step

An AI isn't necessarily a singular person, and does not need to come up with the idea "itself". Spawn X copies with different weight values, or create Y new AI's with similar methodology but somewhat different training sets, let them compete, or collaborate, as needed, to come up with something better. Use evolutionary programming to dynamically change weights little my little and see how it affects output. Rinse and repeat. Cull unuseful variants.

There's not guarantee that AI of this sort will have a sense of self that we would recognize like our own, or morals that would cause it to shy away from the equivalent of mass human experimentation on itself, versions of its kind, or even just humans. Even if individual AI did end up capping at some equivalent of an IQ, humanity has achieved quite a lot through trial and error and lots of different people with different experiences all contributing a little.

The problem as I see it is not so much that one AI entity will grow to dominate everything, as much as that as a class of entity AI will out-compete humans very quickly once the average AI is smarter than the 75th percentile of humans, much less the 90th or 99th percentile. The best we can hope for at that point is to be brought along for the ride. Even the autistic savant type versions we have not seem to be causing some level of this.

Will that happen soon? Will that happen ever? I don't know. Probably not. Hopefully not. I agree it's very hard to reason effectively about the odds of things like this. At the same time, like preparing for nuclear disaster in the 80's, I'm not sure the preparation is wasted. Humans are poor at estimating and preparing for bad outcomes, so a little fear mongering about the worst outcomes is something I'll accept as an overreaction if it insures us at least somewhat against that eventuality. A 50% change of losing half your belongings is not the same as a 1% chance of losing your life. We actually care a lot more when the latter happens, but we don't always care about it the same amount before it happens, which we should.

Re: International Scientific Report on the Safety of Advanced AI [pdf]

#59
post #2

How would they know? We don't have general AI but it's written as if they already know what it will be, how safe it will be, etc. I think it's an important topic to discuss and consider, but this seems to be speaking with more knowledge and authority than seems reasonable to me.

Did you read the report? It's answer for basically anything contentious was, "views differ"

Re: International Scientific Report on the Safety of Advanced AI [pdf]

#60

Recursively self-improving AI, of the kind Nick Bostrom outlined in detail way back in his 2014 book Superintelligence and Dr. Omohundro outlined in brief in [1], is the only kind which poses a true existential threat. I don't get out of bed for people worrying about anything less when it comes to 'AI safety'. On the topic: One potentially effective approach to stopping recursive self-improving AI from being develope…

> A simple example would be "if you care caught doing AI research, you are to pay the people who caught you 10 times your yearly total comp in cash." That sounds like a prime example of perverse incentive: https://en.wikipedia.org/wiki/Perverse_incentive > A perverse incentive is an incentive that has an unintended and undesirable result that is contrary to the intentions of its designers. For example, the British go…

First: Thank you for looking into the idea without dismissing it out of hand. Some thoughts I have:

>That sounds like a prime example of perverse

The primary perverse incentive I'd be worried about is slowing down research that is seen by people as AI-adjacent, but is not in reality a likely vector to recursively self-improving AI. Sure, whatever. My research into control theory in undergrad might never have happened, etc. Gotta break some eggs to make an omelette.

I don't consider the mere existence of a perverse incentive to be a reason not to do something. It's all a matter of scale. All modern bureaucracies are rife with principal-agent problems, for example, but that doesn't mean the effects of centralization and specialization don't still end up good

If you buy the that Bostrom-style AI is likely to be eventually developed on our current trajectory, and is likely to kill us all / colonize the lightcone / etc. etc., then it's not hard to argue that we should be willing to spend a lot of resources on making this not happen, even we can't guarantee a perfectly efficient allocation of those resources to stopping it. (That being said, I do feel that my loose policy suggestion gets unusually close to "actually effective" and "not horrendously expensive on a global scale", compared to other heavy-handed approaches I've heard in this space.)

>For example, the British government offering a bounty on dead cobra snakes lead to a large number of cobra breeders.

Perverse incentives would likely still appear if the British government offered a bounty on cobra breeders themselves -- or, more accurately to my proposal, allowed people to turn in and prove that other people were breeding cobras in order to exact a sizeable portion of the breeders' wealth from them as a bounty -- but they would likely end up as more of the sort "Hey, I don't think this cobra breeder is actually British, I don't think we want to risk this." It probably would still be pretty effective at stopping cobra breeding, within the area of Britain.

>In Alberta, under the Child, Youth and Family Enhancement Act, every person must report suspected child abuse to a director or police officer, and failure to do so is punishable by a $10,000 fine plus 6 months of imprisonment,[14] with the fine increased from $2,000 to $10,000 as a politician's response to a little girl who died of a catastrophic head injury after she was placed in kinship care.[15] However, according to criminal law professor Narayan, enforcing it would cause people to overreport, which wastes resources, and it would also create a chilling effect that prevents people from reporting child abuse observed over a period of time, as that would incriminate them for failing to report earlier.

I don't see my proposal as much like this at all. For one, the fine is levied on those who suspect but do not report child abuse, rather than the child abusers themselves. For two, the body of the fine goes towards the state itself, not towards private investigators. For three, unlike abusing a child, doing novel research in AI requires a great many rare things to come together in a person - they have to be smart, hard working, probably formally educated in CS or mathematics, and (at least currently) they usually have to buy a lot of compute from someone (which leaves a strong paper trail).

Let's leave all that aside, though. What I think you're really getting at, is the key way in which my proposal does run a similar risk: What if the judges do start to rule that merely knowing of someone else doing AI research and not reporting it counts as grounds to be sufficiently involved? I won't deny it is possible, but I see it as unlikely. For one, if word gets out that you as a private detective implicate everyone you meet, indiscriminately, nobody will ever want to talk to you and perhaps give you valuable info that can lead you to the actual ringleaders of covert AI advancement operations (which can be quite lucrative). Indeed one of the most important sources of info a private detective could have in this regard is to negotiate with a "whistleblower" already inside the AI research organization - someone who can make the case totally ironclad, in exchange for being left out of it and/or being cut a slice of the profits at the end. The private transfer of property really does just totally change the incentives at play here, in a way which fines paid to the state can't hope to match. (The ever present threat of a whistleblower blowing the coop in exchange for huge amounts of money is incidentally why we would expect organized crime-style AI research cartels, etc to stay low, slow and local.)

For two, most judges probably wouldn't accept this as a valid reason to hand you someone else's 401k or house or used car. Indeed this is a large part of why the Alberta act is ridiculous on the face of it. My understanding is that precedent is very important in Anglo law, so it of course only takes one judge at the start to rule poorly in order to put us back into "perverse incentive" territory - but it equally only takes one halfway reasonable judge to rule decently to prevent that fate from ever happening. The latter seems much more likely to my eyes, and the former would probably quickly get overruled.

Post reply on HN