Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

International Scientific Report on the Safety of Advanced AI [pdf]

assets.publishing.service.gov.uk

Post reply on HN
Print view View original

41–50 of 66 posts

Re: International Scientific Report on the Safety of Advanced AI [pdf]

#41

Post by swores »

readyman wrote: post #14
jonas21 wrote: post #11

Earlier quoted context omitted.

And if they say "yes, it is legit", what does that tell you?

That the author has risked their reputation on the claim. If you're doubting the author is legit, interrogate their professional associations with an internet search, relying on the domain name system. Nothing about any of this is new or profound. Counterfeit documents have been around for hundreds of years.

The question you replied to wasn't "why should you believe someone who says they are behind a piece of research", it was about the usefulness of receiving an email saying it.

Their point (I assume) was that it would be illogical to worry that the report might be written and released by AI yet consider an email response as evidence against it.

If AI can create and release this report it can also hijack a real person's email or create a fake persona that pretends to be a real person.

Re: International Scientific Report on the Safety of Advanced AI [pdf]

#42

Post by tkwa »

consumer451 wrote: post #22

I understand that this report is not really about AGI, but I would like to again raise my main concern: I am much more worried about the implications of the real threat of dumb humans using dumb "AI" in the near-term, than I am about the theoretical threat of AGI. Example: https://news.ycombinator.com/item?id=39944826 https://news.ycombinator.com/item?id=39918245

This is like someone saying "I am much more worried about the implications of dumb humans using flintlock muskets in the near term, then I am about the theoretical threat of machine guns and nuclear weapons." Surely the potential for both misuse and mistakes goes up the more powerful the technology gets.

Re: International Scientific Report on the Safety of Advanced AI [pdf]

#43

Post by johndough »

hiAndrewQuinn wrote: post #16

Recursively self-improving AI, of the kind Nick Bostrom outlined in detail way back in his 2014 book Superintelligence and Dr. Omohundro outlined in brief in [1], is the only kind which poses a true existential threat. I don't get out of bed for people worrying about anything less when it comes to 'AI safety'. On the topic: One potentially effective approach to stopping recursive self-improving AI from being develope…

> A simple example would be "if you care caught doing AI research, you are to pay the people who caught you 10 times your yearly total comp in cash."

That sounds like a prime example of perverse incentive: https://en.wikipedia.org/wiki/Perverse_incentive

> A perverse incentive is an incentive that has an unintended and undesirable result that is contrary to the intentions of its designers.

For example, the British government offering a bounty on dead cobra snakes lead to a large number of cobra breeders.

Your proposal is conceptionally similar to the Alberta Child, Youth and Family Enhancement Act (fourth bullet point): https://en.wikipedia.org/wiki/Perverse_incentive#Community_s...

Re: International Scientific Report on the Safety of Advanced AI [pdf]

#44

Post by tkwa »

joebob42 wrote: post #2

How would they know? We don't have general AI but it's written as if they already know what it will be, how safe it will be, etc. I think it's an important topic to discuss and consider, but this seems to be speaking with more knowledge and authority than seems reasonable to me.

It seems fine to me. When there is evidence for a certain type of current or future harm they present it, and when there is not they express uncertainty.

Can AI enable phishing? "Research has found that between January to February 2023, there was a 135% increase in ‘novel social engineering attacks’ in a sample of email accounts (343*), which is thought to correspond to the widespread adoption of ChatGPT."

Can AIs make bioweapons? "General-purpose AI systems for biological uses do not present a clear current threat, and future threats are hard to assess and rule out."

Re: International Scientific Report on the Safety of Advanced AI [pdf]

#45

Post by dragonwriter »

hiAndrewQuinn wrote: post #16

Recursively self-improving AI, of the kind Nick Bostrom outlined in detail way back in his 2014 book Superintelligence and Dr. Omohundro outlined in brief in [1], is the only kind which poses a true existential threat. I don't get out of bed for people worrying about anything less when it comes to 'AI safety'. On the topic: One potentially effective approach to stopping recursive self-improving AI from being develope…

> Such a program would incur minimal policing costs

No, it wouldn’t. Just because the policing costs aren't tax-funded doesn't mean they don't exist. (And I'm not talking just about costs voluntarily incurred by bounty seekers, I'm also talking about the cost that the system imposes involuntarily on others who are neither actually guilty nor bounty seekers, because the financial incentives can motivate pursuits imposing costs on targets who are not factually guilty.)

> and could easily scale to handle international threats.

No, it couldn’t, especially for things where a major source of the international threat is governmental, where private bounties aren’t going to work at all.

And it especially can’t work against developing technology, where the evidence needed in litigation and the litigation would itself be a vector for spreading the knowledge that you are attempting to repress.

Re: International Scientific Report on the Safety of Advanced AI [pdf]

#46

Post by ninetyninenine »

123yawaworht456 wrote: post #10
ninetyninenine wrote: post #3

The biggest risk of ai can be simply characterized by one concept: This very report could have been generated by AI. We can’t fully tell.

it could also be written by a communist/terrorist/nazi/russian in Notepad on Windows XP. you can't fully tell.

We can’t. The point is 2 years ago between ai and a human, we 100 percent could tell.

Now we can’t. In the future it will be even harder.

Re: International Scientific Report on the Safety of Advanced AI [pdf]

#47

Post by consumer451 »

tkwa wrote: post #42
consumer451 wrote: post #22

I understand that this report is not really about AGI, but I would like to again raise my main concern: I am much more worried about the implications of the real threat of dumb humans using dumb "AI" in the near-term, than I am about the theoretical threat of AGI. Example: https://news.ycombinator.com/item?id=39944826 https://news.ycombinator.com/item?id=39918245

This is like someone saying "I am much more worried about the implications of dumb humans using flintlock muskets in the near term, then I am about the theoretical threat of machine guns and nuclear weapons." Surely the potential for both misuse and mistakes goes up the more powerful the technology gets.

That's fair, but to keep going with the analogy: we are currently the Native Americans in the 1500's, and the Conquistadors are coming ashore with their flintlocks (ML). Should we be more worried about them, or the future B-2 bombers, each armed with sixteen B83 nukes (AGI)?

I understand that the timeline may be exponentially more compressed in our modern case, but should we ignore the immediate problem?

In this analogy, the flintlocks could be actual ML-powered murder bots, or just ML-powered economic kill bots, both fully controlled by humans.

The flintlocks enable the already powerful to further consolidate their power, to the great detriment of the less powerful. No super AGI is necessary, it just takes a large handful of human Conquistador sociopaths with >1,000x "productivity" gains, to erase our culture.

I don't understand how we could ever get to the point of handling the future B-2 nuke problem, as a civilization, without first figuring out how to properly share the benefits of the flintlock.

Re: International Scientific Report on the Safety of Advanced AI [pdf]

#48

Post by ben_w »

mikpanko wrote: post #6
joebob42 wrote: post #2

How would they know? We don't have general AI but it's written as if they already know what it will be, how safe it will be, etc. I think it's an important topic to discuss and consider, but this seems to be speaking with more knowledge and authority than seems reasonable to me.

I believe by “general-purpose AI” the report doesn’t mean AGI.

Which is one of those cases where I briefly want to reject linguistic descriptivism because to me the "G" in "AGI" is precisely "general".

But then I laugh at myself, because words shift and you have to roll with the changes.

But do be aware that this shift of meanings is not universally acknowledged let alone accepted — there's at least half a dozen different meanings to the term "AGI", except one of them requires "consciousness" and there's loads of different meanings of that too.

Re: International Scientific Report on the Safety of Advanced AI [pdf]

#49

Post by ben_w »

api wrote: post #24
joebob42 wrote: post #2

How would they know? We don't have general AI but it's written as if they already know what it will be, how safe it will be, etc. I think it's an important topic to discuss and consider, but this seems to be speaking with more knowledge and authority than seems reasonable to me.

You’ve hit on a giant thing that bothers me about this discourse: endless rationalistic discourse about systems and phenomena that we have no experience with at all, not even analogous experience. This is not like the atomic bomb. We had tons of experience with big bombs. We just knew atom bombs if they worked could make orders of magnitude larger booms. The implications if real big bombs could be reasoned about with…

> This is not like the atomic bomb. We had tons of experience with big bombs. We just knew atom bombs if they worked could make orders of magnitude larger booms. The implications if real big bombs could be reasoned about with some basis in reality.

Well, we thought we did.

We really didn't fully appreciate the impact of the fallout until we saw it; and Castle Bravo was much bigger than expected because we didn't know what we were doing; and the demon core; and the cold war arms race…

But yeah, my mental framing for this is a rerun of the first stage of the industrial revolution, and it took quite a lot of harm for what is now basic workplace health and safety such as "don't use children to remove things from heavy machinery while it's running", and we're likely to have something that's equally dumb happen even in the relatively good possible futures that don't have paperclip maximisers or malicious humans using AI for evil.

Re: International Scientific Report on the Safety of Advanced AI [pdf]

#50

Post by ben_w »

readyman wrote: post #14
jonas21 wrote: post #11

Earlier quoted context omitted.

And if they say "yes, it is legit", what does that tell you?

That the author has risked their reputation on the claim. If you're doubting the author is legit, interrogate their professional associations with an internet search, relying on the domain name system. Nothing about any of this is new or profound. Counterfeit documents have been around for hundreds of years.

Three people make a tiger[0], and even current LLMs are good at pretending to be a crowd.

[0] https://en.wikipedia.org/wiki/Three_men_make_a_tiger

Post reply on HN