Live data from Hacker News

We need a Butlerian Jihad against AI

erikhoel.substack.com

131–140 of 161 posts

Re: We need a Butlerian Jihad against AI

#131
Every treatment of this topic I see makes the same really fundamental mistake, rendering everything after it pointless.

Is there a danger? Yes. Is it because the AI might escape control? No. Is it too late? Yes.

In cheap SF, somebody always says, "we can always pull the plug." But we too often can't pull the plug. Consider the simplest robot we all obey without thought: the traffic light. What would it mean to pull its plug? What would have to happen for its plug to be pulled?

If the traffic light served no one's interest, somebody would pull it. But few things serve no one's interest. The worse a system is, the more integrated it is into someone's interest. Any AI we would like to unplug is almost certainly protected by law, and by numerous layers of physical protection. Lives may even have come to depend on it, daily.

Any AI will be designed to to serve somebody, and will most likely continue to appear to serve them. Just, not necessarily you. But maybe even you, too, much of the time.

We already have numerous AIs controlling myriad details of modern life. We call them corporations, institutions, agencies. Their parts are made of job descriptions and papers, offices and spreadsheets. They are very inefficient. But they are, nonetheless, terrifyingly effective. At times they consume people by the million, seemingly almost autonomously. Ask Uyghurs whose organs are about to be harvested how much it matters that, in principle, a person could turn off the machine that has ensnared them. Ask everyone hoping for a transplant how many would turn it off.

A strong AI would be, simply, an institution that is not so dependent on fallible organic components. It would be more efficient than what it replaces. But the institutions we have are entirely efficient enough to cause all the unpleasantness we can devise.

We are well into an era where the institutions we built to serve us, or anyway to serve somebody, do not. Who can have much effect on the choices of even such a minor institution as the New York Police Department? Who would be able to cancel the F-35 program? The Chinese Communist Party and Shell Oil have both outlived their creators, and are well equipped to eject any human components that do not serve their implicit charters.

Re: We need a Butlerian Jihad against AI

#132

Earlier quoted context omitted.

Well put. But isn't the concern here with some utilities becoming ... people? Either way could result in disfigured souls. Is AI simply a pursuit of slavery without guilt?

> Is AI simply a pursuit of slavery without guilt? Or simply a pursuit of labor without pay?

Always has been.

Re: We need a Butlerian Jihad against AI

#133
I think we should strive for the Iain Banks vision of the future. Treat the AIS with respect and ensure it has equal rights. I’d be happy with a benevolent AI controlling the government. But I imagine, given the current state of things, we will probably treat it unethically and make it do something like optimise advertisement or bomb random villages.

Re: We need a Butlerian Jihad against AI

#134
post #65

Humans won't become "slaves" to superhuman AI any more than human cells are "slaves" to the human body. Unexamined anthropocentric hogwash.

What if the superhuman AI thinks that humans are the cancer cells to its metaphorical body and decides to eradicate us?

Won’t have to worry then, we’ve not been too successful against cancer

Re: We need a Butlerian Jihad against AI

#135

Earlier quoted context omitted.

I can teach a human to drive in < 1 day. An AI… so far 10+ years. I agree there is crazy progress and investment, but it’s also not entirely certain how far we can push AI.

How long would it take you to teach a human to beat the world champion at Chess or Go?

I’m not saying AI isn’t impressive I’m saying we don’t know the limit… is your point that chess mastery is a sign of superintelligence and driving is not?

Re: We need a Butlerian Jihad against AI

#136

One of the problems the Butlerian Jihad ran up against, aside from the inevitable skirting of the lines from Richese and Ix (many machines on Ix, new machines), is that it runs directly counter to "Thou shalt not disfigure the soul." Replacement of AI with Mentats (as well as other narrow specialities) has done nothing but disfigure the soul. We see few Mentats -- aside from Paul and eventually another -- who are not…

Well put. But isn't the concern here with some utilities becoming ... people? Either way could result in disfigured souls. Is AI simply a pursuit of slavery without guilt?

Why do people have children?

A strong AI would, by most definitions, be the child of its creator(s).

Re: We need a Butlerian Jihad against AI

#137

Earlier quoted context omitted.

If there is literally no possible future with dignity for all consciousnesses... that'd be pretty depressing.

Ian M. Bank's The Culture universe is such a future, which I think is attainble in a few millennia. Powerful super-human AI minds will eventually and inevitably take over; which I think is fine provided they recognize all sentient beings as equals.

My reading of the Culture novels is that the Minds basically regard the non-Minds in their society as pets or curiosities. The society itself is basically that of the Minds, as no other participants really have the capacity to interface with them on that level.

Re: We need a Butlerian Jihad against AI

#139

Earlier quoted context omitted.

That's really not the 10% I was talking about. We don't need that part to follow some lanes. Or the other way to put it is that all the other code is the first 90%, and then the "remaining 10% that's 90% of the effort" is the object recognition that was supposedly easy.

>We don't need that part to follow some lanes. We do! I live near a highway and every time I use a particular on-ramp, the tar stripes make my car think it's going off the road. It never learns, and it has no plan B. It's easy to do the lane-keeping, it's completely terra incognita to decide when to abandon the standard method that works 90% of the time. And what to do next.

For object recognition code, including lane-finding, to be sufficient, it needs to be looking far enough ahead that the car always has a plan B. If we reach that level of quality, and it still fails to find the lane, then the car can simply stop.

Deciding when to stop is a really easy threshold based on how confused the recognizer is and how far away the source of confusion is.

But also those sources of confusion will get rarer and rarer as the object recognition improves, to the point that even very mediocre handling of those situations will still give you an overall good experience.

Re: We need a Butlerian Jihad against AI

#140
post #84

Earlier quoted context omitted.

"AI alarmists" are worried because the worst-case outcomes of AGI are mistakes you cannot ever fix. All the rest of these are bad, but they are problems we can fix given time and thought, because we will still exist. Extinction-level events decrease all future human utility to zero, and so should be treated with extraordinary care.

As you are talking about extinction level events, I'm not very confident that if humans have ultimate say over nuclear weapons, we will continue to not end our species with them. It might in fact be a good idea to establish an AGI overlord which watches over humans and enforces nuclear non proliferation policies. If you look through history, it's full with war, genocide, and similar. Human societies are bound for cha…

> It might in fact be a good idea to establish an AGI overlord which

Does whatever it damn well pleases.

Post reply on HN