Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Software Revolution

blog.samaltman.com

Post reply on HN
Print view View original

271–280 of 397 posts

Re: The Software Revolution

#271

Post by Tenobrus »

aaronem wrote: post #86
mrec wrote: post #55

Earlier quoted context omitted.

Sidetrack, but I'm curious. What do you mean by "a postulated mechanism by which it could actually happen that isn't pure SF"? It reads as if you're asking for a mechanism for a future technological event explained purely in terms of present technology. The whole "existential risk" concerns aren't about what present systems can do, they're about what future self-improving system might do. If we postulate this hypothe…

Not OP, but I agree with the point you question, and my rationale for so doing is that I've yet to see a compelling argument that a self-improving system is other than the software version of a perpetual-motion machine. Those seemed plausible enough, too, when thermodynamics was as ill-understood as information dynamics is now.

So far as I'm aware, most proponents of recursively self improving AIs don't necessarily think they can improve without upper limit (as in perpetual motion). They just think they can improve massively and quickly. Nuclear power lasts a hell of a long time and releases a hell of a lot of energy very fast (see: stars) but that's not perpetual motion/infinite energy either. And prior to those theories being developed it would seem inconceivable for so much energy to be packed into such a small space. But it was. Could be for AI too.

Not saying the parallel actually carries any meaning, just pointing out that you can make multiple analogies to physics and they don't really tell you anything one way or the other.

Re: The Software Revolution

#272

Post by eli_gottlieb »

rayiner wrote: post #260
yellowapple wrote: post #245

Earlier quoted context omitted.

> software can do 90% of what those folks do, but shit the bed when faced with the other 10% So let software handle the 90% and refactor the current jobs to handle the other 10%. > Software generally isn't reliable, predictable, or robust in the face of unusual circumstances Not yet , at least. I agree with you, though, and they're the same reasons why I'm personally paranoid about self-driving cars. Yeah, the occasi…

> So let software handle the 90% and refactor the current jobs to handle the other 10%. If the jobs could be so refactored in a cost-efficient way, they would be.

Two economists are walking down the street. One spots a $100 bill on the ground.

"Hey," he says to his friend, "There's a hundred bucks lying on the ground!"

"Don't be silly," the other replies, "If there were a hundred dollars on the ground, someone would have picked it up already!"

The two economists keep walking down the street.

Re: The Software Revolution

#273

Post by krapp »

JoeAltmaier wrote: post #259
krapp wrote: post #255

Earlier quoted context omitted.

There's nothing about automation that guarantees any solution for the people automated out of a job. The motive behind automation is purely capitalist - it exists not to free people from the burdens of menial labor, but to multiply the value of labor while freeing companies from the moral and financial burden of a human workforce. For most people, jobs - and the availability of jobs - are what allows them to buy food…

Most societies on earth now protect the out-of-work pretty well. There's been steady improvement in standard of living, lifespan, health in most countries for most of a century, to the point where the planet is in pretty good shape. In fact its a puzzle to me why, with this going on, we see an upsurge in terrorism etc. Why aren't people content? What is it that convinces folks to piss away their entire lives on a big…

That's a good point, but in most societies, most people aren't out of work. The safety net depends on people paying in to the system, which depends on people having something to pay.

Re: The Software Revolution

#274

Post by Micaiah_Chang »

tptacek wrote: post #262
Micaiah_Chang wrote: post #258

Earlier quoted context omitted.

Smart is just a shorthand for a complicated series of lower level actions consisting of domain knowledge, raw computational speed and other things yes. I don't think we're really disagreeing about this. However, I do worry that you're confusing the existing constraints on the human brain (that people seem to have tradeoffs between, let's say, charisma and mathematical ability) and constraints that would apply to all…

I sort of am questioning that factor, yes. Stipulate, I don't know, several orders of magnitude of intellectual superiority. Stipulate that human intelligence can be captured in silico , so that when we talk about "intelligence", we are using the most charitable possible definition for the "fear AI" argument. What precise vectors would be available for this system to manipulate the world to harm humans?

A typical example, which I don't really like, is that once it gains some insight into biology that we don't have (a much faster way of figuring out how protein folding works). It can mail a letter to some lab, instructing a lab tech to make some mixture which would create either a deadly virus or a bootstrapping nanomachine factory.

Another one is that perhaps the internet of things is in place by the time such an AI would be possible, at which point it exploits the horrendous lack of security on all such devices to wreck havoc / become stealth miniature factories which make more devistating things.

I mean, there's also the standard "launch ALL the missiles" answer, but I don't know enough about the cybersecurity of missiles. A more indirect way would be to persuade the world leaders to launch them, e.g. show both Russia and American radars that the other one is launching a pre-emptive strike and knock out other forms of communication.

I don't like thinking about this, because people say this is "sci-fi speculation".

Re: The Software Revolution

#275

Post by yellowapple »

HeyLaughingBoy wrote: post #170
enraged_camel wrote: post #103

Earlier quoted context omitted.

>>While I doubt this would happen to me as a software engineer, I would certainly rather work and have my dignity than sit on my ass, collect basic income, and feel worthless. Actually, you say that because you are a software engineer: it's quite a dignified job. Digging ditches out in the desert is not.

I assure you this is not a troll, I'm trying to figure out your thought process. How is one dignified and the other not?

Because the latter is pointless when there already exist tools better suited to that task?

The efforts of those road workers are effectively meaningless. Worthless. Moot. Their efforts would be better expended elsewhere - service jobs, creative occupations, the like - yet here they are, digging ditches for subsistence wages (at best) instead of doing something properly meaningful with their time on Earth and their energy.

The idea in places like India that manual labor takes precedent is a symptom - I'd argue - of exposure to the lingering remnants of a very long-lived caste system; instead of pushing these laborers into more useful fields, there's a preference to relegate them to menial, worthless work under the guise of "public service" or "charity" in order to reinforce that system, whether deliberately or perhaps unconsciously. I can assure you that, of the many reasons to emphasize such manual labor, "dignity" of the laborers is not one of them in this case.

Re: The Software Revolution

#276

Post by eli_gottlieb »

dnautics wrote: post #82
minthd wrote: post #69

Earlier quoted context omitted.

Yes biology is complicated. But we are starting to understand it reasonably well,and more importantly ,we're starting to design it even without understanding. For example, there's a company that produces microbes that create certain materials(genomatica), and uses evolution(with the goal being yield per bacteria) to evolve much more efficient strains - even without deeply understanding how the cell work.And i believe…

I'm just going to say what should be obvious: One is not going to be able to engineer a dangerous plague that will wipe out the human race in one shot. One would have to do experiments, and those experiments will be noticed, because people will get sick and die. I did get lucky and design, from first principles, a 4-fold increase in enzyme activity once, but I am not sure that is something I could repeat.

> One is not going to be able to engineer a dangerous plague that will wipe out the human race in one shot. One would have to do experiments, and those experiments will be noticed, because people will get sick and die.

We should still institute safety protocols suitable for a really-bad-case scenario.

Re: The Software Revolution

#277

Post by tptacek »

JoeAltmaier wrote: post #269
tptacek wrote: post #262

Earlier quoted context omitted.

I sort of am questioning that factor, yes. Stipulate, I don't know, several orders of magnitude of intellectual superiority. Stipulate that human intelligence can be captured in silico , so that when we talk about "intelligence", we are using the most charitable possible definition for the "fear AI" argument. What precise vectors would be available for this system to manipulate the world to harm humans?

I guess the same vectors available to a blogger, or pundit, or politician? To fear-monger; to mislead important decision makers; to spread lies and manipulate the outcomes of important processes. Is it possible to say that such AIs are NOT at work right now, fomenting terrorism, gathering money through clever investment and spending it on potent schemes to upset economies and governments?

The trouble with "persuasion" as the vector of harm from AI is that some of the dumbest people in the world are capable of getting thousands or (in the case of ISIS) millions of people to do their bidding. What contains persuasion as a threat isn't the intellectual limitations of the persuaders: it's the fact that persuasion is a software program that must run at the speed of transmitted human thought.

Re: The Software Revolution

#278

Post by JoeAltmaier »

krapp wrote: post #273
JoeAltmaier wrote: post #259

Earlier quoted context omitted.

Most societies on earth now protect the out-of-work pretty well. There's been steady improvement in standard of living, lifespan, health in most countries for most of a century, to the point where the planet is in pretty good shape. In fact its a puzzle to me why, with this going on, we see an upsurge in terrorism etc. Why aren't people content? What is it that convinces folks to piss away their entire lives on a big…

That's a good point, but in most societies, most people aren't out of work. The safety net depends on people paying in to the system, which depends on people having something to pay.

I think of it differently. As an engineer I'd use a control volume - draw a circle around the economy. Label inputs and outputs. E.g. mining, sunlight to produce food and energy, available land and water. The economy thrives if those things have a positive balance. The money is just a strange way of scorekeeping - imaginary points the people use to regulate their selfishness.

For instance the idea of a Basic Income is proposed once the economy has enough to feed and house everyone insensitive to the exact employment rate.

Re: The Software Revolution

#279

Post by mkempe »

Economic progress in a free market involves technological progress, the accumulation of capital, and the increased productivity of labor thanks to the two previous elements. That's not something to fear. There is no limit to human ingenuity. Labor is not a fixed pie that technology and capital shrink over time. And yes some labor-intensive activities have been and will continue to be replaced by capital-intensive systems -- which is wonderful because it means that it frees people to make ever-more of their time and powers of reason. The demand for labor (hence the wages for labor) increases with the accumulation of capital -- not the opposite. There is a line of economic thinkers who have elaborated on a pro-Capitalist view of economics, from Adam Smith to George Reisman via Jean-Baptiste Say, David Ricardo, Carl Menger, and Ludwig von Mises. It may help to read them.

Re: The Software Revolution

#280

Post by Micaiah_Chang »

tptacek wrote: post #267
Micaiah_Chang wrote: post #265

Earlier quoted context omitted.

Er, sorry for giving the impression that it'd be a supervillain. My intention was to indicate that it'd be a weird intelligence, and that by default weird intelligences don't do what humans want. There are some other examples which I could have given to clarify (e.g. telling it to "make everyone happy" could just result in it giving everyone heroine forever, telling it to preserve people's smiles could result in it f…

So like it's clear to me why you wouldn't want to take a system based on AI-like technology and have it control air traffic or missile response. But it doesn't take a deep appreciation for the dangers of artificial intelligence to see that. You can just understand the concept of a software bug to know why you want humans in the observe/decide/act loop of critical systems. So there must be more to it than that, right?…

The "more to it" is "if the AI is much faster at thinking than humans, then even humans in the observe/decide/act are not secure". AI systems having bugs also imply that protections placed on AI systems would also have bugs.

The fear is that maybe there's no such thing as a "superintelligence proof" system, when the human component is no longer secure.

Note that I don't completely buy into the threat of superintelligence either, but on a different issue. I do believe that it is a problem worthy of consideration, but I think recursive self-improvement is more likely to be on manageable time scales, or at least on time scales slow enough that we can begin substantially ramping up worries about it before it's likely.

Edit: Ah! I see your point about circularity now.

Most of the vectors of attack I've been naming are the more obvious ones. But the fear is that, for a superintelligent being perhaps anything is a vector. Perhaps it can manufacture nanobots independent of a biolab (do we somehow have universal surveillance of every possible place that has proteins?), perhaps it uses mundane household tools to macguyver up a robot army (do we ban all household tools?). Yes, in some sense it's an argument from ignorance, but I find it implausible that every attack vector has been covered.

Also, there are two separate points I want to make, first of all, there's going to be a difference between 'secure enough to defend against human attacks' and 'secure enough to defend against superintelligent attacks'. You are right in that the former is important, but it's not so clear to me that the latter is achievable, or that it wouldn't be cheaper to investigate AI safety rather than upgrade everything from human secure to super AI secure.

Post reply on HN