31–40 of 158 posts
"When you start reading about AI safety, it’s striking how there are two separate communities—the one mostly worried about machine learning perpetuating racial and gender biases, and the one mostly worried about superhuman AI turning the planet into goo" - great quote.
Agreed. Maybe... Short sighted vs. far sighted vs. ignorance is bliss?
So, physics is a dead-end? Given that Scott is running his own research lab, a year is a very long time and him working out of his field is an indication that physics is in a big trouble.
I am confused, how did you reach that conclusion? How does this announcement relate to the future of physics research? Sure, Scott's research is at the intersection of complexity and physics, but he is a CS professor at Texas working in the theoretical computer science department. His work leans far more towards TCS, with some work having connections to cosmology (he cares about physical limits of the universe and th…
https://scottaaronson.blog/?p=6457
I also had the following exchange at my birthday dinner:
Physicist: So I don’t get this, Scott. Are you a physicist who studied computer science, or a computer scientist who studied physics?
Me: I’m a computer scientist who studied computer science.
Physicist: But then you…
Me: Yeah, at some point I learned what a boson was, in order to invent BosonSampling.
Physicist: And your courses in physics…
Me: They ended at thermodynamics. I couldn’t handle PDEs.
Physicist: What are the units of h-bar?
Me: Uhh, well, it’s a conversion factor between energy and time. (*)
Physicist: Good. What’s the radius of the hydrogen atom?
Me: Uhh … not sure … maybe something like 10-15 meters?
Physicist: OK fine, he’s not one of us.
Earlier quoted context omitted.
Could you please explain your reasoning for the comparison to perpetual motion machines? (Keep in mind that biological machines, ie life, have managed to turn the surface of the planet into 'green goo'.)
Self-reproducing machines are capable of covering the surface of the planet, yes. There's one right now (covid). But there's lots of energy and oxygen up here and they rarely displace other such machines (species) or even displace much of any earth and water. And because they're self-contained and self-reproducing, all of their instructions can be lost over time to entropy including the ones we're afraid of. None of…
All that oxygen comes from all the plants.
Yes, life has so far only covered the top of the planet. You are right that a paper clip maximizer would need quite a bit of time to go deeper than life has gone (if it would get there at all).
> And a paperclip factory isn't self-reproducing [...]
Why wouldn't it? If your hypothetical superhuman AGI determined that becoming self-reproducing would be the right thing to do, presumably it would do that.
No perfection required for that. Biological machines aren't perfect either. Just good enough.
You are right that thermodynamics puts a limit on how fast anything can transform the planet into paperclips or grey goo.
Though the limit is probably mostly about waste heat, not necessarily about available energy:
There's enough hydrogen around that an AGI that figured out nuclear fusion would have all the energy it needs. But on a planet wide basis, there's no way to dissipate waste heat faster than via radiation into space.
(Assuming currently known physics, but allowing for advances in technology and engineering.)
---
Of course, when we worry about paperclip maximisers, it's bad enough when they turn the whole biosphere into paperclips. Noticing that they'll have a hard time turning the rest of the earth into paperclips would be scant consolation for humanity.
(But the thermodynamic limits on waste heat still apply even when just turning the biosphere into paperclips.)
Earlier quoted context omitted.
Could you please explain your reasoning for the comparison to perpetual motion machines? (Keep in mind that biological machines, ie life, have managed to turn the surface of the planet into 'green goo'.)
Self-reproducing machines are capable of covering the surface of the planet, yes. There's one right now (covid). But there's lots of energy and oxygen up here and they rarely displace other such machines (species) or even displace much of any earth and water. And because they're self-contained and self-reproducing, all of their instructions can be lost over time to entropy including the ones we're afraid of. None of…
Earlier quoted context omitted.
Could you please explain your reasoning for the comparison to perpetual motion machines? (Keep in mind that biological machines, ie life, have managed to turn the surface of the planet into 'green goo'.)
Self-reproducing machines are capable of covering the surface of the planet, yes. There's one right now (covid). But there's lots of energy and oxygen up here and they rarely displace other such machines (species) or even displace much of any earth and water. And because they're self-contained and self-reproducing, all of their instructions can be lost over time to entropy including the ones we're afraid of. None of…
Earlier quoted context omitted.
Symmetrically someone like him transitioning from quantum computing should imply something negative about the state of quantum computing.
He mostly studies computational complexity. Quantum computing is a part of that, but there's other subfields. Though the kind of AI safety described in this post seems more like an extremely fancy version of program verification, so out of CS bloggers you'd expect John Regehr to get into it.
It kind of is. The field of AI safety is actually much more advanced than most people realise, with actual, real techniques to e.g. make sure neural networks are aligned with certain goals even under fluctuating parameters. Granted, we're still far from soothing an AGI before it can do something bad, but the tools we have today are already pushing in that direction (assuming neural networks are the right way to AGI of course).
"When you start reading about AI safety, it’s striking how there are two separate communities—the one mostly worried about machine learning perpetuating racial and gender biases, and the one mostly worried about superhuman AI turning the planet into goo" - great quote.
AI is not going to become self aware and destroy the world. AI is going to cause something like the industrial revolution of the 19th century: massive changes in who is rich, massive changes in the labor market, massive changes in how people make war, etc. It’s already started really. What worries me most is that as long as society is capitalist, AI will be used to optimize for self-enrichment, likely causing an even…
Earlier quoted context omitted.
He mostly studies computational complexity. Quantum computing is a part of that, but there's other subfields. Though the kind of AI safety described in this post seems more like an extremely fancy version of program verification, so out of CS bloggers you'd expect John Regehr to get into it.
>the kind of AI safety described in this post seems more like an extremely fancy version of program verification It kind of is. The field of AI safety is actually much more advanced than most people realise, with actual, real techniques to e.g. make sure neural networks are aligned with certain goals even under fluctuating parameters. Granted, we're still far from soothing an AGI before it can do something bad, but t…
Earlier quoted context omitted.
Self-reproducing machines are capable of covering the surface of the planet, yes. There's one right now (covid). But there's lots of energy and oxygen up here and they rarely displace other such machines (species) or even displace much of any earth and water. And because they're self-contained and self-reproducing, all of their instructions can be lost over time to entropy including the ones we're afraid of. None of…
I think you are conflating different concepts. It is clearly imaginable that a very intelligent agent could end humanity if its objective would require so. How exactly is "perpetual motion machines can't exist" related to this? How is it going to prevent an agent from engineering 1000 pandemic viruses at the same time?
This is quite possible. Indeed, I don't believe this is exclusive to superintelligence or requires it at all. Compare to the closest thing we have to "inventing AGI" - having babies. People do that all the time and there isn't a mathematical guarantee that baby won't end humanity, but we don't do much to stop it, and that's not considered a problem. Mainly, why would it want to?
https://twitter.com/thejadedguy/status/844352570470645760?la...
I don't think superintelligence even gives them much advantage if they wanted to. Being able to imagine a virus real good doesn't actually have much to do with the ability to create one, since plans tend to fail for surprising reasons in the real world once you start trying to follow them. Unless you define superintelligence as "it's right about everything all the time", but that seems like a magical power, not something we can invent.
> How exactly is "perpetual motion machines can't exist" related to this?
It wouldn't be able to do the particular kind of ending humanity where you turn them all into paperclips, though it could do other things. There's plenty of ways to do it that reduce entropy rather than increase it - nuclear winter is one.