John Carmack's new AGI company, Keen Technologies, has raised a $20M round

https://www.deepmind.com/blog/specification-gaming-the-flip-...

https://vkrakovna.wordpress.com/2018/04/02/specification-gam...

https://arxiv.org/abs/1803.03453 (The Surprising Creativity of Digital Evolution)

We literally don't know how to stop effective optimizing processes, deployed in non-handcrafted environments, from discovering solutions and workarounds that satisfy the letter of our instructions but not the spirit. Even for "dumb" systems, we have to rely on noticing, then post-hoc disincentivizing, unwanted behaviors, because we don't know how to robustly specify objectives.

When you train a system to, for example, "stop saying racist stuff", without actually understanding what you're doing, all you get is a system that "stops saying racist stuff" when measured by the specific standard you've deployed.

Ask any security professional how seriously people take securing a system, let alone how ineffective they are at it. Now consider the same situation but worse because almost no one takes AI safety seriously.

If you nod solemnly at the words "safety" and "reliability" but don't think anything "really bad" can happen, you will be satisfied with a solution that "works on your machine". If you aren't deeply motivated to build a safe system from the start because you can always correct things late, you are not building a safe system.

It will be possible to produce economically viable autonomous agents without robustly specified objectives.

But hey, surely a smart enough system won't even need a robustly specified objective because it knows what we mean and will always want it just as much.

Surely dangerous behavior like "empowerment" isn't an instrumental goal that effective systems will simply rediscover.

Surely the economic incentives of automation won't encourage just training bad behavior out of sight and out of mind.

Surely in the face of overwhelming profit, corporations won't ignore warning signs of recurring dangerous behavior.

Surely the only people capable of building an AGI will always be those guaranteed to prioritize alignment rather than the appearance of it.

Surely you and every single person will be taking AI safety seriously, constantly watching out for strange behavior.

Surely pulling the plug is easy, whether AI runs on millions of unsecured unmonitored devices or across hundreds of money printing server farms.

Surely an AGI can only be dangerous if it explicitly decides to fool humans rather than earnestly pursuing underspecified objectives.

Surely it's easy to build an epistemically humble AGI because epistemic humility is natural or easy enough to specify.

Surely humanity can afford to delay figuring out how best to safely handle something so potentially impactful, because we always handle these things in time.

I now regret spending half an hour writing a response to your earlier comment.

You can't tell why a mass deployment of motivated autonomous "NSO-9000s" might be more dangerous than a virus that just changes its signature to fool a executable scanner? I don't believe you.

If you honestly believe that an "intelligent" (as adaptive as a slime mold) virus is basically as dangerous as a maliciously deployed AGI then there is literally nothing an AGI could do that would make you consider safety is important during the endeavor of building one.

The problem is that we can't know if the AGI is behaving maliciously or not far into the future, because it has potantial to be far more intelligent than us.

As for coercion, let's think of it as a manipulation. If a super intelligent agent has malicious goals, it can be very manipulative and be subtle in the process as to not spook humans. Companionship of intelligence with evilness is scary.

Of course we can (if we won't have distributed machines running AGI, because things like this may complicate things) "unplug" it. But the real problem arises from the fact that us humans can be easily manipulated. That's how I look at the problem.

I really enjoyed John Carmack’s and Lex Firdman’s 5 hour talk/interview.

Anyway, I like to efforts for human values preserving AGI. But, it will be a long time before we see it. I am 71 and I hope that I live to see it, outlying out of intellectual curiosity.

Heh. My "life's work": https://ardour.org ... My "previous life's work": http://amazon.com/

Being an inconsequential gimmick isn't about implementation, it's about semantics, relevance and utility. Only the last of these is really impacted at all by engineering, and even then, the engineering component doesn't matter if the thing just isn't useful.