Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Safety-first AI for autonomous data centre cooling and industrial control

deepmind.com

Post reply on HN
Print view View original

11–20 of 24 posts

Re: Safety-first AI for autonomous data centre cooling and industrial control

#11

Post by candiodari »

Chopsah2 wrote: post #10
candiodari wrote: post #6

This explanation ignores the risk that is posed by sudden absence of the AI. Given that it's a remote system this seems relevant. If, for whatever reason, the AI decides to call it quits, or the network in between disconnects, what happens ? I ask because I've implemented systems that allowed operators of large networks to do exponentially more work, and therefore grow the network while keeping operators down. It ess…

Hi, I'm one of the engineers working on this project at Google. If the AI disconnects or starts to make bad control decisions, the local control system (which has veto power) kicks it out and takes over. We lose some efficiency when this happens, but the cooling system stays safe and operates in a mode that the human operators understand completely.

That was true for this system as well. How long until management fires these people ? How long until nobody in the industry knows anymore ?

Re: Safety-first AI for autonomous data centre cooling and industrial control

#12

Post by Chopsah2 »

candiodari wrote: post #11
Chopsah2 wrote: post #10

Earlier quoted context omitted.

Hi, I'm one of the engineers working on this project at Google. If the AI disconnects or starts to make bad control decisions, the local control system (which has veto power) kicks it out and takes over. We lose some efficiency when this happens, but the cooling system stays safe and operates in a mode that the human operators understand completely.

That was true for this system as well. How long until management fires these people ? How long until nobody in the industry knows anymore ?

Here's an excerpt from https://www.datacenterknowledge.com/google-alphabet/google-s... where our VP Joe Kava addresses that question:

What About the Jobs?

With more and more of the company’s data centers shifting to automated infrastructure control, and with the real possibility that the same will eventually start happening outside of Google, arises the inevitable question of jobs. Are Google’s data center engineers engineering themselves and their colleagues out of work?

So far, Kava hasn’t seen evidence of that happening.

“We still have people there, because they still have to do all the maintenance,” he said. “So, you’re not getting rid of the people, you’re augmenting” the existing team’s capabilities. “Instead of trying to tune the system themselves, they can focus more of their time on preventative maintenance and corrective repairs.”

Besides, AI still does poorly in situations “outside of the envelope of its training,” he said. People are very good at making observations in what Kava likes to call “corner cases” and coming up with a course of action on the spot. AI isn’t.

In other words, it’s a good idea to have AI fine-tune a cooling system to improve efficiency in pre-tornado conditions, but you better have some human engineers around in case a tornado forms.

Re: Safety-first AI for autonomous data centre cooling and industrial control

#13

Post by mlthoughts2018 »

Chopsah2 wrote: post #12
candiodari wrote: post #11

Earlier quoted context omitted.

That was true for this system as well. How long until management fires these people ? How long until nobody in the industry knows anymore ?

Here's an excerpt from https://www.datacenterknowledge.com/google-alphabet/google-s... where our VP Joe Kava addresses that question: What About the Jobs? With more and more of the company’s data centers shifting to automated infrastructure control, and with the real possibility that the same will eventually start happening outside of Google, arises the inevitable question of jobs. Are Google’s data center engineers…

It’s so funny to me how no one working on technology that will wipe out classes of jobs ever self-reports evidence that it will wipe out classes of jobs.

I’m sure early automakers “saw no evidence” in jobs for carriage drivers too.

Re: Safety-first AI for autonomous data centre cooling and industrial control

#14

Post by friday99 »

Chopsah2 wrote: post #12
candiodari wrote: post #11

Earlier quoted context omitted.

That was true for this system as well. How long until management fires these people ? How long until nobody in the industry knows anymore ?

Here's an excerpt from https://www.datacenterknowledge.com/google-alphabet/google-s... where our VP Joe Kava addresses that question: What About the Jobs? With more and more of the company’s data centers shifting to automated infrastructure control, and with the real possibility that the same will eventually start happening outside of Google, arises the inevitable question of jobs. Are Google’s data center engineers…

I think the problem is in the idea that "you better have some human engineers around in case a tornado forms." That is a completely different job than the original job and the folks waiting around "in case a tornado forms" likely won't have the skills to fix that tornado anymore. The issue isn't that "the robots are going to take our jobs!" its that the new jobs are ones humans actually aren't very good at; waiting around ever vigilant until the automated system screws up and then immediately coming up to speed and fixing the system they no longer have any interaction with.

Re: Safety-first AI for autonomous data centre cooling and industrial control

#15

Post by wenc »

3pt14159 wrote: post #3

> It was amazing to see the AI learn to take advantage of winter conditions and produce colder than normal water, which reduces the energy required for cooling within the data centre. Sorry, what? It must be more complex than that. That's something a basic multi-linear optimizer could have accomplished.

I wonder the same.... AI is good for making use of unstructured information, but for structured numerical information from sensors, an advanced optimal control algorithm like MPC will likely perform as well or better.

Plus MPC isn't a black box, and so is amenable to characterization, interpretation and analysis. It also has robustness-enhancing features that have been developed over decades, so the fallbacks are known.

Re: Safety-first AI for autonomous data centre cooling and industrial control

#17

Post by Chopsah2 »

cbhl wrote: post #16

I'd be curious to see what processes are in place for testing the failover of these control systems. (For example, do the teams failover to the local cooling system during DiRT?)

The AI control is architected such that failover isn't really necessary; the local control is always in control, it just gets suggestions (which it can safely ignore) from the AI. If the AI disappears or starts sending bad suggestions, the AI gets kicked out.

We're trying to continually improve the AI so that its time in control is maximized, much like Waymo's early days with self-driving car software + safety drivers.

Re: Safety-first AI for autonomous data centre cooling and industrial control

#18

Post by tomkat0789 »

I did my PhD dissertation on a similar topic to this. The company I worked with despised neural net methods because it wasn't robust to new operating regimes that cropped up frequently and it wasn't explainable enough for anybody to trust it.

That said, it'd be cool if Google opened the data set for everybody. The process monitoring community would be excited!

Re: Safety-first AI for autonomous data centre cooling and industrial control

#19

Post by tree_of_item »

zzzcpan wrote: post #8
candiodari wrote: post #6

This explanation ignores the risk that is posed by sudden absence of the AI. Given that it's a remote system this seems relevant. If, for whatever reason, the AI decides to call it quits, or the network in between disconnects, what happens ? I ask because I've implemented systems that allowed operators of large networks to do exponentially more work, and therefore grow the network while keeping operators down. It ess…

> it's been working so well I wonder if they'd even notice before it's too late I guess the solution is to simulate faults everywhere at all times in production. Something most systems definitely ignore. Fault tolerance is just too different from "normal" way of writing software, it can't be done well without paradigm shift.

You were downvoted but I think you're right. Netflix's Chaos Monkey is a similar idea: https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey

Re: Safety-first AI for autonomous data centre cooling and industrial control

#20

Post by Ajedi32 »

candiodari wrote: post #6

This explanation ignores the risk that is posed by sudden absence of the AI. Given that it's a remote system this seems relevant. If, for whatever reason, the AI decides to call it quits, or the network in between disconnects, what happens ? I ask because I've implemented systems that allowed operators of large networks to do exponentially more work, and therefore grow the network while keeping operators down. It ess…

The blog post already explains this.

> Automatic Failover to a neutral state if the AI control system does violate safety constraints. Smooth Transfer during failovers to prevent sudden changes to the system. Rules and heuristics as backup if we need to exit AI control mode.

Post reply on HN