Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Happened in the UK Blackouts?

mitchoneill.com

Post reply on HN
Print view View original

111–120 of 175 posts

Re: What Happened in the UK Blackouts?

#111

Post by Zenst »

HenryBemis wrote: post #66
londons_explore wrote: post #16

It's notable that this report is completely consistent so far with the way I would design a cyber attack to take out a countries power grid. > The causes provided by RWE for the initiation of the trip of Little Barford steam turbine (ST1C) was due to discrepancies on three independent safety critical speed measurement signals to the generator control system. A cyber attack would most likely, rather than directly take…

> A cyber attack would most likely.. When I was discussing the two interruptions (one on the 9nth and a smaller one on the 24th) with friends, I suggested that this could have been an attempt to map out UK's infrastructure/SCADAs, and they thought that my sandwich had too much consiracy sauce in it. Remembering how Stuxnet interefered with Iran's nuclear centrifuges, this could very well be something similar. Just sc…

I personally knew the chap auditing SCADA in power distribution back in 2001, so I'm fairly confident the level of competence and what governmental bodies handle such issues.

But nothing is perfect, and whilst an edge case, space weather can and does wondrous things to pylons. It's planning for such outages and the layers of vulnerability in systems that such an event can expose in which you want top people on the case and planning around such exposures. After all, you don't want your fallback site all secure but nobody put the electric security doors on their own redundant power. Might end up with such doors opening automaticly and allowing anybody in, or not oipening and preventing access to the big red switch to failover onto some backup system. Things like that you just can't overlook.

But with power-grid distribution systems, their will always be a few weak points, case of if these three transformers suddenly went offline the cascade effect would be horrific. It's those cases you need to plan for. After all, 3 lorry accidents crashing into those transformers, whilst not very likely to happen at the same time - people win the lottery every week with much higher odds.

Re: What Happened in the UK Blackouts?

#112

Post by Ntrails »

aidos wrote: post #108
KyleJ61782 wrote: post #76

Earlier quoted context omitted.

This. Very much this. In the presentations I've seen on the matter, the most vulnerable systems are those measuring and carrying telemetry back from various meters around the power system. One doesn't have to directly compromise the system to have an effect. Simply providing bad data to automated protection systems could be enough to cause instability.

A lesson Knight Capital learnt the hard way. https://en.m.wikipedia.org/wiki/Knight_Capital_Group

There but for the grace of God...

Re: What Happened in the UK Blackouts?

#113

Post by kylegordon »

_def wrote: post #101
std_throwaway wrote: post #63

Earlier quoted context omitted.

An even more stealth method is to make the system work perfectly if everything is running fine. Like correctly reporting net frequency in the range 49-51 Hz but reporting false values when outside of that range. That way there will be no detectable errors until everything starts behaving erratically. The best way for a society to prepare is to have a few small incidents that put the safety systems to the test on a re…

Reminds me of stuxnet

Likewise. Sadly not many people in this thread seem to be aware of this method.

Re: What Happened in the UK Blackouts?

#115

Post by mytailorisrich »

topspin wrote: post #58
pm215 wrote: post #12

Earlier quoted context omitted.

A lot of the headline grabbing stuff wasn't the grid's fault though: * the railways didn't have traction power disconnected -- but one type of (new) train saw the within-specification frequency drop, shut down, and in many cases needed a fitter to go out and restart it. Everything else behind a dead train was stuck. I imagine the train company are discussing this with their supplier... * Ipswich hospital had a power…

I read your list and see a bunch of 'technical debt.' Trains put into operation without the ability to survive a power transient without intervention. Life critical backup power system failures. The set of critical sites is not being maintained properly. In my mind these failures are are not entirely unrelated; reliable power is taken for granted and the testing and maintenance necessary to cope with power outages is…

This is not technical debt. These are signs of oversights or potential poor management. Indeed that's what neglecting these procedures is.

This is why a comprehensive investigation is needed to ensure the details are known and all lessons learned or, unfortunately, re-learned.

Re: What Happened in the UK Blackouts?

#116

Post by NeedMoreTea »

mytailorisrich wrote: post #6
Zenst wrote: post #2

"Frequency drops below 48.8Hz which triggers the Low Frequency Demand Disconnection scheme (LFDD). Approximately 1GW (5% of total) of Great Britain electricity demand is turned off." Is interesting as it highlights the importance of managed failure. Things fail, but the ability to see those failures and control how that fails is important. Had such an action not been taken, the result would of been a far greater outa…

The issue with this specific outage is that it was badly managed. The main gripe people and authorities have with what happened is that key infrastructures immediately lost power. When (only) 5% of demand is turned off this should not impact railways, airports, and hospitals. Beyond the initial loss of power other problems amplified the consequences: For example, trains could not just restart but required specialist…

You're misunderstanding how it went. The 5% that's mentioned as the Low Frequency Demand Disconnection scheme, who were disconnected in managing the incident, is a scheme where very high volume users (those who have a meter reading taken every hour) voluntarily opt to have interruptible supply for better prices. When the grid needs to, they're given a power cut that they signed up for. Commonly used by big manufacturers, smelters etc. It's in their contract, expected and gets them lower electricity bills.

It helps prevent the grid tipping into catastrophic shutdown when there's no backup generation left, or something major has tripped, as part of demand management. If it hadn't happened the consequences would have far more headline-worthy.

Railways, hospitals and the like that encountered issues were part of "the rest". That is not part of the 5% turned off, but the small part of the 95% "rest" that can't cope with some (larger than usual) drop of frequency. If they're buying trains that can't cope with a frequency slump or backup systems that aren't tested that's outwith grid control. That's a sign their systems need to be improved, and tested properly too.

Re: What Happened in the UK Blackouts?

#118

Post by mytailorisrich »

toyg wrote: post #57
pedrocr wrote: post #26

Earlier quoted context omitted.

In a way these events become a blessing in disguise because of those things. We've had to deal with similar crisis situations in our supply chain. It really helps to test your procedures and operations fixing stuff along the way. The team also gains experience and becomes much more prepared for worse events should they come.

I would go as far as suggesting that these things should be routinely tested. In the UK most public buildings test fire alarms once a week; a programmed plan of localised, managed blackouts, to be carried out once every few years, would probably help in the long run - particularly as the grid is rapidly changing as described at the end of the post.

Of course they should be routinely tested. This is not a new idea and is an established best practice.

This is why I am very surprised when I read that hospitals generators failing to take over is just "one of those things".

It's like the 'old' saying that you don't have backups until you test recovery from backups...

Re: What Happened in the UK Blackouts?

#119

Post by growse »

topspin wrote: post #58
pm215 wrote: post #12

Earlier quoted context omitted.

A lot of the headline grabbing stuff wasn't the grid's fault though: * the railways didn't have traction power disconnected -- but one type of (new) train saw the within-specification frequency drop, shut down, and in many cases needed a fitter to go out and restart it. Everything else behind a dead train was stuck. I imagine the train company are discussing this with their supplier... * Ipswich hospital had a power…

I read your list and see a bunch of 'technical debt.' Trains put into operation without the ability to survive a power transient without intervention. Life critical backup power system failures. The set of critical sites is not being maintained properly. In my mind these failures are are not entirely unrelated; reliable power is taken for granted and the testing and maintenance necessary to cope with power outages is…

> Trains put into operation without the ability to survive a power transient without intervention.

The trains were built to a environmental tolerance spec, and shutting down when that spec is exceeded seems like a reasonable thing to do. Having an expert test that all the safety systems are still working properly after they've been exposed to conditions they weren't designed for also seems prudent.

Do you think people should pay higher ticket prices / more tax to fund trains that can tolerate a wider range of input frequencies, given how rarely these sorts of events happen?

Re: What Happened in the UK Blackouts?

#120

Post by Reason077 »

std_throwaway wrote: post #63
londons_explore wrote: post #16

It's notable that this report is completely consistent so far with the way I would design a cyber attack to take out a countries power grid. > The causes provided by RWE for the initiation of the trip of Little Barford steam turbine (ST1C) was due to discrepancies on three independent safety critical speed measurement signals to the generator control system. A cyber attack would most likely, rather than directly take…

An even more stealth method is to make the system work perfectly if everything is running fine. Like correctly reporting net frequency in the range 49-51 Hz but reporting false values when outside of that range. That way there will be no detectable errors until everything starts behaving erratically. The best way for a society to prepare is to have a few small incidents that put the safety systems to the test on a re…

”Like correctly reporting net frequency in the range 49-51 Hz but reporting false values when outside of that range.”

You can’t fake the system frequency like that. Anyone connected to the grid can observe/measure the system frequency trivially - you could do so right now at the nearest 10A socket. The attack you describe would require compromising large numbers of diverse, distributed generation and frequency response assets.

It would be like trying to fool everyone into thinking it’s a hot day by hacking every single thermometer!

Post reply on HN