What Happened in the UK Blackouts? • Hacker News Board

Re: What Happened in the UK Blackouts?

#11

Post by Zenst » Sun, Sep 01, 2019, 8:52 AM UTC

mytailorisrich wrote: post #6

Zenst wrote: post #2

"Frequency drops below 48.8Hz which triggers the Low Frequency Demand Disconnection scheme (LFDD). Approximately 1GW (5% of total) of Great Britain electricity demand is turned off." Is interesting as it highlights the importance of managed failure. Things fail, but the ability to see those failures and control how that fails is important. Had such an action not been taken, the result would of been a far greater outa…

The issue with this specific outage is that it was badly managed. The main gripe people and authorities have with what happened is that key infrastructures immediately lost power. When (only) 5% of demand is turned off this should not impact railways, airports, and hospitals. Beyond the initial loss of power other problems amplified the consequences: For example, trains could not just restart but required specialist…

Most situations/responses to an issue in hindsight can always be managed better, being able to have a cookie-cutter solution that fits most cases is better than none. THough I'm sure such plans will be refined - leasons learned.

Thing is, a solution today might not be best for tomorrow as the grid's usage is not static, new this and that are added so the impact changes.

It is interesting that you mention the inability for trains to restart. That was educational and was due to some new trains and showed that the drivers did not have the skills to in effect - reboot the trains from a cold start. So less a grid fault and more a case of that it never happened before so the result was less prepared. Interestingly the previous generation of trains did not have this issue, which may explain why it got overlooked as something that needed attention until such an event like this.

As for hospitals, they all have backup generators, though if that generator fails to kick in as was the case in one I read about, that can and will cause issues. Again, may be a case of never needing it before and the combination of that and regular checks/tests presented the problem.

Remember - nothing is perfect, and many lessons (as with any outage in all walks of life) will be learned from this as with all previous outages. But equally have to remember that whilst you can have the same combination of events happen in say 5 years time and have a response that works for that combination learned from the last time. Things change, like that new housing estate attached to the grid, small nuances like that could change everything or nothing. You just don't know. Heck, could be newer issues, suddenly find that you have thousands of electric cars being charged at the time you cut them off now feeding back small amounts of electricity back into the grid as they never expected to loose power without a physical disconnect.

Re: What Happened in the UK Blackouts?

#12

Post by pm215 » Sun, Sep 01, 2019, 8:59 AM UTC

mytailorisrich wrote: post #6

Zenst wrote: post #2

"Frequency drops below 48.8Hz which triggers the Low Frequency Demand Disconnection scheme (LFDD). Approximately 1GW (5% of total) of Great Britain electricity demand is turned off." Is interesting as it highlights the importance of managed failure. Things fail, but the ability to see those failures and control how that fails is important. Had such an action not been taken, the result would of been a far greater outa…

The issue with this specific outage is that it was badly managed. The main gripe people and authorities have with what happened is that key infrastructures immediately lost power. When (only) 5% of demand is turned off this should not impact railways, airports, and hospitals. Beyond the initial loss of power other problems amplified the consequences: For example, trains could not just restart but required specialist…

A lot of the headline grabbing stuff wasn't the grid's fault though:

* the railways didn't have traction power disconnected -- but one type of (new) train saw the within-specification frequency drop, shut down, and in many cases needed a fitter to go out and restart it. Everything else behind a dead train was stuck. I imagine the train company are discussing this with their supplier...

* Ipswich hospital had a power outage, but that was because their own systems disconnected from the grid when they saw the frequency drop, and then they had a problem with one of their generators.

*One regional airport seems to have been accidentally not on the list of key sites not to disconnect -- sounds like an admin snafu, and has now been fixed.

None of these seem like "bad management" -- they are a collection of relatively minor, entirely unrelated, problems of the sort you're likely to run into if you have to activate an emergency procedure you haven't needed to use in a decade.

Re: What Happened in the UK Blackouts?

#13

Post by mytailorisrich » Sun, Sep 01, 2019, 8:59 AM UTC

Zenst wrote: post #11

mytailorisrich wrote: post #6

Earlier quoted context omitted.

The issue with this specific outage is that it was badly managed. The main gripe people and authorities have with what happened is that key infrastructures immediately lost power. When (only) 5% of demand is turned off this should not impact railways, airports, and hospitals. Beyond the initial loss of power other problems amplified the consequences: For example, trains could not just restart but required specialist…

Most situations/responses to an issue in hindsight can always be managed better, being able to have a cookie-cutter solution that fits most cases is better than none. THough I'm sure such plans will be refined - leasons learned. Thing is, a solution today might not be best for tomorrow as the grid's usage is not static, new this and that are added so the impact changes. It is interesting that you mention the inabilit…

All the issues I mentioned are obvious, and so are the potential problems you describe. This is not a question of "it's easy to say in hindsight" as these are the key issues that must be covered by standard planning.

This blackout made a lot of noise because of this (apart from the actual disruption, obviously).

Re: What Happened in the UK Blackouts?

#14

Post by mytailorisrich » Sun, Sep 01, 2019, 9:01 AM UTC

pm215 wrote: post #12

mytailorisrich wrote: post #6

Earlier quoted context omitted.

The issue with this specific outage is that it was badly managed. The main gripe people and authorities have with what happened is that key infrastructures immediately lost power. When (only) 5% of demand is turned off this should not impact railways, airports, and hospitals. Beyond the initial loss of power other problems amplified the consequences: For example, trains could not just restart but required specialist…

A lot of the headline grabbing stuff wasn't the grid's fault though: * the railways didn't have traction power disconnected -- but one type of (new) train saw the within-specification frequency drop, shut down, and in many cases needed a fitter to go out and restart it. Everything else behind a dead train was stuck. I imagine the train company are discussing this with their supplier... * Ipswich hospital had a power…

I did mention the railways and indeed the importance of making sure the whole system can cope, which means what you list in addition to the grid in isolation.

> None of these seem like "bad management"

They do seem like bad management because to me it looks like these issues were not taken seriously.

> problems of the sort you're likely to run into if you have to activate an emergency procedure you haven't needed to use in a decade.

That's bad management right there if an organisation's emergency procedure has been collecting dust for a decade.

Re: What Happened in the UK Blackouts?

#15

Post by DanBC » Sun, Sep 01, 2019, 9:03 AM UTC

mytailorisrich wrote: post #14

pm215 wrote: post #12

Earlier quoted context omitted.

A lot of the headline grabbing stuff wasn't the grid's fault though: * the railways didn't have traction power disconnected -- but one type of (new) train saw the within-specification frequency drop, shut down, and in many cases needed a fitter to go out and restart it. Everything else behind a dead train was stuck. I imagine the train company are discussing this with their supplier... * Ipswich hospital had a power…

I did mention the railways and indeed the importance of making sure the whole system can cope, which means what you list in addition to the grid in isolation. > None of these seem like "bad management" They do seem like bad management because to me it looks like these issues were not taken seriously. > problems of the sort you're likely to run into if you have to activate an emergency procedure you haven't needed to…

> They do seem like bad management.

But not bad management by the Grid.

Re: What Happened in the UK Blackouts?

#16

Post by londons_explore » Sun, Sep 01, 2019, 9:04 AM UTC

It's notable that this report is completely consistent so far with the way I would design a cyber attack to take out a countries power grid.

> The causes provided by RWE for the initiation of the trip of Little Barford steam turbine (ST1C) was due to discrepancies on three independent safety critical speed measurement signals to the generator control system.

A cyber attack would most likely, rather than directly take capacity offline, instead make the system less stable by attempting to make safety systems not work properly. That has the benefit that the attack will be triggered at th exact same time as another incident, increasing the impact.

Injecting false values for a sensor reading is a very good way to hide your tracks as an attacker - just inject a bunch of "0" values, and then uninstall whatever malware did it, and probably nobody will ever find out why that sensor randomly acted up.

Re: What Happened in the UK Blackouts?

#17

Post by bonzini » Sun, Sep 01, 2019, 9:09 AM UTC

Zenst wrote: post #8

bonzini wrote: post #5

Earlier quoted context omitted.

The main problem with frequency mismatch is that when two power sources "meet" with different frequency the effect can be catastrophic, because the faster generator pulls the slower (think of the "hares" in 3000-10000 m running competitions) and typically both end up breaking.

Nice article covering frfequncy importance at grid level:- https://www.drax.com/energy-policy/need-whole-country-freque... Though I'm not sure on the details of a frequency drop and impact upon your house in exacting details. Equally I'm not up on substations (did work on IT for large util company many decades ago, but not field engineering). But the first safety control from supply to the home, would be the distribu…

That website has had me nerd-sniped. Another great one is on "black start": https://www.drax.com/technology/black-start-important-back-p...

Re: What Happened in the UK Blackouts?

#18

Post by mytailorisrich » Sun, Sep 01, 2019, 9:11 AM UTC

DanBC wrote: post #15

mytailorisrich wrote: post #14

Earlier quoted context omitted.

I did mention the railways and indeed the importance of making sure the whole system can cope, which means what you list in addition to the grid in isolation. > None of these seem like "bad management" They do seem like bad management because to me it looks like these issues were not taken seriously. > problems of the sort you're likely to run into if you have to activate an emergency procedure you haven't needed to…

> They do seem like bad management. But not bad management by the Grid.

Including by the grid because key infrastructures did lose power.

But in general a worrying aspect of what happened is that there seems to have been oversights all over the place (grid, hospitals, railways, etc).

Re: What Happened in the UK Blackouts?

#19

Post by baxtr » Sun, Sep 01, 2019, 9:17 AM UTC

pm215 wrote: post #12

mytailorisrich wrote: post #6

Earlier quoted context omitted.

The issue with this specific outage is that it was badly managed. The main gripe people and authorities have with what happened is that key infrastructures immediately lost power. When (only) 5% of demand is turned off this should not impact railways, airports, and hospitals. Beyond the initial loss of power other problems amplified the consequences: For example, trains could not just restart but required specialist…

A lot of the headline grabbing stuff wasn't the grid's fault though: * the railways didn't have traction power disconnected -- but one type of (new) train saw the within-specification frequency drop, shut down, and in many cases needed a fitter to go out and restart it. Everything else behind a dead train was stuck. I imagine the train company are discussing this with their supplier... * Ipswich hospital had a power…

Sounds like a good chance to test the systems

Re: What Happened in the UK Blackouts?

#20

Post by pm215 » Sun, Sep 01, 2019, 9:17 AM UTC

The article says near the bottom "At the time [the GT1A generator tripped] there was 4000MW of unused generation in reserve, and if only a fraction of this reserve was instructed to dispatch within 60 seconds of the initial loss it’s very likely that load shedding would have not occurred."

My understanding was that having reserves available within 30-60s was pretty expensive, and that that would have been what was used to handle the initial frequency drop (the interim report PDF says there was 1000MW of this). Drax's page on this (https://www.drax.com/technology/power-systems-super-subs/) says that even spinning reserve can be on the order of two minutes, and the next tier below that (STOR) is on the order of 20 minutes. So does anybody know what the 4000MW available within 60s but not called on was?