Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Happened in the UK Blackouts?

mitchoneill.com

Post reply on HN
Print view View original

21–30 of 175 posts

Re: What Happened in the UK Blackouts?

#21

Post by DanBC »

mytailorisrich wrote: post #18
DanBC wrote: post #15

Earlier quoted context omitted.

> They do seem like bad management. But not bad management by the Grid.

Including by the grid because key infrastructures did lose power. But in general a worrying aspect of what happened is that there seems to have been oversights all over the place (grid, hospitals, railways, etc).

How many 9s do you want from the grid?

There are lots of reasons why hospitals lose power, which is why they all have backup generators.

Ipswich's backup generator had a faulty battery. This is nothing to do with the grid. This is entirely the fault of the hospital trust. Most of the hospital's backups worked.

https://www.ipswichstar.co.uk/news/ipswich-hospital-blackout...

> A spokeswoman said: "Our initial investigation has shown that at 4.52pm on Friday, August 9, we lost our electricity supply until 5pm. We have 11 generators at Ipswich Hospital and all of them worked immediately.

> "What we have found is that the battery which automatically operates one of the switch over systems from mains power to generator power was faulty despite being within its recommended life.

Here's the trust document talking about the new installation when it was put in.

http://www.ipswichhospital.nhs.uk/aboutourhospital/Documents...

> Electrical High Voltage Infrastructure was a critically important project to the Trust in providing a resilient high voltage electrical network to the hospital. Through a range of electrical projects, including the replacement and upgrading of outpatients generator ‘C’, the maternity generator project and the upgrading of the high voltage switchgear for the forthcoming biofuels project, this represented a total overall investment in excess of £2m. Virtually all the Trust’s electrical switchgear has now been improved and modernised. Along with the new backup generators the hospital is protected for years to come and a major significant risk to hospital has been dealt with.

Re: What Happened in the UK Blackouts?

#22

Post by kzrdude »

At one stage, they have to shed load (turn power off), and that sounds totally reasonable. How are these areas divided, how does one turn power off for a whole area, and how is it decided who to cut from power (temporarily)?

Re: What Happened in the UK Blackouts?

#23

Post by tripzilch »

mytailorisrich wrote: post #13
Zenst wrote: post #11

Earlier quoted context omitted.

Most situations/responses to an issue in hindsight can always be managed better, being able to have a cookie-cutter solution that fits most cases is better than none. THough I'm sure such plans will be refined - leasons learned. Thing is, a solution today might not be best for tomorrow as the grid's usage is not static, new this and that are added so the impact changes. It is interesting that you mention the inabilit…

All the issues I mentioned are obvious, and so are the potential problems you describe. This is not a question of "it's easy to say in hindsight" as these are the key issues that must be covered by standard planning. This blackout made a lot of noise because of this (apart from the actual disruption, obviously).

Do you have background in the field of giant power supply fault tolerance backup mishap management strategies, or something? Or are you just stating and assuming all of this is obvious, covered by standard planning, instead of easy to say in hindsight? Cause you're making a lot of statements, saying it is such, without backing it up. Maybe it is super obvious, but personally I'm impressed by the strategies that were in place.

Re: What Happened in the UK Blackouts?

#24

Post by hyperman1 »

I like this article as it gives a lot of advanced insights about what seems at first an easy problem.

Generators are tech from, what, 2 centuries ago? Keeping them in sync sounds boring, just talk together. And then, under the surface, looms an incredibly complex problem about working at scale, having engineering, political, organizational aspects.

Re: What Happened in the UK Blackouts?

#25

Post by mytailorisrich »

DanBC wrote: post #21
mytailorisrich wrote: post #18

Earlier quoted context omitted.

Including by the grid because key infrastructures did lose power. But in general a worrying aspect of what happened is that there seems to have been oversights all over the place (grid, hospitals, railways, etc).

How many 9s do you want from the grid? There are lots of reasons why hospitals lose power, which is why they all have backup generators. Ipswich's backup generator had a faulty battery. This is nothing to do with the grid. This is entirely the fault of the hospital trust. Most of the hospital's backups worked. https://www.ipswichstar.co.uk/news/ipswich-hospital-blackout... > A spokeswoman said: "Our initial investiga…

It's clear that there were issues at hospitals (as expected cynical me thinks, knowing British healthcare...) but the fact that key infrastructure lost power does points to issues at the grid level as well (because sites that lost power did so based on grid procedures).

Re: What Happened in the UK Blackouts?

#26

Post by pedrocr »

pm215 wrote: post #12
mytailorisrich wrote: post #6

Earlier quoted context omitted.

The issue with this specific outage is that it was badly managed. The main gripe people and authorities have with what happened is that key infrastructures immediately lost power. When (only) 5% of demand is turned off this should not impact railways, airports, and hospitals. Beyond the initial loss of power other problems amplified the consequences: For example, trains could not just restart but required specialist…

A lot of the headline grabbing stuff wasn't the grid's fault though: * the railways didn't have traction power disconnected -- but one type of (new) train saw the within-specification frequency drop, shut down, and in many cases needed a fitter to go out and restart it. Everything else behind a dead train was stuck. I imagine the train company are discussing this with their supplier... * Ipswich hospital had a power…

In a way these events become a blessing in disguise because of those things. We've had to deal with similar crisis situations in our supply chain. It really helps to test your procedures and operations fixing stuff along the way. The team also gains experience and becomes much more prepared for worse events should they come.

Re: What Happened in the UK Blackouts?

#28

Post by mitchoneill1 »

pm215 wrote: post #20

The article says near the bottom "At the time [the GT1A generator tripped] there was 4000MW of unused generation in reserve, and if only a fraction of this reserve was instructed to dispatch within 60 seconds of the initial loss it’s very likely that load shedding would have not occurred." My understanding was that having reserves available within 30-60s was pretty expensive, and that that would have been what was us…

Hey! (author of the article here). I'm not happy with how I phrased this with the 4GW of capacity -- and while I suspect generation could have increased by hundreds of MW within 60 seconds I may be wrong. I've updated how I've phrased it in the blog to hopefully give a bit more clarity to my position.

I don't meant to suggest that plants could ramp full within 60 seconds, it's that 4GW means there's a bunch of headroom, and that there was at least the physical possibility that dispatch instructions to increase generation/start-up (the reserves were "available") could have been sent out and plants could have started moving on an upward trajectory, creating even a few extra hundred MW of generation if many of the more flexible plants (gas, hydro) could increase their output even just a bit.

Here in Australia we dispatch on a 5 minute period, and there are some plants that are very slow moving and have to ramp over tens of minutes or hours, but moving grid-wide generation up and down many hundreds of MW within a 5 minute dispatch period is a common occurrence, and so this coupled with the relatively high amount of reserve makes me suspect it would have at least been physically possible to increase output and hasten the return to a normal operating frequency.

How you create both the systems and market structures/rules to oversee dispatch and coordination of something like that in a fast and automated way is very complex and infeasible to effectively do any time soon, but I think it's a system we should be thinking about and moving towards.

Overall though I'm just grumpy about the amount of manual, slow work that's still involved in grid operations (when we lose an inter-connector here in Australia it takes 3-15 minutes for the dispatch engine to become aware that there's no inter-connector anymore as it needs to be reconfigured manually).

Re: What Happened in the UK Blackouts?

#29

Post by jonatron »

It was a few minutes of downtime affecting a fraction of people. The last large UK outage was 10 years ago. The uptime has a lot of 9s already, so even if they ensure this particular problem wouldn't cause downtime again, the next time it'd just be a different rare event that causes it.

Re: What Happened in the UK Blackouts?

#30

Post by Zenst »

mytailorisrich wrote: post #25
DanBC wrote: post #21

Earlier quoted context omitted.

How many 9s do you want from the grid? There are lots of reasons why hospitals lose power, which is why they all have backup generators. Ipswich's backup generator had a faulty battery. This is nothing to do with the grid. This is entirely the fault of the hospital trust. Most of the hospital's backups worked. https://www.ipswichstar.co.uk/news/ipswich-hospital-blackout... > A spokeswoman said: "Our initial investiga…

It's clear that there were issues at hospitals (as expected cynical me thinks, knowing British healthcare...) but the fact that key infrastructure lost power does points to issues at the grid level as well (because sites that lost power did so based on grid procedures).

When they cut the power to an area, they can't selectively keep isolated locations active unless they are isolated in that they operate upon their own spur. That is why Hospitals would be deemed critical and important not to cut out, they are handled by onsight failover power supply in the form of backup generators.

In this case, it was like asking somebody to be able to catch a ball, they say they are ready and yet fail to catch that ball when you thru it to them. Is that the fault of the person who throws the ball or the person that was not prepared to catch the ball when they said they could.

In this instance it falls into the infortunate situation, and from what I can tell, you can't lay blame onto anybody or party for the failure and it is just one of those things. Sure leasons will be learned, but from the grid procedure in this hospital case, there will be no change I can see that would be learned. However the Hospital will be learning from this and may have more frequent testing and checks of their failover, may make changes. So if blame is to be leveraged upon anybody - you could say it would be from the party that had to make changes. But in this instance, would that be fair. Hence I would call this unfortunate and whilst lessons will be learned, you can not lay blame out of those lessons all the time. However - if it happens again and those same mistakes transpire - then, maybe you have some grounds to lay some blame.

Post reply on HN