Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Happened in the UK Blackouts?

mitchoneill.com

Post reply on HN
Print view View original

81–90 of 175 posts

Re: What Happened in the UK Blackouts?

#81

Post by PeterisP »

HenryBemis wrote: post #66
londons_explore wrote: post #16

It's notable that this report is completely consistent so far with the way I would design a cyber attack to take out a countries power grid. > The causes provided by RWE for the initiation of the trip of Little Barford steam turbine (ST1C) was due to discrepancies on three independent safety critical speed measurement signals to the generator control system. A cyber attack would most likely, rather than directly take…

> A cyber attack would most likely.. When I was discussing the two interruptions (one on the 9nth and a smaller one on the 24th) with friends, I suggested that this could have been an attempt to map out UK's infrastructure/SCADAs, and they thought that my sandwich had too much consiracy sauce in it. Remembering how Stuxnet interefered with Iran's nuclear centrifuges, this could very well be something similar. Just sc…

The 2015 attack on Ukraine power grid (https://www.wired.com/2016/03/inside-cunning-unprecedented-h...) may be relevant here.

Re: What Happened in the UK Blackouts?

#82

Post by pedrocr »

toyg wrote: post #57
pedrocr wrote: post #26

Earlier quoted context omitted.

In a way these events become a blessing in disguise because of those things. We've had to deal with similar crisis situations in our supply chain. It really helps to test your procedures and operations fixing stuff along the way. The team also gains experience and becomes much more prepared for worse events should they come.

I would go as far as suggesting that these things should be routinely tested. In the UK most public buildings test fire alarms once a week; a programmed plan of localised, managed blackouts, to be carried out once every few years, would probably help in the long run - particularly as the grid is rapidly changing as described at the end of the post.

Definitely. We use seasonal demand peaks to learn how to cope with extremes. But if we didn't have those plus strikes I'd definitely be planning a few test runs.

Re: What Happened in the UK Blackouts?

#83

Post by slovenlyrobot »

londons_explore wrote: post #16

It's notable that this report is completely consistent so far with the way I would design a cyber attack to take out a countries power grid. > The causes provided by RWE for the initiation of the trip of Little Barford steam turbine (ST1C) was due to discrepancies on three independent safety critical speed measurement signals to the generator control system. A cyber attack would most likely, rather than directly take…

Does that complete consistency include simulating a lightning strike to a transmission line?

Re: What Happened in the UK Blackouts?

#84

Post by Aeolun »

std_throwaway wrote: post #63
londons_explore wrote: post #16

It's notable that this report is completely consistent so far with the way I would design a cyber attack to take out a countries power grid. > The causes provided by RWE for the initiation of the trip of Little Barford steam turbine (ST1C) was due to discrepancies on three independent safety critical speed measurement signals to the generator control system. A cyber attack would most likely, rather than directly take…

An even more stealth method is to make the system work perfectly if everything is running fine. Like correctly reporting net frequency in the range 49-51 Hz but reporting false values when outside of that range. That way there will be no detectable errors until everything starts behaving erratically. The best way for a society to prepare is to have a few small incidents that put the safety systems to the test on a re…

Chaos monkey for your electricity grid!

Re: What Happened in the UK Blackouts?

#85

Post by StreamBright »

Zenst wrote: post #2

"Frequency drops below 48.8Hz which triggers the Low Frequency Demand Disconnection scheme (LFDD). Approximately 1GW (5% of total) of Great Britain electricity demand is turned off." Is interesting as it highlights the importance of managed failure. Things fail, but the ability to see those failures and control how that fails is important. Had such an action not been taken, the result would of been a far greater outa…

It also highlights one of the most important characteristics of energy grids: frequency sensitivity.

Re: What Happened in the UK Blackouts?

#86

Post by Slartie »

aurbano wrote: post #41

It actually sounds like this was really useful though, as a lot of the systems involved will be improved in the future. I'm not sure if the grid is already doing this, but they should trigger failures randomly, including Low Frequency Demand Disconnection schemes from time to time to ensure that all downstream systems are also ready (like the hospital someone mentioned that had an issue with the backup generator, or…

Sorry, but the electrical grid is not built like a Netflix server farm. Its explicit purpose is to provide reliability to the customer, so the customer will not have to invest into mitigations for network failures, because it is much cheaper, more economical and technically much more feasible to make these investments in the network than individually on every single customer location. The only exception are some very specific customers for which the provided amount of reliability is not enough - these need to increase the reliability by investing in additional failsafes, and are of course responsible for testing these by themselves (if they'd like to have random network outages performed, they are free to do so, because any consequences from these harsh tests are in their responsibility anyway, just as any consequences from not doing proper tests and thus having the safeguards potentially fail in an emergency).

Netflix also would never intentionally break my FireTV box, which I use to watch Netflix, just to ensure that I have a backup in place (which I don't, because I don't care THAT much about Netflix availability).

[edit] Now that I think about it...the purpose of the Netflix server farm is also to provide reliability to the customer, so it's actually comparable to a certain degree. It's just that the possible outcome of a mitigation failure in the Netflix server farm is limited to some people not watching Netflix for a few minutes, while the possible outcome of a power outage even for residential customers are usually far worse, let alone industrial or health customers. This allows Netflix to take the limited risk of intentionally breaking internals in their infrastructure, because the worst that can happen is them having to excuse themselves to some customers for a service interruption. Also, no one expects them to submit a report describing how it came to those interruptions, so they don't need to admit that they were the result of intentional self-inflicted disruptions (which sounds really crazy and irresponsible to pretty much all consumers out there). With electrical grid operators, the stakes are infinitely higher, and they are legally required to investigate any bigger problems and report details about their origins to both government authorities and customers. And you can be sure that "oh we just intentionally dropped 5% of the network to see if we get a full blackout or if our failsafes work" would not be an acceptable excuse to any of these two.

Re: What Happened in the UK Blackouts?

#87

Post by pm215 »

danpalmer wrote: post #75
aurbano wrote: post #41

It actually sounds like this was really useful though, as a lot of the systems involved will be improved in the future. I'm not sure if the grid is already doing this, but they should trigger failures randomly, including Low Frequency Demand Disconnection schemes from time to time to ensure that all downstream systems are also ready (like the hospital someone mentioned that had an issue with the backup generator, or…

I assumed that the LFDD scheme was a small number of large power users such as factories that would get cheaper rates, much like AWS spot instances. Another commenter has said this is how it works in Germany. Unfortunately, some quick research[1] suggests this isn’t the case, and actually it’s just a contractual obligation of the power companies to disconnect enough demand to bring power use back down. The amount the…

The UK has the 'pay commercial users who are able to voluntarily reduce demand' system too, and that was part of the initial response to the frequency drop, I think. The problem is that the volume of demand you can get rid of this way isn't necessarily enough to be sufficient -- if you need to dump 10 or 20% of demand and especially if it needs to be spread evenly over the grid then it's going to be hard to do without disconnecting people involuntarily.

I wouldn't be surprised if we could do better in the volume of demand we could reduce on demand, especially with smarter technology, though. At some point the question is whether the extra complexity and infrastructure is cost-effective to avoid once-a-decade events on the scale of this one, which probably 90% of the UK would never have noticed unless they read the news...

Re: What Happened in the UK Blackouts?

#88

Post by PeterisP »

fyfy18 wrote: post #3

What affect does the frequency drop have to home users? Is there protection in the substations near to homes that will cut off the power completely if the frequency goes out of range?

A small frequency drop doesn't have a major impact to home users as well, but it's a symptom of a generation/consumption mismatch - the frequency dropping from 50 to 48.8 is a sign that the consumption is draining energy that's not generated by essentially taking it from the inertia of the major generators, and temporarily reducing consumption as all the large phase-linked motors are suddenly slowing down slightly... both of which are temporary, short-term, unsustainable effects.

The emergency response kicks in not because 49 is innately very bad and 48 is some horrible evil by itself, it kicks in because if you do nothing, then in a few minutes you're going to see 40, 30 and the whole system is going to stop - or, more realistically, every connected subsystem (both generators and consumers) will have their emergency responses kicking in and either shutting down or automatically disconnecting from your system.

Re: What Happened in the UK Blackouts?

#89

Post by yebyen »

mantap wrote: post #69
HenryBemis wrote: post #66

Earlier quoted context omitted.

> A cyber attack would most likely.. When I was discussing the two interruptions (one on the 9nth and a smaller one on the 24th) with friends, I suggested that this could have been an attempt to map out UK's infrastructure/SCADAs, and they thought that my sandwich had too much consiracy sauce in it. Remembering how Stuxnet interefered with Iran's nuclear centrifuges, this could very well be something similar. Just sc…

If that was the intention it backfired because it highlighted several problems to National Grid that they weren't aware of before.

There is some version of reality in which this was the point, if we are more charitable to the attackers motives

Re: What Happened in the UK Blackouts?

#90

Post by pm215 »

guerby wrote: post #47
mitchoneill1 wrote: post #28

Earlier quoted context omitted.

Hey! (author of the article here). I'm not happy with how I phrased this with the 4GW of capacity -- and while I suspect generation could have increased by hundreds of MW within 60 seconds I may be wrong. I've updated how I've phrased it in the blog to hopefully give a bit more clarity to my position. I don't meant to suggest that plants could ramp full within 60 seconds, it's that 4GW means there's a bunch of headro…

I assume battery based grid power stations like the Tesla one in Australia are amongst the best in class to help in this kind of crisis situation? Compared to the gaz turbine the system it is simpler (no mehanical/heat/fluid parts) and depending on design will likely never be completely off since made of dozens or hundreds of independant equipement (batteries and inverters). Thanks for the article and comment!

Battery systems seem likely to play a greater part in the grid in future, but I think it's worth noting that the problem here was "sudden loss of generation exceeded the amount that the system operating specs require to be covered by having alternatives on standby". So unless we're collectively prepared to pay to raise the resilience by having more standby capacity, might we just find ourselves with more batteries and fewer turbines running inefficiently at half-load, so we take the benefit of battery tech in cheaper electricity rather than reduced chances of outage?
Post reply on HN