Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Water system controllers don't belong on the internet, says ex-NSA chief

theregister.com

Post reply on HN
Print view View original

141–150 of 166 posts

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#141

Post by thyristan »

palmotea wrote: post #120
ssl-3 wrote: post #76

Earlier quoted context omitted.

You mean like, just hang a thumb drive on a lanyard inside of the cabinet -- with source code and such? That will probably just disappear. Maybe with good intention or maybe with bad, but it's probably gone either way. If it doesn't disappear, then it's hanging right there for any of the competitors to use. That's a problem for the original installer's ongoing employment. And if it includes the programming software,…

> You mean like, just hang a thumb drive on a lanyard inside of the cabinet -- with source code and such? This guy is very happy when someone literally does that: https://www.youtube.com/@Cursed_Controls (great channel btw), because usually he has to deal with worse. > And if it includes the programming software, then that lets Joe (from over in shipping) have a go at rejiggering the packaging machine. That's a probl…

The PLC is usually in a control cabinet that can be opened by a control cabinet key:

https://www.knipex.de/schaltschrankschluessel?v=2921

Real locks in such systems are rare, sometimes even disallowed. Usually, a lock with a real key is only used to temporarily prevent switching on a machine during maintenance or such.

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#142

Post by duckmysick »

tedd4u wrote: post #86
amluto wrote: post #63

Earlier quoted context omitted.

Sometimes I wonder whether one could design PLC-style software by first expressing the system’s constraints and then writing the actual code and having either (preferably) formal verification that the code will not violate the constraints or (less preferably) enforced assertions that will send the machine into a safe state if the constraints do not hold. The system I mentioned in my other post controlled some VFDs, a…

Seems like, here in the 21st century, one could reasonably have a model of the plant that tests could be executed against. Anything like that exist?

The industry/consulting term is "digital twin". Nothing off the shelf, since every plant is different. Even for a single process it gets complicated really fast. You probably want to track the environment itself too like temperature, humidity, leaks, etc. All the machinery and if they don't have necessary sensors, gotta retrofit them. Oh and human workers too. Stuff like "John Doe pressed a manual override button at 9:25 AM" or "it takes 12 minutes for a forklift operator to move a pallet from zone A to C".

I was involved in a project that made such a twin for a warehouse. It was simpler in theory because there was no manufacturing process. Just moving objects from one place to another. It was still expensive and took a long time. We tracked traffic heatmap but also collected data on and modelled luminosity for safety. On top of that there was a standard warehouse space optimization.

You're going to run into similar problems as with testing and documenting code. Eventually something will change - a process, a machine, a sensor. You have to stay on top of things and update your model too. Otherwise they will drift apart and not be as useful.

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#143

Post by b3orn »

ssl-3 wrote: post #76
polishdude20 wrote: post #70

Earlier quoted context omitted.

The best thing they could do to new PLC's would be to have the src code live alongside the firmware inside. So any new person would be able to open it and reason about it.

You mean like, just hang a thumb drive on a lanyard inside of the cabinet -- with source code and such? That will probably just disappear. Maybe with good intention or maybe with bad, but it's probably gone either way. If it doesn't disappear, then it's hanging right there for any of the competitors to use. That's a problem for the original installer's ongoing employment. And if it includes the programming software,…

If it's a custom industrial installation then the source code is part of the documentation just like the electrical schematics, specifically so that any trained electrician can maintain it. On the other hand Joe from shipping wouldn't even know what a PLC is and if you let untrained staff maintain electrical systems I think you have some more serious problems.

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#144

Post by thyristan »

amelius wrote: post #35
procarch2019 wrote: post #25

Earlier quoted context omitted.

You’d be surprised how insecure some of these facilities are, especially to someone who has working knowledge of what a PLC (or other process controllers) does and how it works. You can easily look like a tech who belongs there either troubleshooting something or working on a project. I’ve been doing industrial controls for 15 years and surprisingly infrastructure is some of the most poorly funded. I believe a lot of…

Forgive my ignorance but isn't a PLC simply a computer with some GPIO ports? I.e., a Raspberry Pi could be called a PLC? Why are we talking about them as if they are something else? Is it an exotic OS that makes them different?

Yes, it is.

All the siblings might talk a lot about "but ruggedized!", "but realtime!", "but supported!".

But in many cases, it is just a case of vendor lock-in. Ruggedized is just a casing and decoupled I/O-ports, you can get that for your Raspberry Pi. Realtime is a concern, but modern PLCs aren't really realtime anymore. Those are usually some big multicore non-realtime ARM, plus some small realtime-capable I/O-processors that do the really time-critical stuff. Or even just a CPLD to implement some trivial ladder logic cycles. Support matters, but if your PLC vendor goes bankrupt you've drained a few millions. Might be better to get something cheap where you can just stock up a few hundred replacement parts for cheaper.

That's why actually, a lot of smaller PLCs are Raspberry Pi CMs now. The vendor won't tell you, but if you open them up, you'll see.

And the big customers also wised up to the PLC scam and are now moving towards putting their control tasks on standard x86 hardware in the shop floor basement plus realtime ethernet with cheap I/O extenders on the shop floor. Which is why the big PLC vendors now try to sell "virtual PLC" to those customers, so they get at least a little money instead of none ;) https://www.siemens.com/en-us/products/simatic/s7-1500v/

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#145

Post by ocschwar »

Kim_Bruning wrote: post #90
tedd4u wrote: post #86

Earlier quoted context omitted.

Seems like, here in the 21st century, one could reasonably have a model of the plant that tests could be executed against. Anything like that exist?

We're often talking about custom plants here, or plants made in very low volume. Making a simulator for that can be prohibitively expensive, since you can't amortize the cost. For particularly important/expensive plants it still sometimes gets done. Though note the simulator might not be able to capture all the nuances of the real world deployment, so it might not be sufficient. Meanwhile having a separate copy of th…

The trouble is the simulation tools are either proprietary and thus have only a small mind share among software developers, or academic, and either way, PLC programmers have not gone near them yet.

But a decent LLM should be able to translate the description of a plant to a TLA+ model (or Alloy, or something similar.)

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#146

Post by hulitu »

pigbearpig wrote: post #58
jacobgold wrote: post #38

With coding agents now being used for hacking, there's a decent chance we'll see a 9/11-scale hacking incident as a result of NSA/DHS negligence in securing American internet-connected services. Similar to how the CIA's negligence allowed 9/11 itself. The USG should be deploying thousands of security engineers armed with the latest coding models and agents, in attempt to secure systems before they're hacked. A few bi…

How is it NSA/DHS negligence? Neither is responsible for securing the infrastructure of state and local governments nor private companies. They should provide guidance, but I’m not sure we really want the NSA inside of networks more than they already are. If voters and CEOs don’t want to spend the money required to secure their infrastructure, that’s on them.

> How is it NSA/DHS negligence?

By mandating backdoors.

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#147

Post by NotMichaelBay »

hommelix wrote: post #118
polishdude20 wrote: post #70

Earlier quoted context omitted.

The best thing they could do to new PLC's would be to have the src code live alongside the firmware inside. So any new person would be able to open it and reason about it.

> The best thing they could do to new PLC's would be to have the src code live alongside the firmware inside. So any new person would be able to open it and reason about it. This is already the case. Some PLC allows you to store and retrieve the source code of the programmed directly in the flash of the PLC. Sometimes this feature is behind a higher access level or password.

Is it version controlled too?

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#148

Post by Kim_Bruning »

closeparen wrote: post #102
Kim_Bruning wrote: post #9

Don't put your PLCs directly on the internet. In fact most industrial stuff is very not made to be directly connected to the internet. But interpose a firewall+VPN solution and you might be ok, if done competently. And remote access to hardware definitely makes management and maintenance a lot easier and quicker. (else you need to drive out for every minor issue)

If we can beat a security-state airgap by sprinkling USB drives in the parking lot, I think the Iranians can beat an "internal" network by getting someone to click on an email attachment or visit the wrong website on their municipal issued Windows machine. It's very common for the proprietary software for interfacing with ancient, expensive machines to break after OS upgrades, so they're probably unpatched... you mig…

Which would then tunnel through the VPN and still hit the PLC. Right. "But we need it on our windows computer" is not an entirely unreasonable ask. But security is only partially served.

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#149

Post by giantg2 »

spauldo wrote: post #114
amluto wrote: post #61

Earlier quoted context omitted.

Another odd observation about the industry. For better or for worse, I inherited a little system controlled by a Fanuc PLC. It had a bug and would occasionally get stuck in a bad state. By some minor miracle, I managed to track down the person who had originally programmed it (now retired and moved out of state), and he emailed me a file and introduced me to his apprentice (who lives in a third state). I found a tria…

A microcontroller based solution is rarely a good replacement for a PLC. You can find integrators willing to come out and troubleshoot your device in the field a lot more easily than embedded progammers and vendor support is usually excellent. When your plant is down, that sort of thing matters. PLCs handle harsh environments for decades, whereas with a microcontroller all the thermal issues and power conditioning ar…

"but you won't need to do it again for another quarter century."

Hmm, so how do they handle the new cyber threats, since most of these are connected? It seems upgrades would need to be performed to address new threats. And just because something is not internet connected doesn't make it immune (eg stuxnet).

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#150

Post by tda »

aliasxneo wrote: post #31

Part of my career encapsulates a period where I was a PLC programmer, installer, commissioner, and troubleshooter for massive build outs (sky scrapers, data centers, laboratories, factories). Interestingly, this was after years of teaching myself software engineering, eventually participating in large open source projects. The clash of entering the PLC world was _extremely_ harsh. Let me give an example: I once worke…

Yep, everything is lacking. No version control, ladder diagrams instead of written code, goto statements as primary control flow in code. A $500 PLC with less processing power than a 1985 086 and 1 whole MB of memory. Upgradable with a $200 license to 1.5MB, only programmable with a 20GB monstrosity of a piece of software in which download means copying a program.e to the PLC, and upload means downloading a backup to your laptop. That world needs some disruption really really bad
Post reply on HN