Live data from Hacker News

Water system controllers don't belong on the internet, says ex-NSA chief

theregister.com

111–120 of 163 posts

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#111
At modern population densities, basic infrastructure breaking on down on a large scale can kill millions in a matter of weeks.

The largest threat isn't bombs falling on our heads, it's incompetent fools leaving the door open to their enemies.

These aren't mistakes that can be excused. Failing in one's duty to steward important infrastructure must mean immediate replacement of leadership.

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#112

Earlier quoted context omitted.

How is it NSA/DHS negligence? Neither is responsible for securing the infrastructure of state and local governments nor private companies. They should provide guidance, but I’m not sure we really want the NSA inside of networks more than they already are. If voters and CEOs don’t want to spend the money required to secure their infrastructure, that’s on them.

DHS has the Cybersecurity & Infrastructure Security Agency. Their job is to help organizations in the US with protecting public services and physical infrastructure. They are partially responsible for the outcomes. https://www.cisa.gov/

CISA was one of the first agencies gutted by the DOGE purge.

https://www.cbsnews.com/news/cybersecurity-agencys-top-recru...

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#113
post #101

Earlier quoted context omitted.

Eh? It's just a thumb drive, on a lanyard, hanging in a cabinet. It's not like it has guards posted. :) Maybe Joe (from shipping) took it back to his desk to have a squiz at it and try to passively learn something new during some otherwise-downtime and never returned it -- years ago. That cabinet is in his area, so he has the key. Maybe Tim (the plant electrician) borrowed it to get something else done, and it uninte…

Things like thumb drives physically connected to cabinets can just disappear in your environment with no followup whatsoever? And that's regular, expected, and desired?

> Things like thumb drives physically connected to cabinets can just disappear in your environment with no followup whatsoever?

Yes. I follow up. No one ever knows.

> And that's regular,

Yes

> expected

Yes

> and desired?

No.

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#114
post #61

Part of my career encapsulates a period where I was a PLC programmer, installer, commissioner, and troubleshooter for massive build outs (sky scrapers, data centers, laboratories, factories). Interestingly, this was after years of teaching myself software engineering, eventually participating in large open source projects. The clash of entering the PLC world was _extremely_ harsh. Let me give an example: I once worke…

Another odd observation about the industry. For better or for worse, I inherited a little system controlled by a Fanuc PLC. It had a bug and would occasionally get stuck in a bad state. By some minor miracle, I managed to track down the person who had originally programmed it (now retired and moved out of state), and he emailed me a file and introduced me to his apprentice (who lives in a third state). I found a tria…

A microcontroller based solution is rarely a good replacement for a PLC. You can find integrators willing to come out and troubleshoot your device in the field a lot more easily than embedded progammers and vendor support is usually excellent. When your plant is down, that sort of thing matters. PLCs handle harsh environments for decades, whereas with a microcontroller all the thermal issues and power conditioning are all on you. When something fails, it's usually just a matter of pulling a spare module off the shelf and plugging it in.

If that's an old GE 90-30 (a common FANUC system), it's time to replace it. You're currently in that period when the CPU has reached end of life but the modules are still supported and the vendor has an easy upgrade path. Emerson owns the brand now, and can sell you a new unit that'll run your old program with minimal changes. You can call them and get a list of integrators in your area. It'll be pricey, but you won't need to do it again for another quarter century.

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#115

He is wrong and right. They should be connected to the Internet when they aren't 30 year old PLCs ripe for abuse. Until then, cut the data lines and do water monitoring the old way.

A data diode can allow monitoring via the internet without risking ingress of control. Commercial units aren't cheap though. Obviously we need open source designs. Perhaps the easiest way would be a Raspberry pi set up with an opto isolated CGA/EGA/VGA/SVGA capture that could be viewed via the internet? (I mean, we're probably talking systems still running MS-DOS or Windows 98 running these systems)

If you want to do that sort of thing in a cheap way, then you can.

I'm told a box with two optical NICs in it with only one fibre in it, TX->RX, works quite well, plus a bit of software to wrap around it.

But these sorts of organisations love a bit of bureaucratic regulation, and "certified" devices and that sort of thing, so there's probably not much of a market for a data diode without the paperwork.

On the opto isolated idea - I think there was a QR-code and camera based data transfer tool posted on HN a few days ago!

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#116

Part of my career encapsulates a period where I was a PLC programmer, installer, commissioner, and troubleshooter for massive build outs (sky scrapers, data centers, laboratories, factories). Interestingly, this was after years of teaching myself software engineering, eventually participating in large open source projects. The clash of entering the PLC world was _extremely_ harsh. Let me give an example: I once worke…

> This industry is in _desperate_ need of strong technical help, but unfortunately the "higher ups" tend to be the same people who are "comfortable" with the way thing are and refuse to move. I literally tried for a decade before giving up and moving into software engineering proper.

As soon as AI can automate the development and deployment of physical infrastructure and the firmware that drives it, watch how willing to embrace the "cutting edge" those same higher-ups become.

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#117
post #98

Part of my career encapsulates a period where I was a PLC programmer, installer, commissioner, and troubleshooter for massive build outs (sky scrapers, data centers, laboratories, factories). Interestingly, this was after years of teaching myself software engineering, eventually participating in large open source projects. The clash of entering the PLC world was _extremely_ harsh. Let me give an example: I once worke…

Storytime: Once upon a time, I was a young grasshopper at a small consulting firm at which every employee was issued a Windows machine, our source control system was "\Customers" on C: drive, and our deployment strategy was "putty + copy-paste + hot-reload". One morning, I heard something terrible while my boss's boss's machine was booting. Something akin to grinding. The thing was angry. I gave boss^2 a heads-up tha…

Nasty looks seems to be the sign of them realizing how you proved them to be incompetent.

Incompetence will cost them their job.

If I had fuck you money I'd just call them incompetent fucks and laugh at them.

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#118
post #61

Earlier quoted context omitted.

Another odd observation about the industry. For better or for worse, I inherited a little system controlled by a Fanuc PLC. It had a bug and would occasionally get stuck in a bad state. By some minor miracle, I managed to track down the person who had originally programmed it (now retired and moved out of state), and he emailed me a file and introduced me to his apprentice (who lives in a third state). I found a tria…

The best thing they could do to new PLC's would be to have the src code live alongside the firmware inside. So any new person would be able to open it and reason about it.

> The best thing they could do to new PLC's would be to have the src code live alongside the firmware inside. So any new person would be able to open it and reason about it.

This is already the case. Some PLC allows you to store and retrieve the source code of the programmed directly in the flash of the PLC. Sometimes this feature is behind a higher access level or password.

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#119
post #35

Earlier quoted context omitted.

You’d be surprised how insecure some of these facilities are, especially to someone who has working knowledge of what a PLC (or other process controllers) does and how it works. You can easily look like a tech who belongs there either troubleshooting something or working on a project. I’ve been doing industrial controls for 15 years and surprisingly infrastructure is some of the most poorly funded. I believe a lot of…

Forgive my ignorance but isn't a PLC simply a computer with some GPIO ports? I.e., a Raspberry Pi could be called a PLC? Why are we talking about them as if they are something else? Is it an exotic OS that makes them different?

The term "PLC" is a lot more specific than "computer with some GPIO ports." No one that works with PLCs would consider a Raspberry Pi to be one.

These things are walled gardens. You never see the operating system. You can only change their behavior using the vendor's software. They generally run a single program (that you write using the vendor's software) on a fixed scan cycle. They read the inputs, run your program, write the outputs, then repeat.

While you're giving up the nearly infinite possibilities that an SBC gives you, the benefits more than make up for the lack of flexibility. They run (and have parts and support available) for decades. Modules are easy to diagnose and replace. An electrician who isn't a programmer can follow ladder logic and troubleshoot problems. Integrators can quickly come up to speed and understand your code.

There's a reason companies will pay tens or even hundreds of thousands of dollars for these things.

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#120
post #76

Earlier quoted context omitted.

The best thing they could do to new PLC's would be to have the src code live alongside the firmware inside. So any new person would be able to open it and reason about it.

You mean like, just hang a thumb drive on a lanyard inside of the cabinet -- with source code and such? That will probably just disappear. Maybe with good intention or maybe with bad, but it's probably gone either way. If it doesn't disappear, then it's hanging right there for any of the competitors to use. That's a problem for the original installer's ongoing employment. And if it includes the programming software,…

> You mean like, just hang a thumb drive on a lanyard inside of the cabinet -- with source code and such?

This guy is very happy when someone literally does that: https://www.youtube.com/@Cursed_Controls (great channel btw), because usually he has to deal with worse.

> And if it includes the programming software, then that lets Joe (from over in shipping) have a go at rejiggering the packaging machine. That's a problem for whoever has to pay someone with a clue to show up and fix it.

You'd think Joe from shipping wouldn't be given a key to the cabinet with the PLC.

Post reply on HN