Water system controllers don't belong on the internet, says ex-NSA chief • Hacker News Board

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#161

Post by KellyCriterion » Sat, Aug 08, 2026, 7:37 PM UTC

aliasxneo wrote: post #31

Part of my career encapsulates a period where I was a PLC programmer, installer, commissioner, and troubleshooter for massive build outs (sky scrapers, data centers, laboratories, factories). Interestingly, this was after years of teaching myself software engineering, eventually participating in large open source projects. The clash of entering the PLC world was _extremely_ harsh. Let me give an example: I once worke…

May I ask: In which year was this?

> I vividly recall coming into the data center for a fortune 50 company, one everyone here would know, and being astonished that they had never heard of Network Attached Storage or RAID The thing is: Big companies, even non critical are usually bonded to some frameworks/regulatory standards/etc.

So, mentioning that they didnt know NAS or even RAID (which is very old, end of 80s), makes me wondering in which decade this happened?

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#162

Post by KellyCriterion » Sat, Aug 08, 2026, 7:40 PM UTC

procarch2019 wrote: post #25

Wowfunhappy wrote: post #18

Earlier quoted context omitted.

Wouldn't the physical facilities themselves have security?

You’d be surprised how insecure some of these facilities are, especially to someone who has working knowledge of what a PLC (or other process controllers) does and how it works. You can easily look like a tech who belongs there either troubleshooting something or working on a project. I’ve been doing industrial controls for 15 years and surprisingly infrastructure is some of the most poorly funded. I believe a lot of…

> so it’s bidded out Na, bro! Today we call this "public private partnership" :-D

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#163

Post by ssl-3 » Sat, Aug 08, 2026, 7:56 PM UTC

b3orn wrote: post #143

ssl-3 wrote: post #76

Earlier quoted context omitted.

You mean like, just hang a thumb drive on a lanyard inside of the cabinet -- with source code and such? That will probably just disappear. Maybe with good intention or maybe with bad, but it's probably gone either way. If it doesn't disappear, then it's hanging right there for any of the competitors to use. That's a problem for the original installer's ongoing employment. And if it includes the programming software,…

If it's a custom industrial installation then the source code is part of the documentation just like the electrical schematics, specifically so that any trained electrician can maintain it. On the other hand Joe from shipping wouldn't even know what a PLC is and if you let untrained staff maintain electrical systems I think you have some more serious problems.

> On the other hand Joe from shipping wouldn't even know what a PLC is

Joe from shipping is undefined. It is possible that Joe knows more about PLCs than you think you know about shipping, and that he has a god complex worse than that of a grizzled Haskell programmer. People all over have jobs that don't align with their interests, education, and prior work experiences. That's pretty normal; the bills still need paid and people find work where they can.

> if you let untrained staff maintain electrical systems I think you have some more serious problems.

Or maybe Joe from shipping is a departmental director who has this role because he is a strong leader who takes the initiative to embrace and overcome new challenges. We don't know.

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#164

Post by structural » Sat, Aug 08, 2026, 8:00 PM UTC

KellyCriterion wrote: post #161

aliasxneo wrote: post #31

Part of my career encapsulates a period where I was a PLC programmer, installer, commissioner, and troubleshooter for massive build outs (sky scrapers, data centers, laboratories, factories). Interestingly, this was after years of teaching myself software engineering, eventually participating in large open source projects. The clash of entering the PLC world was _extremely_ harsh. Let me give an example: I once worke…

May I ask: In which year was this? > I vividly recall coming into the data center for a fortune 50 company, one everyone here would know, and being astonished that they had never heard of Network Attached Storage or RAID The thing is: Big companies, even non critical are usually bonded to some frameworks/regulatory standards/etc. So, mentioning that they didnt know NAS or even RAID (which is very old, end of 80s), ma…

I've seen similar things in this decade. Lots of stuff still running where the last change in design was 20+ years ago, the current team knows how to keep the lights on and replace parts, and calls a vendor if something goes end of life and asks for a replacement part, then configures it the same way that the older one was.

It's quite reliable, and does not require paying for a team of people who wants to try new things. Once this happens for a long while there's no one in the building left who's going to advocate for change, since the only thing all that fancy new technology brings is the next 5 years of nobody getting a bonus because operational metrics weren't being met anymore.

You have to try really hard to convince these people that the thing that they have seen on a daily basis to work just fine for the past twenty years is no longer good enough, because all their evidence says that it is. And you should also listen to them much more than people who just want to use the latest thing typically do, because they do have the decades of experience seeing how things actually fail.

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#165

Post by structural » Sat, Aug 08, 2026, 8:06 PM UTC

tedd4u wrote: post #86

amluto wrote: post #63

Earlier quoted context omitted.

Sometimes I wonder whether one could design PLC-style software by first expressing the system’s constraints and then writing the actual code and having either (preferably) formal verification that the code will not violate the constraints or (less preferably) enforced assertions that will send the machine into a safe state if the constraints do not hold. The system I mentioned in my other post controlled some VFDs, a…

Seems like, here in the 21st century, one could reasonably have a model of the plant that tests could be executed against. Anything like that exist?

This is starting to be done for very high value plants (think chip fabs, large datacenters, that kind of thing. When the infrastructure is a $10b expense it's worth it).

Small plants in the middle of nowhere, especially municipal ones? You're lucky if it was designed this century and not 50-100 years ago. Everyone's just trying to keep old stuff running at absolute minimum cost.

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#166

Post by BrandoElFollito » Sat, Aug 08, 2026, 8:22 PM UTC

aliasxneo wrote: post #31

Part of my career encapsulates a period where I was a PLC programmer, installer, commissioner, and troubleshooter for massive build outs (sky scrapers, data centers, laboratories, factories). Interestingly, this was after years of teaching myself software engineering, eventually participating in large open source projects. The clash of entering the PLC world was _extremely_ harsh. Let me give an example: I once worke…

I was on the cybersecurity side of that, "advising" huge, multinational companies that hold the world together.

Oh boy.

I was told for instance once that my concerns about an open, unencrypted Wi-Fi for SCADA systems were not a big deal because "we are in the desert anyway, who would come here".

I saw terrible things and people who were made "cybersecurity expert" when they did not want to. They were sent to courses which were totally useless and the architecture was completely fucked up anyway.

I often wondered over the years why we have so little significant incidents in the utilities sector.