Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Water system controllers don't belong on the internet, says ex-NSA chief

theregister.com

Post reply on HN
Print view View original

61–70 of 164 posts

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#61

Post by amluto »

aliasxneo wrote: post #31

Part of my career encapsulates a period where I was a PLC programmer, installer, commissioner, and troubleshooter for massive build outs (sky scrapers, data centers, laboratories, factories). Interestingly, this was after years of teaching myself software engineering, eventually participating in large open source projects. The clash of entering the PLC world was _extremely_ harsh. Let me give an example: I once worke…

Another odd observation about the industry. For better or for worse, I inherited a little system controlled by a Fanuc PLC. It had a bug and would occasionally get stuck in a bad state. By some minor miracle, I managed to track down the person who had originally programmed it (now retired and moved out of state), and he emailed me a file and introduced me to his apprentice (who lives in a third state). I found a trial version of the design software, opened the file and found the bug (despite never having seen this style of programming before). Since I had no way to upload a new file (and since it wasn’t clear that uploading it without a matching outdated and not easily available copy of the programming environment would be wise), I convinced the apprentice to come visit while we was nearby for another client. He did not see the bug even when I pointed it out, but I convinced him to make the change and upload the modified file. Now the system works. (He had a Windows laptop with a giant folder of customer files, of course.)

While he was on site, I asked him why there was nothing on the PLC or its enclosure to identify his old boss or give any contact information, and he seemed surprised and told me that no one ever does that. I asked how a new owner is supposed to get support, and he shrugged.

If this thing ever fails, I’ll probably replace it with an Arduino or an ESP32 or something along those lines. Or I’ll just find something off the shelf to replace the entire system.

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#63

Post by amluto »

aliasxneo wrote: post #40
Kim_Bruning wrote: post #36

Earlier quoted context omitted.

Oh I can tell stories too! Meanwhile these guys often have huge amounts of practical experience as electricians or control technicians/engineers to go with their PLC skills, so you can't just say we should get rid of them either. Breaking into the industrial market is tricky if you don't have connections too. And if you're hired as the PLC programmer, it's sometimes an afterthought AFTER the plant is already built. "…

Yes, I don't mean to poke fun solely on the integrators. I've seen the "software first" type land in the same role and flail just as hard, if not harder. I recall one individual, who was quite proud of his "beautiful" ladder logic code, after about 3 months of being at the company ended up destroying a $100k+ chiller plant by cavitating all of the pumps because they didn't understand head pressure. I think that's one…

Sometimes I wonder whether one could design PLC-style software by first expressing the system’s constraints and then writing the actual code and having either (preferably) formal verification that the code will not violate the constraints or (less preferably) enforced assertions that will send the machine into a safe state if the constraints do not hold.

The system I mentioned in my other post controlled some VFDs, and one would occasionally get stuck running at minimum speed forever instead of turning all the off when it should have. Fortunately the only harm done was a stupid waste of power and and nothing was physically damaged. If it had gotten stuck at maximum speed it might have been a different story.

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#64

Post by gopher_space »

lokar wrote: post #24
grebc wrote: post #4

Earlier quoted context omitted.

If it’s connected, it’s compromised. Or will be. Folly to think otherwise.

A lot of water infrastructure is physically spread out. It be very expensive and cumbersome (and probably inefficient) to require staff to by physically present at each site for monitoring and making any changes.

I mean first you need to figure out the cost of a large municipality getting stuxnetted, and then you need to figure out where to hide when your constituents find out you made that kind of call.

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#66

Post by Kim_Bruning »

stackghost wrote: post #53
amelius wrote: post #35

Earlier quoted context omitted.

Forgive my ignorance but isn't a PLC simply a computer with some GPIO ports? I.e., a Raspberry Pi could be called a PLC? Why are we talking about them as if they are something else? Is it an exotic OS that makes them different?

If a Raspberry Pi had industrial environment ratings and was certified hard real time, then yes it could be considered a PLC. All the ones I've encountered in the wild ran VxWorks

> If a Raspberry Pi had industrial environment ratings and was certified hard real time, then yes it could be considered a PLC.

Ostensibly yes, but so far I haven't seen anyone really use a PLC in a way that requires hard real time (so far). The cycle on eg a siemens S7-1200 is anyway much too slow for anything really exciting, and a Pi might very well be more reliable in actual practice, were it not for the very unfortunate tendency to eat SD cards. :-P

(And revolution pi actually ships a hardened Pi for industrial use. So that's one way to go about it. I'm not a big fan of that brand, but it's a data-point. Meanwhile in personal experience some regular pi's left in industrial cabinets for one-off emergency monitoring purposes have managed to stay annoyingly alive over time.)

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#68

Post by mikewarot »

1970-01-01 wrote: post #3

He is wrong and right. They should be connected to the Internet when they aren't 30 year old PLCs ripe for abuse. Until then, cut the data lines and do water monitoring the old way.

A data diode can allow monitoring via the internet without risking ingress of control. Commercial units aren't cheap though.

Obviously we need open source designs.

Perhaps the easiest way would be a Raspberry pi set up with an opto isolated CGA/EGA/VGA/SVGA capture that could be viewed via the internet? (I mean, we're probably talking systems still running MS-DOS or Windows 98 running these systems)

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#69

Post by analog31 »

steve_adams_86 wrote: post #47
aliasxneo wrote: post #31

Part of my career encapsulates a period where I was a PLC programmer, installer, commissioner, and troubleshooter for massive build outs (sky scrapers, data centers, laboratories, factories). Interestingly, this was after years of teaching myself software engineering, eventually participating in large open source projects. The clash of entering the PLC world was _extremely_ harsh. Let me give an example: I once worke…

Everything you're describing mirrors my experience with remote scientific deployments and lab equipment (wet labs in particular). I support a team which manages dozens of these, and while it's shocking how inefficient and unsafe their practices are, it's similarly shocking how poorly the industry has kept up with or even tried to establish better practices. We get brand new equipment that costs in the realm of $10k f…

>>> The pay is also so much worse in my experience.

The result is that anybody who can program well enough to develop software is doing so. Those with any programming skills who remain behind are smart-but-undisciplined programmers, people who have a reason to be in a particular geographic area, too old to think about changing careers, or just plain nuts.

This is what "lack of investment in manufacturing, research, and infrastructure" looks like. Sow what you reap.

Disclosure: I might be one of those people. I came out of grad school in physics research, and programmed an entire plant. Thankfully, that was before it was economical to put every controller on the Internet.

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#70

Post by polishdude20 »

amluto wrote: post #61
aliasxneo wrote: post #31

Part of my career encapsulates a period where I was a PLC programmer, installer, commissioner, and troubleshooter for massive build outs (sky scrapers, data centers, laboratories, factories). Interestingly, this was after years of teaching myself software engineering, eventually participating in large open source projects. The clash of entering the PLC world was _extremely_ harsh. Let me give an example: I once worke…

Another odd observation about the industry. For better or for worse, I inherited a little system controlled by a Fanuc PLC. It had a bug and would occasionally get stuck in a bad state. By some minor miracle, I managed to track down the person who had originally programmed it (now retired and moved out of state), and he emailed me a file and introduced me to his apprentice (who lives in a third state). I found a tria…

The best thing they could do to new PLC's would be to have the src code live alongside the firmware inside. So any new person would be able to open it and reason about it.
Post reply on HN