Live data from Hacker News

Water system controllers don't belong on the internet, says ex-NSA chief

theregister.com

41–50 of 166 posts

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#41
post #5

There are many wireless pump-and-reservoir systems that while not internet connected, use insecure RF links. These local RF (and casting a wider net, Bluetooth) interfaces are also ripe for abuse.

If that means you need to at least be physically present then I'd say that's a lot of protection already. Means someone in a foreign country can't simply get lucky fuzzing.

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#42

Part of my career encapsulates a period where I was a PLC programmer, installer, commissioner, and troubleshooter for massive build outs (sky scrapers, data centers, laboratories, factories). Interestingly, this was after years of teaching myself software engineering, eventually participating in large open source projects. The clash of entering the PLC world was _extremely_ harsh. Let me give an example: I once worke…

Im supposed to be doing endpoint work with people working in this field amd basically have to convince compliance they'll need to be local administrators to do their job. They get onsite and dont know what app they'll need until they look at a plc model. They'll download something from a .tw ftp server and its an unsigned executable that absolutely must run as admin to program these things.

It is amazing how much of the sysadmin community just doesnt believe this is a thing you need to work with, everyone insisting its just security people being lazy and so on.

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#43

With coding agents now being used for hacking, there's a decent chance we'll see a 9/11-scale hacking incident as a result of NSA/DHS negligence in securing American internet-connected services. Similar to how the CIA's negligence allowed 9/11 itself. The USG should be deploying thousands of security engineers armed with the latest coding models and agents, in attempt to secure systems before they're hacked. A few bi…

Unlike the CIA and/or FBI being in charge of preventing attacks like 9/11, NSA and DHS aren't in charge of each state's or city's infrastructure. They could run some opt-in program that local governments could then engage with, but "negligence" is a bit strong for systems they're not in charge of and have no access to.

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#44

Part of my career encapsulates a period where I was a PLC programmer, installer, commissioner, and troubleshooter for massive build outs (sky scrapers, data centers, laboratories, factories). Interestingly, this was after years of teaching myself software engineering, eventually participating in large open source projects. The clash of entering the PLC world was _extremely_ harsh. Let me give an example: I once worke…

> So, anyways, just imagine the most archaic and barbaric set of IT software, controls, and procedures, dumb that down even further, and you've landed on the infrastructure/teams that operate probably half of critical infrastructure. Same with SCADA: just as bad as what you describe.

We once reversed engineered Schneider Electric's ION protocol because they wanted to charge us $10k per "seat" to get software that could interact with it directly. We took home a controller and developed a Python program that could intercept and understand the RS-232 comms. We then developed our own system to interact with it. It was all unencrypted. I just did a quick search and it looks like "Secure ION" is now a thing, introduced in the year 2023. Which just proves my point.

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#45

We could say that about a lot of infrastructure that has been recklessly placed on the open Internet because it was cheaper than more secure solutions. I would say the same about home security systems, for instance.

No infrastructure should be on the open internet the potential for abuse is incredible. At a minimum a VPN should be used to tunnel all connections back to what ever command and control server exist, leaving equipment visible on the public IP should be a crime if not a felony.

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#46

Part of my career encapsulates a period where I was a PLC programmer, installer, commissioner, and troubleshooter for massive build outs (sky scrapers, data centers, laboratories, factories). Interestingly, this was after years of teaching myself software engineering, eventually participating in large open source projects. The clash of entering the PLC world was _extremely_ harsh. Let me give an example: I once worke…

A colleague of mine transitioned from a senior IC role to manager of a PLC group in the same company. Their methods were just as unsophisticated as you describe, and now he is attempting to evangelize/impose the finer parts of software engineering discipline, such as source control and integration tests (which can be trickier when it requires a hardware test bed.) Bringing some new tools to bear can be a fun job, as…

    integration tests (which can be trickier when it requires a hardware test bed.)
I use this as a fizzbuzz-type test when I'm interviewing at hardware companies: do they have development hardware in a rack with programmable power supplies and mini-PCs (or similar)? It's a low, low bar for testing, and rules surprisingly many companies.

They'll often just have The Guy running manual tests instead.

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#47

Part of my career encapsulates a period where I was a PLC programmer, installer, commissioner, and troubleshooter for massive build outs (sky scrapers, data centers, laboratories, factories). Interestingly, this was after years of teaching myself software engineering, eventually participating in large open source projects. The clash of entering the PLC world was _extremely_ harsh. Let me give an example: I once worke…

Everything you're describing mirrors my experience with remote scientific deployments and lab equipment (wet labs in particular). I support a team which manages dozens of these, and while it's shocking how inefficient and unsafe their practices are, it's similarly shocking how poorly the industry has kept up with or even tried to establish better practices. We get brand new equipment that costs in the realm of $10k for a tiny logger, and the thing is barely capable of remotely managed deployment of version controlled firmware. There are no tools in place to allow proper testing of the firmware unless you have the cash to buy extras and build test harnesses from them. The firmware itself is often in languages that don't have proper testing frameworks, type systems, IDEs, etc.

It's HARD to do the right thing. Dragging and dropping files into proprietary hardware management programs is the de facto standard.

Then you layer on top the unfortunate reality that sometimes electricity does weird stuff, people design weird circuits or wire the wrong components in, and firmware tends to have to deal with non-deterministic inputs a lot more often than, say, an API on the web. It's rough.

The pay is also so much worse in my experience.

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#48
post #24
post #4

Earlier quoted context omitted.

If it’s connected, it’s compromised. Or will be. Folly to think otherwise.

A lot of water infrastructure is physically spread out. It be very expensive and cumbersome (and probably inefficient) to require staff to by physically present at each site for monitoring and making any changes.

That’s fine. It’s compromised. Happy with that?

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#49

Earlier quoted context omitted.

Keeping things up to date and holding critical infrastructure to higher standards than consumer tech is not a bad idea. Taking things offline and properly airgapped can also work, but wouldn't the cost of that exceed making specialized things and maintaining them? We got into this situation due to cost, not ignorance. Both choices are higher cost than putting ancient devices on the internet.

I think you're assuming that updating software/hardware to more recent versions is sufficient to prevent a nation-state from wreaking havoc, and I'm not sure that's true. When it comes to something as critical as water infrastructure, maybe just don't connect the system to the internet on the off chance that you're wrong.

In the case of a nation state actor, the most recent update could well be the attack vector.

Re: Water system controllers don't belong on the internet, says ex-NSA chief

#50
post #14

>Other countries start securing their water >nsa: what no, stop that

I sometimes wonder how much damage (and potential damage) to US infrastructure exists simply because intelligence-agencies prioritize being able to exploit it globally over fixing it on defense.

I sometimes think about how much the CIA risked setting back global health when using a fake vaccination program to find Bin Laden.
Post reply on HN