Part of my career encapsulates a period where I was a PLC programmer, installer, commissioner, and troubleshooter for massive build outs (sky scrapers, data centers, laboratories, factories). Interestingly, this was after years of teaching myself software engineering, eventually participating in large open source projects. The clash of entering the PLC world was _extremely_ harsh.
Let me give an example: I once worked with an integrator who was working on an AHU feeding an extremely critical portion of a datacenter (I was a lead by this point and mostly played babysitter). During certain points of the day you couldn't open the door to this room due to negative pressure because the logic was over-ramping the exhaust fans. As I watched this contractor work, I saw him open his laptop, with Windows on it (because Microsoft has had a death grip on this industry for decades now), and proceed to backup the PLC program into a massive folder with God knows how many other "customer projects" he was carrying around in this thing. He then proceeded to go do some physical checks in the field, came back, and prepared to upload the fixed program. As I watched, I noticed he _grabbed a backup from ANOTHER customer_ and I immediately had to intervene. Who knows what untold damage I saved from that single move.
I tell this story to demonstrate just how far into the dark ages this industry is. I vividly recall coming into the data center for a fortune 50 company, one everyone here would know, and being astonished that they had never heard of Network Attached Storage or RAID and why they might want to consider a disaster recovery plan for their multi-million dollar mechanical plant.
This industry is in _desperate_ need of strong technical help, but unfortunately the "higher ups" tend to be the same people who are "comfortable" with the way thing are and refuse to move. I literally tried for a decade before giving up and moving into software engineering proper.
So, anyways, just imagine the most archaic and barbaric set of IT software, controls, and procedures, dumb that down even further, and you've landed on the infrastructure/teams that operate probably half of critical infrastructure.
Everything you're describing mirrors my experience with remote scientific deployments and lab equipment (wet labs in particular). I support a team which manages dozens of these, and while it's shocking how inefficient and unsafe their practices are, it's similarly shocking how poorly the industry has kept up with or even tried to establish better practices. We get brand new equipment that costs in the realm of $10k for a tiny logger, and the thing is barely capable of remotely managed deployment of version controlled firmware. There are no tools in place to allow proper testing of the firmware unless you have the cash to buy extras and build test harnesses from them. The firmware itself is often in languages that don't have proper testing frameworks, type systems, IDEs, etc.
It's HARD to do the right thing. Dragging and dropping files into proprietary hardware management programs is the de facto standard.
Im supposed to be doing endpoint work with people working in this field amd basically have to convince compliance they'll need to be local administrators to do their job. They get onsite and dont know what app they'll need until they look at a plc model. They'll download something from a .tw ftp server and its an unsigned executable that absolutely must run as admin to program these things.
It is amazing how much of the sysadmin community just doesnt believe this is a thing you need to work with, everyone insisting its just security people being lazy and so on.
A colleague of mine transitioned from a senior IC role to manager of a PLC group in the same company. Their methods were just as unsophisticated as you describe, and now he is attempting to evangelize/impose the finer parts of software engineering discipline, such as source control and integration tests (which can be trickier when it requires a hardware test bed.) Bringing some new tools to bear can be a fun job, as long as the team trusts you.
integration tests (which can be trickier when it requires a hardware test bed.)
I use this as a fizzbuzz-type test when I'm interviewing at hardware companies: do they have development hardware in a rack with programmable power supplies and mini-PCs (or similar)? It's a low, low bar for testing, and rules surprisingly many companies.
They'll often just have The Guy running manual tests instead.
Yes, the fortune 50 company I spoke of was running some of the most advanced data centers in the world. There was literally a team of world class IT specialists walking the same corridors as the industrial automation team but they might as well have spoken different languages.
I remember a time where I was beating the drums on security and ended up in a meeting with a senior red team member in the company. This person was absolutely convinced we were not running Windows Server 2008 anywhere in the company (the year was 2019 at the time of that meeting). Needless to say, he was very concerned when I showed him the 50+ servers running it globally, all covering critical infrastructure.
I think eventually Ragnarok will happen and things will improve. I just hope it's not as detrimental as it seems setup to be.
Oh I can tell stories too! Meanwhile these guys often have huge amounts of practical experience as electricians or control technicians/engineers to go with their PLC skills, so you can't just say we should get rid of them either.
Breaking into the industrial market is tricky if you don't have connections too. And if you're hired as the PLC programmer, it's sometimes an afterthought AFTER the plant is already built. "What do you mean it'll take another month? The plant is finished, isn't it?".
Oh, and some projects ban "PC"s to begin with. Which sort of excludes any kind of PC programmer. And it sort of even makes sense. A lot of default PC behaviors (especially commercial software), are no longer user-unfriendly but potentially very expensive or even user-lethal when attached to a physical plant.
Sounds like I could learn some things from you (and maybe vice versa). Poke me on the email in my HN profile!
Yes, I don't mean to poke fun solely on the integrators. I've seen the "software first" type land in the same role and flail just as hard, if not harder. I recall one individual, who was quite proud of his "beautiful" ladder logic code, after about 3 months of being at the company ended up destroying a $100k+ chiller plant by cavitating all of the pumps because they didn't understand head pressure.
I think that's one of the core difficulties with PLC programming. You have to have strong knowledge on traditional science fields like thermal dynamics, material sciences, fluid mechanics, etc., while also understanding the limitations of a 16 bit floating point integer and why overflowing that can be catastrophic.
> So, anyways, just imagine the most archaic and barbaric set of IT software, controls, and procedures, dumb that down even further, and you've landed on the infrastructure/teams that operate probably half of critical infrastructure.
Same with SCADA: just as bad as what you describe.
We once reversed engineered Schneider Electric's ION protocol because they wanted to charge us $10k per "seat" to get software that could interact with it directly. We took home a controller and developed a Python program that could intercept and understand the RS-232 comms. We then developed our own system to interact with it. It was all unencrypted. I just did a quick search and it looks like "Secure ION" is now a thing, introduced in the year 2023. Which just proves my point.
There are many wireless pump-and-reservoir systems that while not internet connected, use insecure RF links. These local RF (and casting a wider net, Bluetooth) interfaces are also ripe for abuse.
If that means you need to at least be physically present then I'd say that's a lot of protection already. Means someone in a foreign country can't simply get lucky fuzzing.
You’d be surprised how insecure some of these facilities are, especially to someone who has working knowledge of what a PLC (or other process controllers) does and how it works. You can easily look like a tech who belongs there either troubleshooting something or working on a project.
I’ve been doing industrial controls for 15 years and surprisingly infrastructure is some of the most poorly funded. I believe a lot of these places are run by operating companies, so it’s bidded out (we all know how bids work I think). I’m not surprised when I walk into these places and see the computers are running EOL operating systems and the networking is essentially flat.
Forgive my ignorance but isn't a PLC simply a computer with some GPIO ports? I.e., a Raspberry Pi could be called a PLC? Why are we talking about them as if they are something else? Is it an exotic OS that makes them different?
Okay but then is the RF connection really you're biggest concern?
I'm kind of worried (probably stupidly) that posting ideas will get me on some list, but it seems like there would be many simpler terrorism opportunities.
You can also find all kinds of interesting water infrastructure, often with no electronics, all around town. It rarely has any sort of security beyond a padlock.
With coding agents now being used for hacking, there's a decent chance we'll see a 9/11-scale hacking incident as a result of NSA/DHS negligence in securing American internet-connected services. Similar to how the CIA's negligence allowed 9/11 itself.
The USG should be deploying thousands of security engineers armed with the latest coding models and agents, in attempt to secure systems before they're hacked. A few billion dollars spent here could save us trillions.
Unlike the CIA and/or FBI being in charge of preventing attacks like 9/11, NSA and DHS aren't in charge of each state's or city's infrastructure. They could run some opt-in program that local governments could then engage with, but "negligence" is a bit strong for systems they're not in charge of and have no access to.
We could say that about a lot of infrastructure that has been recklessly placed on the open Internet because it was cheaper than more secure solutions. I would say the same about home security systems, for instance.
No infrastructure should be on the open internet the potential for abuse is incredible. At a minimum a VPN should be used to tunnel all connections back to what ever command and control server exist, leaving equipment visible on the public IP should be a crime if not a felony.
Don't put your PLCs directly on the internet. In fact most industrial stuff is very not made to be directly connected to the internet. But interpose a firewall+VPN solution and you might be ok, if done competently.
And remote access to hardware definitely makes management and maintenance a lot easier and quicker. (else you need to drive out for every minor issue)
He is wrong and right. They should be connected to the Internet when they aren't 30 year old PLCs ripe for abuse. Until then, cut the data lines and do water monitoring the old way.
Disagree. Why connect them to the internet? They should be super hardened against attacks, and should NOT have a physical connection to the internet. Same with electrical infrastructure. Network access? Possibly, however that network should NOT be accessible from the internet.
The only exception I can think of would be for meter reading, which should be a separate, read only device with no ability to do harm altogether.
People start freaking out at the costs of dedicated fibers to every monitored facility. Hence even 'private' networks still run over the same actual lines as the internet.
The physical cost of installing definitely dominates the conversation but there is a reliability component as well. If you have a dedicated point-to-point fiber it's only one backhoe needed to ruin your day. If you have something like a tunnel over the internet, the death of one router or one link means you will probably just route around it and still be fine if you can tolerate the momentary blip in connectivity.
Rather than "on/off" I think we need to distinguish between at least four things:
1. Connected naively to the internet.
2. Behind a hardened VPN endpoint which is on the internet.
3. Has a separate physical private network.
4. Requires physical access.
I think it's obvious that #1 should be prohibited in favor of #2. After that point we need to ask what the impact is of a Denial of Service attack that prevents anyone from remotely accessing the system.
The difference between #2 and #3 may depend on whether things could be Very Bad if the system is disconnected at a time of the attacker's choosing. For example, disabling access to flood-control valves during a hurricane.
The big question for 2 is what devices have access. If it’s a bunch of employees from loosely managed general use laptops, bad. If it’s a few computers at HQ that are totally locked down and without internet access, probably ok.
A lot of water infrastructure is physically spread out. It be very expensive and cumbersome (and probably inefficient) to require staff to by physically present at each site for monitoring and making any changes.
> Keeping things up to date and holding critical infrastructure to higher standards than consumer tech is not a bad idea.
This is a great theory, but practice (over centuries now if not millennia) tells us that critical infrastructure is rarely properly maintained. "If it ain't broke, don't fix it" is the motto of governments and large organizations everywhere when it comes to proper maintenance. As opposed to improper (keep the existing thing running) maintenance, proper maintenance requires being proactive and is expensive, often requiring partial or full replacements of systems while also keeping the old system running until a hand-off time. In order to get a government or corporation to be proactive, they have to see a problem.
No problem, no worry. That it can be hacked is not a problem from their perspective. That it has been hacked might be a problem to them, but only if their constituents find out. More likely, they'll make it the poor engineer's problem, the engineer who had no budget and no staff to address it beforehand.
When it's time to cut costs, proper maintenance is one of the first places organizations look to because it's not a present problem. Then it becomes normal to not do the work, from an organizational perspective, and all those engineers and technicians are just a bunch of Cassandras.
I think you're assuming that updating software/hardware to more recent versions is sufficient to prevent a nation-state from wreaking havoc, and I'm not sure that's true. When it comes to something as critical as water infrastructure, maybe just don't connect the system to the internet on the off chance that you're wrong.
You can't have both secure infrastructure and Internet exposed infrastructure. I don't know if I'd frame it as incompetence, but it does seem firmly outside the capabilities of current engineering practice.
If you need a computer system to be actually secure, rule #0 is absolutely ensure it cannot receive unauthorized inputs of any kind (airgapped, big Faraday cage, JB Weld all the ports, big scary guys with guns, redundant locks, blast doors, etc). Otherwise you've lost against any sufficiently motivated adversary.
Right, enclose it in meters thick reinforced concrete walls and let no one near it.
While that works for Chernobyl, if you have a real world systems you might want somewhat more practical access.
Of course exposing industrial hardware directly on the internet is the other extreme, and you get what you're asking for.
Do something in between, if you even just apply normal network security you'll be ahead of the pack.
Problem is, a lot of these systems are not built by IT people. While they have a lot of quite admirable skills, it's just not their primary job, and thus they tend to lack the necessary paranoia at times.
The problem is there's no good way to actually enforce this. Every organization has their own idea of what is "good enough". The NSA has some pretty good advice[0]. But as far as I know there's no written-in-stone engineering standard organizations have to meet, just "best practices". If the building inspector finds fault with the construction of your facility, it gets evacuated and shut down until the defect is remedied. There's no inspector for your network security. That's the problem.
I sometimes wonder how much damage (and potential damage) to US infrastructure exists simply because intelligence-agencies prioritize being able to exploit it globally over fixing it on defense.
Part of my career encapsulates a period where I was a PLC programmer, installer, commissioner, and troubleshooter for massive build outs (sky scrapers, data centers, laboratories, factories). Interestingly, this was after years of teaching myself software engineering, eventually participating in large open source projects. The clash of entering the PLC world was _extremely_ harsh.
Let me give an example: I once worked with an integrator who was working on an AHU feeding an extremely critical portion of a datacenter (I was a lead by this point and mostly played babysitter). During certain points of the day you couldn't open the door to this room due to negative pressure because the logic was over-ramping the exhaust fans. As I watched this contractor work, I saw him open his laptop, with Windows on it (because Microsoft has had a death grip on this industry for decades now), and proceed to backup the PLC program into a massive folder with God knows how many other "customer projects" he was carrying around in this thing. He then proceeded to go do some physical checks in the field, came back, and prepared to upload the fixed program. As I watched, I noticed he _grabbed a backup from ANOTHER customer_ and I immediately had to intervene. Who knows what untold damage I saved from that single move.
I tell this story to demonstrate just how far into the dark ages this industry is. I vividly recall coming into the data center for a fortune 50 company, one everyone here would know, and being astonished that they had never heard of Network Attached Storage or RAID and why they might want to consider a disaster recovery plan for their multi-million dollar mechanical plant.
This industry is in _desperate_ need of strong technical help, but unfortunately the "higher ups" tend to be the same people who are "comfortable" with the way thing are and refuse to move. I literally tried for a decade before giving up and moving into software engineering proper.
So, anyways, just imagine the most archaic and barbaric set of IT software, controls, and procedures, dumb that down even further, and you've landed on the infrastructure/teams that operate probably half of critical infrastructure.
Everything you're describing mirrors my experience with remote scientific deployments and lab equipment (wet labs in particular). I support a team which manages dozens of these, and while it's shocking how inefficient and unsafe their practices are, it's similarly shocking how poorly the industry has kept up with or even tried to establish better practices. We get brand new equipment that costs in the realm of $10k for a tiny logger, and the thing is barely capable of remotely managed deployment of version controlled firmware. There are no tools in place to allow proper testing of the firmware unless you have the cash to buy extras and build test harnesses from them. The firmware itself is often in languages that don't have proper testing frameworks, type systems, IDEs, etc.
It's HARD to do the right thing. Dragging and dropping files into proprietary hardware management programs is the de facto standard.
Im supposed to be doing endpoint work with people working in this field amd basically have to convince compliance they'll need to be local administrators to do their job. They get onsite and dont know what app they'll need until they look at a plc model. They'll download something from a .tw ftp server and its an unsigned executable that absolutely must run as admin to program these things.
It is amazing how much of the sysadmin community just doesnt believe this is a thing you need to work with, everyone insisting its just security people being lazy and so on.
A colleague of mine transitioned from a senior IC role to manager of a PLC group in the same company. Their methods were just as unsophisticated as you describe, and now he is attempting to evangelize/impose the finer parts of software engineering discipline, such as source control and integration tests (which can be trickier when it requires a hardware test bed.) Bringing some new tools to bear can be a fun job, as long as the team trusts you.
They'll often just have The Guy running manual tests instead.
Yes, the fortune 50 company I spoke of was running some of the most advanced data centers in the world. There was literally a team of world class IT specialists walking the same corridors as the industrial automation team but they might as well have spoken different languages.
I remember a time where I was beating the drums on security and ended up in a meeting with a senior red team member in the company. This person was absolutely convinced we were not running Windows Server 2008 anywhere in the company (the year was 2019 at the time of that meeting). Needless to say, he was very concerned when I showed him the 50+ servers running it globally, all covering critical infrastructure.
I think eventually Ragnarok will happen and things will improve. I just hope it's not as detrimental as it seems setup to be.
Oh I can tell stories too! Meanwhile these guys often have huge amounts of practical experience as electricians or control technicians/engineers to go with their PLC skills, so you can't just say we should get rid of them either.
Breaking into the industrial market is tricky if you don't have connections too. And if you're hired as the PLC programmer, it's sometimes an afterthought AFTER the plant is already built. "What do you mean it'll take another month? The plant is finished, isn't it?".
Oh, and some projects ban "PC"s to begin with. Which sort of excludes any kind of PC programmer. And it sort of even makes sense. A lot of default PC behaviors (especially commercial software), are no longer user-unfriendly but potentially very expensive or even user-lethal when attached to a physical plant.
Sounds like I could learn some things from you (and maybe vice versa). Poke me on the email in my HN profile!
Yes, I don't mean to poke fun solely on the integrators. I've seen the "software first" type land in the same role and flail just as hard, if not harder. I recall one individual, who was quite proud of his "beautiful" ladder logic code, after about 3 months of being at the company ended up destroying a $100k+ chiller plant by cavitating all of the pumps because they didn't understand head pressure.
I think that's one of the core difficulties with PLC programming. You have to have strong knowledge on traditional science fields like thermal dynamics, material sciences, fluid mechanics, etc., while also understanding the limitations of a 16 bit floating point integer and why overflowing that can be catastrophic.
> So, anyways, just imagine the most archaic and barbaric set of IT software, controls, and procedures, dumb that down even further, and you've landed on the infrastructure/teams that operate probably half of critical infrastructure.
Same with SCADA: just as bad as what you describe.
We once reversed engineered Schneider Electric's ION protocol because they wanted to charge us $10k per "seat" to get software that could interact with it directly. We took home a controller and developed a Python program that could intercept and understand the RS-232 comms. We then developed our own system to interact with it. It was all unencrypted. I just did a quick search and it looks like "Secure ION" is now a thing, introduced in the year 2023. Which just proves my point.
There are many wireless pump-and-reservoir systems that while not internet connected, use insecure RF links. These local RF (and casting a wider net, Bluetooth) interfaces are also ripe for abuse.
If that means you need to at least be physically present then I'd say that's a lot of protection already. Means someone in a foreign country can't simply get lucky fuzzing.
Wouldn't the physical facilities themselves have security?
You’d be surprised how insecure some of these facilities are, especially to someone who has working knowledge of what a PLC (or other process controllers) does and how it works. You can easily look like a tech who belongs there either troubleshooting something or working on a project.
I’ve been doing industrial controls for 15 years and surprisingly infrastructure is some of the most poorly funded. I believe a lot of these places are run by operating companies, so it’s bidded out (we all know how bids work I think). I’m not surprised when I walk into these places and see the computers are running EOL operating systems and the networking is essentially flat.
Forgive my ignorance but isn't a PLC simply a computer with some GPIO ports? I.e., a Raspberry Pi could be called a PLC? Why are we talking about them as if they are something else? Is it an exotic OS that makes them different?
Okay but then is the RF connection really you're biggest concern?
I'm kind of worried (probably stupidly) that posting ideas will get me on some list, but it seems like there would be many simpler terrorism opportunities.
Historically people just shoot pipelines and transformers.
I assume they are talking about things like water towers that are spread around, and linked back to hq via insecure wireless.
You can also find all kinds of interesting water infrastructure, often with no electronics, all around town. It rarely has any sort of security beyond a padlock.
With coding agents now being used for hacking, there's a decent chance we'll see a 9/11-scale hacking incident as a result of NSA/DHS negligence in securing American internet-connected services. Similar to how the CIA's negligence allowed 9/11 itself.
The USG should be deploying thousands of security engineers armed with the latest coding models and agents, in attempt to secure systems before they're hacked. A few billion dollars spent here could save us trillions.
Unlike the CIA and/or FBI being in charge of preventing attacks like 9/11, NSA and DHS aren't in charge of each state's or city's infrastructure. They could run some opt-in program that local governments could then engage with, but "negligence" is a bit strong for systems they're not in charge of and have no access to.
We could say that about a lot of infrastructure that has been recklessly placed on the open Internet because it was cheaper than more secure solutions. I would say the same about home security systems, for instance.
No infrastructure should be on the open internet the potential for abuse is incredible. At a minimum a VPN should be used to tunnel all connections back to what ever command and control server exist, leaving equipment visible on the public IP should be a crime if not a felony.
Don't put your PLCs directly on the internet. In fact most industrial stuff is very not made to be directly connected to the internet. But interpose a firewall+VPN solution and you might be ok, if done competently.
And remote access to hardware definitely makes management and maintenance a lot easier and quicker. (else you need to drive out for every minor issue)
"If done competently" is a bold assumption unfortunately, not just these days but always
He is wrong and right. They should be connected to the Internet when they aren't 30 year old PLCs ripe for abuse. Until then, cut the data lines and do water monitoring the old way.
Disagree. Why connect them to the internet? They should be super hardened against attacks, and should NOT have a physical connection to the internet. Same with electrical infrastructure. Network access? Possibly, however that network should NOT be accessible from the internet.
The only exception I can think of would be for meter reading, which should be a separate, read only device with no ability to do harm altogether.
At a few of our sites, we have webcam-pointing-at-gauge. Deemed secure via air gap and simple to deploy.
That's great! Different take on the opto-isolator.
People start freaking out at the costs of dedicated fibers to every monitored facility. Hence even 'private' networks still run over the same actual lines as the internet.
The physical cost of installing definitely dominates the conversation but there is a reliability component as well. If you have a dedicated point-to-point fiber it's only one backhoe needed to ruin your day. If you have something like a tunnel over the internet, the death of one router or one link means you will probably just route around it and still be fine if you can tolerate the momentary blip in connectivity.
Rather than "on/off" I think we need to distinguish between at least four things:
1. Connected naively to the internet.
2. Behind a hardened VPN endpoint which is on the internet.
3. Has a separate physical private network.
4. Requires physical access.
I think it's obvious that #1 should be prohibited in favor of #2. After that point we need to ask what the impact is of a Denial of Service attack that prevents anyone from remotely accessing the system.
The difference between #2 and #3 may depend on whether things could be Very Bad if the system is disconnected at a time of the attacker's choosing. For example, disabling access to flood-control valves during a hurricane.
The big question for 2 is what devices have access. If it’s a bunch of employees from loosely managed general use laptops, bad. If it’s a few computers at HQ that are totally locked down and without internet access, probably ok.
If it’s connected, it’s compromised. Or will be.
Folly to think otherwise.
A lot of water infrastructure is physically spread out. It be very expensive and cumbersome (and probably inefficient) to require staff to by physically present at each site for monitoring and making any changes.
Keeping things up to date and holding critical infrastructure to higher standards than consumer tech is not a bad idea.
Taking things offline and properly airgapped can also work, but wouldn't the cost of that exceed making specialized things and maintaining them?
We got into this situation due to cost, not ignorance. Both choices are higher cost than putting ancient devices on the internet.
> Keeping things up to date and holding critical infrastructure to higher standards than consumer tech is not a bad idea.
This is a great theory, but practice (over centuries now if not millennia) tells us that critical infrastructure is rarely properly maintained. "If it ain't broke, don't fix it" is the motto of governments and large organizations everywhere when it comes to proper maintenance. As opposed to improper (keep the existing thing running) maintenance, proper maintenance requires being proactive and is expensive, often requiring partial or full replacements of systems while also keeping the old system running until a hand-off time. In order to get a government or corporation to be proactive, they have to see a problem.
No problem, no worry. That it can be hacked is not a problem from their perspective. That it has been hacked might be a problem to them, but only if their constituents find out. More likely, they'll make it the poor engineer's problem, the engineer who had no budget and no staff to address it beforehand.
When it's time to cut costs, proper maintenance is one of the first places organizations look to because it's not a present problem. Then it becomes normal to not do the work, from an organizational perspective, and all those engineers and technicians are just a bunch of Cassandras.
I think you're assuming that updating software/hardware to more recent versions is sufficient to prevent a nation-state from wreaking havoc, and I'm not sure that's true. When it comes to something as critical as water infrastructure, maybe just don't connect the system to the internet on the off chance that you're wrong.
You can't have both secure infrastructure and Internet exposed infrastructure. I don't know if I'd frame it as incompetence, but it does seem firmly outside the capabilities of current engineering practice.
If you need a computer system to be actually secure, rule #0 is absolutely ensure it cannot receive unauthorized inputs of any kind (airgapped, big Faraday cage, JB Weld all the ports, big scary guys with guns, redundant locks, blast doors, etc). Otherwise you've lost against any sufficiently motivated adversary.
Right, enclose it in meters thick reinforced concrete walls and let no one near it.
While that works for Chernobyl, if you have a real world systems you might want somewhat more practical access.
Of course exposing industrial hardware directly on the internet is the other extreme, and you get what you're asking for.
Do something in between, if you even just apply normal network security you'll be ahead of the pack.
Problem is, a lot of these systems are not built by IT people. While they have a lot of quite admirable skills, it's just not their primary job, and thus they tend to lack the necessary paranoia at times.
> normal network security
The problem is there's no good way to actually enforce this. Every organization has their own idea of what is "good enough". The NSA has some pretty good advice[0]. But as far as I know there's no written-in-stone engineering standard organizations have to meet, just "best practices". If the building inspector finds fault with the construction of your facility, it gets evacuated and shut down until the defect is remedied. There's no inspector for your network security. That's the problem.
[0] https://media.defense.gov/2022/Jun/15/2003018261/-1/-1/0/CTR...
>Other countries start securing their water
>nsa: what no, stop that
I sometimes wonder how much damage (and potential damage) to US infrastructure exists simply because intelligence-agencies prioritize being able to exploit it globally over fixing it on defense.
The NSA's job is to collect information, not screw with people's water.
you reap what you sow