As someone who spent over 20 years active duty, and spent a ton of my career in the IT, security, etc. side of the house:
Unlikely to be a hack, more likely to be a misconfiguration or update sent incorrectly.
That said, the timing of the disclosure and the issue are rather concerning.
Regarding the highest value targets to hit with an attack like this, you would want to target Guam, Hawai'i, and other isolated overseas locations where this would have ripple effects in the local economy. Guam specifically would cause catastrophic supply shortages, since DeCA probably supplies around 50% of the groceries on that island (that's a WAG based on my time there).
A couple years ago I worked on a service that had to communicate with a Siemens S7-1500 PLC. Based on my experience with that project, none of what I’ve read recently about unsecured industrial PLCs is surprising.
I opened Siemens TIA Portal and PLCSIM for the first time and thought “wow, I didn’t think the Windows 95 GUI library was still supported.” None of the PLC contractors we had hired knew how to enable TLS on the thing (user/pass eg admin/admin was their usual). Anecdote: I once spent hours reading the docs and clicking around trying to get it to accept an SSL certificate signed by a real CA and it wouldn’t go, but it accepted one I self-signed in openssl.
In all fairness, the people who are experts in the field of Siemens PLC programming are usually mechanical-ish engineers and security is not in their skill set or on their mind.
Most factories I know do not allow their PLC be accessed from the internet. They are usually on a separate Network. However, the "engineering" station (the computer running e.g. TIA Portal) sometimes is.
The PLC engineers I had contact with usually had an electrical engineering background. That's why they like PLCs in the first place with the ladder logic programming languages, grafcet and if they feel fancy a bit of structured text (assembly like) or structured control language (pascal like). They indeed did not know much about software security but a great deal about machine safety.
A real security nightmare are older OPC servers (OPC-DA) which is super reliant on DCOM. OPC is quite important to connect the PLCs to SCADA systems or 3rd party devices.
Isn't this the industry expectation in that kind of equipment? If it was signed by a real CA the cert. could expire and render the equipment unable to communicate.
That’s a tough question. If your PLC is on an airgapped LAN, admin/admin is not great security hygiene but you’ve reduced most of the risk by airgapping.
On my project the service I wrote was doing bidirectional communication with the PLC over OPCUA. The server running this pod was connected to the internet, so it was critical to have proper TLS for the OPCUA client/server. Rotating LetsEncrypt certs on the system every 45 days is a lot of toil, but using a self-signed cert that expires in 2040 from some dev laptop doesn’t pass muster in most organizations either.
That’s just the OPCUA path.. In these projects You also typically see WinCC HMIs that can talk to the PLC from anywhere on the network without TLS. And also SIMATIC Web Server pages - minimally secured by default, and the amount of info you can grok about the system in Chrome Dev Tools is troubling.
I'm not sure if you're speaking from personal experience, but most I've interacted with don't have to worry about the self-signed vs. LetsEncrypt debate. They just don't do it. Also there would be no way to do LetsEncrypt as the system is air gapped.
My mind was blown when I realized that the way tftp works is that as the machine is booting it asks the network if anyone has some software for it to run.
Well, what else can it do, really? It has to boot with pretty much zero knowledge about the external world (maybe except asking the user for the current date and time). Sure, you can hardcode an outdated list of CAs (it's always outdated because the system can be booted 10/20/100 years after it was made) in but that just opens you to unexpected obsolescence, and you usually can't put too much stuff in the bootloader anyway.
Not really dissimilar from the human upbringing: leave a baby with "bad" guardians, and it will grow up corrupted. That's a feature, not a bug: if you knew what behaviour exactly you wanted (other than "whatever Simon says"), you would just bake it in in the first place, right?
The author doesn't really claim it was a hack, just that it is a possibility. But they are charging down the path of the potential hack before asking the more obvious question: How many refrigerators exist in the military at all? And of those, how many are having problems?
Because a half dozen a day sounds plausible as standard maintenance issues, as the author acknowledges. If it were a hack, I'd expect something like 50% of them to have problems. But not knowing how many there are, I don't know how significant these incidents really are.
They are charging down that path because vulnerabilities that effect the refrigerators were disclosed the same day as 14 refrigerators failed in an absurd way. They all turned on the defrost cycle and heated the food.
The problem is the author should have put a few concise bullet points at the top. (14 freezers failed at the same time. They are all internet-controlled, and failed at the same time as a disclosure about a vulnerability. They all failed by turning on the defrost cycle and heating food.)
I really recommend skimming the article to the end.
(Unfortunately, the article really is so verbose it's a borderline rant.)
OTOH how many bases are effected and we didn't hear about it? Those 14 bases are only the ones we know about.
Not just any failure, specifically heating the food (defrost) so it goes bad. Happening overnight, so it wouldn't be caught before it's too late.
All that could still be a coincidence, but the more coincidences start to pile up the more we have to consider other possibilities too. I do agree it would be unusual to 'waste' a vuln like that, but perhaps the implant/CVE was about to be exposed anyway.
Not just that, the position to stick a thermometer into the food before serving was axed as DEI, and the position to clean the food prep surface areas of the kitchen is too beneath the warrior ethos. Buying above single ply is too expense and it's too heavy, so have fun with the ED (dual meaning).
And how many shipboard stores have been affected? Hardly something they’re going to talk about, and a far stronger candidate for attack. This could be spillover.
> I learned that commissaries (of which there are ~235 worldwide) aren’t actually independently operated by whatever military installation or base they happen to sit on.
according to the article, the denominator is ~235.
There are a couple hundred US armed forces bases each with commissaries that would be managed by DeCA.
An attack like the author hypothesized would require a LOTL modus operandi, and doing so on 14 locations wouldn't justify completely blowing up an entire LOTL operation, because it exposes indicators, registers, and tradecraft that is then shared amongst all security vendors.
The way it's framed is clickbait at its worst with the added issue of limited security experience, but the same can be said of HN in general.
Then I would suspect that this is either down to the common control system, or there has been a batch failure of the controllers in the freezers that were presumably ordered and supplied at the same time.
I've seen batch failures in radio equipment where I could predict 100% accurately which devices would fail based on the range of serial numbers.
Yeah I don't know why "hack" is more obvious than this. Central control pushes an update, it bugs out and cooks a dozen commissaries' frozen foods. Smart hack would be to do this randomly and fly under the radar.
I'm in the firmware bug camp too. Over/under on "the remote management server went down and a bug on all the freezers decided to put them back into some form of local control where its first action was to do a defrost cycle then put it back into offline service"?
It's also possible that, because the US is busy bombing Iran, Iran may be busy attacking the US infrastructure in any way possible? The US is also in a tepid war with russia, last I checked.
Some "cyber" crew getting a shell on an outsourced service provider and running a "defrost" command is also a totally explanation for this situation. It's also totally possible that some crew has mapped out a list of PLC entry points for various orgs and has them in a spreadsheet of "if we find a vulnerability, we should X this Y with this prestaged script that our intern / LMM cooked up last year to defrost these freezers"
This would be a bigger deal for the commissary locations outside the US, though I see none are on the list. Many of the very junior enlisted make very little money (~2400USD/month), and the low pricing at the commissary helps quite a lot. In the US, you would typically have some affordable off-base options. Overseas, it depends. Many of the locations are remote, or in places where the local groceries are significantly more expensive.
To summarize for people who TLDR: 14 freezers failed at the same time. They are all internet-controlled, and failed at the same time as a disclosure about a vulnerability. They all failed by turning on the defrost cycle and heating food.
Regardless if this was a hack or a bug, the bigger lesson is that overcomplicated systems fail in catastrophic ways. Why do military commissaries need remote-controlled freezers? It seems like a very fragile, and needless, way to run a freezer.
---
But, there are some options that the author didn't consider:
1: This could be a quickly applied patch that failed.
2: This could be a "script kiddie" hack from someone who isn't a government actor.
I'm less onboard with a state actor. Generally, when a state actor has hacked something, they don't want the victim to know. In this case, if it was a state actor, I would anticipate that they would make a single freezer fail in a way that they could verify using something like a hacked video camera or otherwise by watching public social media feeds. IMO: A state actor would only "make sense" if they knew the hole was closing soon and they don't care if they're discovered, perhaps because their operation is winding down.
These aren’t your typical refrigerator or freezer, these facilities have walk-ins or purpose built cold storage with multiple evaporators and condensing units. A building automation system is pretty standard for most buildings above a certain size, and monitoring and controlling the refrigeration is usually a part of it.
Unfortunately, I would wager that all BAS software is full of flaws and holes, allowing access to it for the public internet seems like a bad idea. I need to be on my company VPN to access our locally hosted BAS front end (which I have authorized access for) which seems like the bare minimum security.
And at some point in hindsight it will be obvious what fractions of problems were
A. This technology is inherently crap, that's our fault
B. A bored teenager broke it. Bored teenagers are a thing, it literally doesn't matter which country they are in, stop building things bored teenagers will blow up, this is also inherently our fault
C. Foreign Adversaries
It suits both mass media audience figures and a narrative of wily enemies rather than incompetence to pin everything on C and it seems eminently possible that a country with as many enemies as the US would attract this sometimes, but the reality is that both A and B are much more likely despite being embarrassing.
There's a far simpler explanation than some outside actor (either state sponsored or otherwise) deciding that the best thing they can do is to muck around with freezers.
We know there's been a severe rot of operational capabilities in the military thanks for Hegseths purges and general stupidity. It's entirely possible and quite likely that over the course of his various drunken binges he decided to get rid of people who were in charge of operational control for stuff like freezers across military bases.
I would agree, but the freezers going into high heat defrost mode seems like an intentional action from someone, whether that be incompetence or malice on the side of DeCA, or malice from a third party. If they got rid of the people commanding the freezers what to do, I feel like they'd just stay on whatever mode they were already on, rather than suddenly command all the freezers to defrost
That's the exact sort of thing the military would want us to think, because 'we were hacked by another nation' sounds a lot better than 'we fired the people responsible for food logistics'. You're not going to be able to divine the real reason this way.
Oops, you're absolutely right to call me out for that. Starting the defrost cycle without emptying the freezer first COULD lead to spoilage. The load-bearing temperature is 0 degrees Celsius — above that point, and frozen food starts to go bad.
It'll take them two weeks and then they'll find the systems were hacked half a year ago and they had industrial robots write messages on a literal chalkboard in order to share progress.
As someone who spent over 20 years active duty, and spent a ton of my career in the IT, security, etc. side of the house:
Unlikely to be a hack, more likely to be a misconfiguration or update sent incorrectly.
That said, the timing of the disclosure and the issue are rather concerning.
Regarding the highest value targets to hit with an attack like this, you would want to target Guam, Hawai'i, and other isolated overseas locations where this would have ripple effects in the local economy. Guam specifically would cause catastrophic supply shortages, since DeCA probably supplies around 50% of the groceries on that island (that's a WAG based on my time there).
A couple years ago I worked on a service that had to communicate with a Siemens S7-1500 PLC. Based on my experience with that project, none of what I’ve read recently about unsecured industrial PLCs is surprising.
I opened Siemens TIA Portal and PLCSIM for the first time and thought “wow, I didn’t think the Windows 95 GUI library was still supported.” None of the PLC contractors we had hired knew how to enable TLS on the thing (user/pass eg admin/admin was their usual). Anecdote: I once spent hours reading the docs and clicking around trying to get it to accept an SSL certificate signed by a real CA and it wouldn’t go, but it accepted one I self-signed in openssl.
In all fairness, the people who are experts in the field of Siemens PLC programming are usually mechanical-ish engineers and security is not in their skill set or on their mind.
Most factories I know do not allow their PLC be accessed from the internet. They are usually on a separate Network. However, the "engineering" station (the computer running e.g. TIA Portal) sometimes is.
The PLC engineers I had contact with usually had an electrical engineering background. That's why they like PLCs in the first place with the ladder logic programming languages, grafcet and if they feel fancy a bit of structured text (assembly like) or structured control language (pascal like). They indeed did not know much about software security but a great deal about machine safety.
A real security nightmare are older OPC servers (OPC-DA) which is super reliant on DCOM. OPC is quite important to connect the PLCs to SCADA systems or 3rd party devices.
Isn't this the industry expectation in that kind of equipment? If it was signed by a real CA the cert. could expire and render the equipment unable to communicate.
That’s a tough question. If your PLC is on an airgapped LAN, admin/admin is not great security hygiene but you’ve reduced most of the risk by airgapping.
On my project the service I wrote was doing bidirectional communication with the PLC over OPCUA. The server running this pod was connected to the internet, so it was critical to have proper TLS for the OPCUA client/server. Rotating LetsEncrypt certs on the system every 45 days is a lot of toil, but using a self-signed cert that expires in 2040 from some dev laptop doesn’t pass muster in most organizations either.
That’s just the OPCUA path.. In these projects You also typically see WinCC HMIs that can talk to the PLC from anywhere on the network without TLS. And also SIMATIC Web Server pages - minimally secured by default, and the amount of info you can grok about the system in Chrome Dev Tools is troubling.
I'm not sure if you're speaking from personal experience, but most I've interacted with don't have to worry about the self-signed vs. LetsEncrypt debate. They just don't do it. Also there would be no way to do LetsEncrypt as the system is air gapped.
> Rotating LetsEncrypt certs on the system every 45 days is a lot of toil
What is unique about your system using LetsEncrypt that you can't automate certbot to handle this task as it was designed and intended to be done?
also you can't pin the user/pw to the machine with a note, because someone might need remote access. better stick with admin/admin
My mind was blown when I realized that the way tftp works is that as the machine is booting it asks the network if anyone has some software for it to run.
Well, what else can it do, really? It has to boot with pretty much zero knowledge about the external world (maybe except asking the user for the current date and time). Sure, you can hardcode an outdated list of CAs (it's always outdated because the system can be booted 10/20/100 years after it was made) in but that just opens you to unexpected obsolescence, and you usually can't put too much stuff in the bootloader anyway.
Not really dissimilar from the human upbringing: leave a baby with "bad" guardians, and it will grow up corrupted. That's a feature, not a bug: if you knew what behaviour exactly you wanted (other than "whatever Simon says"), you would just bake it in in the first place, right?
The author doesn't really claim it was a hack, just that it is a possibility. But they are charging down the path of the potential hack before asking the more obvious question: How many refrigerators exist in the military at all? And of those, how many are having problems?
Because a half dozen a day sounds plausible as standard maintenance issues, as the author acknowledges. If it were a hack, I'd expect something like 50% of them to have problems. But not knowing how many there are, I don't know how significant these incidents really are.
They are charging down that path because vulnerabilities that effect the refrigerators were disclosed the same day as 14 refrigerators failed in an absurd way. They all turned on the defrost cycle and heated the food.
The problem is the author should have put a few concise bullet points at the top. (14 freezers failed at the same time. They are all internet-controlled, and failed at the same time as a disclosure about a vulnerability. They all failed by turning on the defrost cycle and heating food.)
I really recommend skimming the article to the end.
(Unfortunately, the article really is so verbose it's a borderline rant.)
The article has a post that says this happened across 14 bases at the same time.
So what's the denominator? Every base has some kid of refrigerator, and there must be 100s-1000s of bases.
OTOH how many bases are effected and we didn't hear about it? Those 14 bases are only the ones we know about.
Not just any failure, specifically heating the food (defrost) so it goes bad. Happening overnight, so it wouldn't be caught before it's too late.
All that could still be a coincidence, but the more coincidences start to pile up the more we have to consider other possibilities too. I do agree it would be unusual to 'waste' a vuln like that, but perhaps the implant/CVE was about to be exposed anyway.
Interesting times...
Not just that, the position to stick a thermometer into the food before serving was axed as DEI, and the position to clean the food prep surface areas of the kitchen is too beneath the warrior ethos. Buying above single ply is too expense and it's too heavy, so have fun with the ED (dual meaning).
And how many shipboard stores have been affected? Hardly something they’re going to talk about, and a far stronger candidate for attack. This could be spillover.
Oh that's interesting. what if the issues w/ toilet spillover were hacks? Hilarious.
> I learned that commissaries (of which there are ~235 worldwide) aren’t actually independently operated by whatever military installation or base they happen to sit on.
according to the article, the denominator is ~235.
If we limit ourselves to these, then that's a 0.5% known failure rate.
6%
There are a couple hundred US armed forces bases each with commissaries that would be managed by DeCA.
An attack like the author hypothesized would require a LOTL modus operandi, and doing so on 14 locations wouldn't justify completely blowing up an entire LOTL operation, because it exposes indicators, registers, and tradecraft that is then shared amongst all security vendors.
The way it's framed is clickbait at its worst with the added issue of limited security experience, but the same can be said of HN in general.
Then I would suspect that this is either down to the common control system, or there has been a batch failure of the controllers in the freezers that were presumably ordered and supplied at the same time.
I've seen batch failures in radio equipment where I could predict 100% accurately which devices would fail based on the range of serial numbers.
Obvious sabotage would be addressed promptly. Subtle sabotage could persist as a minor torment indefinitely.
Stuxnet was a good example of that.
My only question is, why would all refrigeration be under the remote control of DECA? That seems unnecessarily complicated.
Single source systems provider and integrator and a doom date?
Could be a hack or a design flaw. I await the root cause analysis.
I would suspect a firmware bug. Or a "Service Required" timer that was ignored.
Yeah I don't know why "hack" is more obvious than this. Central control pushes an update, it bugs out and cooks a dozen commissaries' frozen foods. Smart hack would be to do this randomly and fly under the radar.
I'm in the firmware bug camp too. Over/under on "the remote management server went down and a bug on all the freezers decided to put them back into some form of local control where its first action was to do a defrost cycle then put it back into offline service"?
Sure, that's possible?
It's also possible that, because the US is busy bombing Iran, Iran may be busy attacking the US infrastructure in any way possible? The US is also in a tepid war with russia, last I checked.
Some "cyber" crew getting a shell on an outsourced service provider and running a "defrost" command is also a totally explanation for this situation. It's also totally possible that some crew has mapped out a list of PLC entry points for various orgs and has them in a spreadsheet of "if we find a vulnerability, we should X this Y with this prestaged script that our intern / LMM cooked up last year to defrost these freezers"
This would be a bigger deal for the commissary locations outside the US, though I see none are on the list. Many of the very junior enlisted make very little money (~2400USD/month), and the low pricing at the commissary helps quite a lot. In the US, you would typically have some affordable off-base options. Overseas, it depends. Many of the locations are remote, or in places where the local groceries are significantly more expensive.
To summarize for people who TLDR: 14 freezers failed at the same time. They are all internet-controlled, and failed at the same time as a disclosure about a vulnerability. They all failed by turning on the defrost cycle and heating food.
Regardless if this was a hack or a bug, the bigger lesson is that overcomplicated systems fail in catastrophic ways. Why do military commissaries need remote-controlled freezers? It seems like a very fragile, and needless, way to run a freezer.
---
But, there are some options that the author didn't consider:
1: This could be a quickly applied patch that failed.
2: This could be a "script kiddie" hack from someone who isn't a government actor.
I'm less onboard with a state actor. Generally, when a state actor has hacked something, they don't want the victim to know. In this case, if it was a state actor, I would anticipate that they would make a single freezer fail in a way that they could verify using something like a hacked video camera or otherwise by watching public social media feeds. IMO: A state actor would only "make sense" if they knew the hole was closing soon and they don't care if they're discovered, perhaps because their operation is winding down.
>Generally, when a state actor has hacked something, they don't want the victim to know
Could be the Iranians, or someone aligned, conducting anti-morale operations. Could be the start of a series of small but annoying failures.
Or could be a rouge LLM in one of the big labs, that accidentally self-prompt-injected itself with the title of that vulnerability research paper.
These aren’t your typical refrigerator or freezer, these facilities have walk-ins or purpose built cold storage with multiple evaporators and condensing units. A building automation system is pretty standard for most buildings above a certain size, and monitoring and controlling the refrigeration is usually a part of it.
Unfortunately, I would wager that all BAS software is full of flaws and holes, allowing access to it for the public internet seems like a bad idea. I need to be on my company VPN to access our locally hosted BAS front end (which I have authorized access for) which seems like the bare minimum security.
Very interesting article, very neurotically written. Definitely got grating by the end.
The bold text use make me think it was largely LLM-written. Maybe even LLM-researched.
I was getting hints of that as well.
"To be very clear: I do not have evidence that the Defense Commissary Agency was hacked."
Should be much closer to the top of the article. Otherwise this is just weird and potentially dangerously wrong research.
Can you spell out the danger this blog post represents?
Would be hilarious if this was a runaway AI that someone was using to control their own IoT fridge.
> "I'm sorry I'm familiar with that function. Let me research enabling defrost for you."
The vulnerability research paper article mentions has a title that I could imagine an LLM take as an instruction - or a challenge.
Welcome to the internet of shitty unsupported and insecure crap! Are we really this dumb as a society?
As the saying goes, the S in IoT stands for security
Not exactly strong evidence presented here, but it wouldn't be a surprise either
How many people do you know that have always on microphones in their home so that they buy things from amazon or google trivia answers?
Yes
And at some point in hindsight it will be obvious what fractions of problems were
A. This technology is inherently crap, that's our fault
B. A bored teenager broke it. Bored teenagers are a thing, it literally doesn't matter which country they are in, stop building things bored teenagers will blow up, this is also inherently our fault
C. Foreign Adversaries
It suits both mass media audience figures and a narrative of wily enemies rather than incompetence to pin everything on C and it seems eminently possible that a country with as many enemies as the US would attract this sometimes, but the reality is that both A and B are much more likely despite being embarrassing.
it's not AI generated so it must be true
Gilfoyle was here. Everything that has been mocked in the Silicon Valley Show has either already happened or will happen.
There's a far simpler explanation than some outside actor (either state sponsored or otherwise) deciding that the best thing they can do is to muck around with freezers.
We know there's been a severe rot of operational capabilities in the military thanks for Hegseths purges and general stupidity. It's entirely possible and quite likely that over the course of his various drunken binges he decided to get rid of people who were in charge of operational control for stuff like freezers across military bases.
I would agree, but the freezers going into high heat defrost mode seems like an intentional action from someone, whether that be incompetence or malice on the side of DeCA, or malice from a third party. If they got rid of the people commanding the freezers what to do, I feel like they'd just stay on whatever mode they were already on, rather than suddenly command all the freezers to defrost
To be fair, if you want to mess with your adversaries troop morale, screwing up dinner is pretty effective.
These are commissary fridges, not galley fridges.
Still troops, still dinner?
General Stupidity should probably be demoted for this.
This is exactly the sort of thing that a saboteur would want its targets to think.
“If sovereign and subject are in accord, put division between them.” —Sun Tzu, The Art of War
That's the exact sort of thing the military would want us to think, because 'we were hacked by another nation' sounds a lot better than 'we fired the people responsible for food logistics'. You're not going to be able to divine the real reason this way.
There would be far better excuses here than saying “we’re incompetent.”
I'm waiting for the OpenAI report that their agents defrosted everything.
Oops, you're absolutely right to call me out for that. Starting the defrost cycle without emptying the freezer first COULD lead to spoilage. The load-bearing temperature is 0 degrees Celsius — above that point, and frozen food starts to go bad.
It'll take them two weeks and then they'll find the systems were hacked half a year ago and they had industrial robots write messages on a literal chalkboard in order to share progress.
Anthropic: “We opened the pantry door!”
Pandora's Pantry