I’ve mentioned this before, but at my previous employer we set up staged Artifactory so tha production couldn’t pull from anything that hadn’t been through the test stage, and test couldn’t pull from anything that hadn’t been through CI.
Because releases were relatively slow (weekly) compared to other places I worked (continuous), we had a reasonable lead time to have third party packages scanned for vulns before they made it to production.
The setup was very minimal, really just a script to link one stage’s artifacts to the next stage’s repo. But the end effect was production never pulled from the internet and never pulled packages that hadn’t been deployed to the previous stage.
>> But the end effect was production never pulled from the internet
Having production ever pull from the interwebs just seems bonkers to me. Even if you (for some reason?) want to stay up-to-date on every single dependency release you should be pulling to your own repo with some sort of gated workflow. If you're doing continuous deployment you definitely want to put extra control around your external dependencies, and releasing your product quickly after they change is probably the rare exception.
Reminds me of my favorite setup for a React project. We had rotating deployed builds (like three or more of them) for pull requests, the React UI pointed to a specific system so you always had consistent data, devs would immediately be able to test the UI, QA could follow immediately after.
Bun added `trustedDependencies` [1] to package.json and only executes postInstall scripts coming from these dependencies. I think this is something that should be supported across all JS package managers, even more than version cooldowns.
I'm a little bit amused by the inclusion of .NET in this article.
The last time I pulled in more than Dapper I was using .NET Framework 4.8. Batteries are very included now. Perhaps a cooldown on dapper and maybe two other things would protect me to some degree, but when you have 3rd party dependencies you can literally count on one hand, it's hard to lose track of this stuff over time. I'd notice any dependency upgrade like a flashing neon sign because it happens so rarely. It's a high signal event. I've got a lot of time to audit them when they occur.
I think there's a lot more than you might initially realize. A few of the top of my head (beyond your ORM): automapper, 3rd-part json or Polly, logging, server-side validation, many more. Another vector: unlikely a lot of other languages I've found way more .net libraries for 3rd party connectors or systems are community-based.
.NET definitely includes more these days, including lots of the things I've mentioned above, but they're often not as good and you likely have legacy dependencies.
It's a fine balancing act between getting the latest updates and avoiding supply chain attacks.
I completely understand the author here, because I'm actually also leaning more towards avoiding supply chain attacks than jumping on the latest CVEs.
It's just a gut feeling, rooted in 25 years of experience as a sysadmin, but I feel like a supply chain attack can do a lot more damage in general than most unpatched known vulnerabilities.
Just based on my own personal experiences, no real data.
I'll try to put words to it, but a supply chain attack is more focused, higher chance of infilitration. While a CVE very rarely is exploited en masse, and exploitation often comes with many caveats.
That combined with the current state of the world, where supply chain attacks seem to be a very high profile target for state actors.
and those rare zero-days can be treated as the exception, and dealt with quickly. It seems backwards to optimize for dependency change reaction time these days with the supply chain such an attractive target.
This is security through obscurity (which is to say: it's not security). Malware already waits dormant for years in many different attack vectors. There are known, simple fixes for the attacks on package managers. They need to implement those fixes, not ineffective hacks.
- Could you explain what you mean by "security through obscurity"? The mechanism is well explained in the blog.yossarian.net posts linked within. It is simply adding a time filter on a client.
- Also, I'm not sure if package registries (e.g. server) and package managers (e.g. client) are being conflated here regarding "attacks on package managers", this seems to be more of a mitigation a client could do when the upstream content in a registry is compromised.
- Lastly, I agree with the sentiment that this is not a full solution. But I think it can be useful nevertheless, a la Swiss Cheese Safety Model. [1]
It is too bad pkgsrc from NetBSD did not become a thing, a few Linux distros allow for its use and I think Minux also uses it.
Using pkgin(1) you get to know what happens before anything is done. Creating packages is hard, but as an end user pkgsrc shows me what it will do before it does anything.
This example for installing a binary package is from an already active NetBSD workstation, all items needed will be shown for the package you want to install.
# pkgin install gnumeric-1.12.59nb2
calculating dependencies...done.
4 packages to install:
gnumeric-1.12.59nb2 goffice0.10-0.10.59nb2 lasem-0.6.0nb1 libgsf-1.14.54
0 to remove, 0 to refresh, 0 to upgrade, 4 to install
16M to download, 76M of additional disk space will be used
proceed ? [Y/n] n
Has any notable package manager tried the staggered rollout model that approximately every one of us uses when deploying changes at our day jobs? The complexity is, of course, version compatibility/mismatch, but it seems like a solvable problem. Then you could have an automatic canary system.
The closest I've seen to this are opt-in early release channels.
I think the premise is that modern scanners are really good at finding malicious code (and are run by dozens of companies in the industry), but when it gets pushed and installed inside of that 7 day window, the spread is uncontrolled. This basically gives you opportunity to let the machinery in the package ecosystem do it's job.
I've been working on a project lately as my bachelor's dissertation which I later plan on working on long term on this issue.
The basic premise is a secure package registry as an alternative to NPM/PyPi/etc where we use a bunch of different methods to try to minimize risk. So e.g. reproducible builds, tracing execution and finding behavioral differences between release and source, historical behavioral anomalies, behavioral differences with baseline safe package, etc. And then rather than having to install any client side software, just do a `npm config set registry https://reg.example.com/api/packages/secure/npm/`
eBPF traces of high level behavior like network requests & file accesses should catch the most basic mass supply chain attacks like Shai Hulud. The more difficult one is xz-utils style attacks where it's a subtle backdoor. That requires tests that we can run reproducibly across versions & tracing exact behavior.
Hopefully by automating as much as possible, we can make this generally accessible rather than expensive enterprise-only like most security products (really annoys me). Still definitely need a layer of human reviews for anything it flags though since a false positive might as well be defamation.
Won't know if this is the right direction until things are done & we can benchmark against actual case studies, but at least one startup accelerator is interested in funding.
If everyone only starts using a package after 7d, it feels like it just means we don't find out about problematic packages until 7d later . The reason 7d works is because it is the "don't go first" effect (I assert with no evidence). But if everyone does the same delay, there's no longer a benefit to you delaying. This feels like a prisoners dilemma of no one upgrading.
I do think there is some sense in having some cool down. Automated review systems having some time to sound alarms would be good.
I'm not sure what the reporting mechanisms look like for various ecosystems. Being able to declare that there should be a hold is Serious Business, and going through with a hold or removal is a very human-costly decision to make for repo maintainers. With signficiant lag. So we are up to at least 2d, if centralized.
Ideally I'd like to see something on atprotocol, where individuals can create records on their PDS's that declare dangers. This can form a reputation system, that disincentivizes bad actors (false reports), and which can let anyone on the net quickly see incoming dangers, in a distributed fashion, real time.
IIUC the recent high-profile npm backdoors were mostly detected by supply-chain-security firms that ingest all package updates from the registry and look for suspicious code using automated or semi-automated analysis. Dependency cooldowns work great with this kind of thing. I agree that, if malicious packages were mostly detected via user reports, dependency cooldowns would create a prisoners' dilemma.
I don't understand what you're saying about reporting mechanisms; is there something wrong with how this is currently done?
This is theater. I say this every time it comes up, but the solution here IS NOT to just add vain and silly friction between potentially-malicious upstreams and package users. You'll never win that war.
The solution is independent audit. "Package managers" need to be (as they are in the Linux world) human beings responsible for integrating, validating and testing the upstream software for the benefit of their users.
NPM, PyPI, Cargo et. al. continue to think they can short circuit that process and still ship safe software, and they verifiably cannot.
This is silly. Critical security and bug fixes come out and you are going to wait because you think older must be safer just to avoid a supply chain attack issue? Just secure the supply chain. Be critical about your dependencies and update strategy before updating. If you got some 200 translative dependencies and you don't know everything your build, that is a problem and you probably should look into that because assuming waiting is a solution is not going to stop you from getting hurt with that risk of a surface area.
In the age of AI, I reduced my load on small utility libraries and just have the bigger ones that I'll follow semver and update to manager versions when it make sense and always take small patches but still look at the release notes for what changed.
Surprised not to see nix mentioned in connection with this topic!
If you use nix (especially nix flakes), this consideration falls out naturally from the nixpkgs repository reference (SHA, branch, etc) you choose to track. Nixpkgs has various branches for various appetites for the "cutting edge".
Yes! Nixpkgs straddles both worlds, like a system package manager it provides a way to install packages and their dependencies. However, like most language package managers it also imposes a locking mechanism so that every input into the nix expression* is locked to a hash, the mechanism is recursive and handles multiple versions of the same package in parallel.
The recent(ish) concept of "nix flakes" means there are 2 related but different mechanisms for achieving this but the end result is the same.
* In the land of NixOS everything is a nix expression including the system packages, configuration files, install image. It's all locked to hashes of upstream sources and is in theory, fully byte-identical reproducible.
Yeah, definitely. Think of it as what the post calls a "system" package manager. The difference between nix and the SPMs mentioned by the post is that in the former case, control over dependencies lies with you, not with the package manager.
In other words, with nix you decide the spec of the software you want installed on your machine, not the maintainer of your chosen package manager. Depending on your use case and knowledge/experience level, either choice may be preferable.
Also, nixpkgs is definitively the widest-spanning "package manager" on the "market"; see link.
I’ve mentioned this before, but at my previous employer we set up staged Artifactory so tha production couldn’t pull from anything that hadn’t been through the test stage, and test couldn’t pull from anything that hadn’t been through CI.
Because releases were relatively slow (weekly) compared to other places I worked (continuous), we had a reasonable lead time to have third party packages scanned for vulns before they made it to production.
The setup was very minimal, really just a script to link one stage’s artifacts to the next stage’s repo. But the end effect was production never pulled from the internet and never pulled packages that hadn’t been deployed to the previous stage.
>> But the end effect was production never pulled from the internet
Having production ever pull from the interwebs just seems bonkers to me. Even if you (for some reason?) want to stay up-to-date on every single dependency release you should be pulling to your own repo with some sort of gated workflow. If you're doing continuous deployment you definitely want to put extra control around your external dependencies, and releasing your product quickly after they change is probably the rare exception.
Reminds me of my favorite setup for a React project. We had rotating deployed builds (like three or more of them) for pull requests, the React UI pointed to a specific system so you always had consistent data, devs would immediately be able to test the UI, QA could follow immediately after.
Bun added `trustedDependencies` [1] to package.json and only executes postInstall scripts coming from these dependencies. I think this is something that should be supported across all JS package managers, even more than version cooldowns.
[1]: https://bun.com/docs/guides/install/trusted
Can you help me understand why one would ever need a post-install script in the first place, please?
Ime the most reasonable case is an optional compilation of native components when prebuilt ones are not compatible. See also node-gyp
I'm a little bit amused by the inclusion of .NET in this article.
The last time I pulled in more than Dapper I was using .NET Framework 4.8. Batteries are very included now. Perhaps a cooldown on dapper and maybe two other things would protect me to some degree, but when you have 3rd party dependencies you can literally count on one hand, it's hard to lose track of this stuff over time. I'd notice any dependency upgrade like a flashing neon sign because it happens so rarely. It's a high signal event. I've got a lot of time to audit them when they occur.
I think there's a lot more than you might initially realize. A few of the top of my head (beyond your ORM): automapper, 3rd-part json or Polly, logging, server-side validation, many more. Another vector: unlikely a lot of other languages I've found way more .net libraries for 3rd party connectors or systems are community-based.
.NET definitely includes more these days, including lots of the things I've mentioned above, but they're often not as good and you likely have legacy dependencies.
It's a fine balancing act between getting the latest updates and avoiding supply chain attacks.
I completely understand the author here, because I'm actually also leaning more towards avoiding supply chain attacks than jumping on the latest CVEs.
It's just a gut feeling, rooted in 25 years of experience as a sysadmin, but I feel like a supply chain attack can do a lot more damage in general than most unpatched known vulnerabilities.
Just based on my own personal experiences, no real data.
I'll try to put words to it, but a supply chain attack is more focused, higher chance of infilitration. While a CVE very rarely is exploited en masse, and exploitation often comes with many caveats.
That combined with the current state of the world, where supply chain attacks seem to be a very high profile target for state actors.
and those rare zero-days can be treated as the exception, and dealt with quickly. It seems backwards to optimize for dependency change reaction time these days with the supply chain such an attractive target.
This is security through obscurity (which is to say: it's not security). Malware already waits dormant for years in many different attack vectors. There are known, simple fixes for the attacks on package managers. They need to implement those fixes, not ineffective hacks.
A few points/qs:
- Could you explain what you mean by "security through obscurity"? The mechanism is well explained in the blog.yossarian.net posts linked within. It is simply adding a time filter on a client.
- Also, I'm not sure if package registries (e.g. server) and package managers (e.g. client) are being conflated here regarding "attacks on package managers", this seems to be more of a mitigation a client could do when the upstream content in a registry is compromised.
- Lastly, I agree with the sentiment that this is not a full solution. But I think it can be useful nevertheless, a la Swiss Cheese Safety Model. [1]
[1]https://en.wikipedia.org/wiki/Swiss_cheese_model
It is too bad pkgsrc from NetBSD did not become a thing, a few Linux distros allow for its use and I think Minux also uses it.
Using pkgin(1) you get to know what happens before anything is done. Creating packages is hard, but as an end user pkgsrc shows me what it will do before it does anything.
This example for installing a binary package is from an already active NetBSD workstation, all items needed will be shown for the package you want to install.
Has any notable package manager tried the staggered rollout model that approximately every one of us uses when deploying changes at our day jobs? The complexity is, of course, version compatibility/mismatch, but it seems like a solvable problem. Then you could have an automatic canary system.
The closest I've seen to this are opt-in early release channels.
Does the timer start at upload, or does the timer start at mass exploitation?
Not a bad idea, but we'd need to have evidence of the former case to make it mandatory and widespread.
IIRC, many times it's compromised credentials of maintainers, which are caught very quickly, so that's evidence of the former case.
I think the premise is that modern scanners are really good at finding malicious code (and are run by dozens of companies in the industry), but when it gets pushed and installed inside of that 7 day window, the spread is uncontrolled. This basically gives you opportunity to let the machinery in the package ecosystem do it's job.
I've been working on a project lately as my bachelor's dissertation which I later plan on working on long term on this issue.
The basic premise is a secure package registry as an alternative to NPM/PyPi/etc where we use a bunch of different methods to try to minimize risk. So e.g. reproducible builds, tracing execution and finding behavioral differences between release and source, historical behavioral anomalies, behavioral differences with baseline safe package, etc. And then rather than having to install any client side software, just do a `npm config set registry https://reg.example.com/api/packages/secure/npm/`
eBPF traces of high level behavior like network requests & file accesses should catch the most basic mass supply chain attacks like Shai Hulud. The more difficult one is xz-utils style attacks where it's a subtle backdoor. That requires tests that we can run reproducibly across versions & tracing exact behavior.
Hopefully by automating as much as possible, we can make this generally accessible rather than expensive enterprise-only like most security products (really annoys me). Still definitely need a layer of human reviews for anything it flags though since a false positive might as well be defamation.
Won't know if this is the right direction until things are done & we can benchmark against actual case studies, but at least one startup accelerator is interested in funding.
If everyone only starts using a package after 7d, it feels like it just means we don't find out about problematic packages until 7d later . The reason 7d works is because it is the "don't go first" effect (I assert with no evidence). But if everyone does the same delay, there's no longer a benefit to you delaying. This feels like a prisoners dilemma of no one upgrading.
I do think there is some sense in having some cool down. Automated review systems having some time to sound alarms would be good.
I'm not sure what the reporting mechanisms look like for various ecosystems. Being able to declare that there should be a hold is Serious Business, and going through with a hold or removal is a very human-costly decision to make for repo maintainers. With signficiant lag. So we are up to at least 2d, if centralized.
Ideally I'd like to see something on atprotocol, where individuals can create records on their PDS's that declare dangers. This can form a reputation system, that disincentivizes bad actors (false reports), and which can let anyone on the net quickly see incoming dangers, in a distributed fashion, real time.
(Hire me, I'll build it.)
IIUC the recent high-profile npm backdoors were mostly detected by supply-chain-security firms that ingest all package updates from the registry and look for suspicious code using automated or semi-automated analysis. Dependency cooldowns work great with this kind of thing. I agree that, if malicious packages were mostly detected via user reports, dependency cooldowns would create a prisoners' dilemma.
I don't understand what you're saying about reporting mechanisms; is there something wrong with how this is currently done?
This seems like it would cause the https://xkcd.com/989/ effect.
This is theater. I say this every time it comes up, but the solution here IS NOT to just add vain and silly friction between potentially-malicious upstreams and package users. You'll never win that war.
The solution is independent audit. "Package managers" need to be (as they are in the Linux world) human beings responsible for integrating, validating and testing the upstream software for the benefit of their users.
NPM, PyPI, Cargo et. al. continue to think they can short circuit that process and still ship safe software, and they verifiably cannot.
This is silly. Critical security and bug fixes come out and you are going to wait because you think older must be safer just to avoid a supply chain attack issue? Just secure the supply chain. Be critical about your dependencies and update strategy before updating. If you got some 200 translative dependencies and you don't know everything your build, that is a problem and you probably should look into that because assuming waiting is a solution is not going to stop you from getting hurt with that risk of a surface area.
In the age of AI, I reduced my load on small utility libraries and just have the bigger ones that I'll follow semver and update to manager versions when it make sense and always take small patches but still look at the release notes for what changed.
Surprised not to see nix mentioned in connection with this topic!
If you use nix (especially nix flakes), this consideration falls out naturally from the nixpkgs repository reference (SHA, branch, etc) you choose to track. Nixpkgs has various branches for various appetites for the "cutting edge".
Can you explain how this works? Is it different from what's described in the "Language vs. system package managers" section of the post?
Yes! Nixpkgs straddles both worlds, like a system package manager it provides a way to install packages and their dependencies. However, like most language package managers it also imposes a locking mechanism so that every input into the nix expression* is locked to a hash, the mechanism is recursive and handles multiple versions of the same package in parallel.
The recent(ish) concept of "nix flakes" means there are 2 related but different mechanisms for achieving this but the end result is the same.
* In the land of NixOS everything is a nix expression including the system packages, configuration files, install image. It's all locked to hashes of upstream sources and is in theory, fully byte-identical reproducible.
Yeah, definitely. Think of it as what the post calls a "system" package manager. The difference between nix and the SPMs mentioned by the post is that in the former case, control over dependencies lies with you, not with the package manager.
In other words, with nix you decide the spec of the software you want installed on your machine, not the maintainer of your chosen package manager. Depending on your use case and knowledge/experience level, either choice may be preferable.
Also, nixpkgs is definitively the widest-spanning "package manager" on the "market"; see link.
https://en.wikipedia.org/wiki/Nix_(package_manager)#Nixpkgs