Back when GitHub Actions first came out, I used commit hashes rather than tags in all my `uses:` lines. Some of my colleagues disagreed, saying that tags were secure enough. I eventually said, "Well, for well-known actions like actions/checkout, sure; if that one gets compromised it'll be all over the news within minutes." But for all the third-party actions, I kept commit hashes.
I feel rather vindicated now. There's still a small possibility of getting supply-chain attacked via a SHA collision, or a relatively much larger (though still small in absolute terms) possibility of getting supply-chain attacked via NPM dependencies of the action you're relying on.
But if you're not using a commit hash in your `uses:` lines, go switch to it now. And if you're just using major-version-only tags like `v5` then do it RIGHT now, before that action gets a compromised version uploaded with a `v5.2.3` tag.
GitHub Actions doesn't have a lock file, so your repo is still prone to transitive attacks if the SHA-locked actions you use also happen to use other composite actions by tags, which could be compromised in the future.
just noting that pinning within your own actions is not enough, you also need to ensure any composite actions do not use mutable references (for actions, docker images, etc.)
I feel pretty happy we use Renovator (EDIT: It's Renovate) at my current workplace which by default will raise PRs to change any tags for actions with the SHA instead. Then, even when it bumps the version in future PRs, it bumps the SHA (with a comment of which tag version it represents)
Glad to hear you're enjoying Renovate - I'm biased, but I agree that the SHA pinning PR updates are a very nice feature
We recently found (in Renovate) some edge cases with how tags work in GitHub Actions which was fun (https://news.ycombinator.com/item?id=47892740) and there's a few things in there Dependabot doesn't seem to support too
There is no realistic risk of a SHA collision attack. Getting supply chain attacked via NPM dependencies is much more likely. Hopefully the actions creators are also pinning their hashes.
Yup! Still haven't switched off of Github, but considering it at this point. If you're in my shoes, here's some tools we use that help:
- https://github.com/sethvargo/ratchet for pinning external Actions/Workflows to specific commit hashes
- https://www.warpbuild.com/ for much faster runners (also: runs-on/namespace/buildjet/blacksmith/depot/... take your pick)
- soon moving to Buildkite for orchestration of our workflows
I still just need a reasonable alternative for the "store our git repo, allow us to make and merge prs" part of things. Hopefully someone takes all the pieces that the Pierre team is publishing and makes this available soon. The Github UI and the `gh` cli are actually really nice and the existing alternative code storage tools are not great IMO.
Great writeup. Though combined with the lack of lockfiles for transitive actions, relying purely on static analysis is tough. Linter like zizmor are great, but they struggle with deep composite actions trees and runtime template injection.
I got frustrated with the lack of security to started working myself on an open-source runtime sandbox for GHA: https://github.com/electricapp/hasp
The first check was inspired by the trivy attack. hasp enforces SHA pinning AND checks that a comment (# v4.1.2) actaully resolves to its preceding SHA. That grew into a larger suite of checks.
Instead of just statically parsing YAML it hooks into the runner env itself. Some of its runtime checks mirror what zizmor already does including resolving upstream SHAs to canonical branches (no impostor commits) and traversing the transitive dependency tree. I have a PR up with a comparison document here (hasp vs. zizmor): https://github.com/electricapp/hasp/pull/13/changes#diff-aab...
Furthermore, it sandboxes itself to prevent sensitive exfiltration by acting as a token broker which injects the secret at runtime -- the GH token can only ever be used to call the GH API. It uses landlock, seccomp, and eBPF via Rust, so no docker. The token broker sandbox can also be used to wrap a generic executable giving hasp generic applicability beyond GHA context (i.e. agentic or other contexts, where token runtime injection seems quite in vogue)
I'm using this as a stopgap until GH rolls out some of the features on its roadmap. I'm moving torward treating the runner as a zero-trust or actively malicious environment, so this was my small contribution on that front.
I apologize in advance for the plug. I've spent the last 5 years warning of the importance of not leaving CI locked in a black box platform and proprietary DSL. All the while going on a quest to reinvent CI as an open, programmable platform. Honestly it's still a work-in-progress: it turns out that reinvention is hard! But, if you want a glimpse of what CI can be when you shed 30 years of legacy, consider checking out Dagger (https://dagger.io).
Or, if you just want to talk about the future of CI with like-minded systems engineers, without committing to using a particular product, consider joining our Discord: https://discord.com/invite/dagger-io
Programming in YAML has always seemed crazy to me. Actions seem like a great place to create a simple mixed imperative/declarative scripting language (js extension or whatever) with a solid instrumented/observable/debuggable runtime and an OO API that can be run locally against mock infrastructure.
No thanks, Jenkins has three DSL languages and none of it is good. You dont have to inline code in yaml, you can call a script and call it day, write that script in any language you want.
This should really what LLM ought to bring in terms of security. Be able to break things faster considering it is now easier for the maintainers to fix them.
This has downsides of course, moving further into the "everything rot so fast these days" trope, but we will in a adversarial world where the threat is constantly evolving.
Tomorrow (today) the servers and repo won't be scanned by scripts anymore but by increasingly capable models with knowledge about more security issues than many searchers.
Back when GitHub Actions first came out, I used commit hashes rather than tags in all my `uses:` lines. Some of my colleagues disagreed, saying that tags were secure enough. I eventually said, "Well, for well-known actions like actions/checkout, sure; if that one gets compromised it'll be all over the news within minutes." But for all the third-party actions, I kept commit hashes.
I feel rather vindicated now. There's still a small possibility of getting supply-chain attacked via a SHA collision, or a relatively much larger (though still small in absolute terms) possibility of getting supply-chain attacked via NPM dependencies of the action you're relying on.
But if you're not using a commit hash in your `uses:` lines, go switch to it now. And if you're just using major-version-only tags like `v5` then do it RIGHT now, before that action gets a compromised version uploaded with a `v5.2.3` tag.
GitHub Actions doesn't have a lock file, so your repo is still prone to transitive attacks if the SHA-locked actions you use also happen to use other composite actions by tags, which could be compromised in the future.
just noting that pinning within your own actions is not enough, you also need to ensure any composite actions do not use mutable references (for actions, docker images, etc.)
I feel pretty happy we use Renovator (EDIT: It's Renovate) at my current workplace which by default will raise PRs to change any tags for actions with the SHA instead. Then, even when it bumps the version in future PRs, it bumps the SHA (with a comment of which tag version it represents)
Glad to hear you're enjoying Renovate - I'm biased, but I agree that the SHA pinning PR updates are a very nice feature
We recently found (in Renovate) some edge cases with how tags work in GitHub Actions which was fun (https://news.ycombinator.com/item?id=47892740) and there's a few things in there Dependabot doesn't seem to support too
Is it Renovator or Renovate? I'm trying to find it to check it out...
Oops, my bad. We keep calling it Renovator internally but the name is RenovateBot or Renovate.
https://docs.renovatebot.com/
There is no realistic risk of a SHA collision attack. Getting supply chain attacked via NPM dependencies is much more likely. Hopefully the actions creators are also pinning their hashes.
Yup! Still haven't switched off of Github, but considering it at this point. If you're in my shoes, here's some tools we use that help:
- https://github.com/sethvargo/ratchet for pinning external Actions/Workflows to specific commit hashes - https://www.warpbuild.com/ for much faster runners (also: runs-on/namespace/buildjet/blacksmith/depot/... take your pick) - soon moving to Buildkite for orchestration of our workflows
I still just need a reasonable alternative for the "store our git repo, allow us to make and merge prs" part of things. Hopefully someone takes all the pieces that the Pierre team is publishing and makes this available soon. The Github UI and the `gh` cli are actually really nice and the existing alternative code storage tools are not great IMO.
Great writeup. Though combined with the lack of lockfiles for transitive actions, relying purely on static analysis is tough. Linter like zizmor are great, but they struggle with deep composite actions trees and runtime template injection.
I got frustrated with the lack of security to started working myself on an open-source runtime sandbox for GHA: https://github.com/electricapp/hasp
The first check was inspired by the trivy attack. hasp enforces SHA pinning AND checks that a comment (# v4.1.2) actaully resolves to its preceding SHA. That grew into a larger suite of checks.
Instead of just statically parsing YAML it hooks into the runner env itself. Some of its runtime checks mirror what zizmor already does including resolving upstream SHAs to canonical branches (no impostor commits) and traversing the transitive dependency tree. I have a PR up with a comparison document here (hasp vs. zizmor): https://github.com/electricapp/hasp/pull/13/changes#diff-aab...
Furthermore, it sandboxes itself to prevent sensitive exfiltration by acting as a token broker which injects the secret at runtime -- the GH token can only ever be used to call the GH API. It uses landlock, seccomp, and eBPF via Rust, so no docker. The token broker sandbox can also be used to wrap a generic executable giving hasp generic applicability beyond GHA context (i.e. agentic or other contexts, where token runtime injection seems quite in vogue)
I'm using this as a stopgap until GH rolls out some of the features on its roadmap. I'm moving torward treating the runner as a zero-trust or actively malicious environment, so this was my small contribution on that front.
I apologize in advance for the plug. I've spent the last 5 years warning of the importance of not leaving CI locked in a black box platform and proprietary DSL. All the while going on a quest to reinvent CI as an open, programmable platform. Honestly it's still a work-in-progress: it turns out that reinvention is hard! But, if you want a glimpse of what CI can be when you shed 30 years of legacy, consider checking out Dagger (https://dagger.io).
Or, if you just want to talk about the future of CI with like-minded systems engineers, without committing to using a particular product, consider joining our Discord: https://discord.com/invite/dagger-io
Programming in YAML has always seemed crazy to me. Actions seem like a great place to create a simple mixed imperative/declarative scripting language (js extension or whatever) with a solid instrumented/observable/debuggable runtime and an OO API that can be run locally against mock infrastructure.
No thanks, Jenkins has three DSL languages and none of it is good. You dont have to inline code in yaml, you can call a script and call it day, write that script in any language you want.
You can do the same in jenkins, but a bit of scripting is probably more readable in Groovy than whatever Yaml dsl.
But I totally agree that the Jenkins langs are terrible, the errors even worse, somehow they managed to make jvm backtraces even more unreadable.
This should really what LLM ought to bring in terms of security. Be able to break things faster considering it is now easier for the maintainers to fix them.
This has downsides of course, moving further into the "everything rot so fast these days" trope, but we will in a adversarial world where the threat is constantly evolving.
Tomorrow (today) the servers and repo won't be scanned by scripts anymore but by increasingly capable models with knowledge about more security issues than many searchers.
<tangent>
Github actions is running like treacle now. Even when our company pays lots of money for cloud and private Github runners.
I know its the go-to punchbag but I think enabling Copilot reviews globally for a large proportion of Github was a bit hasty.
The security problems aside, if it continues this way, people won't be able to ship and deploy code from Github actions.
We might dare I say it, have to go back to self hosted Jenkins or Travis CI.
shameless self plug, but please check out RWX! (rwx.com)
I just have a Spot instance we use for our builds. It's turned on via serverless, runs it's job with a timeout and exits.
Lately i don't use any managed services and life couldn't be any simpler.
My team has been using https://runs-on.com/ for AWS instance runners, had a few glitches but largely been great for using AWS instances for runners.
This aligns nicely with today's/current GitHub Actions outage
Github outage? Must be a Y in the day