SRE/Sysadmin/DevOps/Whatever here, while blog didn't talk about doing anything difficult but setting ENVVAR standards, I will point out all replacements are just as frustrating especially when talking about secrets.
Anything involving vaults where Application reaches out to specific secret vault like Hashicorp Vault/OpenBao/Secrets Manager quickly becomes massive vendor lock in where replacement is very difficult due to library replacement and makes vault uptime extremely important. This puts Ops in extremely difficult place when it becomes time to upgrade or do maintenance.
Config files have problem of you have secrets, how do you get them into config file since config files are generally kept in public systems? Most of the time it's some form of either "Template replacement by privileged system before handing it to application" or "Entire Config File gets loaded into secret vault and passed into application". Templating can be error prone and loading entire config files into secret manager is frustrating as well since someone could screw up the load.
Speaking of config files, since most systems are running containers, and unless you are at Ops discipline company, these config files are never in the right place, it becomes error prone for Ops to screw up the mounting. Also, whatever format you use, JSON/YAML/TOML is ripe for some weird config file bug to emerge like Norway problem in YAML.
Getting secrets from Kubernetes Secrets API I've seen done but lock in again. I'd strongly not recommend this approach unless you are designing a Kubernetes operator or other such type system.
I will say I've seen Subprocess thing bite people but I've seen less and less subprocess generation these days. Most teams go with message bus type system instead of sub processing since it's more robust and allows independent scaling.
This is why I continue to use env vars and dotenv for configuration. They are extremely simple, work well, and are compatible with secrets managers and other secrets tooling.
Though lately I've been veering into sOps the last few years. YAML is just so nice for expressing how an app should be configured, and sops makes encrypting chunks of it so easy. Dealing with GPG keys can be challenging though, which Vault/OpenBao solve, but then lock-in becomes an issue (though less so with OpenBao).
How is the kubernetes secret API lock in? Genuinely wondering - were you trying to use that deployment yaml for something other than a kubernetes deployment? For most applications, you should be mounting the secret on your application, then you can inject it as either an environment variable or a json file that your application reads in an environment agnostic way.
Then, on the backend, you can configure etcd to use whatever KMS provider you like for encryption.
Because you can't run the container, even for development outside Kubernetes.
Yes, you can mount Secrets as Volumes or Env Var in Kubernetes which is fine but I'm not talking about "How you get env var/secret" but "Methods of dealing with config."
Yes you can? The container should be completely agnostic to the fact that it's running in kubernetes. You can do config the same way. Configmaps are mounted as regular files and environment variables. The application doesn't care if the configmap came from the cluster resource or a file your created on your dev machine with dev credentials. You can mount local files into the container yourself. It's docker run -v "source:destination" I think.
One of you is talking about mapping a secret to an environment variable and the other one of you is talking about having the work load make an API call to retrieve the secret. You’re not even talking about the same thing.
sigh I’m extremely competent Ops type and I know. If you mount secrets as Volume or Env Var, that’s Config file or Env var from Application PoV. We are looking at this from Application PoV.
I’ve seen Applications that do direct calls to Kubernetes API and retrieve the secret from it. So they have custom role with bindings and service account and Kubernetes client libraries.
If you're not developing k8s operators, you're calling the api server directly, then complaining about lock in, then that's a skill issue. If you're developing k8s operators, then you should use a tool like kind for integration tests and dependency injection for other stuff and the concept of lock in doesn't make sense. You can also deploy your helm chart directly to kind.
This is where I like things like Tilt. If you're deploying to a k8s cluster, it's probably a good idea to do local dev in as close to a similar environment as possible.
Bit more of an initial hurdle than "just run the docker image"; however.
I've look at Tilt and it's another abstraction for Kubernetes which rarely ends well at scale.
However, most of time, Devs don't need to develop on Kubernetes since it's just Container Runtime and Networking Layer they don't care about. They run container, they connect to HTTP endpoint to talk to other containers, they are happy. Details are left to us Ops people.
I think parent is referring to something like SOPS [0], which can pass secrets via FIFO. That way, there’s nothing on disk, the pipe is cleared after first read, and /proc/cmdline doesn’t reveal anything.
You have a config file, it needs to have secrets so likely you are going to run some templating system where you replace dbpassword: ${dbPassword} with password from some secret system. Hopefuly you understand possible issues with any templating system that could result in replacement failures.
String manipulating is one of those "This is easy" until it's not.
I don't mean to hardcode secrets into the config file either. I was suggesting to put a command into the configuration file that the application then calls to get the secret. This way the secret is only ever passed over a file descriptor between two processes.
I do something similar. I usually have a flag, something like --password-file. It can only be used to specify a file containing the secret and at startup the application reads it.
Yes, this is also possible, but which the approach I stated, the secret can be generated by another program or received from the network, it isn't just limited to a file.
My initial Ops gut feeling says "This is as lock in and error prone as Application Vault Libraries" but if Dev wanted to propose this, I'd be willing to see it in real operation.
It isn't lock in, because all the application depends on is that it gets a string it can pass to exec/the shell and then reads all data from stdout until EOF as the secret.
You can always pass echo "SECRET" as the command, so it is a strict superset of the config file. Also programs that tend to provide a command option also tend to provide a simple string option.
Interesting read. Another interesting fact is that `setenv()` is fundamentally broken on POSIX, and should essentially never be called in library code. In application code, it should be called only in the absence of any alternative, and certainly before any threads have started.
The reason is that `getenv()` hands out raw pointers to the variables, so overwriting a variable using `setenv()` is impossible to guard against. Treat with extreme caution.
I think it's worth specifically calling out that the right way to set environment variables is in execve() only, as communication across an exec() is the precise niche for which they are the right tool.
Well yeah of course. What you could do though would be to have e.g. --secrets-file and at startup time the application reads that file to get the secret. Then you could use file permissions to make sure only the (application) user running the application can read that file (or even more extreme, the application destroys the file after reading it).
I think that would still be better than env vars, which are more likely to leak somewhere you didn't intend them to.
Sure. But practically the amount you leak is infinitesimal.
For a lot of applications that's the right call given the rest of the posix semantics you're constrained to and the kinds and frequency of data you pass via env vars.
Practically, if you're moving enough data through setenv that the memory leaked versus the steady state fluctuation of the program is at all visible, you've got much bigger problems.
It would be a problem, except that the behaviour you're moving away from is a stale pointer. So surely any application that'd be leaking under that new behaviour would be crashing today.
The same advice applies to a correct program regardless of the implementation: using setenv is a bug. Only use getenv.
Getenv in a program without setenv is fine in both implementations. Setenv is unusable with all conforming implementations.
To pass environments to children, use execve.
The Linux behavior allows a careful single threaded program to use setenv correctly. The BSD/Solaris behavior makes all usage incorrect, but the incorrectness comes in the form of a memory leak, which is preferable to a security issue, usually.
There's no correct, portable use of setenv. If you call it, it's a bug.
IIRC the Solaris implementation of getenv()/setenv() leaks memory, because they want it to still be POSIX-compatible and the POSIX API has a "obey the spec, don't leak memory, be thread-safe -- pick 2" thing going on and you can't actually have a bug-free solution that keeps compatibility with decades of Unix applications.
Arguably that's less harmful than the thread-safety issues on Linux, but there's no perfect solution here as long as POSIX stays what it is.
NetBSD has had getenv_r() for ages, [0] and I believe FreeBSD has very recently copied it. [1] If FreeBSD has it, macOS might eventually adopt it too. People tried already to get it into both glibc and POSIX, and it was rejected, but if it becomes more common on other platforms, that increases the chance they’ll both eventually accept it.
You still don’t know if some library you use calls getenv() and stores the returned pointer (or, potentially, runs in a different thread). Using getenv_r() protects your code against setenv(), but other code that may be using getenv() is still unprotected against your (or anyone else’s) use of setenv().
> You still don’t know if some library you use calls getenv()
Well, you can know, if you check their imported symbols. (I suppose they could be getting it via dlsym, but what are the odds of that...)
So if getenv_r() is added to the C library, and over time third party libraries started adopting it, you could get to the point that you could know no code in your process is calling getenv(), because none of the libraries your process loads import that symbol.
They could even add a glibc tunable to make getenv() call abort()... then you could be very sure nobody is calling getenv(), because if anyone ever did, boom
Environment variables are often used to pass secrets around. But, despite its ubiquity, I believe that's a bad practice:
- On Linux systems, any user process can inspect any other process of that same user for it's environment variables. We can argue about threat model but, especially for a developer's system, there are A LOT of processes running as the same user as the developer.
- IMO, this has become an even more pronounced problem with the popularity of running non-containerized LLM agents in the same user space as the developer's primary OS user. It's a secret exfiltration exploiter's dream.
- Environment variables are usually passed down to other spawned processes instead of being contained to the primary process which is often the only one that needs it.
- Systemd exposes unit environment variables to all system clients through DBUS and warns against using environment variables for secrets[1]. I believe this means non-root users have access to environment variables set for root-only units/services. I could be wrong, I haven't tested it yet. But if this is the case, I bet that's a HUGE surprise to many system admins.
I think ephemeral file sharing between a secret managing process (e.g. 1Password's op cli tool) and the process that needs the secrets (flask, terraform, etc.) is likely the only solution that keeps secrets out of files and also out of environment variables. This is how Systemd's credentials system works. But it's far from widely supported.
Any good solutions for passing secrets around that don't involve environment variables or regular plain text files?
Edit: I think 1Password's op client has a good start in that each new "session" requires my authorization. So I can enable that tool for a cli sessions where I need some secrets but a rogue process that just tries to use the `op` binary isn't going to get to piggyback on that authorization. I'd get a new popup. But this is only step 1. Step 2 is...how to share that secret with the process that needs it and we are now back to the discussion above.
Since at least 2012, environment variables have been at least as secure as ordinary memory:
commit b409e578d9a4ec95913e06d8fea2a33f1754ea69
Author: Cong Wang <xiyou.wangcong@gmail.com>
Date: Thu May 31 16:26:17 2012 -0700
proc: clean up /proc/<pid>/environ handling
You can't read another process's environment unless you can ptrace-read the process, and if you can ptrace-read the process you know all its secrets anyway.
Just to clarify, that means that by default every process of the same user can access the variables. But it doesn't really matter because by default every process of the same user can read any secret directly from the target process anyway, right?
And that the right thing to do if you want to harden your system is to disallow ptrace-read, and not bother changing software that uses environment variables?
Because I think most people that just try will be able to read the variables of any process on their computer.
Yes, you can consider all processes running under the same user as able to peek at each other's data. This is the point of running under the same uid: sharing data.One uid should be considered one security domain, with any separators inside it being guardrails, not brick walls.
If you want to prevent other processes from peeking into your process, run it under a different uid. Again. that's the point. A bunch of good software does that, running only small privileged bits in separate processes, and running some / bulk of the processes under an uid with severely limited privileges. See e.g. Postfix MTA, or the typical API server + reverse proxy split.
I don't think that this part of Unix is somehow problematic, or needs changing.
I think this part of Unix is exceedingly problematic. I want to be able to run program that don't have the ability to do anything that I, personally, can do. Ideally this would be doable without nasty kludges or root's help.
Linux has some ways to accomplish this, for example:
- seccomp. It can be done quite securely, but running general purpose software in seccomp is not necessarily a good way to prevent it from acting like the running user.
- Namespaces. Unless user namespaces are in use, only root can set this up. With user namespaces, anyone can do it, but the attack surface against the kernel is huge, mount namespaces are somewhat porous (infamously, /proc/self/exe is a hole), and they can be a bit resource-intensive, especially if you want to use network namespaces for anything interesting. Also, user namespaces have kind of obnoxious interactions with ordinary filesystem uids (if I'm going to run some complex containerized stack with access to a specific directory within my home directory, who should own the files in there, and how do I achieve that without root's help?). And userns without subuid doesn't isolate very strongly, and subuid need's root's help.
- Landlock. Kind of cool, kind of limited.
- Tools like Yama. On Ubuntu, with Yama enabled (the default), programs can't generally ptrace (or read /proc/self/environ) from other programs running as the same user.
In any case, once you've done something to prevent a process from accessing /proc/fd/PID for other pids belonging to the same user (e.g. pidns) and/or restricted ptrace (e.g. PR_SET_DUMPABLE), then environ gets protected.
It's not your (unprivileged user account's) place to decide the security posture of the entire system, that's why you're running into issues with root. Even Yama, an LSM, requires root (or de facto equivalent) for initial setup (as it should).
Namespaces, if done incorrectly, can significantly increase the attack surface of the entire system (mount namespaces especially need to be treated with care) and same with regards to anything to do with user accounts.
The real security barrier on most operating systems to date is the user account and if you want full isolation (modulo kernel bugs), run a process as a separate user if you're that concerned about leakage (or ideally, don't run it at all).
> It's not your (unprivileged user account's) place to decide the security posture of the entire system, that's why you're running into issues with root.
Tell that to literally any unprivileged user who would like to run any sort of software without exposing their entire account to any possible code execution bug or malicious code in the software they're running.
> The real security barrier on most operating systems to date is the user account and if you want full isolation (modulo kernel bugs), run a process as a separate user if you're that concerned about leakage (or ideally, don't run it at all).
That a very 1980s or maybe 1990s perspective, I think. It's 2025. I have some computers. They are mine. I trust myself to administer them and do pretty much anything to them, but I would prefer not to trust all my software. Sure, I can, very awkwardly, try to isolate things by using multiple uids, but it's extremely awkward and I see no technical reason why uids are a good solution here.
And there are no shortage of technical reasons why uids are a horrible solution. I would like to run unprivileged software and delegate some degree of privilege to them -- imagine a web browser or an IDE or whatever. I, as the owner of the machine, can make a uid. But I would also like my IDE to isolate various things it runs from each other and from the IDE as a whole, and it cannot do that because it does not own the machine and should absolutely not be running as root.
I've never tried to use ptrace-read to read secrets from a running process's memory so I can't comment on that part.
I had always assumed getting secrets from a running process' memory was non-trivial and/or required root permissions but maybe that's a bad assumption.
However, reading a process' environment is as trivial as:
`cat /proc/123456/environ`
as long as it's ran by the same user. No ptrace-read required.
You do need PTRACE access to pid 123456 in order to access that file. It is transparent to you, but the kernel will use the current task's PTRACE_ATTACH access when attempting to get that information.
By default, on most distributions, a user has PTRACE_ATTACH to all processes owned by it. This can be configured with ptrace_scope:
> - On Linux systems, any user process can inspect any other process of that same user for it's environment variables. We can argue about threat model but, especially for a developer's system, there are A LOT of processes running as the same user as the developer.
Here's the thing though, the security model of most operating systems means that running a process as a user is acting as that user. There are some caveats (FreeBSD has capsicum, Linux has landlock, SELinux, AppArmor, Windows has integrity labels), but in the general case, if you can convince someone to run something, the program has delegated authority to act on behalf of that user. (And some user accounts on many systems may have authority to impersonate others.)
While it's by no means the only security model (there exists fully capability based operating systems out there), it's the security model that is used for most forms of computing, for better or worse. One of the consequences of this is that you can control anything within your domain. I can kill my own processes, put them to sleep, and most importantly for this, debug them. Anything I own that has secrets can grab them my ptrace/process_vm_readv/ReadProcessMemory/etc.
I haven't seen much language support for it, though. On one part maybe because it's Linux only.
People that write in Rust (and maybe Go, depends how easy FFI is) should give it a try.
I wanted for a time to get some support for it in PHP, since wrapping a C function should be easy, but the thought of having to also modify php-fpm put a dent in that. I can't and don't want to hack on C code.
In practice it'd be great if the process manager spawn children after opening a secret file descriptor, and pass those on. Not in visible memory, not in /proc/*/environ
You should be able to build up a nice capability model to get access to those memfds from daemon too rather than having to spawn out of a process manager if that model fits your use case a bit better.
You've described the classic Unix security model, with minor improvements. While decent for the time it is showing its age. Specifically the difficulty in adapting to cheap, ubiquitous computing which it wasn't designed for.
If you need to keep secrets from other processes—don't run them under the same user account. Or access them remotely, although that brings other tradeoffs and difficulties.
> Environment variables are often used to pass secrets around. But, despite its ubiquity, I believe that's a bad practice:
I think environment variables are recommended to pass configuration parameters, and also secrets, in containerized applications managed by container orchestration systems.
By design, other processes cannot inspect what environment variables are running in a container.
Also, environment variables are passed to child processes because, by design, the goal is to run child processes in the same environment (i.e., same values) as the parent process, or with minor tweaks. Also, the process that spawns child processes is the one responsible for set it's environment variables, which means it already has at least read access to those secrets.
All in all I think all your concerns are based on specious reasoning, but I'll gladly discuss them in finer detail.
Please note that in my OP, I never mentioned containers.
Let's say, as a developer, I need to do some API interactions with GitHub. So, in a terminal, using 1Password's cli tool `op`, I load my GH API token and pass it to my Python script that is doing the API calls.
Presumably, the reason I use that process is because I want to keep that token's exposure isolated to that script and I don't want the token exposed on the filesystem in plaintext. There is no reason for every process running as my user on my laptop to have access to that token.
But, that's not how things work. The Claude agent running in a different CLI session (as an example) also now has access to my GitHub token. Or, any extension I've ever loaded in VS Code also has access. Etc.
It's better than having it in plain text on the file system but not by much.
Additionally, unless I've misunderstood the systemd docs, if you are passing secrets through environment variables using unit's `Environment` config, ANY USER on a server can read those values. So now, we don't even have user isolation in effect.
My reasoning is pretty plain. It could be wrong, but it's hardly specious.
> By design, other processes cannot inspect what environment variables are running in a container.
That’s not exactly true. If a process is running in a container, and someone is running bash outside of that container, reading that processes environment variables is as simple as “cat /proc/<pid>/environ”. If you meant that someone in one container cannot inspect the environment variables of a process running in a different container, that’s more true. That said, containers should not be considered a security boundary in the same way a hypervisor is.
I think in the context of containers you're right, there's a level of isolation and secrets are probably fine. But I think under other contexts that lack that isolation (e.g. bare-metal processes, local dev tooling) there are extra concerns.
(inb4: container env-vars are isolated from other containers, not from processes on the host system)
Point to a file?
E.g. `CONFIG_PATH=/etc/myapp/config.ini /opt/myapp`
That being said, I still use env vars and don't plan on stopping. I just haven't (yet?) seen any exploits or threat models relating to it that would keep me up at night.
Is that file more secure than the environment variables it's replacing? On Linux I think you can secure it to just your service with SELinux. Not sure about Windows
Until when? Secrets in applications in many cases (I would probably wager majority of the cases) are only useful if they're in plaintext at some point, for example if you're constructing a HTTP client or authenticating to some other remote system.
As far as high-level language constructs go, there were similarish things like SecureString (in .NET) or GuardedString (in Java), although as best as I can tell they're relatively unused mostly because the ergonomics around them make them pretty annoying to use.
Linux security model is pretty broken without namespaces. systemd has some bells and whistles that help but if you want something better than environment variables you're naturally going to reach for cgroups.
Linux/Unix security model is that different actors are represented by different OS users. When you don't do that, it's already like inviting the burglars in your living room.
Is there consensus around that? Given how containers and bubblewrap/flatpak are considered (to some degree) security boundaries, I approach statements like this with skepticism.
You answered your own question. Bubblewrap uses namespaces, but not by themselves. They're used in conjunction with other tools to provide a security boundary. Even then, it's not a very good security boundary. A serious security boundary should hold up even to privilege escalation, which is not true for bubblewrap.
Each kind of namespace provides its own kind of boundary. Yes, you need something like bubblewrap to stitch that all together. And it is kinda leaky, especially without taking some degree of care.
What're you saying about privilege escalation? I don't see how a user namespace does not prevent/limit privilege escalation.
More than that, I'm interested if there is some broader consensus I'm missing on the shortcomings of namespaces.
> I don't see how a user namespace does not prevent/limit privilege escalation.
They only prevent a single class of privilege escalation from setuid usage within the namespace. You can still obtain root using race conditions, heap overflows, side-channels, etc. or by coordinating with something outside of the namespace like a typical chroot escape.
Here's an old (patched) example of escaping the sandbox:
> More than that, I'm interested if there is some broader consensus I'm missing on the shortcomings of namespaces.
The consensus is that they're not security features on Linux. I'm not sure who sold you on the idea that they were, because that was not handed down by the kernel devs.
If I was actually serious about the security of a system, I'd explicitly use virtualization at-minimum. If all I wanted was a mall cop and was happy to take the risk of misconfiguration or attacks on the kernel I'd probably say something that adds a lot of extra sanity to namespaces is "good enough", like bubblewrap. I would never trust it or say that it's secure though.
cgroups aren't relevant here, I think. Not sure if that was just a typo, since you did mention namespaces in the first sentence. PID and user namespaces in particular are relevant here.
> Any good solutions for passing secrets around that don't involve environment variables or regular plain text files?
Plain text files are fine, it's the permissions on that file that is a problem.
The best way is to be in control of the source to the program you want to run then you can make a change to always protect against leaking secrets by starting the program as a user that can read the secrets field. After startup the program reads the whole file and immediately drops privileges by switching to a user that cannot read the secrets file.
> On Linux systems, any user process can inspect any other process of that same user for it's environment variables. We can argue about threat model but, especially for a developer's system, there are A LOT of processes running as the same user as the developer.
This is a very good point I'd never realised! I'm not sure how you get around it, though, as if that program can even find a credential and decrypt a file, if it runs as the user then everything else can go find that credential as well.
I added a note to my original post about how 1Password's cli tool work. Access to the binary doesn't mean automatic access to any authorized session. I think that's a good start. In my case, I have it tied into my fingerprint reader, so if something want's access to a secret using `op` I get a prompt to authorize. If I haven't just called `op` myself, I know it's a suspect request.
--passphrase-fd was always the correct thing to do (gpg has had it for ages), but support for that is very meager. I mean, you can’t even kubectl login using environment variables, you pretty much have to pass tokens through command-line arguments and we’ve known that’s terrible for around forty-five years.
> Any good solutions for passing secrets around that don't involve environment variables or regular plain text files?
Honestly, my answer is still systemd-creds. It's easy to use and avoids the problem that plain environment variables have. It's a few years old by now, should be available on popular distros. Although credential support for user-level systemd services was added just a few weeks ago.
A TL;DR example of systemd-creds for anyone reading this:
# Run the initial setup
systemd-creds setup
# This dir should have permissions set to 700 (rwx------).
credstore_dir=/etc/credstore.encrypted
# For user-level services:
# credstore_dir="$HOME/.config/credstore.encrypted"
# Set the secret.
secret=$(systemd-ask-password -n)
# Encrypt the secret.
# For user-level services, add `--user --uid uidhere`.
# A TPM2 chip is used for encryption by default if available.
echo "$secret" | systemd-creds encrypt \
--name mypw - "$credstore_dir/mypw.cred"
chmod 600 "$credstore_dir/mypw.cred"
The process you start in the service will then be able to read the decrypted credential from the ephemeral file `$CREDENTIALS_DIR/mypw`. The environment variable is set automatically by systemd. You can also use the command `systemd-creds cat mypw` to get the value in a shell script.
At least systemd v250 is required for this. v258 for user-level service support.
I can't recommend Varlock [1] enough. A great way to manage env vars in a project. It lets you define which ones are necessary or optional, their types, and where they should be fetched from.
One of the worst things about Environment variables among others discussed here is the implicit and opaque nature of them. Majority of applications rely on them in the *nix world. Even if more explicit and obvious ways of configuration files or remote services (consul/etcd, et al.) and command line arguments are supported env vars are traditionally supported as well.
But as mentioned in the article it is just a global hashmap that can be cloned and extended for child processes. Maybe in 1979 it was a good design decision. Today it sometimes hurts.
For example, kubernetes by default literally pollutes the container’s environment with so-called service links. And you will have fun time debugging a broken application if any of those “default” env vars conflict with the env vars that your app might expect to see.
They are ubiquitous and we are living in the world of neo-conservatism in IT where legacy corner cuts are treated as a standard and never challenged (hello /bin, /usr/bin, /lib, /usr/lib)[0]
At a past firm, I was trying to debug how a particular ENV var was getting set. I started out thinking it was something simple like the user's .bashrc or equivalent.
I quickly realized that there were roughly 10 "layers" of env var loadings happening where the first couple layers were:
- firm wide
- region
- business unit
- department
- team etc etc
I ended up having to turn on a bash debug flag so that I could see exactly where the var was getting set in the layer stack.
Not sure if any other high level languages have it, but Node.js has just added command line flags to trace precisely all env var accesses and modifications:
I get anxiety whenever I need to set an environment var on Linux. There are (somewhat distro-specific) ways that work properly, but the usual procedures you find online stops working once you reboot (or close the terminal I think?). They should add a simple env var GUI like Windows has that just works, and isn't terminal-specific. Windows has the annoyance of needing to restart the the terminal (or open a new one) for changes to take effect, but works well other than that.
> but the usual procedures you find online stops working once you reboot (or close the terminal I think?)
The environment isn't persistent between sessions. That means you need to make the change in a way that runs on every new session (login or new terminal window).
Depending on how your system is configured:
.bash_profile gets run once on every login
.bashrc gets run once on every non-login new session (i.e. a new terminal window)
It's typical, if using these files, to do something like this:
if [ -f ~/.bashrc ]; then
source ~/.bashrc
fi
in the .bash_profile file, putting most other things in .bashrc, such that you don't have to worry about the distinction.
If you're not even using bash or bash-likes at all, but instead something like Zsh, fish, etc you'll need to set things the way they want to be set there.
> They should add a simple env var GUI like Windows has that just works, and isn't terminal-specific
This doesn't exist in linux, because there isn't "one place" that all possible terminals draw from. Conceivably, it's possible to write a GUI tool that reads your .bashrc file, looks for anything that resembles a variable, parses the bash code to look for ways it is set, and then present a GUI tool that could update it in common cases, but... it's way easier to just write the change in a text editor.
That applies if you use bash. If you use another shell, you'll need to use that shell's methods to set environment variables in the config(s). And probably still need to set them for bash so that bash scripts get the correct env vars.
NixOS & home-manager do have functions that can set environment variables across all shells enabled via either of those systems. So I've got `programs.bash.enable = true;` && `programs.fish.enable = true;` && `home.sessionVariables = { EDITOR = "nvim"; };` in my home-manager config for example. A GUI editor for that would certainly be possible.
A lot of programs and tools will try to be cute and write into your bash files for you. I was mostly waxing philosophical at that point having gotten nerd sniped by "write a gui tool for environment variables" and "environment variables like $PATH in bash files" coming together in my head.
Funny, as a primarily linux user the windows behaviour irks me to no end, it's the cause of so many recurrent problems on end-user machines with so many apps that pollute environment.. and then you wonder why something doesn't work and then it turns out that for some reason you're using $SOFTWARE from c:\Perl64\bin instead of its proper place
On systemd systems you can just either set KEY=VALUE pairs in `/etc/environment` or any file in `/etc/environment.d/` (and technically a few other places [0]).
In theory it should be relatively easy to write a GUI for it by manually parsing the files.
The application restarting part can't really be fixed, since environment variables aren't ever injected to a running process and can only be changed by the process itself (terms and conditions may apply) and even changes during runtime could be ignored since the program itself may have cached some already computed value based on the variable.
static secrets that you can even extract and set in a config file (or store in a secrets manager) need to go. TPM secrets that you can't extract, and public keys that you can pin without considering them "secret", along with Oauth / IAM roles, are the way to go.
> POSIX-specified utilities use uppercase envvars, but that’s not prescriptive for your programs. Quite the contrary: you’re encouraged to use lowercase for your envvars so they don’t collide with the standard tools.
> But in reality, not many applications use lowercase. The proper etiquette in software development is to use ALL_UPPERCASE
I always prefer lower case for env variables in scripts. Thanks for pointing out that it can help reduce clashes with standard tools.
...and people have to find special ways to set them. I didn't even know that pam_env existed until a recent security vulnerability announcement. It's never come to my attention before, I don't think I've ever seen it utilized. I've now made a runbook item to disable it. It's a shame that "configuration" includes undoing the fetishes and helpfulness [0][1] of others.
pam-config -d --env
[0] Crowded elevator atrium. Multiple elevators running. Elevator wants to close, another one is coming (oh! I heard it "ding"!). Somebody is holding the elevator which wants to depart and trying to wave me in. Why doesn't somebody push them out?
[1] I'm at a stop sign. Some complete idiot is trying to turn left onto the street I'm leaving and waving me to turn left in front of them. Fuck no! I turn in front of you, somebody rearends you, you fly forward into me: my fault! You should be able to make this turn, if you can't go around the block! [2]
[2] I could go out of my way and turn right. Or I can just wait and see what happens.
Change your browser settings. You can tell the browser what font you prefer to use, and set minimum font sizes. You can also turn off remote fonts, so that pages can only use fonts you already have. Then you can just uninstall any fonts you don’t like or that you cannot read easily.
Environment variables also suck due to having zero typo existence. Write FOO_BAR instead of FOO_BARS? Silent failure. V1 of your package recognizes FOO_BAR and V2 changes it to FOO_BARS (because now we can have more than one)? Silent failure.
Won't help. I'm _running a program_ want to configure it with environment variables, not _writing a program_ that I expect the user to configure for himself.
> falsey
There are more than two programming languages in the world.
Eh, they are just more command line arguments, ones that go on the left side of the command instead of the right. I guess an alternative is something the Windows registry, but I'm not seeing that as a great improvement since it's less direct.
Command-line arguments aren't passed to subprocesseses, can't be inspected by arbitrary functions in the same process, and don't leak memory if they are changed.
Yes, I'm just highlighting that it's just as convenient to change them as command line parameters on a per-invocation basis. If there is a proposal for something else, it should retain this capability. Config files and something like the windows registry are much less convenient.
> can't be inspected by arbitrary functions in the same process
You can not really rely on that. The initial stack layout is well-defined at least on linux, so digging up argv is not difficult. Or just open /proc/self/cmdline
>"Wow, I really enjoyed writing this…
…and I hope it wasn’t a boring read."
No, it was very interesting actually!
An excellent deep-dive into the murky area of Unix/Linux environment variables, how they are set, how they are passed, what's really going on behind the scenes.
Basically a must-read for any present or future OS designer...
Observation (if I might!) -- environment variables are to programs (especially chains of programs, parent processes, sub processes and sub-sub processes, etc.) what parameters are to functions -- and/or what command-line parameters are... they're sort of like global variables that get passed around a lot...
They also can influence program behavior -- and thus the determinism of a program -- and thus the determinism of a chain of programs...
Phrased another way -- software that works on one developer's machine might not work on another developer's machine and/or in a production environment because one crucial environment variable was either set or not set, or set to the wrong value...
(NixOS seems to understand this pretty well... that "hermetically sealing" or closing or "performing a closure around" (that's probably slightly the wrong language/terminology in "Nix-speak" but bear with me!) a software environment, including the environment variables is the way to create deterministic software builds that run on any machine... but here I'm digressing...)
But my main point: I complete agree with the article's author -- environment variables are a legacy mess!
Although, if we think about it, environment variables (if we broaden the definition) are a sub-pattern of anything that affects the state of an individual machine or runtime environment -- in other words, things such as the Windows Registry, at least the global aspects of it -- are also in the same category.
Future OS's, when they offer environments for programs or chains of programs to run -- should be completely containerized -- that is, the view of the system -- what data/settings/environment variables/registry variables/files/syscalls/global variables it has access to -- should be completely determinable by the user, completely logabble, completely transparent, and completely able to be compared, one environment to another.
In this way, software that either a) fails to work at all b) works in a non-deterministic way -- can be more easily debugged/diagnosed/fixed (I'm looking at you, future AI's that will assist humans in doing this!) -- then if all of that information is in various different places, broken, and/or opaque...
To reiterate:
>"Wow, I really enjoyed writing this…
…and I hope it wasn’t a boring read."
No, I really enjoyed reading it(!), it's a brilliant article, and thank you for writing it! :-)
We're almost rid of dotfiles in $HOME. Can we please also get rid of the abuse of environment variables for configuration? There is no reason your program needs to muck up the environment table of every process by defining MY_PROGRAM_API_KEY instead of taking a command line argument or reading a configuration file. It's not "secure" just because it's not on the command line. And it will mess up for users because it's nigh impossible to ensure you have the same environment variables over all login types. The variable might be there when you run locally, but not in an ssh session or cron. Some ephemeral configuration such as LD_LIBRARY_PATH and PWD is difficult to handle without environment variables, but those are rare exceptions and not the norm.
Contribute to a clean environment! Don't use environment variables for configuration! </rant>
Depending on your system, passing secrets via environment can be more secure than passing secrets via a command line. Command line arguments of all running processes can be visible to other users of the system, environment variables are usually not.
"MY_APP_SECRET_KEY=bla myapp" hardly accomplishes anything substantial over "myapp --secret-key=blah" It's just a less robust and less well-supported command-line interface. It's not supported by most gui launchers, PowerShell, nor many cron implementations, for example.
SRE/Sysadmin/DevOps/Whatever here, while blog didn't talk about doing anything difficult but setting ENVVAR standards, I will point out all replacements are just as frustrating especially when talking about secrets.
Anything involving vaults where Application reaches out to specific secret vault like Hashicorp Vault/OpenBao/Secrets Manager quickly becomes massive vendor lock in where replacement is very difficult due to library replacement and makes vault uptime extremely important. This puts Ops in extremely difficult place when it becomes time to upgrade or do maintenance.
Config files have problem of you have secrets, how do you get them into config file since config files are generally kept in public systems? Most of the time it's some form of either "Template replacement by privileged system before handing it to application" or "Entire Config File gets loaded into secret vault and passed into application". Templating can be error prone and loading entire config files into secret manager is frustrating as well since someone could screw up the load.
Speaking of config files, since most systems are running containers, and unless you are at Ops discipline company, these config files are never in the right place, it becomes error prone for Ops to screw up the mounting. Also, whatever format you use, JSON/YAML/TOML is ripe for some weird config file bug to emerge like Norway problem in YAML.
Getting secrets from Kubernetes Secrets API I've seen done but lock in again. I'd strongly not recommend this approach unless you are designing a Kubernetes operator or other such type system.
I will say I've seen Subprocess thing bite people but I've seen less and less subprocess generation these days. Most teams go with message bus type system instead of sub processing since it's more robust and allows independent scaling.
+1 to all of this.
This is why I continue to use env vars and dotenv for configuration. They are extremely simple, work well, and are compatible with secrets managers and other secrets tooling.
Though lately I've been veering into sOps the last few years. YAML is just so nice for expressing how an app should be configured, and sops makes encrypting chunks of it so easy. Dealing with GPG keys can be challenging though, which Vault/OpenBao solve, but then lock-in becomes an issue (though less so with OpenBao).
How is the kubernetes secret API lock in? Genuinely wondering - were you trying to use that deployment yaml for something other than a kubernetes deployment? For most applications, you should be mounting the secret on your application, then you can inject it as either an environment variable or a json file that your application reads in an environment agnostic way.
Then, on the backend, you can configure etcd to use whatever KMS provider you like for encryption.
Because you can't run the container, even for development outside Kubernetes.
Yes, you can mount Secrets as Volumes or Env Var in Kubernetes which is fine but I'm not talking about "How you get env var/secret" but "Methods of dealing with config."
Yes you can? The container should be completely agnostic to the fact that it's running in kubernetes. You can do config the same way. Configmaps are mounted as regular files and environment variables. The application doesn't care if the configmap came from the cluster resource or a file your created on your dev machine with dev credentials. You can mount local files into the container yourself. It's docker run -v "source:destination" I think.
One of you is talking about mapping a secret to an environment variable and the other one of you is talking about having the work load make an API call to retrieve the secret. You’re not even talking about the same thing.
sigh I’m extremely competent Ops type and I know. If you mount secrets as Volume or Env Var, that’s Config file or Env var from Application PoV. We are looking at this from Application PoV.
I’ve seen Applications that do direct calls to Kubernetes API and retrieve the secret from it. So they have custom role with bindings and service account and Kubernetes client libraries.
If you're not developing k8s operators, you're calling the api server directly, then complaining about lock in, then that's a skill issue. If you're developing k8s operators, then you should use a tool like kind for integration tests and dependency injection for other stuff and the concept of lock in doesn't make sense. You can also deploy your helm chart directly to kind.
Don’t use live system secrets and credentials when running your application locally. Then you don’t need to access the same secrets.
Keep it simple and design your applications so they’re agnostic to that fact.
It’s really not that hard, I’ve been doing this for at least 6 or 7 years. A little bit of good engineering goes a long way!
This is where I like things like Tilt. If you're deploying to a k8s cluster, it's probably a good idea to do local dev in as close to a similar environment as possible.
Bit more of an initial hurdle than "just run the docker image"; however.
I've look at Tilt and it's another abstraction for Kubernetes which rarely ends well at scale.
However, most of time, Devs don't need to develop on Kubernetes since it's just Container Runtime and Networking Layer they don't care about. They run container, they connect to HTTP endpoint to talk to other containers, they are happy. Details are left to us Ops people.
You can have a command setting that is invoked to get the string. This way you don't have vendor login, but also don't need a separate template step.
You still need to present that to Application.
So Command Line leaks worse than Env Var.
Config file, see original post for problems.
Env Var, see blog for problems.
I think parent is referring to something like SOPS [0], which can pass secrets via FIFO. That way, there’s nothing on disk, the pipe is cleared after first read, and /proc/cmdline doesn’t reveal anything.
0: https://github.com/getsops/sops
No I meant a property in the application config.
For example mbsync/isync does this.
You have a config file, it needs to have secrets so likely you are going to run some templating system where you replace dbpassword: ${dbPassword} with password from some secret system. Hopefuly you understand possible issues with any templating system that could result in replacement failures.
String manipulating is one of those "This is easy" until it's not.
Apparently I'm still unclear.
I don't mean to hardcode secrets into the config file either. I was suggesting to put a command into the configuration file that the application then calls to get the secret. This way the secret is only ever passed over a file descriptor between two processes.
I do something similar. I usually have a flag, something like --password-file. It can only be used to specify a file containing the secret and at startup the application reads it.
Yes, this is also possible, but which the approach I stated, the secret can be generated by another program or received from the network, it isn't just limited to a file.
My initial Ops gut feeling says "This is as lock in and error prone as Application Vault Libraries" but if Dev wanted to propose this, I'd be willing to see it in real operation.
It isn't lock in, because all the application depends on is that it gets a string it can pass to exec/the shell and then reads all data from stdout until EOF as the secret.
Sure, but let's say you have 5 secrets to get or maybe the new vault CLI does not support just stdout printing but prints JSON only.
I still think this is worse than config file/Env Vars.
You can always pass echo "SECRET" as the command, so it is a strict superset of the config file. Also programs that tend to provide a command option also tend to provide a simple string option.
Interesting read. Another interesting fact is that `setenv()` is fundamentally broken on POSIX, and should essentially never be called in library code. In application code, it should be called only in the absence of any alternative, and certainly before any threads have started.
The reason is that `getenv()` hands out raw pointers to the variables, so overwriting a variable using `setenv()` is impossible to guard against. Treat with extreme caution.
I think it's worth specifically calling out that the right way to set environment variables is in execve() only, as communication across an exec() is the precise niche for which they are the right tool.
Why are they the right tool for that instead of passing data with program args or another way of IPC?
Program args can be seen with tools like 'ps', so passing credentials that way is a poor choice.
FWIW, environment variables (edit: of your own processes) can also be seen with `ps`.
Right, you can see your own environment variables. Root can see your environment variables. Another user can not.
edit: submitted before I saw your edit. :-)
Well yeah of course. What you could do though would be to have e.g. --secrets-file and at startup time the application reads that file to get the secret. Then you could use file permissions to make sure only the (application) user running the application can read that file (or even more extreme, the application destroys the file after reading it).
I think that would still be better than env vars, which are more likely to leak somewhere you didn't intend them to.
They can also be caught by bash and system audit logs.
AFAIK Solaris solved that problem but Linux refuses to copy their solution.
Their solution was that setenv leaks memory rather than overwriting in place.
FreeBSD does the same. See here for discussion: https://freebsd-current.freebsd.narkive.com/NwqZQDWm/fix-for...
Sure. But practically the amount you leak is infinitesimal.
For a lot of applications that's the right call given the rest of the posix semantics you're constrained to and the kinds and frequency of data you pass via env vars.
An infinitesimal leak becomes a problem if it is done an infinite number of times...
execve seems like the preferable choice on a lot of grounds.
Which is why I said
> frequency of data
Practically, if you're moving enough data through setenv that the memory leaked versus the steady state fluctuation of the program is at all visible, you've got much bigger problems.
It would be a problem, except that the behaviour you're moving away from is a stale pointer. So surely any application that'd be leaking under that new behaviour would be crashing today.
The same advice applies to a correct program regardless of the implementation: using setenv is a bug. Only use getenv.
Getenv in a program without setenv is fine in both implementations. Setenv is unusable with all conforming implementations.
To pass environments to children, use execve.
The Linux behavior allows a careful single threaded program to use setenv correctly. The BSD/Solaris behavior makes all usage incorrect, but the incorrectness comes in the form of a memory leak, which is preferable to a security issue, usually.
There's no correct, portable use of setenv. If you call it, it's a bug.
No it always leaks, regardless if your program now correctly invalidates all derived pointers upon calling setenv.
IIRC the Solaris implementation of getenv()/setenv() leaks memory, because they want it to still be POSIX-compatible and the POSIX API has a "obey the spec, don't leak memory, be thread-safe -- pick 2" thing going on and you can't actually have a bug-free solution that keeps compatibility with decades of Unix applications.
Arguably that's less harmful than the thread-safety issues on Linux, but there's no perfect solution here as long as POSIX stays what it is.
Link?
Why would you ever call setenv in library code to begin with?
NetBSD has had getenv_r() for ages, [0] and I believe FreeBSD has very recently copied it. [1] If FreeBSD has it, macOS might eventually adopt it too. People tried already to get it into both glibc and POSIX, and it was rejected, but if it becomes more common on other platforms, that increases the chance they’ll both eventually accept it.
[0] https://man.netbsd.org/getenv_r.3
[1] https://github.com/freebsd/freebsd-src/commit/873420ca1e6e8a...
You still don’t know if some library you use calls getenv() and stores the returned pointer (or, potentially, runs in a different thread). Using getenv_r() protects your code against setenv(), but other code that may be using getenv() is still unprotected against your (or anyone else’s) use of setenv().
> You still don’t know if some library you use calls getenv()
Well, you can know, if you check their imported symbols. (I suppose they could be getting it via dlsym, but what are the odds of that...)
So if getenv_r() is added to the C library, and over time third party libraries started adopting it, you could get to the point that you could know no code in your process is calling getenv(), because none of the libraries your process loads import that symbol.
They could even add a glibc tunable to make getenv() call abort()... then you could be very sure nobody is calling getenv(), because if anyone ever did, boom
Environment variables are often used to pass secrets around. But, despite its ubiquity, I believe that's a bad practice:
- On Linux systems, any user process can inspect any other process of that same user for it's environment variables. We can argue about threat model but, especially for a developer's system, there are A LOT of processes running as the same user as the developer.
- IMO, this has become an even more pronounced problem with the popularity of running non-containerized LLM agents in the same user space as the developer's primary OS user. It's a secret exfiltration exploiter's dream.
- Environment variables are usually passed down to other spawned processes instead of being contained to the primary process which is often the only one that needs it.
- Systemd exposes unit environment variables to all system clients through DBUS and warns against using environment variables for secrets[1]. I believe this means non-root users have access to environment variables set for root-only units/services. I could be wrong, I haven't tested it yet. But if this is the case, I bet that's a HUGE surprise to many system admins.
I think ephemeral file sharing between a secret managing process (e.g. 1Password's op cli tool) and the process that needs the secrets (flask, terraform, etc.) is likely the only solution that keeps secrets out of files and also out of environment variables. This is how Systemd's credentials system works. But it's far from widely supported.
Any good solutions for passing secrets around that don't involve environment variables or regular plain text files?
Edit: I think 1Password's op client has a good start in that each new "session" requires my authorization. So I can enable that tool for a cli sessions where I need some secrets but a rogue process that just tries to use the `op` binary isn't going to get to piggyback on that authorization. I'd get a new popup. But this is only step 1. Step 2 is...how to share that secret with the process that needs it and we are now back to the discussion above.
1: https://www.freedesktop.org/software/systemd/man/latest/syst...
Since at least 2012, environment variables have been at least as secure as ordinary memory:
You can't read another process's environment unless you can ptrace-read the process, and if you can ptrace-read the process you know all its secrets anyway.cmdline is a different story.
Just to clarify, that means that by default every process of the same user can access the variables. But it doesn't really matter because by default every process of the same user can read any secret directly from the target process anyway, right?
And that the right thing to do if you want to harden your system is to disallow ptrace-read, and not bother changing software that uses environment variables?
Because I think most people that just try will be able to read the variables of any process on their computer.
Yes, you can consider all processes running under the same user as able to peek at each other's data. This is the point of running under the same uid: sharing data.One uid should be considered one security domain, with any separators inside it being guardrails, not brick walls.
If you want to prevent other processes from peeking into your process, run it under a different uid. Again. that's the point. A bunch of good software does that, running only small privileged bits in separate processes, and running some / bulk of the processes under an uid with severely limited privileges. See e.g. Postfix MTA, or the typical API server + reverse proxy split.
I don't think that this part of Unix is somehow problematic, or needs changing.
I think this part of Unix is exceedingly problematic. I want to be able to run program that don't have the ability to do anything that I, personally, can do. Ideally this would be doable without nasty kludges or root's help.
Linux has some ways to accomplish this, for example:
- seccomp. It can be done quite securely, but running general purpose software in seccomp is not necessarily a good way to prevent it from acting like the running user.
- Namespaces. Unless user namespaces are in use, only root can set this up. With user namespaces, anyone can do it, but the attack surface against the kernel is huge, mount namespaces are somewhat porous (infamously, /proc/self/exe is a hole), and they can be a bit resource-intensive, especially if you want to use network namespaces for anything interesting. Also, user namespaces have kind of obnoxious interactions with ordinary filesystem uids (if I'm going to run some complex containerized stack with access to a specific directory within my home directory, who should own the files in there, and how do I achieve that without root's help?). And userns without subuid doesn't isolate very strongly, and subuid need's root's help.
- Landlock. Kind of cool, kind of limited.
- Tools like Yama. On Ubuntu, with Yama enabled (the default), programs can't generally ptrace (or read /proc/self/environ) from other programs running as the same user.
In any case, once you've done something to prevent a process from accessing /proc/fd/PID for other pids belonging to the same user (e.g. pidns) and/or restricted ptrace (e.g. PR_SET_DUMPABLE), then environ gets protected.
It's not your (unprivileged user account's) place to decide the security posture of the entire system, that's why you're running into issues with root. Even Yama, an LSM, requires root (or de facto equivalent) for initial setup (as it should).
Namespaces, if done incorrectly, can significantly increase the attack surface of the entire system (mount namespaces especially need to be treated with care) and same with regards to anything to do with user accounts.
The real security barrier on most operating systems to date is the user account and if you want full isolation (modulo kernel bugs), run a process as a separate user if you're that concerned about leakage (or ideally, don't run it at all).
> It's not your (unprivileged user account's) place to decide the security posture of the entire system, that's why you're running into issues with root.
Tell that to literally any unprivileged user who would like to run any sort of software without exposing their entire account to any possible code execution bug or malicious code in the software they're running.
> The real security barrier on most operating systems to date is the user account and if you want full isolation (modulo kernel bugs), run a process as a separate user if you're that concerned about leakage (or ideally, don't run it at all).
That a very 1980s or maybe 1990s perspective, I think. It's 2025. I have some computers. They are mine. I trust myself to administer them and do pretty much anything to them, but I would prefer not to trust all my software. Sure, I can, very awkwardly, try to isolate things by using multiple uids, but it's extremely awkward and I see no technical reason why uids are a good solution here.
And there are no shortage of technical reasons why uids are a horrible solution. I would like to run unprivileged software and delegate some degree of privilege to them -- imagine a web browser or an IDE or whatever. I, as the owner of the machine, can make a uid. But I would also like my IDE to isolate various things it runs from each other and from the IDE as a whole, and it cannot do that because it does not own the machine and should absolutely not be running as root.
I've never tried to use ptrace-read to read secrets from a running process's memory so I can't comment on that part.
I had always assumed getting secrets from a running process' memory was non-trivial and/or required root permissions but maybe that's a bad assumption.
However, reading a process' environment is as trivial as:
`cat /proc/123456/environ`
as long as it's ran by the same user. No ptrace-read required.
You do need PTRACE access to pid 123456 in order to access that file. It is transparent to you, but the kernel will use the current task's PTRACE_ATTACH access when attempting to get that information.
By default, on most distributions, a user has PTRACE_ATTACH to all processes owned by it. This can be configured with ptrace_scope:
https://www.kernel.org/doc/Documentation/security/Yama.txt
> - On Linux systems, any user process can inspect any other process of that same user for it's environment variables. We can argue about threat model but, especially for a developer's system, there are A LOT of processes running as the same user as the developer.
Here's the thing though, the security model of most operating systems means that running a process as a user is acting as that user. There are some caveats (FreeBSD has capsicum, Linux has landlock, SELinux, AppArmor, Windows has integrity labels), but in the general case, if you can convince someone to run something, the program has delegated authority to act on behalf of that user. (And some user accounts on many systems may have authority to impersonate others.)
While it's by no means the only security model (there exists fully capability based operating systems out there), it's the security model that is used for most forms of computing, for better or worse. One of the consequences of this is that you can control anything within your domain. I can kill my own processes, put them to sleep, and most importantly for this, debug them. Anything I own that has secrets can grab them my ptrace/process_vm_readv/ReadProcessMemory/etc.
> Any good solutions for passing secrets around that don't involve environment variables or regular plain text files?
memfd_secret comes to mind https://man7.org/linux/man-pages/man2/memfd_secret.2.html
I haven't seen much language support for it, though. On one part maybe because it's Linux only.
People that write in Rust (and maybe Go, depends how easy FFI is) should give it a try.
I wanted for a time to get some support for it in PHP, since wrapping a C function should be easy, but the thought of having to also modify php-fpm put a dent in that. I can't and don't want to hack on C code.
In practice it'd be great if the process manager spawn children after opening a secret file descriptor, and pass those on. Not in visible memory, not in /proc/*/environ
For go, it's available in /x/sys/unix: https://pkg.go.dev/golang.org/x/sys/unix#MemfdSecret
Oh, I like that.
You should be able to build up a nice capability model to get access to those memfds from daemon too rather than having to spawn out of a process manager if that model fits your use case a bit better.
You've described the classic Unix security model, with minor improvements. While decent for the time it is showing its age. Specifically the difficulty in adapting to cheap, ubiquitous computing which it wasn't designed for.
If you need to keep secrets from other processes—don't run them under the same user account. Or access them remotely, although that brings other tradeoffs and difficulties.
> Environment variables are often used to pass secrets around. But, despite its ubiquity, I believe that's a bad practice:
I think environment variables are recommended to pass configuration parameters, and also secrets, in containerized applications managed by container orchestration systems.
By design, other processes cannot inspect what environment variables are running in a container.
Also, environment variables are passed to child processes because, by design, the goal is to run child processes in the same environment (i.e., same values) as the parent process, or with minor tweaks. Also, the process that spawns child processes is the one responsible for set it's environment variables, which means it already has at least read access to those secrets.
All in all I think all your concerns are based on specious reasoning, but I'll gladly discuss them in finer detail.
Please note that in my OP, I never mentioned containers.
Let's say, as a developer, I need to do some API interactions with GitHub. So, in a terminal, using 1Password's cli tool `op`, I load my GH API token and pass it to my Python script that is doing the API calls.
Presumably, the reason I use that process is because I want to keep that token's exposure isolated to that script and I don't want the token exposed on the filesystem in plaintext. There is no reason for every process running as my user on my laptop to have access to that token.
But, that's not how things work. The Claude agent running in a different CLI session (as an example) also now has access to my GitHub token. Or, any extension I've ever loaded in VS Code also has access. Etc.
It's better than having it in plain text on the file system but not by much.
Additionally, unless I've misunderstood the systemd docs, if you are passing secrets through environment variables using unit's `Environment` config, ANY USER on a server can read those values. So now, we don't even have user isolation in effect.
My reasoning is pretty plain. It could be wrong, but it's hardly specious.
> By design, other processes cannot inspect what environment variables are running in a container.
That’s not exactly true. If a process is running in a container, and someone is running bash outside of that container, reading that processes environment variables is as simple as “cat /proc/<pid>/environ”. If you meant that someone in one container cannot inspect the environment variables of a process running in a different container, that’s more true. That said, containers should not be considered a security boundary in the same way a hypervisor is.
I think in the context of containers you're right, there's a level of isolation and secrets are probably fine. But I think under other contexts that lack that isolation (e.g. bare-metal processes, local dev tooling) there are extra concerns.
(inb4: container env-vars are isolated from other containers, not from processes on the host system)
Any good cross-platform and easy ways to share secrets without using environment variables?
SOPS can be part of the solution. It takes care of encrypting and decrypting config files.
https://github.com/getsops/sops
Point to a file? E.g. `CONFIG_PATH=/etc/myapp/config.ini /opt/myapp`
That being said, I still use env vars and don't plan on stopping. I just haven't (yet?) seen any exploits or threat models relating to it that would keep me up at night.
Is that file more secure than the environment variables it's replacing? On Linux I think you can secure it to just your service with SELinux. Not sure about Windows
How do you get that file?
I also use env vars.
Not an answer, but I do wish there was a low level primitive and a corresponding high level language construct to pass around secrets.
Something like: my_secret = create_secret(value)
Then ideally it's an opaque value from that point on
Until when? Secrets in applications in many cases (I would probably wager majority of the cases) are only useful if they're in plaintext at some point, for example if you're constructing a HTTP client or authenticating to some other remote system.
As far as high-level language constructs go, there were similarish things like SecureString (in .NET) or GuardedString (in Java), although as best as I can tell they're relatively unused mostly because the ergonomics around them make them pretty annoying to use.
my_secret = getenv ("VALUE");
:-)
Linux security model is pretty broken without namespaces. systemd has some bells and whistles that help but if you want something better than environment variables you're naturally going to reach for cgroups.
Linux/Unix security model is that different actors are represented by different OS users. When you don't do that, it's already like inviting the burglars in your living room.
Linux's namespaces are not a security mechanism. Please do not use them as one.
Is there consensus around that? Given how containers and bubblewrap/flatpak are considered (to some degree) security boundaries, I approach statements like this with skepticism.
You answered your own question. Bubblewrap uses namespaces, but not by themselves. They're used in conjunction with other tools to provide a security boundary. Even then, it's not a very good security boundary. A serious security boundary should hold up even to privilege escalation, which is not true for bubblewrap.
Each kind of namespace provides its own kind of boundary. Yes, you need something like bubblewrap to stitch that all together. And it is kinda leaky, especially without taking some degree of care.
What're you saying about privilege escalation? I don't see how a user namespace does not prevent/limit privilege escalation.
More than that, I'm interested if there is some broader consensus I'm missing on the shortcomings of namespaces.
> I don't see how a user namespace does not prevent/limit privilege escalation.
They only prevent a single class of privilege escalation from setuid usage within the namespace. You can still obtain root using race conditions, heap overflows, side-channels, etc. or by coordinating with something outside of the namespace like a typical chroot escape.
Here's an old (patched) example of escaping the sandbox:
https://lwn.net/Articles/543273/
> More than that, I'm interested if there is some broader consensus I'm missing on the shortcomings of namespaces.
The consensus is that they're not security features on Linux. I'm not sure who sold you on the idea that they were, because that was not handed down by the kernel devs.
https://lwn.net/Articles/657744/
I've said exactly the same thing on HN myself. However for this problem what do you suggest? Using namespaces is better than using no namespaces.
If I was actually serious about the security of a system, I'd explicitly use virtualization at-minimum. If all I wanted was a mall cop and was happy to take the risk of misconfiguration or attacks on the kernel I'd probably say something that adds a lot of extra sanity to namespaces is "good enough", like bubblewrap. I would never trust it or say that it's secure though.
cgroups aren't relevant here, I think. Not sure if that was just a typo, since you did mention namespaces in the first sentence. PID and user namespaces in particular are relevant here.
Yeah seems you're right, I thought namespaces came under the cgroups umbrella.
> Any good solutions for passing secrets around that don't involve environment variables or regular plain text files?
Plain text files are fine, it's the permissions on that file that is a problem.
The best way is to be in control of the source to the program you want to run then you can make a change to always protect against leaking secrets by starting the program as a user that can read the secrets field. After startup the program reads the whole file and immediately drops privileges by switching to a user that cannot read the secrets file.
Works for more than just secrets too.
> On Linux systems, any user process can inspect any other process of that same user for it's environment variables. We can argue about threat model but, especially for a developer's system, there are A LOT of processes running as the same user as the developer.
This is a very good point I'd never realised! I'm not sure how you get around it, though, as if that program can even find a credential and decrypt a file, if it runs as the user then everything else can go find that credential as well.
If a root exploit is leveraged, then /proc/*/environ of all processes is visible to the adversary.
The classical alternative has been to store (FTP) credentials in a .netrc file (also used by curl).
I have some custom code to pull passwords out of a SQLite database.
For people who are really concerned with this, a "secrecy manger" is more appropriate, such as Cyberark conjur and summon, or Hashicorp Vault.
If a full root exploit is leveraged, it's already game over already, basically anything at that point is just going to be rearranging deck chairs.
I added a note to my original post about how 1Password's cli tool work. Access to the binary doesn't mean automatic access to any authorized session. I think that's a good start. In my case, I have it tied into my fingerprint reader, so if something want's access to a secret using `op` I get a prompt to authorize. If I haven't just called `op` myself, I know it's a suspect request.
I've used keyring, but ultimately does not solve the issue of every application running under the same user.
https://pypi.org/project/keyring/
--passphrase-fd was always the correct thing to do (gpg has had it for ages), but support for that is very meager. I mean, you can’t even kubectl login using environment variables, you pretty much have to pass tokens through command-line arguments and we’ve known that’s terrible for around forty-five years.
> Any good solutions for passing secrets around that don't involve environment variables or regular plain text files?
Honestly, my answer is still systemd-creds. It's easy to use and avoids the problem that plain environment variables have. It's a few years old by now, should be available on popular distros. Although credential support for user-level systemd services was added just a few weeks ago.
A TL;DR example of systemd-creds for anyone reading this:
You can now configure your unit file, e.g.: The process you start in the service will then be able to read the decrypted credential from the ephemeral file `$CREDENTIALS_DIR/mypw`. The environment variable is set automatically by systemd. You can also use the command `systemd-creds cat mypw` to get the value in a shell script.At least systemd v250 is required for this. v258 for user-level service support.
If are like most users you have secrets stored in read-protected files in ~/.ssh. There's nothing wrong with that.
They are (should be) encrypted though, and not usable by someone who might somehow gain access to your ~/.ssh files.
I can't recommend Varlock [1] enough. A great way to manage env vars in a project. It lets you define which ones are necessary or optional, their types, and where they should be fetched from.
[1] https://varlock.dev/
One of the worst things about Environment variables among others discussed here is the implicit and opaque nature of them. Majority of applications rely on them in the *nix world. Even if more explicit and obvious ways of configuration files or remote services (consul/etcd, et al.) and command line arguments are supported env vars are traditionally supported as well.
But as mentioned in the article it is just a global hashmap that can be cloned and extended for child processes. Maybe in 1979 it was a good design decision. Today it sometimes hurts.
For example, kubernetes by default literally pollutes the container’s environment with so-called service links. And you will have fun time debugging a broken application if any of those “default” env vars conflict with the env vars that your app might expect to see.
https://kubernetes.io/docs/tutorials/services/connect-applic...
They are ubiquitous and we are living in the world of neo-conservatism in IT where legacy corner cuts are treated as a standard and never challenged (hello /bin, /usr/bin, /lib, /usr/lib)[0]
[0] - https://askubuntu.com/a/135679
To give an illustration of how bad this can get:
At a past firm, I was trying to debug how a particular ENV var was getting set. I started out thinking it was something simple like the user's .bashrc or equivalent.
I quickly realized that there were roughly 10 "layers" of env var loadings happening where the first couple layers were:
- firm wide
- region
- business unit
- department
- team etc etc
I ended up having to turn on a bash debug flag so that I could see exactly where the var was getting set in the layer stack.
Not sure if any other high level languages have it, but Node.js has just added command line flags to trace precisely all env var accesses and modifications:
https://nodejs.org/api/cli.html#--trace-env
Since you can set/unset/modify env vars in so many ways with different APIs, this sounds super useful in complex debugging scenarios.
I gave up on environment variables long ago. Now my compilers read a dmd.conf file that is in the same directory as the compiler executable.
I get anxiety whenever I need to set an environment var on Linux. There are (somewhat distro-specific) ways that work properly, but the usual procedures you find online stops working once you reboot (or close the terminal I think?). They should add a simple env var GUI like Windows has that just works, and isn't terminal-specific. Windows has the annoyance of needing to restart the the terminal (or open a new one) for changes to take effect, but works well other than that.
> but the usual procedures you find online stops working once you reboot (or close the terminal I think?)
The environment isn't persistent between sessions. That means you need to make the change in a way that runs on every new session (login or new terminal window).
Depending on how your system is configured:
.bash_profile gets run once on every login
.bashrc gets run once on every non-login new session (i.e. a new terminal window)
It's typical, if using these files, to do something like this:
in the .bash_profile file, putting most other things in .bashrc, such that you don't have to worry about the distinction.If you're not even using bash or bash-likes at all, but instead something like Zsh, fish, etc you'll need to set things the way they want to be set there.
> They should add a simple env var GUI like Windows has that just works, and isn't terminal-specific
This doesn't exist in linux, because there isn't "one place" that all possible terminals draw from. Conceivably, it's possible to write a GUI tool that reads your .bashrc file, looks for anything that resembles a variable, parses the bash code to look for ways it is set, and then present a GUI tool that could update it in common cases, but... it's way easier to just write the change in a text editor.
That applies if you use bash. If you use another shell, you'll need to use that shell's methods to set environment variables in the config(s). And probably still need to set them for bash so that bash scripts get the correct env vars.
NixOS & home-manager do have functions that can set environment variables across all shells enabled via either of those systems. So I've got `programs.bash.enable = true;` && `programs.fish.enable = true;` && `home.sessionVariables = { EDITOR = "nvim"; };` in my home-manager config for example. A GUI editor for that would certainly be possible.
> Conceivably, it's possible to write a GUI tool that reads your .bashrc file
What's wrong with an envfile or envdir? The envdir is kind of annoying but at least you can set permissions on the files inside it.
A lot of programs and tools will try to be cute and write into your bash files for you. I was mostly waxing philosophical at that point having gotten nerd sniped by "write a gui tool for environment variables" and "environment variables like $PATH in bash files" coming together in my head.
Funny, as a primarily linux user the windows behaviour irks me to no end, it's the cause of so many recurrent problems on end-user machines with so many apps that pollute environment.. and then you wonder why something doesn't work and then it turns out that for some reason you're using $SOFTWARE from c:\Perl64\bin instead of its proper place
On systemd systems you can just either set KEY=VALUE pairs in `/etc/environment` or any file in `/etc/environment.d/` (and technically a few other places [0]). In theory it should be relatively easy to write a GUI for it by manually parsing the files.
The application restarting part can't really be fixed, since environment variables aren't ever injected to a running process and can only be changed by the process itself (terms and conditions may apply) and even changes during runtime could be ignored since the program itself may have cached some already computed value based on the variable.
[0]: https://www.freedesktop.org/software/systemd/man/latest/envi...
Good info. I may write a script or tiny program to help with this.
https://xkcd.com/927/ - Standards
Situation: There are 14 competing ways to set environment variables on Linux
We should create a universal way that just works and isn't terminal specific to set environment variables!
Situation: There are 15 competing ways to set environment variables on Linux
OP: that wasn't a boring read :) thanks
Unix is a legacy mess. The sooner we rid ourselves of this scourge, the better.
Posted from a productive Windows workstation.
You couldn't pay me enough to go back to doing dev work on Windows.
As if Windows isn’t its own legacy mess..
static secrets that you can even extract and set in a config file (or store in a secrets manager) need to go. TPM secrets that you can't extract, and public keys that you can pin without considering them "secret", along with Oauth / IAM roles, are the way to go.
> POSIX-specified utilities use uppercase envvars, but that’s not prescriptive for your programs. Quite the contrary: you’re encouraged to use lowercase for your envvars so they don’t collide with the standard tools.
> But in reality, not many applications use lowercase. The proper etiquette in software development is to use ALL_UPPERCASE
I always prefer lower case for env variables in scripts. Thanks for pointing out that it can help reduce clashes with standard tools.
Another legacy mess:
It's absolutely crazy that this isn't a dynamically resizable vector.It is, there’s just a limit set by the kernel for the number of pages the command line as a whole can occupy or something like that.
No, it's still an artificial limit, even if you can change it (usually after your command has failed).
The xargs command was designed to address this. From this perspective, it is a kludge.
...and people have to find special ways to set them. I didn't even know that pam_env existed until a recent security vulnerability announcement. It's never come to my attention before, I don't think I've ever seen it utilized. I've now made a runbook item to disable it. It's a shame that "configuration" includes undoing the fetishes and helpfulness [0][1] of others.
[0] Crowded elevator atrium. Multiple elevators running. Elevator wants to close, another one is coming (oh! I heard it "ding"!). Somebody is holding the elevator which wants to depart and trying to wave me in. Why doesn't somebody push them out?[1] I'm at a stop sign. Some complete idiot is trying to turn left onto the street I'm leaving and waving me to turn left in front of them. Fuck no! I turn in front of you, somebody rearends you, you fly forward into me: my fault! You should be able to make this turn, if you can't go around the block! [2]
[2] I could go out of my way and turn right. Or I can just wait and see what happens.
If you need realtime scheduling pam_env was the only way to do it until rtkit came out (and it still does it better than rtkit does, annoyingly).
That was a hard to read font.
Huh... Guess it's what you're used to.
First thing that struck me about the site is how beautiful I found it. I even inspected the font: Iosevka Web, apparently.
Change your browser settings. You can tell the browser what font you prefer to use, and set minimum font sizes. You can also turn off remote fonts, so that pages can only use fonts you already have. Then you can just uninstall any fonts you don’t like or that you cannot read easily.
I'm surprised you say that. Iosevka is quite beloved as a monospace font. I use it for all my terminals, etc.
Prose in monospace is harder to read regardless of the specific font.
On another note, does anyone know how this blog/website was created? I am in general very curious about ssgs hence why I am asking.
My guess would be either Jekyll or hand rolled, due to the url structure.
Environment variables also suck due to having zero typo existence. Write FOO_BAR instead of FOO_BARS? Silent failure. V1 of your package recognizes FOO_BAR and V2 changes it to FOO_BARS (because now we can have more than one)? Silent failure.
In a shell script, set -u. It’ll terminate if you reference an unset variable.
In other languages, check that the var you pulled in isn’t falsey before proceeding.
> In a shell script, set -u
Won't help. I'm _running a program_ want to configure it with environment variables, not _writing a program_ that I expect the user to configure for himself.
> falsey
There are more than two programming languages in the world.
Eh, they are just more command line arguments, ones that go on the left side of the command instead of the right. I guess an alternative is something the Windows registry, but I'm not seeing that as a great improvement since it's less direct.
Command-line arguments aren't passed to subprocesseses, can't be inspected by arbitrary functions in the same process, and don't leak memory if they are changed.
Yes, I'm just highlighting that it's just as convenient to change them as command line parameters on a per-invocation basis. If there is a proposal for something else, it should retain this capability. Config files and something like the windows registry are much less convenient.
> can't be inspected by arbitrary functions in the same process
You can not really rely on that. The initial stack layout is well-defined at least on linux, so digging up argv is not difficult. Or just open /proc/self/cmdline
>"Wow, I really enjoyed writing this… …and I hope it wasn’t a boring read."
No, it was very interesting actually!
An excellent deep-dive into the murky area of Unix/Linux environment variables, how they are set, how they are passed, what's really going on behind the scenes.
Basically a must-read for any present or future OS designer...
Observation (if I might!) -- environment variables are to programs (especially chains of programs, parent processes, sub processes and sub-sub processes, etc.) what parameters are to functions -- and/or what command-line parameters are... they're sort of like global variables that get passed around a lot...
They also can influence program behavior -- and thus the determinism of a program -- and thus the determinism of a chain of programs...
Phrased another way -- software that works on one developer's machine might not work on another developer's machine and/or in a production environment because one crucial environment variable was either set or not set, or set to the wrong value...
(NixOS seems to understand this pretty well... that "hermetically sealing" or closing or "performing a closure around" (that's probably slightly the wrong language/terminology in "Nix-speak" but bear with me!) a software environment, including the environment variables is the way to create deterministic software builds that run on any machine... but here I'm digressing...)
But my main point: I complete agree with the article's author -- environment variables are a legacy mess!
Although, if we think about it, environment variables (if we broaden the definition) are a sub-pattern of anything that affects the state of an individual machine or runtime environment -- in other words, things such as the Windows Registry, at least the global aspects of it -- are also in the same category.
Future OS's, when they offer environments for programs or chains of programs to run -- should be completely containerized -- that is, the view of the system -- what data/settings/environment variables/registry variables/files/syscalls/global variables it has access to -- should be completely determinable by the user, completely logabble, completely transparent, and completely able to be compared, one environment to another.
In this way, software that either a) fails to work at all b) works in a non-deterministic way -- can be more easily debugged/diagnosed/fixed (I'm looking at you, future AI's that will assist humans in doing this!) -- then if all of that information is in various different places, broken, and/or opaque...
To reiterate:
>"Wow, I really enjoyed writing this… …and I hope it wasn’t a boring read."
No, I really enjoyed reading it(!), it's a brilliant article, and thank you for writing it! :-)
Upvoted and favorited!
We're almost rid of dotfiles in $HOME. Can we please also get rid of the abuse of environment variables for configuration? There is no reason your program needs to muck up the environment table of every process by defining MY_PROGRAM_API_KEY instead of taking a command line argument or reading a configuration file. It's not "secure" just because it's not on the command line. And it will mess up for users because it's nigh impossible to ensure you have the same environment variables over all login types. The variable might be there when you run locally, but not in an ssh session or cron. Some ephemeral configuration such as LD_LIBRARY_PATH and PWD is difficult to handle without environment variables, but those are rare exceptions and not the norm.
Contribute to a clean environment! Don't use environment variables for configuration! </rant>
Depending on your system, passing secrets via environment can be more secure than passing secrets via a command line. Command line arguments of all running processes can be visible to other users of the system, environment variables are usually not.
This problem is also removed by just not exporting the variables. Also you can just pass an envp to execvE.
"MY_APP_SECRET_KEY=bla myapp" hardly accomplishes anything substantial over "myapp --secret-key=blah" It's just a less robust and less well-supported command-line interface. It's not supported by most gui launchers, PowerShell, nor many cron implementations, for example.
The command line can be read by any user on the host (with `ps auxww` for example) while the environment cannot.
You should never pass secrets on the command line.
They're the standard for Docker container configuration tho. Otherwise I do agree.