Note: Another techie post. Sorry! This conundrum has really taken over my spotted mindβ¦
I’ve searched high and low for months trying to figure this one out, dear reader, and I’m now asking the general Internet for any ideas on how to make container secrets less dangerous. Especially in the age of daily supply chain exploits and privilege escalation bugs, this feels like a weak link.
It feels so odd to have a well-known directory, /run/secrets, sitting in every container, waiting for a ne’er-do-well to come in and read it all out. I feel like the files in there should be read-once then somehow inaccessible, but I’ve been unable to figure out how to convince the Linux kernel to do that with files. It’s driving me mad!
I’ve asked a few security-minded friends and colleagues, both at work and in my broader network, and all of them have suggested workarounds but none of them have felt really secure. One widely circulated idea is to use environment variables instead. But then all the attacker has to do is read out ENV β sounds to me like that would be even easier than convincing a typical app server to display an attacker-controlled file from the file system!
One thing I personally thought of was to mount an empty tmpfs on /run/secrets after container initialisation in a custom entry point. This still feels like a hack, but it’d be much harder to get the app server to umount (since you are dropping to non-root, right?) and the file descriptors shouldn’t remain open after they’re read in. Still, this feels like a really cheesy hack.
Has anyone else thought of an ingenious solution to this problem? Is anyone else even thinking this is a problem? I’d love to have a discussion about this, either in the comments or, if you’re feeling old-school, find me on Libera IRC as Spots.
I understand the concern but the reality of the situation is running compute needs read access to the contents of your secrets if you remove access while running then you application can not access either. Aside from limiting egress and certificate pinning in your container there is not real way to block ex-filtration of data when an attacker is in your process.
The secrets are read in to the process and stored in memory, so there’s definitely value in removing the files after they’re already read. They’re just sitting there doing nothing.
In the past, when workers could get reaped and a process manager like unicorn would respawn them, then yes, you’d need the secret files to have long lifetimes. But these days, Docker *is* the process manager – if the worker dies, the container is respawned. Docker healthchecks are really useful for making self-healing containers, btw.
This is a thing that pushes me away from using docker increasingly, not that I use it for anything that matters in the first place.. The fact that they didn’t solve the “running everything as root all the time” problem until very late in the project, and the fact that they didn’t design “secrets are always in the same place and frequently exploited” out of existence is Deeply Concerning. lmao
Maybe I’m misunderstanding but how hard would it have been to force dockerfiles to declare their access, per-command, to any secret? Like “GRANT s1 / RUN whatever -f $s1” ? And then outside of that specific call the container has no access to the secret (except via however “whatever” might internally persist it?
But yeah for this and other reasons I’m probably switching towards something more bare-bones, probably libvirt/kvm/qemu or incus.. but I haven’t figured out an alternative yet π
SUID (setuid) script for the umount you want.
Donβt have secrets. Use workload identity either through a proxy or a service mesh if you have one, leak nothing inside your application containers. If youβre calling external APIs control identity through egress proxies.
I canβt answer on https://lobste.rs/s/66gclj/surely_there_must_be_way_make_container
I donβt have any account. I would love a invitation by the way π
You need to write your threat model. Whatβs the worst that can happen.
Generally, as you said if your system is compromised their will be a scan for secrets.
What I use most is a second level secret. Ie secret that permits to get secrets. Ie the exposed secret is no worth without knowing how and where to retrieve them.
A must read is this blog that explains threat model and all very clearly
https://www.macchaffee.com/blog/2022/k8s-secrets/
All the best
What if those secrets were merely echoed into (named) FIFOs? Then you can only read the data out of them once.
I was privately messaged a similar suggestion and I do think this is an interesting way forward, though I really like the egress proxy idea more.
Publish your secrets into a sidecar container, put named pipes in `/run/secrets` have you sidecar only server the secret on first read.
[…] μμΈμ΄λ Dalmatian.Lifeμ κΈμ μ°Έκ³ νμ¬ […]