Subject: A pile of problems with unprivileged Guix daemon and 'guix gc'

From: "Zack Weinberg" <[email-masked]>
Date: Tue, 26 Aug 2025 17:17:22 -0400

# guix gc
finding garbage collector roots...
cannot read potential root `/var/guix/gcroots/auto/idj3k6kjlqi7y8sc4c5xschqh8zkhfvb'
cannot read potential root `/var/guix/gcroots/auto/5si4fzk79j7v27rqaic4lc2qfpk42ilb'
cannot read potential root `/var/guix/gcroots/auto/825grbfhqdfav4g6827d2d3hb8hyhhzl'
cannot read potential root `/var/guix/gcroots/auto/6l77c1c97vij2gg3p95d9zi2k7l0yx29'
cannot read potential root `/var/guix/gcroots/auto/r2x8d211bfp2y3y6wvgp8740ram26ipv'
cannot read potential root `/var/guix/gcroots/auto/jbrxxz57056g8393kh9zyyj325lwq5c5'
guix gc: error: program `/gnu/store/6px1m9n904j8s4hyrmlds707sfnq52d9-guix-1.4.0-41.826e305/bin/guix' failed with exit code 1

So first off, these error messages fail to comply with the first law of
Unix error messages; they don't print strerror(errno), and they don't
name the actual system call that failed, so they don't tell me *why* the
GC roots can't be read.  But leave that aside for now...

# guix gc 2>&1 |
  sed -ne 's:^cannot read potential root `\([a-z0-9/]*\)'\''$:\1:p' >
  /tmp/bad-roots
# ls -l $(cat /tmp/bad-roots)
lrwxrwxrwx 1 guix-daemon guix-daemon 80 Aug 10 01:41 /var/guix/gcroots/auto/5si4fzk79j7v27rqaic4lc2qfpk42ilb -> /root/.cache/guix/inferiors/bpo6zmuuzeya74vbpqn2innq7vw4xzxn7azgjarsmg756jdrsika
lrwxrwxrwx 1 guix-daemon guix-daemon 79 Mar 16 22:20 /var/guix/gcroots/auto/6l77c1c97vij2gg3p95d9zi2k7l0yx29 -> /root/.cache/guix/profiles/simr3ylizyyss24c25azsqfl4vjtw2t4ywvgpbh3iinbrsljgfea
lrwxrwxrwx 1 guix-daemon guix-daemon 80 Jul 27 02:02 /var/guix/gcroots/auto/825grbfhqdfav4g6827d2d3hb8hyhhzl -> /root/.cache/guix/inferiors/zy7a627k6aubd32iun2ibyoy4ulbj4xas55yaibwaayctx6qehta
lrwxrwxrwx 1 guix-daemon guix-daemon 80 Jul 13 01:41 /var/guix/gcroots/auto/idj3k6kjlqi7y8sc4c5xschqh8zkhfvb -> /root/.cache/guix/inferiors/72tvmmz43muzwd4lml3xsfdxw55idd742433w4kylm7yyyohed6a
lrwxrwxrwx 1 guix-daemon guix-daemon 80 Aug  3 01:39 /var/guix/gcroots/auto/jbrxxz57056g8393kh9zyyj325lwq5c5 -> /root/.cache/guix/inferiors/qgxsppsml7olednljz273sdygm5zsxjrrpey2q7ysh5on6evneza
lrwxrwxrwx 1 guix-daemon guix-daemon 80 Jul 20 01:41 /var/guix/gcroots/auto/r2x8d211bfp2y3y6wvgp8740ram26ipv -> /root/.cache/guix/inferiors/whqagcgua6af2zpw3xpaiiifny6pvevcpque3kstsu74ufx6rrda

# ls -ld /root /root/.cache /root/.cache/guix /root/.cache/guix/{inferiors,profiles}
drwx------ 5 root root 4096 Aug 26 20:46 /root/
drwxr-xr-x 4 root root 4096 Jul 22  2024 /root/.cache/
drwxr-xr-x 6 root root 4096 Mar 16 22:19 /root/.cache/guix/
drwxr-xr-x 2 root root 4096 Aug 10 01:41 /root/.cache/guix/inferiors/
drwxr-xr-x 2 root root 4096 Mar 16 22:21 /root/.cache/guix/profiles/

After seeing this I suspected the problem might be that the *Guix daemon*,
which is running unprivileged, cannot access these files.  And indeed, if
I do `chmod 711 /root`, then `guix gc` stops printing the "cannot read
potential root" messages.  But it still doesn't _work_:

# guix gc
finding garbage collector roots...
guix gc: error: program `/gnu/store/6px1m9n904j8s4hyrmlds707sfnq52d9-guix-1.4.0-41.826e305/bin/guix' failed with exit code 1

So that's _really_ bad UX, but again, not the immediate problem.  Since I
do now know that it's the daemon that's having problems, I check the logs:

# tail -3 /var/log/guix-daemon.log
2025-08-26 20:56:21 accepted connection from pid 172, user root
2025-08-26 20:56:21 accepted connection from pid 176, user guix-daemon
2025-08-26 20:56:21 guix gc: error: creating directory `/var/guix/profiles/per-user/guix-daemon': Permission denied

Well, that's suggestive...

# ls -la /var/guix/profiles/per-user
total 28
drwxr-xr-x 7 root         root         4096 Apr 25 20:03 ./
drwxr-xr-x 3 root         root         4096 Aug 26 20:25 ../
drwxr-xr-x 2 root         root         4096 Aug 26 20:25 root/
drwxr-xr-x 2 user1        user1        4096 Apr 25 20:03 user1/
drwxr-xr-x 2 user2        user2        4096 Apr 25 20:03 user2/
drwxr-xr-x 2 user3        user3        4096 Apr 25 20:03 user3/

(actual user names redacted)

# mkdir /var/guix/profiles/per-user/guix-daemon
# chown guix-daemon:guix-daemon /var/guix/profiles/per-user/guix-daemon
# guix gc
finding garbage collector roots...
deleting garbage...
[7 MiB] deleting '/gnu/store/r993z4wdyqqwzxlif1hvqzp6cqhqr2bw-rustc-1.76.0-src.tar.zst.drv'
[7 MiB] deleting '/gnu/store/afnyx8a8qj4wlhywv0zsf57lmk8yskzc-rustc-1.76.0-src.tar.gz.drv'
...
[38344 MiB] deleting '/gnu/store/yd1hpyjjmzmq5qmlv6q2ycqlymsma9rh-freeglut-3.4.0-builder'
[38344 MiB] deleting '/gnu/store/3z0np2ad898193wws74k54rzppr356cv-ipxe-qemu-1.21.1-3.24db39f-builder'
deleting `/gnu/store/trash'
guix gc: error: making `var/empty' writable: Operation not permitted

Well yeah! /var/empty is supposed to be empty at all times!  Is this
not how it's supposed to be?

# ls -ld /var/empty
dr-xr-xr-x 2 root root 4096 Jul 22  2024 /var/empty/

But if I set it to be world-writable I still get the same error.  In
fact, even if I make it *owned by guix-daemon*, I still get the same
error!

And now I'm stuck.  guix gc moved all the trash to /gnu/store/trash,
but it didn't actually delete any of it.  There's 11G in there, and
I can't delete it by hand because the store is mounted read-only
and I don't know how to temporarily override that for this kind of
manual repair job.  Any advice would be most appreciated.

zw