#79321 A pile of problems with unprivileged Guix daemon and 'guix gc'

Message #8 received at 79321@debbugs.gnu.org (full text, mbox, reply):

Received: (at 79321) by debbugs.gnu.org; 27 Aug 2025 06:23:54 +0000
From debbugs-submit-bounces@debbugs.gnu.org Wed Aug 27 02:23:54 2025
Received: from localhost ([127.0.0.1]:58513 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces@debbugs.gnu.org>)
	id 1ur9Za-0003nj-25
	for submit@debbugs.gnu.org; Wed, 27 Aug 2025 02:23:54 -0400
Received: from ditigal.xyz ([2a01:4f8:1c1b:6a1c::]:48010 helo=mail.ditigal.xyz)
 by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.84_2) (envelope-from <rutherther@ditigal.xyz>)
 id 1ur9ZW-0003nO-Gf
 for 79321@debbugs.gnu.org; Wed, 27 Aug 2025 02:23:51 -0400
Received: by cerebrum (OpenSMTPD) with ESMTPSA id 8693f6fb
 (TLSv1.3:TLS_CHACHA20_POLY1305_SHA256:256:NO); 
 Wed, 27 Aug 2025 06:23:43 +0000 (UTC)
Date: Wed, 27 Aug 2025 08:23:41 +0200
From: Rutherther <rutherther@ditigal.xyz>
To: help-guix@gnu.org, Zack Weinberg <zack@owlfolio.org>, 79321@debbugs.gnu.org
Subject: Re: A pile of problems with unprivileged Guix daemon and 'guix gc'
User-Agent: K-9 Mail for Android
In-Reply-To: <9e3bad11-bae9-456f-93ac-c813d52c6ca9@app.fastmail.com>
References: <9e3bad11-bae9-456f-93ac-c813d52c6ca9@app.fastmail.com>
Message-ID: <62B5E93F-E05F-4FF1-A7CC-1D74A397E890@ditigal.xyz>
MIME-Version: 1.0
Content-Type: text/plain;
 charset=utf-8
Content-Transfer-Encoding: quoted-printable
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ditigal.xyz;
 i=@ditigal.xyz; q=dns/txt; s=20240917; t=1756275823; h=date : from :
 to : subject : in-reply-to : references : message-id : mime-version :
 content-type : content-transfer-encoding : from;
 bh=WigxraGMBnPFrL+cxu6UAf1y9rNv4VvK0QvvyLKdi4M=;
 b=kwbG5d4F3GM0IWfXmprHtwhoSfJPSDWqKMR36wvyqn2N0mFZl8ph8cWoMWWUd3CzcHvJP
 HnXgDj57a/FwLjjoW7A3hjAWMLVczZvfpxrJP+JJfhqHwQuTD96gExjpB6AOFIIrAW6SvdP
 vqHE2iI9OUO/HKQPRR8cDZOWf0+MZYE=
X-Spam-Score: 2.5 (++)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 the administrator of that system for details.
 Content preview:  Hi, On August 26, 2025 11:17:22 PM GMT+02:00, Zack Weinberg
 <zack@owlfolio.org> wrote: ># guix gc >finding garbage collector roots...
 >cannot read potential root `/var/guix/gcroots/auto/idj3k6kjlqi7y8sc4c [...]
 Content analysis details:   (2.5 points, 10.0 required)
 pts rule name              description
 ---- ---------------------- --------------------------------------------------
 2.0 PDS_OTHER_BAD_TLD      Untrustworthy TLDs
 [URI: ditigal.xyz (xyz)]
 0.0 SPF_HELO_NONE          SPF: HELO does not publish an SPF Record
 -0.0 SPF_PASS               SPF: sender matches SPF record
 0.5 FROM_SUSPICIOUS_NTLD   From abused NTLD
X-Debbugs-Envelope-To: 79321
X-BeenThere: debbugs-submit@debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request@debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit@debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request@debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request@debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces@debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces@debbugs.gnu.org>
X-Spam-Score: 2.5 (++)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 the administrator of that system for details.
 
 Content preview:  Hi, On August 26, 2025 11:17:22 PM GMT+02:00, Zack Weinberg
    <zack@owlfolio.org> wrote: ># guix gc >finding garbage collector roots...
    >cannot read potential root `/var/guix/gcroots/auto/idj3k6kjlqi7y8sc4c [...]
    
 
 Content analysis details:   (2.5 points, 10.0 required)
 
  pts rule name              description
 ---- ---------------------- --------------------------------------------------
  2.0 PDS_OTHER_BAD_TLD      Untrustworthy TLDs
                             [URI: ditigal.xyz (xyz)]
  0.0 SPF_HELO_NONE          SPF: HELO does not publish an SPF Record
 -0.0 SPF_PASS               SPF: sender matches SPF record
  0.5 FROM_SUSPICIOUS_NTLD   From abused NTLD
  1.0 BULK_RE_SUSP_NTLD      Precedence bulk and RE: from a suspicious TLD
 -1.0 MAILING_LIST_MULTI     Multiple indicators imply a widely-seen list
                             manager

Hi, 

On August 26, 2025 11:17:22 PM GMT+02:00, Zack Weinberg <zack@owlfolio.org> wrote:
># guix gc
>finding garbage collector roots...
>cannot read potential root `/var/guix/gcroots/auto/idj3k6kjlqi7y8sc4c5xschqh8zkhfvb'
>cannot read potential root `/var/guix/gcroots/auto/5si4fzk79j7v27rqaic4lc2qfpk42ilb'
>cannot read potential root `/var/guix/gcroots/auto/825grbfhqdfav4g6827d2d3hb8hyhhzl'
>cannot read potential root `/var/guix/gcroots/auto/6l77c1c97vij2gg3p95d9zi2k7l0yx29'
>cannot read potential root `/var/guix/gcroots/auto/r2x8d211bfp2y3y6wvgp8740ram26ipv'
>cannot read potential root `/var/guix/gcroots/auto/jbrxxz57056g8393kh9zyyj325lwq5c5'
>guix gc: error: program `/gnu/store/6px1m9n904j8s4hyrmlds707sfnq52d9-guix-1.4.0-41.826e305/bin/guix' failed with exit code 1
>
>So first off, these error messages fail to comply with the first law of
>Unix error messages; they don't print strerror(errno), and they don't
>name the actual system call that failed, so they don't tell me *why* the
>GC roots can't be read.  But leave that aside for now...
>
># guix gc 2>&1 |
>  sed -ne 's:^cannot read potential root `\([a-z0-9/]*\)'\''$:\1:p' >
>  /tmp/bad-roots
># ls -l $(cat /tmp/bad-roots)
>lrwxrwxrwx 1 guix-daemon guix-daemon 80 Aug 10 01:41 /var/guix/gcroots/auto/5si4fzk79j7v27rqaic4lc2qfpk42ilb -> /root/.cache/guix/inferiors/bpo6zmuuzeya74vbpqn2innq7vw4xzxn7azgjarsmg756jdrsika
>lrwxrwxrwx 1 guix-daemon guix-daemon 79 Mar 16 22:20 /var/guix/gcroots/auto/6l77c1c97vij2gg3p95d9zi2k7l0yx29 -> /root/.cache/guix/profiles/simr3ylizyyss24c25azsqfl4vjtw2t4ywvgpbh3iinbrsljgfea
>lrwxrwxrwx 1 guix-daemon guix-daemon 80 Jul 27 02:02 /var/guix/gcroots/auto/825grbfhqdfav4g6827d2d3hb8hyhhzl -> /root/.cache/guix/inferiors/zy7a627k6aubd32iun2ibyoy4ulbj4xas55yaibwaayctx6qehta
>lrwxrwxrwx 1 guix-daemon guix-daemon 80 Jul 13 01:41 /var/guix/gcroots/auto/idj3k6kjlqi7y8sc4c5xschqh8zkhfvb -> /root/.cache/guix/inferiors/72tvmmz43muzwd4lml3xsfdxw55idd742433w4kylm7yyyohed6a
>lrwxrwxrwx 1 guix-daemon guix-daemon 80 Aug  3 01:39 /var/guix/gcroots/auto/jbrxxz57056g8393kh9zyyj325lwq5c5 -> /root/.cache/guix/inferiors/qgxsppsml7olednljz273sdygm5zsxjrrpey2q7ysh5on6evneza
>lrwxrwxrwx 1 guix-daemon guix-daemon 80 Jul 20 01:41 /var/guix/gcroots/auto/r2x8d211bfp2y3y6wvgp8740ram26ipv -> /root/.cache/guix/inferiors/whqagcgua6af2zpw3xpaiiifny6pvevcpque3kstsu74ufx6rrda
>
># ls -ld /root /root/.cache /root/.cache/guix /root/.cache/guix/{inferiors,profiles}
>drwx------ 5 root root 4096 Aug 26 20:46 /root/
>drwxr-xr-x 4 root root 4096 Jul 22  2024 /root/.cache/
>drwxr-xr-x 6 root root 4096 Mar 16 22:19 /root/.cache/guix/
>drwxr-xr-x 2 root root 4096 Aug 10 01:41 /root/.cache/guix/inferiors/
>drwxr-xr-x 2 root root 4096 Mar 16 22:21 /root/.cache/guix/profiles/
>
>After seeing this I suspected the problem might be that the *Guix daemon*,
>which is running unprivileged, cannot access these files.  And indeed, if
>I do `chmod 711 /root`, then `guix gc` stops printing the "cannot read
>potential root" messages.  But it still doesn't _work_:
>
># guix gc
>finding garbage collector roots...
>guix gc: error: program `/gnu/store/6px1m9n904j8s4hyrmlds707sfnq52d9-guix-1.4.0-41.826e305/bin/guix' failed with exit code 1
>
>So that's _really_ bad UX, but again, not the immediate problem.  Since I
>do now know that it's the daemon that's having problems, I check the logs:
>
># tail -3 /var/log/guix-daemon.log
>2025-08-26 20:56:21 accepted connection from pid 172, user root
>2025-08-26 20:56:21 accepted connection from pid 176, user guix-daemon
>2025-08-26 20:56:21 guix gc: error: creating directory `/var/guix/profiles/per-user/guix-daemon': Permission denied
>
>Well, that's suggestive...
>
># ls -la /var/guix/profiles/per-user
>total 28
>drwxr-xr-x 7 root         root         4096 Apr 25 20:03 ./
>drwxr-xr-x 3 root         root         4096 Aug 26 20:25 ../
>drwxr-xr-x 2 root         root         4096 Aug 26 20:25 root/
>drwxr-xr-x 2 user1        user1        4096 Apr 25 20:03 user1/
>drwxr-xr-x 2 user2        user2        4096 Apr 25 20:03 user2/
>drwxr-xr-x 2 user3        user3        4096 Apr 25 20:03 user3/
>
>(actual user names redacted)

This doesn't look okay, I think both /var/guix/profiles and /var/guix/profiles/per-user should be owned by guix-daemon. This goes basically for everything under /var/guix, except for profiles/per-user/X, where the owner should be X. Though now looking into guix-ownership service it seems it doesn't try to change this ownership, only of /var/guix. While on the other hand the guix-install.sh script does - it chowns everything and then reverts root's profile to root. An oversight? 

>
># mkdir /var/guix/profiles/per-user/guix-daemon
># chown guix-daemon:guix-daemon /var/guix/profiles/per-user/guix-daemon
># guix gc
>finding garbage collector roots...
>deleting garbage...
>[7 MiB] deleting '/gnu/store/r993z4wdyqqwzxlif1hvqzp6cqhqr2bw-rustc-1.76.0-src.tar.zst.drv'
>[7 MiB] deleting '/gnu/store/afnyx8a8qj4wlhywv0zsf57lmk8yskzc-rustc-1.76.0-src.tar.gz.drv'
>...
>[38344 MiB] deleting '/gnu/store/yd1hpyjjmzmq5qmlv6q2ycqlymsma9rh-freeglut-3.4.0-builder'
>[38344 MiB] deleting '/gnu/store/3z0np2ad898193wws74k54rzppr356cv-ipxe-qemu-1.21.1-3.24db39f-builder'
>deleting `/gnu/store/trash'
>guix gc: error: making `var/empty' writable: Operation not permitted
>
>Well yeah! /var/empty is supposed to be empty at all times!  Is this
>not how it's supposed to be?
>
># ls -ld /var/empty
>dr-xr-xr-x 2 root root 4096 Jul 22  2024 /var/empty/
>
>But if I set it to be world-writable I still get the same error.  In
>fact, even if I make it *owned by guix-daemon*, I still get the same
>error!

I think that resolving the /var/guix permissions might solve this issue as well. The error says var/empty, not /var/empty, implying it is under whatever the guix daemon's pwd is. Well, I unfortunately am just on phone and since recently I decided to try update emacs on it and ended up with emacs without git or anything, I cannot check now what file it is actually trying to create from source like I usually would (with search feature). So best I can tell you, if solving permissions in /var/guix doesnt work, to start guix-daemon with strace and see what file it is actually talking about, not just the relative path. 

>
>And now I'm stuck.  guix gc moved all the trash to /gnu/store/trash,
>but it didn't actually delete any of it.  There's 11G in there, and
>I can't delete it by hand because the store is mounted read-only
>and I don't know how to temporarily override that for this kind of
>manual repair job.  Any advice would be most appreciated.

It is bind mounted, that means you can just umount it. But I would strongly advise against it if it is possible to  let guix solve it on its own. Not sure if stuff in the trash is somehow tracked, if not, it would be safe to do that, yeah. 

Rutherther

PS: please dont send same emails both to guix help and bug guix. This means anyone who replies all will make a new bug report! Omitting the fact that this bug tracker is deprecated in favor of codeberg issues, you should rather use X-Debbugs-Cc to let debbugs send the email with proper email address to reply to - id of the bug. 

>
>zw

Send a report that this bug log contains spam.