#41484 guix gc fails catastrophically if the disk has no space left

Message #5 received at submit@debbugs.gnu.org (full text, mbox, reply):

Received: (at submit) by debbugs.gnu.org; 23 May 2020 14:49:17 +0000
From debbugs-submit-bounces@debbugs.gnu.org Sat May 23 10:49:17 2020
Received: from localhost ([127.0.0.1]:35297 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces@debbugs.gnu.org>)
	id 1jcVSf-0006mM-3K
	for submit@debbugs.gnu.org; Sat, 23 May 2020 10:49:17 -0400
Received: from lists.gnu.org ([209.51.188.17]:32862)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <stefan-guix@vodafonemail.de>) id 1jcVSd-0006mF-8E
 for submit@debbugs.gnu.org; Sat, 23 May 2020 10:49:15 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:34892)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <stefan-guix@vodafonemail.de>)
 id 1jcVSd-0003SC-0V
 for bug-guix@gnu.org; Sat, 23 May 2020 10:49:15 -0400
Received: from mx009.vodafonemail.xion.oxcs.net ([153.92.174.39]:9590)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <stefan-guix@vodafonemail.de>)
 id 1jcVSb-0003f6-Ah
 for bug-guix@gnu.org; Sat, 23 May 2020 10:49:14 -0400
Received: from vsmx002.vodafonemail.xion.oxcs.net (unknown [192.168.75.192])
 by mta-6-out.mta.xion.oxcs.net (Postfix) with ESMTP id D45C76058C2
 for <bug-guix@gnu.org>; Sat, 23 May 2020 14:49:05 +0000 (UTC)
Received: from macbook-pro.kuh-wiese.my-router.de (unknown [2.206.141.243])
 by mta-6-out.mta.xion.oxcs.net (Postfix) with ESMTPA id A22D4605BBE
 for <bug-guix@gnu.org>; Sat, 23 May 2020 14:49:03 +0000 (UTC)
From: Stefan <stefan-guix@vodafonemail.de>
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Subject: guix gc fails catastrophically if the disk has no space left
Message-Id: <A33D3EFC-90DD-4E78-AFE5-AB0BF146373A@vodafonemail.de>
Date: Sat, 23 May 2020 16:49:02 +0200
To: bug-guix@gnu.org
Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\))
X-Mailer: Apple Mail (2.3124)
X-VADE-STATUS: LEGIT
Received-SPF: pass client-ip=153.92.174.39;
 envelope-from=stefan-guix@vodafonemail.de;
 helo=mx009.vodafonemail.xion.oxcs.net
X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/23 10:49:06
X-ACL-Warn: Detected OS   = Linux 2.2.x-3.x (no timestamps) [generic]
X-Spam_score_int: -41
X-Spam_score: -4.2
X-Spam_bar: ----
X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3,
 RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001,
 SPF_PASS=-0.001 autolearn=_AUTOLEARN
X-Spam_action: no action
X-Spam-Score: -1.3 (-)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit@debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request@debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit@debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request@debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request@debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces@debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces@debbugs.gnu.org>
X-Spam-Score: -2.3 (--)

Hi!

While trying to build guix, my disk ran out of space.

building /gnu/store/w5l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv...
100% [#########################################################################]note: build failure may have been caused by lack of free disk space
builder for `/gnu/store/w5l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv' failed with exit code 1
build of /gnu/store/w5l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv failed
View build log at '/var/log/guix/drvs/w5/l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv.bz2'.
guix system: error: build of `/gnu/store/w5l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv' failed

stefan@guix ~$ df -h /
Dateisystem                                       Größe Benutzt Verf. Verw% Eingehängt auf
:/volume5/RaspberryPi/Betriebssysteme/guix-system  1,4T    1,4T     0  100% /

OK, well, there’s certainly a lot of garbage to collect. Due to bad experiences in the past with the same situation, I first try to collect only a little garbage.

stefan@guix ~$ guix gc -F 100M
guix gc: 100 MiB werden freigegeben
finding garbage collector roots...
note: can't create trash directory: creating directory `/gnu/store/trash': No space left on device
deleting garbage...
[0%] deleting '/gnu/store/g30aknl5yxk9jbpx2w7r5d0ak6xa5dl2-grub.cfg.lock'
[0%] deleting '/gnu/store/gidc7jrgbnvfm38qx4sc012kn9llgsvz-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-collection.lock'
[0%] deleting '/gnu/store/9cpj3ir8fnsg5fiqs3x8alai713xzfz0-grub-keymap.de.lock'
[0%] deleting '/gnu/store/z08s3kwlnpqq6aa1v6ydni5qp2k7gkkb-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04'
[0%] deleting '/gnu/store/wv7iwd66q6n5775qa4hgaq9j9f2vyvl4-guix-1.1.0-4.bdc801e.lock'
[0%] deleting '/gnu/store/pab2ylsvas12824qw6j8q2h1qm3gh4x6-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-union.lock'
[0%] deleting '/gnu/store/3ldyga22s6ym9g2pc9ycjwdwqb4i41v2-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04.lock'
[0%] deleting '/gnu/store/nfxgqhjyzwngiadn0l180hsbfnw3h4zy-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-collection.lock'
[0%] deleting '/gnu/store/pab2ylsvas12824qw6j8q2h1qm3gh4x6-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-union'
[0%] deleting '/gnu/store/dahbiixsbi1xhjiim9phl1vginm83bbm-grub.cfg.lock'
[0%] deleting '/gnu/store/61mwka1h77kkhylcs9jj4ij167qgfj39-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04.lock'
[0%] deleting '/gnu/store/q829h36vnwhbdlnzkgdrx2b51kp2mvdv-grub.cfg.lock'
[0%] deleting '/gnu/store/ragpi42x39sf5qwcmcmdn751fp4jfhmv-grub.cfg.lock'
[0%] deleting '/gnu/store/ncdsjfl245k6r7zym6w4sr9fyxrwhb38-grub.cfg.lock'
[0%] deleting '/gnu/store/z08s3kwlnpqq6aa1v6ydni5qp2k7gkkb-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04.lock'
[0%] deleting '/gnu/store/gidc7jrgbnvfm38qx4sc012kn9llgsvz-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-collection'
[0%] deleting '/gnu/store/61mwka1h77kkhylcs9jj4ij167qgfj39-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04'
[0%] deleting '/gnu/store/hmy4ni005digrilzjf93wr9gaj5cq536-computed-files.lock'
[0%] deleting '/gnu/store/hmy4ni005digrilzjf93wr9gaj5cq536-computed-files'
error (ignored): aborting transaction: cannot rollback - no transaction is active
guix gc: error: committing transaction: disk I/O error
stefan@guix ~$ 

And that’s it. Repeating the guix gc command now results in this immediate error:

stefan@guix ~$ guix gc -F 100M
warning: SQLite database is busy (SQLITE_PROTOCOL)
guix gc: error: setting synchronous mode: locking protocol
stefan@guix ~$ guix gc -F 100M
warning: SQLite database is busy (SQLITE_PROTOCOL)
guix gc: error: setting synchronous mode: locking protocol

A reboot is needed to solve this SQLite error. But retrying all this leads to the same situation again and again. It is kind of a deadlock.

Well, as my guix system boots from an NFS share, in the past I solved this by removing other stuff – not related to guix – from the same disk. But right now there isn't unrelated stuff left to be deleted. 

I intend to remove the listed /gnu/store/… files by hand from the NFS server side (which is much easier than removing the read-only bind-mount.) Is this save at least?

This is all on a Raspberry Pi 3b aarch64 machine, if it matters.


Bye

Stefan

Send a report that this bug log contains spam.