guix gc fails catastrophically if the disk has no space left

Hi!

While trying to build guix, my disk ran out of space.

building /gnu/store/w5l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv...

100% [#########################################################################]note: build failure may have been caused by lack of free disk space

builder for `/gnu/store/w5l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv' failed with exit code 1

build of /gnu/store/w5l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv failed

View build log at '/var/log/guix/drvs/w5/l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv.bz2'.

guix system: error: build of `/gnu/store/w5l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv' failed

stefan@guix ~$ df -h /

Dateisystem Größe Benutzt Verf. Verw% Eingehängt auf

:/volume5/RaspberryPi/Betriebssysteme/guix-system 1,4T 1,4T 0 100% /

OK, well, there’s certainly a lot of garbage to collect. Due to bad experiences in the past with the same situation, I first try to collect only a little garbage.

stefan@guix ~$ guix gc -F 100M

guix gc: 100 MiB werden freigegeben

finding garbage collector roots...

note: can't create trash directory: creating directory `/gnu/store/trash': No space left on device

deleting garbage...

[0%] deleting '/gnu/store/g30aknl5yxk9jbpx2w7r5d0ak6xa5dl2-grub.cfg.lock'

[0%] deleting '/gnu/store/gidc7jrgbnvfm38qx4sc012kn9llgsvz-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-collection.lock'

[0%] deleting '/gnu/store/9cpj3ir8fnsg5fiqs3x8alai713xzfz0-grub-keymap.de.lock'

[0%] deleting '/gnu/store/z08s3kwlnpqq6aa1v6ydni5qp2k7gkkb-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04'

[0%] deleting '/gnu/store/wv7iwd66q6n5775qa4hgaq9j9f2vyvl4-guix-1.1.0-4.bdc801e.lock'

[0%] deleting '/gnu/store/pab2ylsvas12824qw6j8q2h1qm3gh4x6-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-union.lock'

[0%] deleting '/gnu/store/3ldyga22s6ym9g2pc9ycjwdwqb4i41v2-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04.lock'

[0%] deleting '/gnu/store/nfxgqhjyzwngiadn0l180hsbfnw3h4zy-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-collection.lock'

[0%] deleting '/gnu/store/pab2ylsvas12824qw6j8q2h1qm3gh4x6-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-union'

[0%] deleting '/gnu/store/dahbiixsbi1xhjiim9phl1vginm83bbm-grub.cfg.lock'

[0%] deleting '/gnu/store/61mwka1h77kkhylcs9jj4ij167qgfj39-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04.lock'

[0%] deleting '/gnu/store/q829h36vnwhbdlnzkgdrx2b51kp2mvdv-grub.cfg.lock'

[0%] deleting '/gnu/store/ragpi42x39sf5qwcmcmdn751fp4jfhmv-grub.cfg.lock'

[0%] deleting '/gnu/store/ncdsjfl245k6r7zym6w4sr9fyxrwhb38-grub.cfg.lock'

[0%] deleting '/gnu/store/z08s3kwlnpqq6aa1v6ydni5qp2k7gkkb-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04.lock'

[0%] deleting '/gnu/store/gidc7jrgbnvfm38qx4sc012kn9llgsvz-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-collection'

[0%] deleting '/gnu/store/61mwka1h77kkhylcs9jj4ij167qgfj39-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04'

[0%] deleting '/gnu/store/hmy4ni005digrilzjf93wr9gaj5cq536-computed-files.lock'

[0%] deleting '/gnu/store/hmy4ni005digrilzjf93wr9gaj5cq536-computed-files'

error (ignored): aborting transaction: cannot rollback - no transaction is active

guix gc: error: committing transaction: disk I/O error

stefan@guix ~$

And that’s it. Repeating the guix gc command now results in this immediate error:

stefan@guix ~$ guix gc -F 100M

warning: SQLite database is busy (SQLITE_PROTOCOL)

guix gc: error: setting synchronous mode: locking protocol

stefan@guix ~$ guix gc -F 100M

warning: SQLite database is busy (SQLITE_PROTOCOL)

guix gc: error: setting synchronous mode: locking protocol

A reboot is needed to solve this SQLite error. But retrying all this leads to the same situation again and again. It is kind of a deadlock.

Well, as my guix system boots from an NFS share, in the past I solved this by removing other stuff – not related to guix – from the same disk. But right now there isn't unrelated stuff left to be deleted.

I intend to remove the listed /gnu/store/… files by hand from the NFS server side (which is much easier than removing the read-only bind-mount.) Is this save at least?

This is all on a Raspberry Pi 3b aarch64 machine, if it matters.

Bye

Stefan

Hi!

After this trouble with guix gc I did a sudo reboot, and it got worse. At startup I now see these errors:

GC Warning: pthread_getattr_up or pthread_attr_getstack failed for main thread

GC Warning: Couldn't read /proc/stat

Welcome, this is GNU's early boot Guile.

Use --repl for an initrd REPL.

loading kernel modules…

loading '/gnu/store/…-system/boot'...

making '/gnu/store/…-system' the current system...

setting up setuid programs in '/run/setuid-programs'...

warning: failed to make '/gnu/store/…-shadow-4.8.1/bin/newuidmap' setuid-root: No space left on device

warning: failed to make '/gnu/store/…' setuid-root: No space left on device

…

populating /etc from /gnu/store/…-etc...

ERROR: In procedure fsync:

In procedure fsync: No space left on device

Entering a new prompt. Type `,bt' for a backtrace or `,q' to continue.

GNU Guile 2.2.6

Copyright …

…

Enter `help' for help.

scheme@(guile-user)> ,bt

In gnu/build/linux-boot.scm:

561:14 10 (_)

In unknown file:

9 (primitive-load "/gnu/store/…")

In ice-9/eval.scm:

619:8 (_ #f)

In unknown file:

7 (primitive-load "/gnu/store/…")

In ice-9/boot-9.scm:

260:13 6 (for-each #<procedure primitive-load (_)> _)

In unknown file:

5 (primitive-load "/gnu/store/_")

In gnu/build/activation.scm:

136:2 4 (activate-users+groups _ _)

In guix/build/syscalls.scm:

1089:4 3 (call-with-file-lock _ _)

In gnu/build/activation.scm:

139:6 2 (_)

In gnu/build/accounts.scm:

253:14 1 (_ (#<<group-entry> name: "root" password: #f gid: …> …) …)

In unknown file:

0 (fsync #<input-output: /etc/group.bJpcke 18>)

scheme@(guile-user)>

Now this is the very end. Even selecting an older system generation is obviously not an option any more.

I’m happy that all this is on my NFS share to which I have access via my server.

Bye

Stefan

On Sat, 23 May 2020 16:49:02 +0200

Stefan <stefan-guix@vodafonemail.de> wrote:

Try deleting /var/log. That's what I did on my tiny Pentium II desktop

with 4 gigs of HDD space.

severity 41484 important

quit

merge 45266 41484

quit