#41484 guix gc fails catastrophically if the disk has no space left

Message #5 received at submit@debbugs.gnu.org (full text, mbox, reply):

From: Stefan <stefan-guix@vodafonemail.de>

To: bug-guix@gnu.org

Subject: guix gc fails catastrophically if the disk has no space left

Date: Sat, 23 May 2020 16:49:02 +0200

Hi!

While trying to build guix, my disk ran out of space.

building /gnu/store/w5l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv...
100% [#########################################################################]note: build failure may have been caused by lack of free disk space
builder for `/gnu/store/w5l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv' failed with exit code 1
build of /gnu/store/w5l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv failed
View build log at '/var/log/guix/drvs/w5/l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv.bz2'.
guix system: error: build of `/gnu/store/w5l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv' failed

stefan@guix ~$ df -h /
Dateisystem                                       Größe Benutzt Verf. Verw% Eingehängt auf
:/volume5/RaspberryPi/Betriebssysteme/guix-system  1,4T    1,4T     0  100% /

OK, well, there’s certainly a lot of garbage to collect. Due to bad experiences in the past with the same situation, I first try to collect only a little garbage.

stefan@guix ~$ guix gc -F 100M
guix gc: 100 MiB werden freigegeben
finding garbage collector roots...
note: can't create trash directory: creating directory `/gnu/store/trash': No space left on device
deleting garbage...
[0%] deleting '/gnu/store/g30aknl5yxk9jbpx2w7r5d0ak6xa5dl2-grub.cfg.lock'
[0%] deleting '/gnu/store/gidc7jrgbnvfm38qx4sc012kn9llgsvz-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-collection.lock'
[0%] deleting '/gnu/store/9cpj3ir8fnsg5fiqs3x8alai713xzfz0-grub-keymap.de.lock'
[0%] deleting '/gnu/store/z08s3kwlnpqq6aa1v6ydni5qp2k7gkkb-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04'
[0%] deleting '/gnu/store/wv7iwd66q6n5775qa4hgaq9j9f2vyvl4-guix-1.1.0-4.bdc801e.lock'
[0%] deleting '/gnu/store/pab2ylsvas12824qw6j8q2h1qm3gh4x6-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-union.lock'
[0%] deleting '/gnu/store/3ldyga22s6ym9g2pc9ycjwdwqb4i41v2-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04.lock'
[0%] deleting '/gnu/store/nfxgqhjyzwngiadn0l180hsbfnw3h4zy-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-collection.lock'
[0%] deleting '/gnu/store/pab2ylsvas12824qw6j8q2h1qm3gh4x6-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-union'
[0%] deleting '/gnu/store/dahbiixsbi1xhjiim9phl1vginm83bbm-grub.cfg.lock'
[0%] deleting '/gnu/store/61mwka1h77kkhylcs9jj4ij167qgfj39-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04.lock'
[0%] deleting '/gnu/store/q829h36vnwhbdlnzkgdrx2b51kp2mvdv-grub.cfg.lock'
[0%] deleting '/gnu/store/ragpi42x39sf5qwcmcmdn751fp4jfhmv-grub.cfg.lock'
[0%] deleting '/gnu/store/ncdsjfl245k6r7zym6w4sr9fyxrwhb38-grub.cfg.lock'
[0%] deleting '/gnu/store/z08s3kwlnpqq6aa1v6ydni5qp2k7gkkb-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04.lock'
[0%] deleting '/gnu/store/gidc7jrgbnvfm38qx4sc012kn9llgsvz-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-collection'
[0%] deleting '/gnu/store/61mwka1h77kkhylcs9jj4ij167qgfj39-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04'
[0%] deleting '/gnu/store/hmy4ni005digrilzjf93wr9gaj5cq536-computed-files.lock'
[0%] deleting '/gnu/store/hmy4ni005digrilzjf93wr9gaj5cq536-computed-files'
error (ignored): aborting transaction: cannot rollback - no transaction is active
guix gc: error: committing transaction: disk I/O error
stefan@guix ~$ 

And that’s it. Repeating the guix gc command now results in this immediate error:

stefan@guix ~$ guix gc -F 100M
warning: SQLite database is busy (SQLITE_PROTOCOL)
guix gc: error: setting synchronous mode: locking protocol
stefan@guix ~$ guix gc -F 100M
warning: SQLite database is busy (SQLITE_PROTOCOL)
guix gc: error: setting synchronous mode: locking protocol

A reboot is needed to solve this SQLite error. But retrying all this leads to the same situation again and again. It is kind of a deadlock.

Well, as my guix system boots from an NFS share, in the past I solved this by removing other stuff – not related to guix – from the same disk. But right now there isn't unrelated stuff left to be deleted. 

I intend to remove the listed /gnu/store/… files by hand from the NFS server side (which is much easier than removing the read-only bind-mount.) Is this save at least?

This is all on a Raspberry Pi 3b aarch64 machine, if it matters.


Bye

Stefan

Message #8 received at 41484@debbugs.gnu.org (full text, mbox, reply):

From: Stefan <stefan-guix@vodafonemail.de>

To: 41484@debbugs.gnu.org

Subject: guix gc fails catastrophically if the disk has no space left

Date: Sat, 23 May 2020 17:53:28 +0200

Hi!

After this trouble with guix gc I did a sudo reboot, and it got worse. At startup I now see these errors:

GC Warning: pthread_getattr_up or pthread_attr_getstack failed for main thread
GC Warning: Couldn't read /proc/stat
Welcome, this is GNU's early boot Guile.
Use --repl for an initrd REPL.

loading kernel modules…
loading '/gnu/store/…-system/boot'...
making '/gnu/store/…-system' the current system...
setting up setuid programs in '/run/setuid-programs'...
warning: failed to make '/gnu/store/…-shadow-4.8.1/bin/newuidmap' setuid-root: No space left on device
warning: failed to make '/gnu/store/…' setuid-root: No space left on device
…
populating /etc from /gnu/store/…-etc...
ERROR: In procedure fsync:
In procedure fsync: No space left on device

Entering a new prompt.  Type `,bt' for a backtrace or `,q' to continue.
GNU Guile 2.2.6
Copyright …
…

Enter `help' for help.
scheme@(guile-user)> ,bt
In gnu/build/linux-boot.scm:
   561:14 10 (_)
In unknown file:
           9 (primitive-load "/gnu/store/…")
In ice-9/eval.scm:
   619:8 (_ #f)
In unknown file:
           7 (primitive-load "/gnu/store/…")
In ice-9/boot-9.scm:
   260:13  6 (for-each #<procedure primitive-load (_)> _)
In unknown file:
           5 (primitive-load "/gnu/store/_")
In gnu/build/activation.scm:
    136:2  4 (activate-users+groups _ _)
In guix/build/syscalls.scm:
   1089:4  3 (call-with-file-lock _ _)
In gnu/build/activation.scm:
    139:6  2 (_)
In gnu/build/accounts.scm:
   253:14  1 (_ (#<<group-entry> name: "root" password: #f gid: …> …) …)
In unknown file:
           0 (fsync #<input-output: /etc/group.bJpcke 18>)
scheme@(guile-user)>


Now this is the very end. Even selecting an older system generation is obviously not an option any more. 

I’m happy that all this is on my NFS share to which I have access via my server.


Bye

Stefan

Message #11 received at submit@debbugs.gnu.org (full text, mbox, reply):

From: raingloom <raingloom@riseup.net>

To: bug-guix@gnu.org

Subject: Re: bug#41484: guix gc fails catastrophically if the disk has no space left

Date: Sat, 23 May 2020 21:57:33 +0200

On Sat, 23 May 2020 16:49:02 +0200
Stefan <stefan-guix@vodafonemail.de> wrote:

> Hi!
> 
> While trying to build guix, my disk ran out of space.
> 
> building
> /gnu/store/w5l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv...
> 100%
> [#########################################################################]note:
> build failure may have been caused by lack of free disk space builder
> for
> `/gnu/store/w5l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv'
> failed with exit code 1 build of
> /gnu/store/w5l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv
> failed View build log at
> '/var/log/guix/drvs/w5/l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv.bz2'.
> guix system: error: build of
> `/gnu/store/w5l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv'
> failed
> 
> stefan@guix ~$ df -h /
> Dateisystem                                       Größe Benutzt Verf.
> Verw% Eingehängt auf
> :/volume5/RaspberryPi/Betriebssysteme/guix-system  1,4T    1,4T     0
>  100% /
> 
> OK, well, there’s certainly a lot of garbage to collect. Due to bad
> experiences in the past with the same situation, I first try to
> collect only a little garbage.
> 
> stefan@guix ~$ guix gc -F 100M
> guix gc: 100 MiB werden freigegeben
> finding garbage collector roots...
> note: can't create trash directory: creating directory
> `/gnu/store/trash': No space left on device deleting garbage...
> [0%] deleting
> '/gnu/store/g30aknl5yxk9jbpx2w7r5d0ak6xa5dl2-grub.cfg.lock' [0%]
> deleting
> '/gnu/store/gidc7jrgbnvfm38qx4sc012kn9llgsvz-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-collection.lock'
> [0%] deleting
> '/gnu/store/9cpj3ir8fnsg5fiqs3x8alai713xzfz0-grub-keymap.de.lock'
> [0%] deleting
> '/gnu/store/z08s3kwlnpqq6aa1v6ydni5qp2k7gkkb-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04'
> [0%] deleting
> '/gnu/store/wv7iwd66q6n5775qa4hgaq9j9f2vyvl4-guix-1.1.0-4.bdc801e.lock'
> [0%] deleting
> '/gnu/store/pab2ylsvas12824qw6j8q2h1qm3gh4x6-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-union.lock'
> [0%] deleting
> '/gnu/store/3ldyga22s6ym9g2pc9ycjwdwqb4i41v2-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04.lock'
> [0%] deleting
> '/gnu/store/nfxgqhjyzwngiadn0l180hsbfnw3h4zy-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-collection.lock'
> [0%] deleting
> '/gnu/store/pab2ylsvas12824qw6j8q2h1qm3gh4x6-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-union'
> [0%] deleting
> '/gnu/store/dahbiixsbi1xhjiim9phl1vginm83bbm-grub.cfg.lock' [0%]
> deleting
> '/gnu/store/61mwka1h77kkhylcs9jj4ij167qgfj39-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04.lock'
> [0%] deleting
> '/gnu/store/q829h36vnwhbdlnzkgdrx2b51kp2mvdv-grub.cfg.lock' [0%]
> deleting '/gnu/store/ragpi42x39sf5qwcmcmdn751fp4jfhmv-grub.cfg.lock'
> [0%] deleting
> '/gnu/store/ncdsjfl245k6r7zym6w4sr9fyxrwhb38-grub.cfg.lock' [0%]
> deleting
> '/gnu/store/z08s3kwlnpqq6aa1v6ydni5qp2k7gkkb-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04.lock'
> [0%] deleting
> '/gnu/store/gidc7jrgbnvfm38qx4sc012kn9llgsvz-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-collection'
> [0%] deleting
> '/gnu/store/61mwka1h77kkhylcs9jj4ij167qgfj39-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04'
> [0%] deleting
> '/gnu/store/hmy4ni005digrilzjf93wr9gaj5cq536-computed-files.lock'
> [0%] deleting
> '/gnu/store/hmy4ni005digrilzjf93wr9gaj5cq536-computed-files' error
> (ignored): aborting transaction: cannot rollback - no transaction is
> active guix gc: error: committing transaction: disk I/O error
> stefan@guix ~$ 
> 
> And that’s it. Repeating the guix gc command now results in this
> immediate error:
> 
> stefan@guix ~$ guix gc -F 100M
> warning: SQLite database is busy (SQLITE_PROTOCOL)
> guix gc: error: setting synchronous mode: locking protocol
> stefan@guix ~$ guix gc -F 100M
> warning: SQLite database is busy (SQLITE_PROTOCOL)
> guix gc: error: setting synchronous mode: locking protocol
> 
> A reboot is needed to solve this SQLite error. But retrying all this
> leads to the same situation again and again. It is kind of a deadlock.
> 
> Well, as my guix system boots from an NFS share, in the past I solved
> this by removing other stuff – not related to guix – from the same
> disk. But right now there isn't unrelated stuff left to be deleted. 
> 
> I intend to remove the listed /gnu/store/… files by hand from the NFS
> server side (which is much easier than removing the read-only
> bind-mount.) Is this save at least?
> 
> This is all on a Raspberry Pi 3b aarch64 machine, if it matters.
> 
> 
> Bye
> 
> Stefan
> 
> 
> 
> 

Try deleting /var/log. That's what I did on my tiny Pentium II desktop
with 4 gigs of HDD space.

Send a report that this bug log contains spam.