guix gc fails catastrophically if the disk has no space left

  • Open
  • quality assurance status badge
Details
3 participants
  • Ludovic Courtès
  • raingloom
  • Stefan
Owner
unassigned
Submitted by
Stefan
Severity
important
Merged with

Debbugs page

S
S
Stefan wrote on 23 May 2020 07:49
(address . bug-guix@gnu.org)
A33D3EFC-90DD-4E78-AFE5-AB0BF146373A@vodafonemail.de
Hi!

While trying to build guix, my disk ran out of space.

building /gnu/store/w5l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv...
100% [#########################################################################]note: build failure may have been caused by lack of free disk space
builder for `/gnu/store/w5l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv' failed with exit code 1
build of /gnu/store/w5l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv failed
View build log at '/var/log/guix/drvs/w5/l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv.bz2'.
guix system: error: build of `/gnu/store/w5l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv' failed

stefan@guix ~$ df -h /
Dateisystem Größe Benutzt Verf. Verw% Eingehängt auf
:/volume5/RaspberryPi/Betriebssysteme/guix-system 1,4T 1,4T 0 100% /

OK, well, there’s certainly a lot of garbage to collect. Due to bad experiences in the past with the same situation, I first try to collect only a little garbage.

stefan@guix ~$ guix gc -F 100M
guix gc: 100 MiB werden freigegeben
finding garbage collector roots...
note: can't create trash directory: creating directory `/gnu/store/trash': No space left on device
deleting garbage...
[0%] deleting '/gnu/store/g30aknl5yxk9jbpx2w7r5d0ak6xa5dl2-grub.cfg.lock'
[0%] deleting '/gnu/store/gidc7jrgbnvfm38qx4sc012kn9llgsvz-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-collection.lock'
[0%] deleting '/gnu/store/9cpj3ir8fnsg5fiqs3x8alai713xzfz0-grub-keymap.de.lock'
[0%] deleting '/gnu/store/z08s3kwlnpqq6aa1v6ydni5qp2k7gkkb-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04'
[0%] deleting '/gnu/store/wv7iwd66q6n5775qa4hgaq9j9f2vyvl4-guix-1.1.0-4.bdc801e.lock'
[0%] deleting '/gnu/store/pab2ylsvas12824qw6j8q2h1qm3gh4x6-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-union.lock'
[0%] deleting '/gnu/store/3ldyga22s6ym9g2pc9ycjwdwqb4i41v2-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04.lock'
[0%] deleting '/gnu/store/nfxgqhjyzwngiadn0l180hsbfnw3h4zy-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-collection.lock'
[0%] deleting '/gnu/store/pab2ylsvas12824qw6j8q2h1qm3gh4x6-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-union'
[0%] deleting '/gnu/store/dahbiixsbi1xhjiim9phl1vginm83bbm-grub.cfg.lock'
[0%] deleting '/gnu/store/61mwka1h77kkhylcs9jj4ij167qgfj39-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04.lock'
[0%] deleting '/gnu/store/q829h36vnwhbdlnzkgdrx2b51kp2mvdv-grub.cfg.lock'
[0%] deleting '/gnu/store/ragpi42x39sf5qwcmcmdn751fp4jfhmv-grub.cfg.lock'
[0%] deleting '/gnu/store/ncdsjfl245k6r7zym6w4sr9fyxrwhb38-grub.cfg.lock'
[0%] deleting '/gnu/store/z08s3kwlnpqq6aa1v6ydni5qp2k7gkkb-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04.lock'
[0%] deleting '/gnu/store/gidc7jrgbnvfm38qx4sc012kn9llgsvz-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-collection'
[0%] deleting '/gnu/store/61mwka1h77kkhylcs9jj4ij167qgfj39-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04'
[0%] deleting '/gnu/store/hmy4ni005digrilzjf93wr9gaj5cq536-computed-files.lock'
[0%] deleting '/gnu/store/hmy4ni005digrilzjf93wr9gaj5cq536-computed-files'
error (ignored): aborting transaction: cannot rollback - no transaction is active
guix gc: error: committing transaction: disk I/O error
stefan@guix ~$

And that’s it. Repeating the guix gc command now results in this immediate error:

stefan@guix ~$ guix gc -F 100M
warning: SQLite database is busy (SQLITE_PROTOCOL)
guix gc: error: setting synchronous mode: locking protocol
stefan@guix ~$ guix gc -F 100M
warning: SQLite database is busy (SQLITE_PROTOCOL)
guix gc: error: setting synchronous mode: locking protocol

A reboot is needed to solve this SQLite error. But retrying all this leads to the same situation again and again. It is kind of a deadlock.

Well, as my guix system boots from an NFS share, in the past I solved this by removing other stuff – not related to guix – from the same disk. But right now there isn't unrelated stuff left to be deleted.

I intend to remove the listed /gnu/store/… files by hand from the NFS server side (which is much easier than removing the read-only bind-mount.) Is this save at least?

This is all on a Raspberry Pi 3b aarch64 machine, if it matters.


Bye

Stefan
S
S
Stefan wrote on 23 May 2020 08:53
(address . 41484@debbugs.gnu.org)
1DFE09AD-D999-4CAC-8362-72772D791380@vodafonemail.de
Hi!

After this trouble with guix gc I did a sudo reboot, and it got worse. At startup I now see these errors:

GC Warning: pthread_getattr_up or pthread_attr_getstack failed for main thread
GC Warning: Couldn't read /proc/stat
Welcome, this is GNU's early boot Guile.
Use --repl for an initrd REPL.

loading kernel modules…
loading '/gnu/store/…-system/boot'...
making '/gnu/store/…-system' the current system...
setting up setuid programs in '/run/setuid-programs'...
warning: failed to make '/gnu/store/…-shadow-4.8.1/bin/newuidmap' setuid-root: No space left on device
warning: failed to make '/gnu/store/…' setuid-root: No space left on device
populating /etc from /gnu/store/…-etc...
ERROR: In procedure fsync:
In procedure fsync: No space left on device

Entering a new prompt. Type `,bt' for a backtrace or `,q' to continue.
GNU Guile 2.2.6
Copyright …

Enter `help' for help.
scheme@(guile-user)> ,bt
In gnu/build/linux-boot.scm:
561:14 10 (_)
In unknown file:
9 (primitive-load "/gnu/store/…")
In ice-9/eval.scm:
619:8 (_ #f)
In unknown file:
7 (primitive-load "/gnu/store/…")
In ice-9/boot-9.scm:
260:13 6 (for-each #<procedure primitive-load (_)> _)
In unknown file:
5 (primitive-load "/gnu/store/_")
In gnu/build/activation.scm:
136:2 4 (activate-users+groups _ _)
In guix/build/syscalls.scm:
1089:4 3 (call-with-file-lock _ _)
In gnu/build/activation.scm:
139:6 2 (_)
In gnu/build/accounts.scm:
253:14 1 (_ (#<<group-entry> name: "root" password: #f gid: …> …) …)
In unknown file:
0 (fsync #<input-output: /etc/group.bJpcke 18>)
scheme@(guile-user)>


Now this is the very end. Even selecting an older system generation is obviously not an option any more.

I’m happy that all this is on my NFS share to which I have access via my server.


Bye

Stefan
R
R
raingloom wrote on 23 May 2020 12:57
(address . bug-guix@gnu.org)
20200523215733.2ed2f0de@riseup.net
On Sat, 23 May 2020 16:49:02 +0200
Stefan <stefan-guix@vodafonemail.de> wrote:

Toggle quote (107 lines)
> Hi!
>
> While trying to build guix, my disk ran out of space.
>
> building
> /gnu/store/w5l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv...
> 100%
> [#########################################################################]note:
> build failure may have been caused by lack of free disk space builder
> for
> `/gnu/store/w5l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv'
> failed with exit code 1 build of
> /gnu/store/w5l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv
> failed View build log at
> '/var/log/guix/drvs/w5/l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv.bz2'.
> guix system: error: build of
> `/gnu/store/w5l7acwbfvn0n8zj2v9by4j3jwbiimj9-guix-1.1.0-4.bdc801e.drv'
> failed
>
> stefan@guix ~$ df -h /
> Dateisystem Größe Benutzt Verf.
> Verw% Eingehängt auf
> :/volume5/RaspberryPi/Betriebssysteme/guix-system 1,4T 1,4T 0
> 100% /
>
> OK, well, there’s certainly a lot of garbage to collect. Due to bad
> experiences in the past with the same situation, I first try to
> collect only a little garbage.
>
> stefan@guix ~$ guix gc -F 100M
> guix gc: 100 MiB werden freigegeben
> finding garbage collector roots...
> note: can't create trash directory: creating directory
> `/gnu/store/trash': No space left on device deleting garbage...
> [0%] deleting
> '/gnu/store/g30aknl5yxk9jbpx2w7r5d0ak6xa5dl2-grub.cfg.lock' [0%]
> deleting
> '/gnu/store/gidc7jrgbnvfm38qx4sc012kn9llgsvz-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-collection.lock'
> [0%] deleting
> '/gnu/store/9cpj3ir8fnsg5fiqs3x8alai713xzfz0-grub-keymap.de.lock'
> [0%] deleting
> '/gnu/store/z08s3kwlnpqq6aa1v6ydni5qp2k7gkkb-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04'
> [0%] deleting
> '/gnu/store/wv7iwd66q6n5775qa4hgaq9j9f2vyvl4-guix-1.1.0-4.bdc801e.lock'
> [0%] deleting
> '/gnu/store/pab2ylsvas12824qw6j8q2h1qm3gh4x6-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-union.lock'
> [0%] deleting
> '/gnu/store/3ldyga22s6ym9g2pc9ycjwdwqb4i41v2-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04.lock'
> [0%] deleting
> '/gnu/store/nfxgqhjyzwngiadn0l180hsbfnw3h4zy-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-collection.lock'
> [0%] deleting
> '/gnu/store/pab2ylsvas12824qw6j8q2h1qm3gh4x6-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-union'
> [0%] deleting
> '/gnu/store/dahbiixsbi1xhjiim9phl1vginm83bbm-grub.cfg.lock' [0%]
> deleting
> '/gnu/store/61mwka1h77kkhylcs9jj4ij167qgfj39-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04.lock'
> [0%] deleting
> '/gnu/store/q829h36vnwhbdlnzkgdrx2b51kp2mvdv-grub.cfg.lock' [0%]
> deleting '/gnu/store/ragpi42x39sf5qwcmcmdn751fp4jfhmv-grub.cfg.lock'
> [0%] deleting
> '/gnu/store/ncdsjfl245k6r7zym6w4sr9fyxrwhb38-grub.cfg.lock' [0%]
> deleting
> '/gnu/store/z08s3kwlnpqq6aa1v6ydni5qp2k7gkkb-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04.lock'
> [0%] deleting
> '/gnu/store/gidc7jrgbnvfm38qx4sc012kn9llgsvz-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04-collection'
> [0%] deleting
> '/gnu/store/61mwka1h77kkhylcs9jj4ij167qgfj39-package-collection-grub-efi-raspberrypi-firmware-u-boot-rpi-3-2.04'
> [0%] deleting
> '/gnu/store/hmy4ni005digrilzjf93wr9gaj5cq536-computed-files.lock'
> [0%] deleting
> '/gnu/store/hmy4ni005digrilzjf93wr9gaj5cq536-computed-files' error
> (ignored): aborting transaction: cannot rollback - no transaction is
> active guix gc: error: committing transaction: disk I/O error
> stefan@guix ~$
>
> And that’s it. Repeating the guix gc command now results in this
> immediate error:
>
> stefan@guix ~$ guix gc -F 100M
> warning: SQLite database is busy (SQLITE_PROTOCOL)
> guix gc: error: setting synchronous mode: locking protocol
> stefan@guix ~$ guix gc -F 100M
> warning: SQLite database is busy (SQLITE_PROTOCOL)
> guix gc: error: setting synchronous mode: locking protocol
>
> A reboot is needed to solve this SQLite error. But retrying all this
> leads to the same situation again and again. It is kind of a deadlock.
>
> Well, as my guix system boots from an NFS share, in the past I solved
> this by removing other stuff – not related to guix – from the same
> disk. But right now there isn't unrelated stuff left to be deleted.
>
> I intend to remove the listed /gnu/store/… files by hand from the NFS
> server side (which is much easier than removing the read-only
> bind-mount.) Is this save at least?
>
> This is all on a Raspberry Pi 3b aarch64 machine, if it matters.
>
>
> Bye
>
> Stefan
>
>
>
>

Try deleting /var/log. That's what I did on my tiny Pentium II desktop
with 4 gigs of HDD space.
L
L
Ludovic Courtès wrote on 16 Jan 2021 05:51
control message for bug #41484
(address . control@debbugs.gnu.org)
87ft31yopu.fsf@gnu.org
severity 41484 important
quit
L
L
Ludovic Courtès wrote on 16 Jan 2021 06:23
control message for bug #45266
(address . control@debbugs.gnu.org)
87czy5yn90.fsf@gnu.org
merge 45266 41484
quit
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 41484@patchwise.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 41484
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch