GNU bug report logs

#77086 Filesystems not unmounted on reboot

Package	Source(s)		Maintainer(s)
guix		PTS Buildd Popcon

Reported by Ian Eure <ian@retrospec.tv>
Date 2025-03-17T21:11:02
Severity grave

Reply or subscribe to this bug. View this bug as an mbox, status mbox, or maintainer mbox

Report forwarded to bug-guix@gnu.org:
bug#77086; Package guix. (Mon, 17 Mar 2025 21:11:03 GMT) (full text, mbox, link).

Acknowledgement sent to Ian Eure <ian@retrospec.tv>:
New bug report received and forwarded. Copy sent to bug-guix@gnu.org. (Mon, 17 Mar 2025 21:11:03 GMT) (full text, mbox, link).

Message #5 received at submit@debbugs.gnu.org (full text, mbox, reply):

Starting recently (last 2-3 weeks), my Guix System machines have
failed to unmount filesystems when restarted.  ex. inside an Emacs
EXWM X11 session, running `sudo reboot' in a shell causes the 
system
to perform a lengthy fsck on next boot, saying "recovering 
journal."

Others on IRC have noticed this as well.  Some think it’s 
correlated
to running `sudo guix system reconfigure', but I observe it 
whether
I’ve reconfigured or not.

I’m using LUKS1 whole-disk encryption.

Here’s the `guix system describe' output from a laptop (ThinkPad 
X280)
which is exhibiting the problem:

  Generation 46	Mar 17 2025 07:22:11	(current)
    file name: /var/guix/profiles/system-46-link
    canonical file name:
    /gnu/store/zjk9w5jwdrlsc6kr8s4iq3317ys17shf-system
    label: GNU with Linux 6.13.7
    bootloader: grub-efi
    root device: /dev/mapper/cryptroot
    kernel:
    /gnu/store/q8vjcaaykh0p7769xk1rz29di1axry07-linux-6.13.7/bzImage
    channels:
      guix:
        repository URL:
        https://git.savannah.gnu.org/git/guix.git
        branch: master
        commit: 98be320183579b3d09cf4059e86a9781485628b4
      nonguix:
        repository URL: https://gitlab.com/nonguix/nonguix
        branch: master
        commit: fa416ebdf9e4d5c3b9676ded8829c5875bcf4f0e
      atomized:
        repository URL:
        https://codeberg.org/ieure/atomized-guix.git
        branch: main
        commit: bd4ef7fad637b7213e1a96885b61a6ca471eeb0b
    configuration file:
    /gnu/store/dx2hzsqh47iz2pbiwiqsd8w7lpc6f4jy-configuration.scm

-- Ian

Information forwarded to bug-guix@gnu.org:
bug#77086; Package guix. (Tue, 18 Mar 2025 05:44:01 GMT) (full text, mbox, link).

Message #8 received at 77086@debbugs.gnu.org (full text, mbox, reply):

I have also seen this on a netbook where I have never used "guix 
system reconfigure"; I only ever use "guix deploy" from another 
machine to upgrade that system.

This is the version of Guix used:

 guix 9212459
   repository URL: https://codeberg.org/guix/guix-mirror
   branch: master
   commit: 92124591eedf27e988c84f75acd4b4d99ff43122
 rosenthal 5172fac
   repository URL: https://codeberg.org/hako/rosenthal.git
   branch: trunk
   commit: 5172fac369025bc98cbfb925b728b2cae20ac2c5

The kernel is linux-libre-lts.

-- 
Ricardo

Severity set to 'grave' from 'normal' Request was from Felix Lechner <felix.lechner@lease-up.com> to control@debbugs.gnu.org. (Thu, 20 Mar 2025 20:49:02 GMT) (full text, mbox, link).

Information forwarded to bug-guix@gnu.org:
bug#77086; Package guix. (Fri, 21 Mar 2025 13:45:04 GMT) (full text, mbox, link).

Message #13 received at 77086@debbugs.gnu.org (full text, mbox, reply):

[Message part 1 (text/plain, inline)]

Hello guys,

so I started looking into this a bit (not promising any results though),
and I can pretty confidently say that there is indeed an issue in the
unmount.

I have created a VM system, tried rebooting a few times and it was fine,
however then I tried reconfiguring, and for that run I got an error upon
reboot. Not only that, I can't boot it anymore :) the filesystem got
corrupted in a way that prevents boots. Welp.

I can't really say at current moment what is causing this, but the
problem is that the root device is busy so it can't be unmounted.
I have a hypothesis given the log I see: that the root filesystem
is being unmounted first rather than last like it should be.
Could the reconfigure throw shepherd off? I am also CCing Ludovic,
I hope he won't mind.

I would like to also point out an e-mail in the
guix-devel sent recently that someone got a timer service running after
reconfigure, but not after reboot, where after reboot the timer module
is not imported by default unless put to service's modules, but after
reconfigure it works, so this leaves me with yet another point for the
impression that shepherd behaves differently on reboot as opposed to
reconfigure reload.

I will try digging more, but I am not that knowledgeable about
shepherd yet, so it will take longer time.

I am attaching both log (starting after reboot command)
and the configuration used for the vm.

[reboot_log (text/plain, attachment)]

[simple.scm (text/plain, attachment)]

[Message part 4 (text/plain, inline)]

Regards,
Rutherther

Information forwarded to bug-guix@gnu.org:
bug#77086; Package guix. (Fri, 21 Mar 2025 16:52:02 GMT) (full text, mbox, link).

Message #16 received at 77086@debbugs.gnu.org (full text, mbox, reply):

[Message part 1 (text/plain, inline)]

Hello guys,

I am writing with an update, because not everything I wrote in the first
e-mail was true exactly. Please let me know if updates distrub you and I
won't be CCing you again.

"Rutherther" <rutherther@ditigal.xyz> writes:

>
> I have created a VM system, tried rebooting a few times and it was fine,
> however then I tried reconfiguring, and for that run I got an error upon
> reboot. Not only that, I can't boot it anymore :) the filesystem got
> corrupted in a way that prevents boots. Welp.

This wasn't the case, I just haven't realized the correct partition to
boot from is /dev/vda2, not /dev/vda1, I copied the image from guix repo
and haven't realized it's wrong (I suppose it haven't used efi before,
but switched to it?). Since the images get the FS replaced by custom
partitions, this error was visible only after reconfigure. After
changing it to /dev/vda2 I am able to boot, but fsck is ran to repair
damage done.

>
> I am attaching both log (starting after reboot command)
> and the configuration used for the vm.
>

Since this was the first reconfigure after the system obtained by guix
system image, the log isn't exactly accurate to what would happen on
subsequent reconfigures.

The previous time the error was that / is busy. Maybe it was because of a
change in the filesystem services from the initial image to custom
config with `file-systems`
While that is definitely an issue as well, it might not be completely
related to the issue reported here by others. (imo there should be a way
for cases where the user is making big changes like changing file
systems to tell reconfigure to not apply it to the running system, to
only apply the bootloader and switch only after boot)

I am attaching log of reboot/halt after subsequent reconfigures where
the issue is manifested a bit differently. It is not / that would be
busy, but /run/user that is busy. I still think it could be because of
wrong order of stopping services, /run/user/0 is unmounted last,
but root file system tries to be unmounted prior to that.

Note that this doesn't happen without a reconfigure, I get this behavior
only after a reconfigure. The reconfigure can happen with no changes to
the config nor guix instance ran. I get this behavior consistently
on every reconfigure ran!

[  202.675565] shepherd[1]: Ignoring error while stopping root-file-system: (system-error "umount" "~S: ~A" ("/run/user" "Device or resource busy") (16))

Regards,
and apologies for two messages when it could've been one if I paid
more attention.
Rutherther

[reboot_log (text/plain, attachment)]

Information forwarded to bug-guix@gnu.org:
bug#77086; Package guix. (Fri, 21 Mar 2025 18:58:02 GMT) (full text, mbox, link).

Message #19 received at 77086@debbugs.gnu.org (full text, mbox, reply):

Hello,

I was testing on commmit Ian sent to reproduce, and now moved to
newest guix, seems to have been solved (at least the specific error I
was getting),
possibly by shepherd 1.0.3 update.

With newest guix both the first reconfigure of guix system image
and subsequent ones are fine.

Let me know if you're still experiencing this issue after updating
and I might try harder to reproduce if I got a different issue on first
try. I am afraid this will be hard to debug on real hw as you don't
really get the log in /var/log/messages for shutting down the system, I
had to get it through a serial line via stdout of qemu.

Also I was able to reproduce the issue on the older
guix just by running the shepherd services upgrade scm script,
no need for full reconfigure, this shows that something has gone
wrong when shepherd was reloading the services. Do we have some kind of
a test for this in guix / shepherd so it can't happen anymore in the future?

Information forwarded to bug-guix@gnu.org:
bug#77086; Package guix. (Sat, 22 Mar 2025 04:51:01 GMT) (full text, mbox, link).

Message #22 received at 77086@debbugs.gnu.org (full text, mbox, reply):

On Fri, Mar 21, 2025 at 05:50:51PM +0100, Rutherther via Bug reports for GNU Guix wrote:
> I am writing with an update, because not everything I wrote in the first
> e-mail was true exactly. Please let me know if updates distrub you and I
> won't be CCing you again.

Please don't hesitate to send these kinds of investigations again! It's
helpful.

Information forwarded to bug-guix@gnu.org:
bug#77086; Package guix. (Sat, 22 Mar 2025 04:52:02 GMT) (full text, mbox, link).

Information forwarded to bug-guix@gnu.org:
bug#77086; Package guix. (Fri, 28 Mar 2025 15:45:02 GMT) (full text, mbox, link).

Message #28 received at 77086@debbugs.gnu.org (full text, mbox, reply):

I’m still seeing this problem on at least one machine.

Here’s the system configuration: https://paste.debian.net/1365931/

It depends on my personal channel: 
https://codeberg.org/ieure/atomized-guix

I captured the state of the machine just now, then ran `sync' and 
`sudo reboot'.  It fsck’d on boot.

Script started on 2025-03-28 08:32:12-07:00 [TERM="dumb" 
TTY="/dev/pts/1" COLUMNS="191" LINES="39"]
isot0pe!ieure:~$ w
08:32:13 up 2 days, 19:43,  2 users,  load average: 1.23, 0.34, 
0.12
USER     TTY        LOGIN@   IDLE   JCPU   PCPU WHAT
ieure    seat0     Tue13    0.00s  0.00s  0.03s 
/gnu/store/wpdr1pfczc5a5yn7ii80qjgcgdv42jr2-gdm-46.2/libexec/gdm-x-session 
--register-session --run-script 
/gnu/store/mmw3bf5wachcd96i84770yzf
ieure    :0        Tue13   ?xdm?   1:59   0.03s 
/gnu/store/wpdr1pfczc5a5yn7ii80qjgcgdv42jr2-gdm-46.2/libexec/gdm-x-session 
--register-session --run-script 
/gnu/store/mmw3bf5wachcd96i84770yzf
isot0pe!ieure:~$ guix describe
Generation 62	Mar 25 2025 11:59:36	(current)
 guix dbef60e
   repository URL: https://git.savannah.gnu.org/git/guix.git
   branch: master
   commit: dbef60edb356246855ad6749936ee511fc1a9b4b
 atomized f6fba0e
   repository URL: https://codeberg.org/ieure/atomized-guix.git
   branch: main
   commit: f6fba0e2b207b69ab093f2be47b306242c748ab8
 nonguix a96e245
   repository URL: https://gitlab.com/nonguix/nonguix
   branch: master
   commit: a96e2451bda5aaf9b48339edee392c6a3017d730
isot0pe!ieure:~$ guix system describe
Generation 49	Mar 25 2025 12:05:36	(current)
 file name: /var/guix/profiles/system-49-link
 canonical file name: 
 /gnu/store/wkzxc2nbiydwr42v8a0702xl128n65b0-system
 label: GNU with Linux 6.13.7
 bootloader: grub-efi
 root device: /dev/mapper/cryptroot
 kernel: 
 /gnu/store/5mac890q3yhfi6pvxdl8mmpyngmra5s0-linux-6.13.7/bzImage
 channels:
   guix:
     repository URL: https://git.savannah.gnu.org/git/guix.git
     branch: master
     commit: dbef60edb356246855ad6749936ee511fc1a9b4b
   atomized:
     repository URL: https://codeberg.org/ieure/atomized-guix.git
     branch: main
     commit: f6fba0e2b207b69ab093f2be47b306242c748ab8
   nonguix:
     repository URL: https://gitlab.com/nonguix/nonguix
     branch: master
     commit: a96e2451bda5aaf9b48339edee392c6a3017d730
 configuration file: 
 /gnu/store/dhmn4825hvfvkbfbvb7xg6rkhdcj38ii-configuration.scm
isot0pe!ieure:~$ exit

Script done on 2025-03-28 08:32:30-07:00 [COMMAND_EXIT_CODE="0"]

-- Ian

Information forwarded to bug-guix@gnu.org:
bug#77086; Package guix. (Sat, 29 Mar 2025 18:11:02 GMT) (full text, mbox, link).

Message #31 received at 77086@debbugs.gnu.org (full text, mbox, reply):

Hi all,

Here’s a less dead link to the config for my system exhibiting the 
problem: https://paste.debian.net/1366307/

I set it to never expire this time.

The FS stuff is very pedestrian, it’s more or less what the 
graphical installer generates, since that’s how I installed that 
system.

Thanks,
 -- Ian

Information forwarded to bug-guix@gnu.org:
bug#77086; Package guix. (Tue, 01 Apr 2025 07:26:02 GMT) (full text, mbox, link).

Message #34 received at 77086@debbugs.gnu.org (full text, mbox, reply):

Hi,

Rutherther <rutherther@ditigal.xyz> skribis:

> I was testing on commmit Ian sent to reproduce, and now moved to
> newest guix, seems to have been solved (at least the specific error I
> was getting),
> possibly by shepherd 1.0.3 update.
>
> With newest guix both the first reconfigure of guix system image
> and subsequent ones are fine.

I believe I’m still experiencing it on my laptop (but it’s harmless, I
only briefly see e2fsck saying “recovering journal” at boot time), but
of course, not the slightest clue in /var/log/messages.

> Let me know if you're still experiencing this issue after updating
> and I might try harder to reproduce if I got a different issue on first
> try. I am afraid this will be hard to debug on real hw as you don't
> really get the log in /var/log/messages for shutting down the system, I
> had to get it through a serial line via stdout of qemu.
>
> Also I was able to reproduce the issue on the older
> guix just by running the shepherd services upgrade scm script,
> no need for full reconfigure, this shows that something has gone
> wrong when shepherd was reloading the services. Do we have some kind of
> a test for this in guix / shepherd so it can't happen anymore in the future?

There’s a system test, “root-unmount” in (gnu tests base), but it
succeeds: <https://ci.guix.gnu.org/build/9790935/details>.

Perhaps the problem only shows up with more complex system configs?
My root partition is on a LUKS device, but I think the problem is more
something like EBUSY upon ‘umount’ due to stale processes.

Thanks,
Ludo’.

Information forwarded to bug-guix@gnu.org:
bug#77086; Package guix. (Wed, 02 Apr 2025 15:32:01 GMT) (full text, mbox, link).

Message #37 received at 77086@debbugs.gnu.org (full text, mbox, reply):

Hello Ludo,

Ludovic Courtès <ludo@gnu.org> writes:

> Hi,
>
> Rutherther <rutherther@ditigal.xyz> skribis:
>
>> I was testing on commmit Ian sent to reproduce, and now moved to
>> newest guix, seems to have been solved (at least the specific error I
>> was getting),
>> possibly by shepherd 1.0.3 update.
>>
>> With newest guix both the first reconfigure of guix system image
>> and subsequent ones are fine.
>
> I believe I’m still experiencing it on my laptop (but it’s harmless, I
> only briefly see e2fsck saying “recovering journal” at boot time), but
> of course, not the slightest clue in /var/log/messages.

To make it more clear, what I was able to get in the VM was full disk
recovery that was clearing wrong inodes, every time and I even got disk
corruptions for the files produced during reconfigure (which was the
last thing I did before reboot). So I am not sure if this is the same
thing, do you know, is this one line printed only when there is
something wrong with the journal, or is it printed every time like sort
of a 'welcome' message?

>
>> Let me know if you're still experiencing this issue after updating
>> and I might try harder to reproduce if I got a different issue on first
>> try. I am afraid this will be hard to debug on real hw as you don't
>> really get the log in /var/log/messages for shutting down the system, I
>> had to get it through a serial line via stdout of qemu.
>>
>> Also I was able to reproduce the issue on the older
>> guix just by running the shepherd services upgrade scm script,
>> no need for full reconfigure, this shows that something has gone
>> wrong when shepherd was reloading the services. Do we have some kind of
>> a test for this in guix / shepherd so it can't happen anymore in the future?
>
> There’s a system test, “root-unmount” in (gnu tests base), but it
> succeeds: <https://ci.guix.gnu.org/build/9790935/details>.

Thanks for pointing out this test, good to know about it, although it's
not exactly what I had in mind. I was able to reproduce previously by
running the upgrade-shepherd-services.scm that is ran upon reconfigure,
without it, root fs unmounted cleanly. So the test I had in mind would
be to test that when shepherd is upgraded like this, there aren't
changes to how the services are stopped.

>
> Perhaps the problem only shows up with more complex system configs?
> My root partition is on a LUKS device, but I think the problem is more
> something like EBUSY upon ‘umount’ due to stale processes.

I've tried reproducing with bjoli's config last friday, they were
experiencing it and sent their config to IRC chat. I was unable to
reproduce (though I have to confess I changed the fs to ext4 for my
convenience).
So I am not sure if it is reproducible even given a config (though maybe
something is relevant on the vm vs real machine boot, but I wouldn't
expect it...)

I have yet to try Ian's config, but it's going to take me some time to
get to it as their config is more complicated, mainly the network disk,
which I would like to not skip any disk related stuff (and I think Ian
is missing file-systems requirement on their shepherd autofs service,
but it shouldn't really cause this issue as root filesystem unmount
should be recursive).

Regards,
Rutherther

Information forwarded to bug-guix@gnu.org:
bug#77086; Package guix. (Thu, 03 Apr 2025 09:21:02 GMT) (full text, mbox, link).

Message #40 received at 77086@debbugs.gnu.org (full text, mbox, reply):

Hello,

Rutherther <rutherther@ditigal.xyz> skribis:

>> There’s a system test, “root-unmount” in (gnu tests base), but it
>> succeeds: <https://ci.guix.gnu.org/build/9790935/details>.
>
> Thanks for pointing out this test, good to know about it, although it's
> not exactly what I had in mind. I was able to reproduce previously by
> running the upgrade-shepherd-services.scm that is ran upon reconfigure,
> without it, root fs unmounted cleanly. So the test I had in mind would
> be to test that when shepherd is upgraded like this, there aren't
> changes to how the services are stopped.

I’m not sure I understand: do you have a reproducer in a VM, either
manual or automatic?  If you do, please share; if it’s manual, you can
share the steps you followed.

Thanks in advance! :-)

Ludo’.

Information forwarded to bug-guix@gnu.org:
bug#77086; Package guix. (Thu, 03 Apr 2025 09:39:02 GMT) (full text, mbox, link).

Message #43 received at 77086@debbugs.gnu.org (full text, mbox, reply):

Hello, yes, I shared a (manual) reproducer earlier in this thread. But as I was saying, it was only for commits from Ian's initial email. I was unable to reproduce with same steps after updating to newest (which among other things has shepherd 1.0.3, but I havent bisected to see what commit has fixed the case I got into). Unfortunately people still seem to experience it, so I must have seen other issue / just one possible cause of this one when there are several. I havent sent the final config, one change to it I shared was just mentioned in text. I can send the final config later. I was able to get error on unmounting every time I run reconfigure or just the upgrade shepherd services script.

On April 3, 2025 11:19:45 AM GMT+02:00, "Ludovic Courtès" <ludo@gnu.org> wrote:
>Hello,
>
>Rutherther <rutherther@ditigal.xyz> skribis:
>
>>> There’s a system test, “root-unmount” in (gnu tests base), but it
>>> succeeds: <https://ci.guix.gnu.org/build/9790935/details>.
>>
>> Thanks for pointing out this test, good to know about it, although it's
>> not exactly what I had in mind. I was able to reproduce previously by
>> running the upgrade-shepherd-services.scm that is ran upon reconfigure,
>> without it, root fs unmounted cleanly. So the test I had in mind would
>> be to test that when shepherd is upgraded like this, there aren't
>> changes to how the services are stopped.
>
>I’m not sure I understand: do you have a reproducer in a VM, either
>manual or automatic?  If you do, please share; if it’s manual, you can
>share the steps you followed.
>
>Thanks in advance! :-)
>
>Ludo’.

Information forwarded to bug-guix@gnu.org:
bug#77086; Package guix. (Thu, 03 Apr 2025 13:27:02 GMT) (full text, mbox, link).

Message #46 received at 77086@debbugs.gnu.org (full text, mbox, reply):

Hi,

On Tue, Apr 01 2025, Ludovic Courtès wrote:

> I believe I’m still experiencing it on my laptop

Unmounting properly is great, but would it also make sense to add a call
to sync(2) after a reconfigure since some people reboot quickly?

Kind regards
Felix

Information forwarded to bug-guix@gnu.org:
bug#77086; Package guix. (Tue, 08 Apr 2025 09:46:02 GMT) (full text, mbox, link).

Message #49 received at 77086@debbugs.gnu.org (full text, mbox, reply):

Felix Lechner <felix.lechner@lease-up.com> skribis:

> On Tue, Apr 01 2025, Ludovic Courtès wrote:
>
>> I believe I’m still experiencing it on my laptop
>
> Unmounting properly is great, but would it also make sense to add a call
> to sync(2) after a reconfigure since some people reboot quickly?

The ‘root-file-system’ service calls ‘sync’ when shutting down.

Ludo’.

Display info messages

Send a report that this bug log contains spam.

debbugs.gnu.org maintainers <help-debbugs@gnu.org>. Last modified: Fri Apr 18 12:45:34 2025; Machine Name: wallace-server

GNU bug tracking system

Debbugs is free software and licensed under the terms of the GNU Public License version 2. The current version can be obtained from https://bugs.debian.org/debbugs-source/.