#28772 guix system reconfigure after kernel panic user or group not created

Message #5 received at submit@debbugs.gnu.org (full text, mbox, reply):

From: Oleg Pykhalov <go.wigust@gmail.com>

To: bug-guix@gnu.org

Subject: guix system reconfigure after kernel panic user or group not created

Date: Tue, 10 Oct 2017 08:51:15 +0300

Hello Guix,

During 'guix system reconfigure' I got a kernel panic.

Getting it quite often while 'guix system reconfigure' at Linux magnolia
4.13.4-gnu #1 SMP 1 x86_64 GNU/Linux.  But it's not a subject of current
report.

After reboot I tried another attempt to make a 'guix system
reconfigure', but because of /etc/group.lock and /etc/passwd.lock
operating-system groups and users fields didn't applied.

I tried to get a creation of adbusers group and add my user to it, but
because of those files I cannot do it, until manually removed them.

So, do we need to check if *.lock files are exist and remove them at the
start of 'guix system reconfigure' OR abort reconfigure and creation of
Grub boot entry OR something else?

Thanks,
Oleg.

Message #8 received at 28772@debbugs.gnu.org (full text, mbox, reply):

From: Oleg Pykhalov <go.wigust@gmail.com>

To: 28772@debbugs.gnu.org

Subject: Re: bug#28772: guix system reconfigure after kernel panic user or group not created

Date: Tue, 10 Oct 2017 17:55:39 +0300

Oleg Pykhalov <go.wigust@gmail.com> writes:

[...]

> Getting it quite often while 'guix system reconfigure' at Linux magnolia
> 4.13.4-gnu #1 SMP 1 x86_64 GNU/Linux.  But it's not a subject of current
> report.

Kernel panic was caused by my miss configuring.  I used a Guile module
on reconfigure via GUILE_LOAD_PATH and (use-modles (my-module) in
config.scm.  I guess it's also required after reconfigure, according to
some times getting errors after reconfigure: cannot find my-module.  So
I need to add my-module globally on system.

[...]

Message #11 received at 28772@debbugs.gnu.org (full text, mbox, reply):

From: ludo@gnu.org (Ludovic Courtès)

To: Oleg Pykhalov <go.wigust@gmail.com>

Cc: 28772@debbugs.gnu.org

Subject: Re: bug#28772: guix system reconfigure after kernel panic user or group not created

Date: Tue, 10 Oct 2017 17:37:39 +0200

Hello,

Oleg Pykhalov <go.wigust@gmail.com> skribis:

> During 'guix system reconfigure' I got a kernel panic.

Can you show the exact command and its output?

A user-land program is not supposed to be able to cause a kernel panic;
if it does, that’s a kernel bug.

But perhaps you got the kernel panic *after* rebooting in the
reconfigured system?  That could well be a GuixSD bug, indeed.

> After reboot I tried another attempt to make a 'guix system
> reconfigure', but because of /etc/group.lock and /etc/passwd.lock
> operating-system groups and users fields didn't applied.
>
> I tried to get a creation of adbusers group and add my user to it, but
> because of those files I cannot do it, until manually removed them.
>
> So, do we need to check if *.lock files are exist and remove them at the
> start of 'guix system reconfigure' OR abort reconfigure and creation of
> Grub boot entry OR something else?

I’m not sure I understand what happened.  Could you copy/paste commands
and outputs, or transcribe what happened?

TIA,
Ludo’.

Message #14 received at 28772@debbugs.gnu.org (full text, mbox, reply):

From: ludo@gnu.org (Ludovic Courtès)

To: Oleg Pykhalov <go.wigust@gmail.com>

Cc: 28772@debbugs.gnu.org

Subject: Re: bug#28772: guix system reconfigure after kernel panic user or group not created

Date: Wed, 11 Oct 2017 15:27:18 +0200

Oleg Pykhalov <go.wigust@gmail.com> skribis:

> Oleg Pykhalov <go.wigust@gmail.com> writes:
>
> [...]
>
>> Getting it quite often while 'guix system reconfigure' at Linux magnolia
>> 4.13.4-gnu #1 SMP 1 x86_64 GNU/Linux.  But it's not a subject of current
>> report.
>
> Kernel panic was caused by my miss configuring.  I used a Guile module
> on reconfigure via GUILE_LOAD_PATH and (use-modles (my-module) in
> config.scm.  I guess it's also required after reconfigure, according to
> some times getting errors after reconfigure: cannot find my-module.  So
> I need to add my-module globally on system.

Without seeing the config, I cannot tell whether it’s a bug on our side
or not.

However, if you’re confident that the bug is in your code, that’s good.
;-)  Should we close this bug?

Thanks,
Ludo’.

Message #17 received at 28772@debbugs.gnu.org (full text, mbox, reply):

From: Oleg Pykhalov <go.wigust@gmail.com>

To: ludo@gnu.org (Ludovic Courtès)

Cc: 28772@debbugs.gnu.org

Subject: Re: bug#28772: guix system reconfigure after kernel panic user or group not created

Date: Wed, 11 Oct 2017 20:19:36 +0300

[Message part 1 (text/plain, inline)]

Hello Ludovic,

apologies for not adding logs before.  It's hard to do when I do guix
commands from Xterm and not from Emacs.  Emacs *shell* or *compilation*
buffers will eat all memory if they get too much text.

Probably need to redirect STDOUT STDERR in file when Xterm do guix.

I heard Guix folks work on implementing tiny log output to console and
redirect everything else to a log file.  This will be my life saver.

ludo@gnu.org (Ludovic Courtès) writes:

> Hello,
>
> Oleg Pykhalov <go.wigust@gmail.com> skribis:
>
>> During 'guix system reconfigure' I got a kernel panic.
>
> Can you show the exact command and its output?

Sorry, as I said this is not a topic and I don't want to do it again and
I caution to make it on my current system.

I will setup a specific Guix VM for this, where I could make a 'system
reconfigure'.  Then I'll create a new bug report with full log.

Neverless I'll leave a how-to reproduce it below for at least for myself
TODO list.

The problem
===========

The bigger problem from my view are files like /etc/group.lock and
/etc/passwd.lock.  For example:

    sudo touch /etc/group.lock

/etc/config.scm

    (operating-system
      ;; …
      (groups (cons
         (user-group (name "test"))
         %base-groups)))

reconfigure log

    

[report.send (text/plain, attachment)]

[Message part 3 (text/plain, inline)]

The new system generation was produced without "test" group, so you
could reboot into it.  And it could lead to problems if we will have a
tiny output to console and big output to a log file, I guess.

> A user-land program is not supposed to be able to cause a kernel panic;
> if it does, that’s a kernel bug.

                       How to make a kernel panic

The problem will be No defined variable IPTABLES-SSH after 'guix system
reconfigure' and kernel crash after.

$HOME/src/iptables/iptables/ru.scm

    (define-module (iptables ru)
      ;; …
      )

    (define %iptables-ssh
       "-A INPUT -p tcp --dport 22 \
    -m state --state NEW -m recent --set --name SSH -j ACCEPT")

/etc/config.scm

    (use-modules ;; …
                 (iptables ru))

    (define start-firewall
      #~(let ((iptables
               (lambda (str)
                 (zero? (system (string-join `(,#$(file-append iptables
                                                               "/sbin/iptables")
                                               ,str) " "))))))
          (format #t "Install iptables rules.~%")
          (and
           ;; …
           (iptables %iptables-ssh))))

    (define firewall-service
      (simple-service 'firewall shepherd-root-service-type
                      (list
                       (shepherd-service
                        (provision '(firewall))
                        (requirement '())
                        (start #~(lambda _
                                   #$start-firewall))
                        (respawn? #f)
                        (stop #~(lambda _
                                  (zero?
                                   (system* #$(file-append iptables
                                                           "/sbin/iptables")
                                            "-F"))))))))

    (operating-system
      ;; …
        (services (cons* ;; …
                         firewall-service)))


Make a kernel panic

    sudo GUILE_LOAD_PATH=\"$HOME/src/iptables\
    :$GUILE_LOAD_PATH\" guix system reconfigure \
    $HOME/dotfiles/guix/system-magnolia.scm

    # Run above again and kernel will panic.

> But perhaps you got the kernel panic *after* rebooting in the
> reconfigured system?  That could well be a GuixSD bug, indeed.

No, it happens after second 'guix system reconfigure' with howto above.

[...]

Thanks,
Oleg.

Message #20 received at 28772@debbugs.gnu.org (full text, mbox, reply):

From: ludo@gnu.org (Ludovic Courtès)

To: Oleg Pykhalov <go.wigust@gmail.com>

Cc: 28772@debbugs.gnu.org

Subject: Re: bug#28772: guix system reconfigure after kernel panic user or group not created

Date: Thu, 12 Oct 2017 09:57:09 +0200

[Message part 1 (text/plain, inline)]

Hi Oleg,

Oleg Pykhalov <go.wigust@gmail.com> skribis:

> apologies for not adding logs before.  It's hard to do when I do guix
> commands from Xterm and not from Emacs.  Emacs *shell* or *compilation*
> buffers will eat all memory if they get too much text.

I sympathize…

> The problem
> ===========
>
> The bigger problem from my view are files like /etc/group.lock and
> /etc/passwd.lock.  For example:
>
>     sudo touch /etc/group.lock
>
> /etc/config.scm
>
>     (operating-system
>       ;; …
>       (groups (cons
>          (user-group (name "test"))
>          %base-groups)))
>
> reconfigure log

I think we can avoid the problem by forcefully removing these two lock
files at boot time:

[Message part 2 (text/x-patch, inline)]

diff --git a/gnu/services.scm b/gnu/services.scm
index 329b7b151..2ef1d8530 100644
--- a/gnu/services.scm
+++ b/gnu/services.scm
@@ -368,6 +368,8 @@ boot."
                                                 #t))))
                     ;; Ignore I/O errors so the system can boot.
                     (fail-safe
+                     (delete-file "/etc/group.lock")
+                     (delete-file "/etc/passwd.lock")
                      (delete-file-recursively "/tmp")
                      (delete-file-recursively "/var/run")
                      (mkdir "/tmp")

[Message part 3 (text/plain, inline)]

>                        How to make a kernel panic
>
> The problem will be No defined variable IPTABLES-SSH after 'guix system
> reconfigure' and kernel crash after.

[...]

> Make a kernel panic
>
>     sudo GUILE_LOAD_PATH=\"$HOME/src/iptables\
>     :$GUILE_LOAD_PATH\" guix system reconfigure \
>     $HOME/dotfiles/guix/system-magnolia.scm
>
>     # Run above again and kernel will panic.

I tried to reproduce it with my user’s shepherd, but that didn’t work:

--8<---------------cut here---------------start------------->8---
ludo@ribbon ~/src/guix$ cat ,t.scm
(define s
  (make <service>
    #:provides '(nothing)
    #:start (lambda _ unbound)))

(register-services s)
(start s)
ludo@ribbon ~/src/guix$ herd load root ,t.scm
Loading ,t.scm.
herd: exception caught while executing 'load' on service 'root':
ERROR: Unbound variable: foo
ludo@ribbon ~/src/guix$ herd load root ,t.scm
Loading ,t.scm.
herd: exception caught while executing 'start' on service 'nothing':
ERROR: Unbound variable: unbound
ludo@ribbon ~/src/guix$ herd load root ,t.scm
Loading ,t.scm.
Assertion (null? (lookup-services (canonical-name new))) failed.
herd: exception caught while executing 'load' on service 'root':
ERROR: Throw to key `assertion-failed' with args `()'.
ludo@ribbon ~/src/guix$ echo $?
1
ludo@ribbon ~/src/guix$ herd status

[...]

ludo@ribbon ~/src/guix$ echo $?
0
--8<---------------cut here---------------end--------------->8---

IOW, shepherd caught the exceptions and didn’t die.

What am I missing?

Ludo’.

Message #23 received at 28772@debbugs.gnu.org (full text, mbox, reply):

From: Oleg Pykhalov <go.wigust@gmail.com>

To: ludo@gnu.org (Ludovic Courtès)

Cc: 28772@debbugs.gnu.org

Subject: Re: bug#28772: guix system reconfigure after kernel panic user or group not created

Date: Thu, 12 Oct 2017 11:15:48 +0300

Hello Ludovic,

ludo@gnu.org (Ludovic Courtès) writes:

[...]

> I think we can avoid the problem by forcefully removing these two lock
> files at boot time:
>
> diff --git a/gnu/services.scm b/gnu/services.scm
> index 329b7b151..2ef1d8530 100644
> --- a/gnu/services.scm
> +++ b/gnu/services.scm
> @@ -368,6 +368,8 @@ boot."
>                                                  #t))))
>                      ;; Ignore I/O errors so the system can boot.
>                      (fail-safe
> +                     (delete-file "/etc/group.lock")
> +                     (delete-file "/etc/passwd.lock")
>                       (delete-file-recursively "/tmp")
>                       (delete-file-recursively "/var/run")
>                       (mkdir "/tmp")
>
>

There is also a '/etc/.pwd.lock'.  Info about this file
https://lists.debian.org/debian-user/2005/07/msg02949.html


I'm not sure if any files are exist.  Days past after reconfigure
failure.

$ sudo find /etc -name '*.lock' # Shows nothing.

[...]

> IOW, shepherd caught the exceptions and didn’t die.
>
> What am I missing?

I'll try to make a reproducible thing later.

Thanks,
Oleg.

Message #26 received at 28772@debbugs.gnu.org (full text, mbox, reply):

From: ludo@gnu.org (Ludovic Courtès)

To: Oleg Pykhalov <go.wigust@gmail.com>

Cc: 28772@debbugs.gnu.org

Subject: Re: bug#28772: guix system reconfigure after kernel panic user or group not created

Date: Fri, 13 Oct 2017 10:25:52 +0200

Hi Oleg,

Oleg Pykhalov <go.wigust@gmail.com> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
> [...]
>
>> I think we can avoid the problem by forcefully removing these two lock
>> files at boot time:
>>
>> diff --git a/gnu/services.scm b/gnu/services.scm
>> index 329b7b151..2ef1d8530 100644
>> --- a/gnu/services.scm
>> +++ b/gnu/services.scm
>> @@ -368,6 +368,8 @@ boot."
>>                                                  #t))))
>>                      ;; Ignore I/O errors so the system can boot.
>>                      (fail-safe
>> +                     (delete-file "/etc/group.lock")
>> +                     (delete-file "/etc/passwd.lock")
>>                       (delete-file-recursively "/tmp")
>>                       (delete-file-recursively "/var/run")
>>                       (mkdir "/tmp")
>>
>>
>
> There is also a '/etc/.pwd.lock'.  Info about this file
> https://lists.debian.org/debian-user/2005/07/msg02949.html
>
>
> I'm not sure if any files are exist.  Days past after reconfigure
> failure.
>
> $ sudo find /etc -name '*.lock' # Shows nothing.

I’ve pushed it as aad8a143000600abec5c8ebfadec4c09f34f1b73.

> [...]
>
>> IOW, shepherd caught the exceptions and didn’t die.
>>
>> What am I missing?
>
> I'll try to make a reproducible thing later.

Awesome.

Thanks,
Ludo’.

Message #29 received at 28772@debbugs.gnu.org (full text, mbox, reply):

From: ludo@gnu.org (Ludovic Courtès)

To: guix-devel@gnu.org

Cc: 28772@debbugs.gnu.org, Maxim Cournoyer <maxim.cournoyer@gmail.com>

Subject: Re: 02/02: doc: Add an example to the documentation of the udev-service.

Date: Fri, 13 Oct 2017 10:35:43 +0200

Hi!

rekado@elephly.net (Ricardo Wurmus) skribis:

> commit e0c1d080b520c1bbd2dcd7bc90a750f5ce580486
> Author: Ricardo Wurmus <rekado@elephly.net>
> Date:   Mon Oct 9 23:03:56 2017 +0200
>
>     doc: Add an example to the documentation of the udev-service.
>     
>     * doc/guix.texi (Base Services): Update 'udev-service' documentation.

Good idea.

> +@example
> +(define %example-udev-rule
> +  (udev-rule "90-usb-thing.rules"
> +             "ACTION==\"add\", SUBSYSTEM==\"usb\", ATTR@{product@}==\"Example\", RUN+=\"/path/to/script\""))
> +
> +(operating-system
> +  ;; @dots{}
> +  (services (modify-services %desktop-services
> +              (udev-service-type config =>
> +                (udev-configuration (inherit config)
> +                  (rules (append (udev-configuration-rules config)
> +                                 (list %example-udev-rule))))))))
> +@end example

<https://bugs.gnu.org/28647> is somewhat related.

Fundamentally though, to simplify this use case, we should have:

  (define (additional-udev-rules . rules)
    "Add RULES, a list of file-like object, as a udev rules."
    (simple-service 'udev-rule udev-service-type rules))

so one can write:

  (operating-system
     ;; …
     (services (cons (additional-udev-rules %example-udev-rule)
                     %desktop-services)))

Thoughts?

Ludo’.

Message #32 received at 28772@debbugs.gnu.org (full text, mbox, reply):

From: ludo@gnu.org (Ludovic Courtès)

To: Oleg Pykhalov <go.wigust@gmail.com>

Cc: 28772@debbugs.gnu.org

Subject: Re: bug#28772: guix system reconfigure after kernel panic user or group not created

Date: Fri, 20 Oct 2017 18:02:08 +0200

Hi,

Oleg Pykhalov <go.wigust@gmail.com> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
> [...]
>
>> I think we can avoid the problem by forcefully removing these two lock
>> files at boot time:
>>
>> diff --git a/gnu/services.scm b/gnu/services.scm
>> index 329b7b151..2ef1d8530 100644
>> --- a/gnu/services.scm
>> +++ b/gnu/services.scm
>> @@ -368,6 +368,8 @@ boot."
>>                                                  #t))))
>>                      ;; Ignore I/O errors so the system can boot.
>>                      (fail-safe
>> +                     (delete-file "/etc/group.lock")
>> +                     (delete-file "/etc/passwd.lock")
>>                       (delete-file-recursively "/tmp")
>>                       (delete-file-recursively "/var/run")
>>                       (mkdir "/tmp")
>>
>>
>
> There is also a '/etc/.pwd.lock'.  Info about this file
> https://lists.debian.org/debian-user/2005/07/msg02949.html
>
>
> I'm not sure if any files are exist.  Days past after reconfigure
> failure.
>
> $ sudo find /etc -name '*.lock' # Shows nothing.
>
> [...]
>
>> IOW, shepherd caught the exceptions and didn’t die.
>>
>> What am I missing?
>
> I'll try to make a reproducible thing later.

Did you eventually gather more info?

Thanks,
Ludo’.

Message #37 received at 28772@debbugs.gnu.org (full text, mbox, reply):

From: Oleg Pykhalov <go.wigust@gmail.com>

To: ludo@gnu.org (Ludovic Courtès)

Cc: 28772@debbugs.gnu.org

Subject: Re: bug#28772: guix system reconfigure after kernel panic user or group not created

Date: Sun, 22 Oct 2017 17:41:34 +0300

[Message part 1 (text/plain, inline)]

Hello Ludovic,

Apologies for the late reply.

ludo@gnu.org (Ludovic Courtès) writes:

[...]

> Did you eventually gather more info?

Yes, I got an undefined %iptables-rst variable in reconfigure output
at first run.

But at second run substitutions didn't work.  GuixSD rebuilds the world.
I didn't wait for this.

I also didn't make guix pull.

/tmp/guixsd
├── bootstrap.sh
├── iptables
│   └── iptables.scm
├── panic.sh
└── vm-image.scm

1 directory, 4 files

[bootstrap.sh (application/x-sh, attachment)]

[panic.sh (application/x-sh, attachment)]

[vm-image.scm (application/octet-stream, attachment)]

[iptables.scm (application/octet-stream, attachment)]

Message #40 received at 28772@debbugs.gnu.org (full text, mbox, reply):

From: Oleg Pykhalov <go.wigust@gmail.com>

To: ludo@gnu.org (Ludovic Courtès)

Cc: 28772@debbugs.gnu.org

Subject: Re: bug#28772: guix system reconfigure after kernel panic user or group not created

Date: Sun, 22 Oct 2017 23:39:55 +0300

ludo@gnu.org (Ludovic Courtès) writes:

[...]

> Did you eventually gather more info?

In addition to previous message.

--- /home/natsu/r/bootstrap.sh	2017-10-22 04:48:42.992394510 +0300
+++ ./bootstrap.sh	2017-10-22 23:33:54.113495658 +0300
@@ -11,7 +11,7 @@
 SIG=guixsd-vm-image-$VERSION.$KERNEL.xz.sig
 
 pull_release () {
-    if [ ! -f $RELEASE ] || [ ! -f $RELEASE.xz ] || ! gpg --verify $SIG
+    if [ ! -f $RELEASE ] || [ ! -f $RELEASE.xz ]
     then
         gpg --keyserver pgp.mit.edu --recv-keys $GPG_KEY
         wget --output-document=$RELEASE.xz https://alpha.gnu.org/gnu/guix/$RELEASE.xz 
@@ -23,6 +23,7 @@
 
 if pull_release
 then qemu-system-x86_64 \
+     -enable-kvm \
      -daemonize \
      -m 1024 \
      -virtfs local,path=$PWD,security_model=none,mount_tag=TAG_pwd \

Also need to increase a size of qemu image.  I don't know how I
reconfigured successfully in previous message.

Message #43 received at 28772@debbugs.gnu.org (full text, mbox, reply):

From: ludo@gnu.org (Ludovic Courtès)

To: Oleg Pykhalov <go.wigust@gmail.com>

Cc: 28772@debbugs.gnu.org

Subject: Re: bug#28772: guix system reconfigure after kernel panic user or group not created

Date: Sun, 22 Oct 2017 16:01:39 -0700

Hi Oleg,

Oleg Pykhalov <go.wigust@gmail.com> skribis:

> ludo@gnu.org (Ludovic Courtès) writes:
>
> [...]
>
>> Did you eventually gather more info?
>
> Yes, I got an undefined %iptables-rst variable in reconfigure output
> at first run.
>
> But at second run substitutions didn't work.  GuixSD rebuilds the world.
> I didn't wait for this.

In my previous message I showed a way to (attempt to) reproduce the
problem by directly using ‘herd load’.

Could you try to reproduce the problem in this way?

You should even be able to check with a shepherd instance not running as
PID 1, which is more convenient.

Thanks in advance,
Ludo’.

Message #46 received at 28772@debbugs.gnu.org (full text, mbox, reply):

From: zimoun <zimon.toutoune@gmail.com>

To: Oleg Pykhalov <go.wigust@gmail.com>

Cc: 28772@debbugs.gnu.org

Subject: Re: bug#28772: guix system reconfigure after kernel panic user or group not created

Date: Fri, 26 Nov 2021 02:19:09 +0100

Hi,

On Tue, 10 Oct 2017 at 08:51, Oleg Pykhalov <go.wigust@gmail.com> wrote:

> During 'guix system reconfigure' I got a kernel panic.
>
> Getting it quite often while 'guix system reconfigure' at Linux magnolia
> 4.13.4-gnu #1 SMP 1 x86_64 GNU/Linux.  But it's not a subject of current
> report.
>
> After reboot I tried another attempt to make a 'guix system
> reconfigure', but because of /etc/group.lock and /etc/passwd.lock
> operating-system groups and users fields didn't applied.
>
> I tried to get a creation of adbusers group and add my user to it, but
> because of those files I cannot do it, until manually removed them.
>
> So, do we need to check if *.lock files are exist and remove them at the
> start of 'guix system reconfigure' OR abort reconfigure and creation of
> Grub boot entry OR something else?


The last message for bug 28772 [1] is 22 Oct 2017 and it is marked
’moreinfo’ since 2017.  I propose to close or provide more details.

1: <http://issues.guix.gnu.org/issue/28772>


Cheers,
simon

Send a report that this bug log contains spam.