#65446 [berlin] guix gc throws: "error: build daemon out of memory"

Message #5 received at submit@debbugs.gnu.org (full text, mbox, reply):

From: Maxim Cournoyer <maxim.cournoyer@gmail.com>

To: bug-guix <bug-guix@gnu.org>

Subject: [cuirass] guix gc throws: "error: build daemon out of memory"

Date: Mon, 21 Aug 2023 22:19:25 -0400

Hi,

Since a few days on Berlin, the following appears in /var/log/mcron.log:

--8<---------------cut here---------------start------------->8---
2023-08-22 04:00:29 28215 /gnu/store/4lmpqf8dnk86avq5xa2l6bznxnmrz27x-guix-1.4.0-7.44bbfc2/bin/guix gc: guix gc: error: build daemon out of memory
--8<---------------cut here---------------end--------------->8---

It's not just a fluke, according to the kernel's dmesg:

--8<---------------cut here---------------start------------->8---
[7190243.475312] __vm_enough_memory: pid: 28300, comm: guix-daemon, not enough memory for the allocation
--8<---------------cut here---------------end--------------->8---

Once it appears, it can be triggered simply by running 'guix gc'.  The
problem can be worked around by restarting the Guix daemon, with:

--8<---------------cut here---------------start------------->8---
herd restart guix-daemon
--8<---------------cut here---------------end--------------->8---

Any ideas what could cause such a condition?

The machine hasn't been rebooted in 83 days, perhaps we should start
with that.

-- 
Thanks,
Maxim

Message #12 received at 65446@debbugs.gnu.org (full text, mbox, reply):

From: Ludovic Courtès <ludo@gnu.org>

To: Maxim Cournoyer <maxim.cournoyer@gmail.com>

Cc: 65446@debbugs.gnu.org, Leo Famulari <leo@famulari.name>

Subject: Re: bug#65446: [berlin] guix gc throws: "error: build daemon out of memory"

Date: Sat, 02 Sep 2023 22:33:44 +0200

Hi!

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> Since a few days on Berlin, the following appears in /var/log/mcron.log:
>
> 2023-08-22 04:00:29 28215 /gnu/store/4lmpqf8dnk86avq5xa2l6bznxnmrz27x-guix-1.4.0-7.44bbfc2/bin/guix gc: guix gc: error: build daemon out of memory
>
>
> It's not just a fluke, according to the kernel's dmesg:
>
> [7190243.475312] __vm_enough_memory: pid: 28300, comm: guix-daemon, not enough memory for the allocation
>
>
> Once it appears, it can be triggered simply by running 'guix gc'.  The
> problem can be worked around by restarting the Guix daemon, with:
>
> herd restart guix-daemon
>
> Any ideas what could cause such a condition?

No idea!

I ran “guix gc -C0” minutes ago, which reproduced the problem;
/var/log/guix-daemon.log reads this:

--8<---------------cut here---------------start------------->8---
2023-09-02 21:40:31 accepted connection from pid 12853, user ludo
2023-09-02 21:40:50 unexpected build daemon error: std::bad_alloc
--8<---------------cut here---------------end--------------->8---

and the kernel writes what you wrote above:

--8<---------------cut here---------------start------------->8---
Sep  2 21:40:50 localhost vmunix: [8204269.198144] __vm_enough_memory: pid: 12873, comm: guix-daemon, not enough memory for the allocation
Sep  2 21:40:50 localhost vmunix: [8204269.207396] __vm_enough_memory: pid: 12873, comm: guix-daemon, not enough memory for the allocation
Sep  2 21:40:50 localhost vmunix: [8204269.216639] __vm_enough_memory: pid: 12873, comm: guix-daemon, not enough memory for the allocation
--8<---------------cut here---------------end--------------->8---

My reading of the Linux code is ‘__vm_enough_memory’ is used by Linux
Security Modules (LSM), so perhaps something changed in our kernel
config, be it compile-time or run-time?  (Cc: Leo just in case.)

This is with Linux-libre 6.3.4-gnu.

Ludo’.

Message #15 received at 65446@debbugs.gnu.org (full text, mbox, reply):

From: "Collin J. Doering" <collin@rekahsoft.ca>

To: 65446@debbugs.gnu.org

Subject: [berlin] guix gc throws: "error: build daemon out of memory"

Date: Mon, 16 Dec 2024 23:11:00 -0500

[Message part 1 (text/plain, inline)]

Perhaps worth noting that I've seen this numerous times on cuirass.genenetwork.org.

Usually restarting guix-daemon will allow 'guix gc' to complete without error. I see dmesg errors that match what Ludo provided.

Kind regards

-- 
Collin J. Doering

http://rekahsoft.ca
http://blog.rekahsoft.ca
http://git.rekahsoft.ca

[signature.asc (application/pgp-signature, inline)]

Message #18 received at submit@debbugs.gnu.org (full text, mbox, reply):

From: Maxim Cournoyer <maxim.cournoyer@gmail.com>

To: "Collin J. Doering" via Bug reports for GNU Guix <bug-guix@gnu.org>

Cc: 65446@debbugs.gnu.org, "Collin J. Doering" <collin@rekahsoft.ca>

Subject: Re: bug#65446: [berlin] guix gc throws: "error: build daemon out of memory"

Date: Wed, 18 Dec 2024 11:05:55 +0900

Hello,

"Collin J. Doering" via Bug reports for GNU Guix <bug-guix@gnu.org>
writes:

> Perhaps worth noting that I've seen this numerous times on cuirass.genenetwork.org.
>
> Usually restarting guix-daemon will allow 'guix gc' to complete
> without error. I see dmesg errors that match what Ludo provided.

Thanks for tipping in; it's a useful thing to know the bug is
reproducible in other environments.

-- 
Thanks,
Maxim

Send a report that this bug log contains spam.