GNU bug report logs

#59365 make-dynamic-linker-cache OOMs for LLVM 15 on i686-linux

Package	Source(s)		Maintainer(s)
guix		PTS Buildd Popcon

Reported by Marius Bakke <marius@gnu.org>
Date 2022-11-18T18:42:02
Severity important

Reply or subscribe to this bug. View this bug as an mbox, status mbox, or maintainer mbox

Report forwarded to bug-guix@gnu.org:
bug#59365; Package guix. (Fri, 18 Nov 2022 18:42:02 GMT) (full text, mbox, link).

Acknowledgement sent to Marius Bakke <marius@gnu.org>:
New bug report received and forwarded. Copy sent to bug-guix@gnu.org. (Fri, 18 Nov 2022 18:42:02 GMT) (full text, mbox, link).

Message #5 received at submit@debbugs.gnu.org (full text, mbox, reply):

[Message part 1 (text/plain, inline)]

Hello,

LLVM 15.0.4 fails on i686-linux:

  https://ci.guix.gnu.org/build/1702995/details

Because the 'make-dynamic-linker-cache' phase runs out of memory:

  starting phase `make-dynamic-linker-cache'
  GC Warning: Repeated allocation of very large block (appr. size 268439552):
	May lead to memory leak and poor performance
  GC Warning: Repeated allocation of very large block (appr. size 134221824):
	May lead to memory leak and poor performance
  GC Warning: Repeated allocation of very large block (appr. size 268439552):
  	May lead to memory leak and poor performance
  GC Warning: Failed to expand heap by 285216768 bytes
  GC Warning: Failed to expand heap by 268439552 bytes
  GC Warning: Out of Memory! Heap size: 3620 MiB. Returning NULL!
  Warning: Unwind-only out of memory exception; skipping pre-unwind handler.
  Warning: Unwind-only out of memory exception; skipping pre-unwind handler.
  Warning: Unwind-only out of memory exception; skipping pre-unwind handler.

(excerpt from https://ci.guix.gnu.org/build/1702995/log/raw)

Not sure why this phase uses so much memory.  Ideas?

[signature.asc (application/pgp-signature, inline)]

Information forwarded to bug-guix@gnu.org:
bug#59365; Package guix. (Tue, 22 Nov 2022 08:43:02 GMT) (full text, mbox, link).

Message #8 received at 59365@debbugs.gnu.org (full text, mbox, reply):

[Message part 1 (text/plain, inline)]

Hi,

(Cc: Maxim and Greg for LLVM packaging questions below.)

Marius Bakke <marius@gnu.org> skribis:

> LLVM 15.0.4 fails on i686-linux:
>
>   https://ci.guix.gnu.org/build/1702995/details
>
> Because the 'make-dynamic-linker-cache' phase runs out of memory:
>
>   starting phase `make-dynamic-linker-cache'
>   GC Warning: Repeated allocation of very large block (appr. size 268439552):
> 	May lead to memory leak and poor performance
>   GC Warning: Repeated allocation of very large block (appr. size 134221824):
> 	May lead to memory leak and poor performance
>   GC Warning: Repeated allocation of very large block (appr. size 268439552):
>   	May lead to memory leak and poor performance
>   GC Warning: Failed to expand heap by 285216768 bytes
>   GC Warning: Failed to expand heap by 268439552 bytes
>   GC Warning: Out of Memory! Heap size: 3620 MiB. Returning NULL!
>   Warning: Unwind-only out of memory exception; skipping pre-unwind handler.
>   Warning: Unwind-only out of memory exception; skipping pre-unwind handler.
>   Warning: Unwind-only out of memory exception; skipping pre-unwind handler.
>
> (excerpt from https://ci.guix.gnu.org/build/1702995/log/raw)
>
> Not sure why this phase uses so much memory.  Ideas?

Yes: the gremlin.scm code uses ‘file-dynamic-info’, which loads the
whole file in memory.  Ridiculous.

We should instead mmap it (but there are no ‘mmap’ bindings in Guile,
yet) or arrange to load just the relevant parts (we’ll have to check but
maybe ‘file-dynamic-info’ can find everything it needs at the beginning
of a file, the PT_DYNAMIC segment.)

For example, with the patch below, things still appear to be fine with
LLVM:

--8<---------------cut here---------------start------------->8---
scheme@(guix build gremlin)> (file-dynamic-info "/gnu/store/mj14k58lfc88jhcn6va0s2fpwkv3s35c-llvm-13.0.1/lib/libLLVMScalarOpts.so")
$11 = #<<elf-dynamic-info> soname: "libLLVMScalarOpts.so.13" needed: ("libLLVMAggressiveInstCombine.so.13" "libLLVMInstCombine.so.13" "libLLVMTransformUtils.so.13" "libLLVMAnalysis.so.13" "libLLVMCore.so.13" "libLLVMSupport.so.13" "libstdc++.so.6" "libm.so.6" "libgcc_s.so.1" "libc.so.6" "ld-linux-x86-64.so.2") rpath: () runpath: ("/gnu/store/mj14k58lfc88jhcn6va0s2fpwkv3s35c-llvm-13.0.1/lib" "/gnu/store/5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.33/lib" "/gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib" "/gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/../../..")>
scheme@(guix build gremlin)> (file-dynamic-info "/gnu/store/mj14k58lfc88jhcn6va0s2fpwkv3s35c-llvm-13.0.1/lib/libLLVMX86CodeGen.so.13")
$12 = #<<elf-dynamic-info> soname: "libLLVMX86CodeGen.so.13" needed: ("libLLVMAsmPrinter.so.13" "libLLVMX86Desc.so.13" "libLLVMX86Info.so.13" "libLLVMGlobalISel.so.13" "libLLVMCFGuard.so.13" "libLLVMSelectionDAG.so.13" "libLLVMCodeGen.so.13" "libLLVMTarget.so.13" "libLLVMTransformUtils.so.13" "libLLVMAnalysis.so.13" "libLLVMProfileData.so.13" "libLLVMMC.so.13" "libLLVMCore.so.13" "libLLVMSupport.so.13" "libstdc++.so.6" "libm.so.6" "libgcc_s.so.1" "libc.so.6" "ld-linux-x86-64.so.2") rpath: () runpath: ("/gnu/store/mj14k58lfc88jhcn6va0s2fpwkv3s35c-llvm-13.0.1/lib" "/gnu/store/5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.33/lib" "/gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib" "/gnu/store/094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib/lib/gcc/x86_64-unknown-linux-gnu/10.3.0/../../..")>
--8<---------------cut here---------------end--------------->8---

We could temporarily delete this phase for all 32-bit builds of LLVM.

But the crux of the problem is that llvm@15 has a single huge shared
library, unlike previous versions:

--8<---------------cut here---------------start------------->8---
$ du -hL  /gnu/store/bgqdvvi7k6l255332rfawgjmn2hpn13r-llvm-15.0.4/lib/*.so
133M    /gnu/store/bgqdvvi7k6l255332rfawgjmn2hpn13r-llvm-15.0.4/lib/libLLVM-15.0.4.so
96K     /gnu/store/bgqdvvi7k6l255332rfawgjmn2hpn13r-llvm-15.0.4/lib/libLTO.so
16K     /gnu/store/bgqdvvi7k6l255332rfawgjmn2hpn13r-llvm-15.0.4/lib/libRemarks.so
--8<---------------cut here---------------end--------------->8---

(It also has tons of .a files, which shouldn’t be there.)

Is that big LLVM.so due to different build options on our side?  Or is
it a radical upstream change (sounds unlikely, but who knows)?

Thanks,
Ludo’.

[Message part 2 (text/x-patch, inline)]

diff --git a/guix/build/gremlin.scm b/guix/build/gremlin.scm
index 2a74d51dd9..8a38dde1eb 100644
--- a/guix/build/gremlin.scm
+++ b/guix/build/gremlin.scm
@@ -250,7 +250,11 @@ (define (file-dynamic-info file)
 info."
   (call-with-input-file file
     (lambda (port)
-      (elf-dynamic-info (parse-elf (get-bytevector-all port))))))
+      (elf-dynamic-info (parse-elf
+                         ;; Read at most 10 MiB in memory, which should be
+                         ;; enough to get the PT_DYNAMIC segment.
+                         ;; TODO: mmap the whole file instead.
+                         (get-bytevector-n port (* 10 (expt 2 20))))))))
 
 (define (file-runpath file)
   "Return the DT_RUNPATH dynamic entry of FILE as a list of strings, or #f if

Information forwarded to bug-guix@gnu.org:
bug#59365; Package guix. (Tue, 22 Nov 2022 10:23:02 GMT) (full text, mbox, link).

Message #11 received at 59365@debbugs.gnu.org (full text, mbox, reply):

Hi Ludovic,

Ludovic Courtès <ludo@gnu.org> writes:

> Hi,
>
> (Cc: Maxim and Greg for LLVM packaging questions below.)
>
> Marius Bakke <marius@gnu.org> skribis:
>
>> LLVM 15.0.4 fails on i686-linux:
>>
>>   https://ci.guix.gnu.org/build/1702995/details
>>
>> Because the 'make-dynamic-linker-cache' phase runs out of memory:
>>
>>   starting phase `make-dynamic-linker-cache'
>>   GC Warning: Repeated allocation of very large block (appr. size 268439552):
>> 	May lead to memory leak and poor performance
>>   GC Warning: Repeated allocation of very large block (appr. size 134221824):
>> 	May lead to memory leak and poor performance
>>   GC Warning: Repeated allocation of very large block (appr. size 268439552):
>>   	May lead to memory leak and poor performance
>>   GC Warning: Failed to expand heap by 285216768 bytes
>>   GC Warning: Failed to expand heap by 268439552 bytes
>>   GC Warning: Out of Memory! Heap size: 3620 MiB. Returning NULL!
>>   Warning: Unwind-only out of memory exception; skipping pre-unwind handler.
>>   Warning: Unwind-only out of memory exception; skipping pre-unwind handler.
>>   Warning: Unwind-only out of memory exception; skipping pre-unwind handler.
>>
>> (excerpt from https://ci.guix.gnu.org/build/1702995/log/raw)
>>
>> Not sure why this phase uses so much memory.  Ideas?
>
> Yes: the gremlin.scm code uses ‘file-dynamic-info’, which loads the
> whole file in memory.  Ridiculous.

If it loaded just that file, it should be fine, no?  It weighs 133 MiB,
as you've shown below:

> But the crux of the problem is that llvm@15 has a single huge shared
> library, unlike previous versions:
>
> $ du -hL  /gnu/store/bgqdvvi7k6l255332rfawgjmn2hpn13r-llvm-15.0.4/lib/*.so
> 133M    /gnu/store/bgqdvvi7k6l255332rfawgjmn2hpn13r-llvm-15.0.4/lib/libLLVM-15.0.4.so
> 96K     /gnu/store/bgqdvvi7k6l255332rfawgjmn2hpn13r-llvm-15.0.4/lib/libLTO.so
> 16K     /gnu/store/bgqdvvi7k6l255332rfawgjmn2hpn13r-llvm-15.0.4/lib/libRemarks.so
> (It also has tons of .a files, which shouldn’t be there.)

The static files are needed at least to build the clang runtime.  I had
tried to get rid of them without success.  Perhaps they could be moved
to a "static" output if they're needed only at that time.

> Is that big LLVM.so due to different build options on our side?  Or is
> it a radical upstream change (sounds unlikely, but who knows)?

It's caused by -DLLVM_LINK_LLVM_DYLIB=ON and -DLLVM_BUILD_LLVM_DYLIB=ON,
which is the supported configuration to build a shared library of LLVM
(-DBUILD_SHARED_LIBS=ON is obsolete/deprecated) [0].

It also makes things conveniently easy to link to LLVM; you just need to
link to '-lLLVM', and everything it needs is available.

[0]  https://llvm.org/docs/BuildingADistribution.html#special-notes-for-library-only-distributions

-- 
Thanks,
Maxim

Information forwarded to bug-guix@gnu.org:
bug#59365; Package guix. (Mon, 20 Nov 2023 19:34:02 GMT) (full text, mbox, link).

Message #14 received at 59365@debbugs.gnu.org (full text, mbox, reply):

Hi,

This still happens:

--8<---------------cut here---------------start------------->8---
starting phase `make-dynamic-linker-cache'
GC Warning: Repeated allocation of very large block (appr. size 16781312):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 67112960):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 134221824):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 67112960):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 67112960):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 134221824):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 134221824):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 134221824):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 268439552):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 33558528):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 67112960):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 67112960):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 268439552):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 268439552):
	May lead to memory leak and poor performance
GC Warning: Repeated allocation of very large block (appr. size 268439552):
	May lead to memory leak and poor performance
GC Warning: Failed to expand heap by 285216768 bytes
GC Warning: Failed to expand heap by 268439552 bytes
GC Warning: Out of Memory! Heap size: 3362 MiB. Returning NULL!
Warning: Unwind-only out of memory exception; skipping pre-unwind handler.
Warning: Unwind-only out of memory exception; skipping pre-unwind handler.
Warning: Unwind-only out of memory exception; skipping pre-unwind handler.
builder for
`/gnu/store/j4w1wrhgpjjcfqf2jskklr95r6hpy51i-llvm-15.0.7.drv' failed
with exit code 1
--8<---------------cut here---------------end--------------->8---

See this recent build: https://ci.guix.gnu.org/build/2602334/details

-- 
Thanks,
Maxim

Information forwarded to bug-guix@gnu.org:
bug#59365; Package guix. (Fri, 12 Apr 2024 09:37:03 GMT) (full text, mbox, link).

Message #17 received at 59365@debbugs.gnu.org (full text, mbox, reply):

Hello,

Ludovic Courtès <ludo@gnu.org> skribis:

>>   GC Warning: Failed to expand heap by 285216768 bytes
>>   GC Warning: Failed to expand heap by 268439552 bytes
>>   GC Warning: Out of Memory! Heap size: 3620 MiB. Returning NULL!
>>   Warning: Unwind-only out of memory exception; skipping pre-unwind handler.
>>   Warning: Unwind-only out of memory exception; skipping pre-unwind handler.
>>   Warning: Unwind-only out of memory exception; skipping pre-unwind handler.
>>
>> (excerpt from https://ci.guix.gnu.org/build/1702995/log/raw)
>>
>> Not sure why this phase uses so much memory.  Ideas?
>
> Yes: the gremlin.scm code uses ‘file-dynamic-info’, which loads the
> whole file in memory.  Ridiculous.
>
> We should instead mmap it (but there are no ‘mmap’ bindings in Guile,
> yet) or arrange to load just the relevant parts (we’ll have to check but
> maybe ‘file-dynamic-info’ can find everything it needs at the beginning
> of a file, the PT_DYNAMIC segment.)

Another instance of the problem that we just stumbled upon is ‘guix pack -RR’:
that too tries to load entire ELF files in memory, in
‘elf-loader-compile-flags’.

Mmap!

Ludo’.

Severity set to 'important' from 'normal' Request was from Ludovic Courtès <ludo@gnu.org> to control@debbugs.gnu.org. (Fri, 12 Apr 2024 09:37:04 GMT) (full text, mbox, link).

Information forwarded to bug-guix@gnu.org:
bug#59365; Package guix. (Thu, 03 Apr 2025 21:31:02 GMT) (full text, mbox, link).

Message #22 received at 59365@debbugs.gnu.org (full text, mbox, reply):

Hi,

Ludovic Courtès <ludo@gnu.org> skribis:

> diff --git a/guix/build/gremlin.scm b/guix/build/gremlin.scm
> index 2a74d51dd9..8a38dde1eb 100644
> --- a/guix/build/gremlin.scm
> +++ b/guix/build/gremlin.scm
> @@ -250,7 +250,11 @@ (define (file-dynamic-info file)
>  info."
>    (call-with-input-file file
>      (lambda (port)
> -      (elf-dynamic-info (parse-elf (get-bytevector-all port))))))
> +      (elf-dynamic-info (parse-elf
> +                         ;; Read at most 10 MiB in memory, which should be
> +                         ;; enough to get the PT_DYNAMIC segment.
> +                         ;; TODO: mmap the whole file instead.
> +                         (get-bytevector-n port (* 10 (expt 2 20))))))))
>  

I was pondering whether to include it on the ‘core-packages-team’
branch, and the answer is no: I quickly found a counterexample where
this hack breaks badly:

--8<---------------cut here---------------start------------->8---
scheme@(guix build gremlin)> (file-dynamic-info "/home/ludo/.guix-home/profile/lib/libgs.so")
ice-9/boot-9.scm:1683:22: In procedure raise-exception:
ERROR:
  1. &invalid-segment-size:
      elf: #<<elf> bytes: #vu8(127 69 76 70 2 1 1 0 0 0 0 0 0 0 0 0 3 0 62 0 1 0 0 0 0 0 …
--8<---------------cut here---------------end--------------->8---

… where:

--8<---------------cut here---------------start------------->8---
$ du -hsL ~/.guix-home/profile/lib/libgs.so
22M	/home/ludo/.guix-home/profile/lib/libgs.so
$ readlink -f ~/.guix-home/profile/lib/libgs.so
/gnu/store/h63c568fnky1kjdk00q5clsba9icg88n-ghostscript-9.56.1/lib/libgs.so.9.56
--8<---------------cut here---------------end--------------->8---

Ludo’.

Display info messages

Send a report that this bug log contains spam.

debbugs.gnu.org maintainers <help-debbugs@gnu.org>. Last modified: Wed Apr 9 15:52:57 2025; Machine Name: wallace-server

GNU bug tracking system

Debbugs is free software and licensed under the terms of the GNU Public License version 2. The current version can be obtained from https://bugs.debian.org/debbugs-source/.