Emacs native compilation on startup can crash the system

After updating to a commit after the introduction of native compilation

in Emacs, I cannot start Emacs any more. It launches an ever increasing

number of async processes for native compilation, which rapidly makes

kswapd the main CPU user on my machine, and ultimately crashes the

kernel.

Some experiments showed that it's loading the site-lisp/site-start.el that

causes this avalanche of processes:

emacs -Q --batch --eval="(print load-path)"

works fine. As does:

emacs -Q --batch --load=~/.guix-profile/share/emacs/site-lisp/guix-emacs.el --eval="(print load-path)"

But not:

emacs -Q --batch --load=~/.guix-profile/share/emacs/site-lisp/site-start.el --eval="(print load-path)"

which starts the avalanche of compilation.

Possibly related: in my user-dir (~/.emacs.d/) I have a directory

"eln-cache" containing a directory "28.1-f1a30909", but that directory

remains empty. Perhaps all native compilation attempts fail and that's

what causes the problem. Just guessing!

Cheers,

Konrad.

My understanding of site-start.el is that it actually loads all the

Emacs packages that are listed in my Guix profile. Which means that my

package list matters. Here is a minimal containerized example that

creates a process avalanche:

guix shell -C emacs emacs-ido-completing-read+ \

-- emacs --batch --eval="(print load-path)"

ido-completing-read+ is a rather small Emacs package, and I don't see

anything obvious in it that could cause trouble with native compilation.

But more importantly, I think no package should be able to cause the

behavior that I observe.

Cheers

Konrad.

Konrad Hinsen <konrad.hinsen@fastmail.net> writes:

I went through all my emacs packages. The only one that starts

the process avalanche is emacs-ido-completing-read+.

Cheers

Konrad.

Am Samstag, dem 17.09.2022 um 17:45 +0200 schrieb Konrad Hinsen:

I think you can prevent native-compilation entirely by setting no-

native-compile to t in your early-init.el (I'm playing with the idea of

doing this anyway, because I'm annoyed by the clutter that falls

through the cracks). That being said, this looks like a real breakage

in the emacs-ido-completing-read package, does it not? Should we add

"no-native-compile" to some local variables?

WDYT?

Am Sonntag, dem 18.09.2022 um 01:19 +0200 schrieb Liliana Marie

Prikler:

Okay, it turns out that disabling native compilation is not so simple

after all. Other ideas are welcome.

Cheers

Liliana Marie Prikler <liliana.prikler@gmail.com> writes:

That's what I discovered as well. What you can do is set

native-comp-speed to -1, reducing native compilation to byte code

compilation. But that doesn't deactivate the machinery.

Cheers,

Konrad.

I did one more test: run native-compile by hand on each elisp file in

the package ido-completing-read+. Works fine.

The next question that I consider relevant: is this an upstream (Emacs)

issue, or an issue created by packaging in Guix? That's not easy to

answer.

A related question: what are appropriate debugging techniques?

Cheers,

Konrad.

Hello Konrad and Liliana,

On Mon, Sep 19, 2022 at 5:16 AM Konrad Hinsen

<konrad.hinsen@fastmail.net> wrote:

I just got bit by this and lost a bunch of time trying to get my

system stable again. I am leaning towards this being an upstream

issue as it is also affecting Debian users [0]. Unfortunately, they

do not seem to have found a solution. I haven't checked the Emacs bug

tracker/mailing lists so I don't know if upstream is working on or has

a fix for this. I will be sticking to a pre-commit

dbcba75c0e96d8eb2b0bf9dbb3a49d15b38f80d2 version of Guix for the time

being. There may be many more people affected that have not reported

in. Is disabling native compilation an option on the table?

- Dave

[0] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1017817

Am Samstag, dem 01.10.2022 um 20:15 -0400 schrieb Thompson, David:

I don't want to disable native compilation for everyone, but I'd very

much disable deferred compilation and replace it with Guix packages.

In other words, as long as your change doesn't break the current "guix

build --with-input=emacs-minimal=emacs" flow, I'm leaning towards

accepting a patch.

As for disabling native compilation, using 

(setq native-comp-deferred-compilation nil)

removes most, but sadly not all compilation from personal experience.

I do wonder what causes the remaining ones to still pop up.

Cheers

Hi David and Liliana,

Thanks David for the added observations. This does indeed look like an

upstream bug, but it also seems to have a context dependence because the

Debian bug reports list some packages are affected that cause no problem

under Guix (e.g. git-timemachine).

As for Liliana's idea of disabling deferred compilation : shouldn't it

be sufficient to have all Emacs Lisp packages in Guix AOT-compiled?

There would be nothing left to compile in deferred mode. A quick scan of

the relevant page on Emacs Wiki

(https://www.emacswiki.org/emacs/GccEmacs)suggests that some package

manager do this.

Cheers,

Konrad.

Hi,

On Sat, 01 Oct 2022 at 20:15, "Thompson, David" <dthompson2@worcester.edu> wrote:

Heavy discussions are currently happening upstream; see this loooong

thread [1]. From my understanding, no fix yet.

1: https://yhetil.org/emacs-devel/87bkqxf1ij.fsf@tethera.net/#r

Cheers.

simon

Am Sonntag, dem 02.10.2022 um 10:25 +0200 schrieb Konrad Hinsen:

From personal experience, no. Even if you compile code ahead of time,

there seem to be some leftovers that are deferred. guix-emacs.el is an

oversight, but apart from that I also other leftovers (possibly from

init.el?)

In Guix, this is more or less a user choice – we advertised the

transformation by which you can opt-in to AOT compilation in a news

entry. Also, enforcing ahead-of-time compilation does not fix the more

pressing issue of packages breaking with native compilation ;)

To quote Eli:

I think this reasoning really falls flat in presence of any non-Emacs

package manager. Like, obviously wanting to natively compile packages

managed by (dpkg, rpm, pacman, emerge, guix), but not natively

compiling a random elisp script you just downloaded from the web is a

legitimate use case.

Cheers

Liliana Marie Prikler <liliana.prikler@gmail.com> writes:

AFAIU, the compiler is invoked any time a function is redefined or

advised, see (info "(elisp) Advising Functions"). In Emacs it is a very

common paractice to put advices around a function to modify its

behaviour. Since this happens during runtime (for example, in response

to a hook, or after you manually run some command), you should accept

the fact that native compiler may be invoked at any time as you use

Emacs.

How pressing is this issue? There are literally no reports of packages

breaking due to native compilation. So this is probably the reason why

Eli considers it as a non-issue.

If security is a concern, you should not load random Elisp in the first

place. It is much easier to just directly run harmful elisp, then to

exploit native compiler, which stays silent until after you evaluate

some (possibly harmful) elisp.

I am new to Guix.

My issue is related to the native compilation too, however it manifests

itself differently.

The build of emacs-next goes well.

However, when I start Emacs it throws lots of errors, most of which are

like this:

Deleting /tmp/comp-lambda-RCGJQI.eln

comp--native-compile: Native compiler error: (lambda (&rest arg1) (let ((f #'make-process)) (apply f arg1))), "Compiling /tmp/comp-lambda-RCGJQI.eln...

x86_64-unknown-linux-gnu-gcc-10.3.0: fatal error: cannot execute ‘as’: execvp: No such file or directory

compilation terminated.

So while it builds successfully I can't really run it.

What confuses me is that everyone else in this thread has a differnt

issue. For some, if I understand it correctly, the problem is that the

compiler runs to aggressively and makes Emacs unusable. Is it right?

But in my case the native compiler just plainly fails to execute,

because assembler program is not found.

Why isn't it deterministic, meaning that going through the same build

process, the resulting build of my Emacs is different from the others,

reporting related, but different issues?

And this is what makes me think that the problem is actually in

packaging.

This sounds like the problem with dependencies, libgccjit in particular,

doesn't it?

severity 57878 important

quit

Am Donnerstag, dem 13.10.2022 um 12:31 +0300 schrieb Max Brieiev:

The nature of compiled code being compiled makes it much easier to

exploit, however. Assume you have a genuine dash.el, but a malicious

person delivers you a dash.eln with some backdoor. Unless you know how

to read x86 assembly, you won't debug the latter, whereas you could

reasonably find the former if you're an Elisp hacker.

This is typically not a concern for Guix, where the challenge mechanism

provides tools to highlight that something is going wrong, but it might

be a concern for traditional distros. Then again, the same applies to

bytecode too, and here as well the solution is to typically use a

trusted package manager.

Cheers

Hi Liliana,

On mer., 12 oct. 2022 at 21:42, Liliana Marie Prikler <liliana.prikler@gmail.com> wrote:

Indeed, the news entry reads,

Maybe it could be worth to have that in the manual too, no? For

example, under ’Application setup, Emacs packages’ [1].

Because it appears to me more than just a simple news if it is more or

less an user choice.

1: https://guix.gnu.org/manual/devel/en/guix.html#Emacs-Packages-1

Cheers,

simon

Am Freitag, dem 14.10.2022 um 18:07 +0200 schrieb zimoun:

Would you prefer a paragraph, a note, or a footnote?

Hi Liliana,

On Fri, 14 Oct 2022 at 20:22, Liliana Marie Prikler <liliana.prikler@gmail.com> wrote:

As you would prefer. Maybe a note with,

Emacs can now compile packages natively. Under the default

configuration, this means that Emacs packages will now be

just-in-time (JIT) compiled as you use them, and the results

stored in a subdirectory of your @code{user-emacs-directory}.

Furthermore, the build system for Emacs packages transparently

supports native compilation, but note, that

@code{emacs-minimal}---the default Emacs for building

packages---has been configured without native compilation. To

natively compile your emacs packages ahead of time, use a

transformation like @option{--with-input=emacs-minimal=emacs}.

BTW, thanks for your efforts to explain upstream the situation. :-)

Cheers,

simon

Am Samstag, dem 15.10.2022 um 12:11 +0200 schrieb zimoun:

Done. Since I just copypasta'd the wording, I made you the author.

As for this bug: I've bumped emacs-next to a version that can inhibit

almost all native-compilation (trampolines are still compiled, but not

written to the cache), fixed the linker issue, and disabled native

compilation for emacs-ido-completing-read+. Any remaining native-comp

issues or can this be closed?

Cheers

Hi Liliana,

On Sat, 15 Oct 2022 at 16:40, Liliana Marie Prikler <liliana.prikler@gmail.com> wrote:

Oh, thanks. (It was the news’ wording. :-))

Closing is fine with me; well I do not have a opinion on that. :-)

Just to be sure, do you mean an emacs-next version which includes the

upstream Lars’s patches? The ones that Eli and Andrea are calling to be

reverted?

from https://yhetil.org/emacs-devel/xjfk05e5k9t.fsf@ma.sdf.org/

Cheers,

simon

Am Samstag, dem 15.10.2022 um 17:40 +0200 schrieb zimoun:

Ehm, yes... best to keep this open until I can bump to a version that's

approved and solves the problem.

Liliana Marie Prikler <liliana.prikler@gmail.com> writes:

Do the recent changes suppose to fix the issue?

Today I did:

guix pull

sudo guix system reconfigure /etc/system-config.scm

guix package -u emacs-next

After starting Emacs I see lots of errors.

It is not mere compilation warnings. They render Emacs unusable, because

trying to run a command gives me "function is void" errors.

So I switched back to previous generation again.

Please, see the screenshot.

Am Dienstag, dem 25.10.2022 um 19:23 +0300 schrieb Max Brieiev:

While I can't really reproduce the one you showed me in a pure

environment, I do see

Warning (comp): /gnu/store/hjciw32mj05yz1m6r6nzwdi12waz81ni-emacs-next-

29.0.50-2.4aeb80c/share/emacs/29.0.50/lisp/emacs-lisp/cl-

loaddefs.el.gz: Error: error Uncompression program `sh' not found

which is equally concerning.

Hi Liliana,

On Tue, 25 Oct 2022 at 20:31, Liliana Marie Prikler <liliana.prikler@gmail.com> wrote:

I have reported something similar the Emacs package ESS. See [1].

1: http://issues.guix.gnu.org/issue/58690

Cheers,

simon

Hi everyone,

On the occasion of a "guix pull", I looked into this issue again,

because magit without ido-completing-read+ is less fun.

Loading this package runs lots of compilation threads, but it does

ultimately terminate, with an error:

Debugger entered--Lisp error: (native-compiler-error (lambda (arg3 &optional arg4 arg5) (let ((f #'call-interactively)) (funcall f arg3 arg4 arg5))) "Debugger entered--Lisp error: (native-compiler-err...")

signal(native-compiler-error ((lambda (arg3 &optional arg4 arg5) (let ((f #'call-interactively)) (funcall f arg3 arg4 arg5))) "Debugger entered--Lisp error: (native-compiler-err..."))

comp--native-compile((lambda (arg3 &optional arg4 arg5) (let ((f #'call-interactively)) (funcall f arg3 arg4 arg5))) nil "/home/hinsen/.emacs.d/eln-cache/28.2-16da12a1/subr...")

comp-trampoline-compile(call-interactively)

comp-subr-trampoline-install(call-interactively)

advice--add-function(:around (#f(compiled-function () #<bytecode 0x32e14019df7e91>) . #f(compiled-function (gv--val) #<bytecode 0x9f608bbba7cb3c2>)) call-interactively@ido-cr+-record-current-command nil)

advice-add(call-interactively :around call-interactively@ido-cr+-record-current-command)

load-with-code-conversion("/gnu/store/abvhh3kcdpbg9vm2jz5q6zqhmfdpr4m7-emacs-..." "/gnu/store/abvhh3kcdpbg9vm2jz5q6zqhmfdpr4m7-emacs-..." nil nil)

load("/gnu/store/abvhh3kcdpbg9vm2jz5q6zqhmfdpr4m7-emacs-...")

(progn (load "/gnu/store/abvhh3kcdpbg9vm2jz5q6zqhmfdpr4m7-emacs-..."))

(setq elisp--eval-defun-result (progn (load "/gnu/store/abvhh3kcdpbg9vm2jz5q6zqhmfdpr4m7-emacs-...")))

elisp--eval-defun()

eval-defun(nil)

funcall-interactively(eval-defun nil)

command-execute(eval-defun)

What I notice here is "advice-add". It makes me wonder whether the

problem is perhaps not native-compiling ido-completing-read+, but

native-compiling the ido code that is getting adviced here.

Cheers,

Konrad.

Hello,

Konrad, Emacs team: is this bug still happening today?

https://issues.guix.gnu.org/57878

TIA,

Ludo’.

Liliana Marie Prikler <liliana.prikler@gmail.com> skribis:

Ludovic Courtès <ludo@gnu.org> writes:

I did some quick tests, and at least for my usage of Emacs, there are no

more problems with native compilation in current Guix (including Emacs

29.1). Great!

Konrad