Redundant library grafts leads to breakage

Hi Guix,

When reconfiguring my system, the build of

/gnu/store/0yf1b1l19h7c3jj1zkhxjmq4sb3yysjq-grub-image.png.drv failed with

the following:

```

Backtrace:

2 (primitive-load "/gnu/store/larqpc2wjhnc6jmj4885k8lynd1?")

In gnu/build/svg.scm:

53:6 1 (svg->png _ "/gnu/store/xadbzis4pvmxib4fk55jrag4fmn55w?" ?)

In unknown file:

0 (rsvg-handle-render-cairo #<rsvg-handle 7ffff5b60150> #)

ERROR: In procedure rsvg-handle-render-cairo:

Wrong type (expecting finalized smob): #<cairo-context 7ffff5b60090>

```

This is with Guix bb5d84a0489a629d30bc2e978807caf20f46e329. My last

successful reconfigure was with 80739ea480a7db667b83b45e3a08be740449f689.

The output of the reconfigure run is attached. Reconfiguring without

grafts succeeds.

Best,

Jack

On Fri, Mar 12, 2021 at 05:58:27PM -0500, Jack Hill wrote:

I wonder if it's related to the changes in the recent Cairo graft, from

commit bc16eacc99e801ac30cbe2aa649a2be3ca5c102a?

On Fri, 12 Mar 2021, Leo Famulari wrote:

Yes, that seems to be it. The previous commit

sudo -E guix time-machine --commit=453e101fc3f7dac9aabcd6122cf05fb7925103c7 -- system reconfigure /config.scm

works, but

sudo -E guix time-machine --commit=bc16eacc99e801ac30cbe2aa649a2be3ca5c102a -- system reconfigure /config.scm

does not.

Best,

Jack

Leo Famulari <leo@famulari.name> writes:

Is anyone else seeing this? FWIW, I tested reconfiguring my Guix system

with the grafts I recently pushed, and grub-img.png built successfully

for me. I'm using the resulting system now.

Also, the changes between the original cairo and its replacement are

quite minimal. I'm having trouble imagining how this graft could have

led to that error, but of course my imagination is limited. :)

Jack: is the problem reproducible, or could it have been a sporadic

failure?

Mark

On Fri, 12 Mar 2021, Mark H Weaver wrote:

So far I've only reconfigured beyond the graft on the one VM, but with

multiple commits. I'll try it on another host shortly.

Best,

Jack

On Fri, 12 Mar 2021, Jack Hill wrote:

I was not able to reproduce the problem on my desktop. Both systems are

using the bios grub-bootloader.

Best,

Jack

Hi Jack,

Jack Hill <jackhill@jackhill.us> writes:

Thanks. Given this, and the lack of similar reports from others, my

guess is that you hit a non-deterministic Guile bug, possibly the same

one as https://bugs.gnu.org/46879 (Non-deterministic failures while

building Guix with Guile 3.0.5).

If the problem happens reproducibly on that one VM only, that suggests

that the bug might have led to a corrupted store item, i.e. a store item

containing a .go file with bad code. If so, running "guix gc" might be

sufficient to clear the corrupted items.

Regards,

Mark

On Sat, 13 Mar 2021, Mark H Weaver wrote:

guix gc and the reconfigure again (now with

373c7b5791acd8f377455be47260948b843dd5db) still results in the same error.

Of course, if the miscompilation was in something that couldn't get

collected… However, I have been able to reproduce it on that one VM across

multiple Guix commits.

Best,

Jack

In an effort to clear out more of the potentially problematic store items,

I switched to an older generation of the system as well as guix pull and

user profiles. I then ran guix gc. At this point, I was running guix from

commit 373e5fc96724fd38bb1263e4af90932ea36f596b and the system profile was

created with guix f3eecfd36cb537a1febc30eea1f6aa448203ba40.

I then pulled, bringing me up to guix

8154beffd8c121e953a7c4cd75c3eebfcc073a9a. Reconfiguring results in the

same error. Any thoughts on how to recover? Should I try building guix

against an older guile version?

Best,

Jack

Still on the same VM, I've been able to reproduce the problem while

building a different derivation:

/gnu/store/07xw2pp63xin4c4y8ndrcdn3n8z1vmx2-grub-image.png.drv from Guix

commit d4e29f3628ad0c7576d7cab659d7fcc19d21999a. I can still build the new

derivation on my desktop.

Hrm, it's a pretty spooky problem.

Best,

Jack

Trying to document some more information in the hopes that others can

reproduce this bug.

On the host that fails:

$ guix describe

Generation 7 Mar 14 2021 16:14:58 (current)

guix d4e29f3

repository URL: https://git.savannah.gnu.org/git/guix.git

branch: master

commit: d4e29f3628ad0c7576d7cab659d7fcc19d21999a

jackhill@kalessin ~$ cat /proc/cpuinfo

processor : 0

vendor_id : GenuineIntel

cpu family : 6

model : 58

model name : Intel Xeon E3-12xx v2 (Ivy Bridge, IBRS)

stepping : 9

microcode : 0x1

cpu MHz : 2599.990

cache size : 16384 KB

physical id : 0

siblings : 1

core id : 0

cpu cores : 1

apicid : 0

initial apicid : 0

fpu : yes

fpu_exception : yes

cpuid level : 13

wp : yes

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca

cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm

constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq

vmx ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes

xsave avx f16c rdrand hypervisor lahf_lm cpuid_fault pti ssbd ibrs ibpb

tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust smep erms

xsaveopt arat md_clear

vmx flags : vnmi preemption_timer posted_intr invvpid ept_x_only

ept_1gb flexpriority apicv tsc_offset vtpr mtf vapic ept vpid

unrestricted_guest vapic_reg vid

bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass

l1tf mds swapgs itlb_multihit srbds

bogomips : 5199.98

clflush size : 64

cache_alignment : 64

address sizes : 40 bits physical, 48 bits virtual

power management:

processor : 1

vendor_id : GenuineIntel

cpu family : 6

model : 58

model name : Intel Xeon E3-12xx v2 (Ivy Bridge, IBRS)

stepping : 9

microcode : 0x1

cpu MHz : 2599.990

cache size : 16384 KB

physical id : 1

siblings : 1

core id : 0

cpu cores : 1

apicid : 1

initial apicid : 1

fpu : yes

fpu_exception : yes

cpuid level : 13

wp : yes

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca

cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm

constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq

vmx ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes

xsave avx f16c rdrand hypervisor lahf_lm cpuid_fault pti ssbd ibrs ibpb

tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust smep erms

xsaveopt arat md_clear

vmx flags : vnmi preemption_timer posted_intr invvpid ept_x_only

ept_1gb flexpriority apicv tsc_offset vtpr mtf vapic ept vpid

unrestricted_guest vapic_reg vid

bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass

l1tf mds swapgs itlb_multihit srbds

bogomips : 5199.98

clflush size : 64

cache_alignment : 64

address sizes : 40 bits physical, 48 bits virtual

power management:

jackhill@kalessin ~$ cat /config.scm

;; This is an operating system configuration for a VM image.

;; Modify it as you see fit and instantiate the changes by running:

;;

;; guix system reconfigure /etc/config.scm

;;

(use-modules (gnu) (guix))

(use-service-modules networking ssh)

(use-package-modules bootloaders certs linux

package-management)

(define vm-image-motd (plain-file "motd" "

\x1b[1;37mThis is the GNU system. Welcome!\x1b[0m

This instance of Guix is a template for virtualized environments.

You can reconfigure the whole system by adjusting /etc/config.scm

and running:

guix system reconfigure /etc/config.scm

Run '\x1b[1;37minfo guix\x1b[0m' to browse documentation.

\x1b[1;33mConsider setting a password for the 'root' and 'guest' \

accounts.\x1b[0m

"))

(operating-system

(host-name "kalessin")

(timezone "America/New_York")

(locale "en_US.utf8")

(initrd-modules (cons "virtio_scsi" %base-initrd-modules))

;; Label for the GRUB boot menu.

(label (string-append "GNU Guix " (package-version guix)))

(firmware '())

;; Below we assume /dev/vda is the VM's hard disk.

;; Adjust as needed.

(bootloader (bootloader-configuration

(bootloader grub-bootloader)

(target "/dev/vda")

(terminal-outputs '(console))))

(file-systems (cons (file-system

(mount-point "/")

(device (file-system-label "kalessin-btrfs"))

(type "btrfs")

(options "compress=zstd"))

%base-file-systems))

(users (cons* (user-account

(name "jackhill")

(comment "Jack Hill")

(group "users")

(supplementary-groups '("wheel" "netdev")))

%base-user-accounts))

;; Our /etc/sudoers file. Since 'guest' initially has an empty password,

;; allow for password-less sudo.

(sudoers-file (plain-file "sudoers" "\

root ALL=(ALL) ALL

%wheel ALL=NOPASSWD: ALL\n"))

(packages (append (list btrfs-progs nss-certs)

%base-packages))

(services

(append (list (service openssh-service-type

(openssh-configuration

(password-authentication? #f)

(authorized-keys

`(("jackhill" ,(local-file "/id_ed25519.pub")

,(local-file "/home/jackhill/tamago.ssh-key")

,(local-file "/home/jackhill/id_ed25519.pub"))))))

;; Use the DHCP client service rather than NetworkManager.

(service dhcp-client-service-type))

(modify-services %base-services

(guix-service-type config =>

(guix-configuration

(inherit config)

(extra-options

'("--disable-deduplication"))

(authorized-keys

(cons

(local-file "/home/jackhill/alperton-guix-key.pub")

%default-authorized-guix-keys))))))))

ckhill@kalessin ~$ sudo -E guix system -v3 reconfigure /config.scm

The following derivations will be built:

/gnu/store/cnl0pbld58rq4zn0l347ssawdxpcs2hg-grub.cfg.drv

/gnu/store/07xw2pp63xin4c4y8ndrcdn3n8z1vmx2-grub-image.png.drv

building /gnu/store/07xw2pp63xin4c4y8ndrcdn3n8z1vmx2-grub-image.png.drv...

Backtrace:

2 (primitive-load "/gnu/store/larqpc2wjhnc6jmj4885k8lynd1?")

In gnu/build/svg.scm:

53:6 1 (svg->png _ "/gnu/store/vmldvxllh07k641wmbnlz3migga29r?" ?)

In unknown file:

0 (rsvg-handle-render-cairo #<rsvg-handle 7ffff5b60150> #)

ERROR: In procedure rsvg-handle-render-cairo:

Wrong type (expecting finalized smob): #<cairo-context 7ffff5b60090>

builder for `/gnu/store/07xw2pp63xin4c4y8ndrcdn3n8z1vmx2-grub-image.png.drv' failed with exit code 1

build of /gnu/store/07xw2pp63xin4c4y8ndrcdn3n8z1vmx2-grub-image.png.drv failed

View build log at '/var/log/guix/drvs/07/xw2pp63xin4c4y8ndrcdn3n8z1vmx2-grub-image.png.drv.bz2'.

cannot build derivation `/gnu/store/cnl0pbld58rq4zn0l347ssawdxpcs2hg-grub.cfg.drv': 1 dependencies couldn't be built

guix system: error: build of `/gnu/store/cnl0pbld58rq4zn0l347ssawdxpcs2hg-grub.cfg.drv' failed

jackhill@kalessin ~$ free -m

total used free shared buff/cache available

Mem: 2994 122 505 0 2366 2783

Swap: 0 0 0

Now on the host where it suceeds:

$ guix describe

Generation 112 Mar 14 2021 16:30:34 (current)

guix-at-duke 2a57b7c

repository URL: https://gitlab.oit.duke.edu/jackhill/guix-at-duke.git

branch: master

commit: d4e29f3628ad0c7576d7cab659d7fcc19d21999a

nonguix 54b8358

repository URL: https://gitlab.com/nonguix/nonguix

branch: master

commit: 54b83587669b5df5fe36bce058f4f2cf34d8a63c

guix d4e29f3

repository URL: https://git.savannah.gnu.org/git/guix.git

branch: master

commit: d4e29f3628ad0c7576d7cab659d7fcc19d21999a

jackhill@alperton ~$ cat /proc/cpuinfo

processor : 0

vendor_id : GenuineIntel

cpu family : 6

model : 37

model name : Intel(R) Core(TM) i7 CPU L 640 @ 2.13GHz

stepping : 5

microcode : 0x7

cpu MHz : 2623.174

cache size : 4096 KB

physical id : 0

siblings : 2

core id : 0

cpu cores : 2

apicid : 0

initial apicid : 0

fpu : yes

fpu_exception : yes

cpuid level : 11

wp : yes

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 popcnt aes lahf_lm pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid dtherm ida arat flush_l1d

vmx flags : vnmi preemption_timer invvpid ept_x_only flexpriority tsc_offset vtpr mtf vapic ept vpid unrestricted_guest

bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit

bogomips : 4256.16

clflush size : 64

cache_alignment : 64

address sizes : 36 bits physical, 48 bits virtual

power management:

processor : 1

vendor_id : GenuineIntel

cpu family : 6

model : 37

model name : Intel(R) Core(TM) i7 CPU L 640 @ 2.13GHz

stepping : 5

microcode : 0x7

cpu MHz : 2416.981

cache size : 4096 KB

physical id : 0

siblings : 2

core id : 2

cpu cores : 2

apicid : 4

initial apicid : 4

fpu : yes

fpu_exception : yes

cpuid level : 11

wp : yes

flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 popcnt aes lahf_lm pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid dtherm ida arat flush_l1d

vmx flags : vnmi preemption_timer invvpid ept_x_only flexpriority tsc_offset vtpr mtf vapic ept vpid unrestricted_guest

bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit

bogomips : 4256.16

clflush size : 64

cache_alignment : 64

address sizes : 36 bits physical, 48 bits virtual

power management:

jackhill@alperton ~$ cat repos/guix-configs/alperton/config.scm

;; This is an operating system configuration template

;; for a "desktop" setup with GNOME and Xfce where the

;; root partition is encrypted with LUKS.

(use-modules (gnu) (gnu system nss) (gnu services xorg)

(gnu packages linux)

(nongnu packages linux)

(nongnu system linux-initrd)

(srfi srfi-1))

(use-service-modules ;; afs

cups desktop docker kerberos sddm)

(use-package-modules certs gnome kerberos printers

scanner security-token

wm)

(operating-system

(host-name "alperton")

(timezone "America/New_York")

(locale "en_US.utf8")

(bootloader (bootloader-configuration

(bootloader grub-bootloader)

(target "/dev/sda")))

;; Specify a mapped device for the encrypted root partition.

;; The UUID is that returned by 'cryptsetup luksUUID'.

(mapped-devices

(list (mapped-device

(source (uuid "9cfdc1d9-d062-4269-9cbb-9cb518c4cf4c"))

(target "alperton_btrfs")

(type luks-device-mapping))))

(file-systems (cons

(file-system

(device (uuid "179969de-85a9-4e95-ba44-79566c492eb5"))

(mount-point "/")

(type "btrfs")

(flags '(no-atime))

(options "compress=zstd")

(dependencies mapped-devices))

%base-file-systems))

(swap-devices (list "/root/swap"))

(users (cons (user-account

(name "jackhill")

(comment "Jack Hill")

(group "users")

(supplementary-groups '("wheel" "netdev"

;"docker"

"audio" "lp" "video"))

(home-directory "/home/jackhill"))

%base-user-accounts))

;; This is where we specify system-wide packages.

(packages (cons* nss-certs ;for HTTPS access

btrfs-progs

fuse-exfat

bluez

mit-krb5

sway

gvfs ;for user mounts

%base-packages))

;; Add GNOME and/or Xfce---we can choose at the log-in

;; screen with F1. Use the "desktop" services, which

;; include the X11 log-in service, networking with

;; NetworkManager, and more.

(services (cons* (service gnome-desktop-service-type)

(bluetooth-service)

;; (service docker-service-type)

(simple-service 'custom-udev-rules udev-service-type (list sane-backends libu2f-host))

(screen-locker-service swaylock)

(service sddm-service-type)

(service cups-service-type

(cups-configuration

(web-interface? #true)))

(service krb5-service-type

(krb5-configuration

(default-realm "HCOOP.NET")

(forwardable? #t)

))

;; (service afs-client-service-type)

(modify-services (remove (lambda (service)

(eq? (service-kind service) gdm-service-type))

%desktop-services)

(guix-service-type

config =>

(guix-configuration

(inherit config)

(authorized-keys

(cons*

(local-file "../keys/libre-01-guix-key.pub")

(local-file "../keys/libre-02-guix-key.pub")

%default-authorized-guix-keys)))))))

(kernel linux)

(kernel-arguments '("quite"

"zswap.enabled=1" "zswap.compressor=zstd"

"zswap.max_pool_percent=50" "zswap.zpool=z3fold"))

(initrd-modules (cons* "zstd" "z3fold"

%base-initrd-modules))

(initrd microcode-initrd)

(firmware (append (list linux-firmware iwlwifi-firmware broadcom-bt-firmware)

%base-firmware))

;; Allow resolution of '.local' host names with mDNS.

(name-service-switch %mdns-host-lookup-nss))

jackhill@alperton ~$ free -m

total used free shared buff/cache available

Mem: 3735 2087 209 368 1438 1076

Swap: 8191 2696 5495

Clearly there as some differences between these hosts. However, the same

derivation and png file:

jackhill@alperton ~$ guix build /gnu/store/07xw2pp63xin4c4y8ndrcdn3n8z1vmx2-grub-image.png.drv

/gnu/store/vmldvxllh07k641wmbnlz3migga29rfn-grub-image.png

jackhill@alperton ~$ cat /gnu/store/07xw2pp63xin4c4y8ndrcdn3n8z1vmx2-grub-image.png.drv

Derive([("out","/gnu/store/vmldvxllh07k641wmbnlz3migga29rfn-grub-image.png","","")],[("/gnu/store/6k0b8k9cl9gcrg603cxva0qnwbxv55xs-guile-rsvg-2.18.1-0.05c6a2f.drv",["out"]),("/gnu/store/b5nnbpgkvgdpzgvj67539ylcaqacj90l-guile-3.0.2.drv",["out"]),("/gnu/store/hb2q1683r8x8n28dyvr4gvdgkhmssq8q-guix-artwork-2f2fe74-checkout.drv",["out"]),("/gnu/store/kvpdmjknxqjm9k6gi2c9bijkrmk9n944-module-import-compiled.drv",["out"]),("/gnu/store/rcl324yiq7a56rwkqwgqx097dwc5mgni-guile-cairo-1.11.2.drv",["out"])],["/gnu/store/ih9cbxl2qwn9bn2yfmr2g40w7p7yafic-module-import","/gnu/store/larqpc2wjhnc6jmj4885k8lynd19fl4m-grub-image.png-builder"],"x86_64-linux","/gnu/store/0m0vd873jp61lcm4xa3ljdgx381qa782-guile-3.0.2/bin/guile",["--no-auto-compile","-L","/gnu/store/ih9cbxl2qwn9bn2yfmr2g40w7p7yafic-module-import","-L","/gnu/store/0b39xp6kndr95k6rccbp8ijwvsrkygvd-guile-rsvg-2.18.1-0.05c6a2f/share/guile/site/3.0","-L","/gnu/store/vjn7ygzzqshvsfzck8hq5lp5pfrr2xp5-guile-cairo-1.11.2/share/guile/site/3.0","-C","/gnu/store/pk1r70b4gxn9fsd53glr8alqz5h1kk65-module-import-compiled","-C","/gnu/store/0b39xp6kndr95k6rccbp8ijwvsrkygvd-guile-rsvg-2.18.1-0.05c6a2f/lib/guile/3.0/site-ccache","-C","/gnu/store/vjn7ygzzqshvsfzck8hq5lp5pfrr2xp5-guile-cairo-1.11.2/lib/guile/3.0/site-ccache","/gnu/store/larqpc2wjhnc6jmj4885k8lynd19fl4m-grub-image.png-builder"],[("out","/gnu/store/vmldvxllh07k641wmbnlz3migga29rfn-grub-image.png"),("preferLocalBuild","1")])

is used on both systems, and after copying the successfully built png file

to "bad" host, it is used successfully.

This has been a difficult problem for me to track down, since the only way

I know to reproduce it is with guix system reconfigure (guix system build

isn't even enough). I think it would make troubleshooting easier if I

could generate the problematic derivations outside of guix system

reconfigure. At this point, I'm not sure what additional information would

be relevant. Are there any additional places I should look?

Thanks!

Jack

Okay, I've started looking at the builder a little more:

jackhill@alperton ~$ cat /gnu/store/larqpc2wjhnc6jmj4885k8lynd19fl4m-grub-image.png-builder

(if (string-suffix? ".svg" "/gnu/store/83qplqmavzphd30hm1maxwlh166ylpwr-guix-artwork-2f2fe74-checkout/grub/GuixSD-fully-black-4-3.svg") (begin (use-modules (gnu build svg)) (svg->png "/gnu/store/83qplqmavzphd30hm1maxwlh166ylpwr-guix-artwork-2f2fe74-checkout/grub/GuixSD-fully-black-4-3.svg" ((@ (guile) getenv) "out") #:width 1024 #:height 768)) (copy-file "/gnu/store/83qplqmavzphd30hm1maxwlh166ylpwr-guix-artwork-2f2fe74-checkout/grub/GuixSD-fully-black-4-3.svg" ((@ (guile) getenv) "out")))

The problem appears to be in the svg->png procedure or at least in the

svg.scm file. On the "bad" system:

jackhill@kalessin ~$ guix environment --ad-hoc guile guile-rsvg guile-readline

jackhill@kalessin ~ [env]$ guile

GNU Guile 3.0.5

Copyright (C) 1995-2021 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.

This program is free software, and you are welcome to redistribute it

under certain conditions; type `,show c' for details.

Enter `,help' for help.

scheme@(guile-user)> ,use (gnu build svg)

;;; note: auto-compilation is enabled, set GUILE_AUTO_COMPILE=0

;;; or pass the --no-auto-compile argument to disable.

;;; compiling /run/current-system/profile/share/guile/site/3.0/gnu/build/svg.scm

;;; WARNING: compilation of /run/current-system/profile/share/guile/site/3.0/gnu/build/svg.scm failed:

;;; failed to create path for auto-compiled file "/run/current-system/profile/share/guile/site/3.0/gnu/build/svg.scm"

scheme@(guile-user)> (svg->png "/gnu/store/83qplqmavzphd30hm1maxwlh166ylpwr-guix-artwork-2f2fe74-checkout/grub/GuixSD-fully-black-4-3.svg" "/tmp/test.png")

ice-9/boot-9.scm:1669:16: In procedure raise-exception:

Wrong type (expecting finalized smob): #<cairo-context 7fb032a3e6b0>

Entering a new prompt. Type `,bt' for a backtrace or `,q' to continue.

On the good system

ckhill@alperton ~$ guix environment --ad-hoc guile guile-rsvg guile-readline

jackhill@alperton ~ [env]$ guile

GNU Guile 3.0.5

Copyright (C) 1995-2021 Free Software Foundation, Inc.

Guile comes with ABSOLUTELY NO WARRANTY; for details type `,show w'.

This program is free software, and you are welcome to redistribute it

under certain conditions; type `,show c' for details.

Enter `,help' for help.

scheme@(guile-user)> ,use (gnu build svg)

;;; note: auto-compilation is enabled, set GUILE_AUTO_COMPILE=0

;;; or pass the --no-auto-compile argument to disable.

;;; compiling /run/current-system/profile/share/guile/site/3.0/gnu/build/svg.scm

;;; compiled /home/jackhill/.cache/guile/ccache/3.0-LE-8-4.4/gnu/store/0j6w61vjjvp4zqzrqvyhqm6254ppzh8y-guix-1.2.0-16.c8887a5/share/guile/site/3.0/gnu/build/svg.scm.go

scheme@(guile-user)> (svg->png "/gnu/store/83qplqmavzphd30hm1maxwlh166ylpwr-guix-artwork-2f2fe74-checkout/grub/GuixSD-fully-black-4-3.svg" "/tmp/test.png")

and a png file is produced. Particularly relivant seems the

auto-compilation failure.

To be continued…

Best,

Jack

Hi Jack,

Jack Hill <jackhill@jackhill.us> writes:

Rolling back to an earlier system generation and running "guix gc" again

was a good idea, but you might have missed one or two crucial steps in

between:

(1) You must *delete* the "older" system generations and user profiles

e.g. by running "guix system delete-generations" and "guix package

--delete-generations", or else "guix gc" won't clear them from your

store. It is not enough to merely switch to an older system

generation and profiles.

(2) You'll also need to actually reboot into the older system

generation, because /run/booted-system will continue to protect

(from GC) the system that you last booted into, even after you

switch systems.

Did you do those things before running "guix gc"?

I'm sorry that you've hit this nasty bug.

Regards,

Mark

On Fri, Mar 12, 2021 at 07:24:16PM -0500, Mark H Weaver wrote:

I wasn't able to reproduce this on either a Core 2 Duo or a newer AMD

EPYC machine.

On Sun, 14 Mar 2021, Mark H Weaver wrote:

Oops, I left out those details. Yes, I did both those things.

Thanks. For me, being the only one that can reproduce or experience a

problem can be a frustrating and lonely experience, so really appreciate

the time you and Leo have spent looking at it.

Best,

Jack

Hi Jack,

Jack Hill <jackhill@jackhill.us> writes:

It occurs to me that we missed something: the profiles in

~/.config/guix/current that are managed by "guix pull". It might be

that code within Guix itself was miscompiled (e.g. gnu/build/svg.scm),

or else that a profile in ~/.config/guix/current is still holding a

reference to something else that was miscompiled, (e.g. guile-cairo).

I suggest "guix pull --commit=453e101fc3f7dac9aabcd6122cf05fb7925103c7",

and then "guix package -p ~/.config/guix/current --delete-generations"

to delete any generations of Guix at commits that came after the Cairo

graft (use "guix pull --list-generations" to list them). Do this for

all user accounts (including root) that have a ~/.config/guix/current

directory. Then, try "guix gc" again.

Thanks,

Mark

On Sun, 14 Mar 2021, Mark H Weaver wrote:

Thanks Mark. I've done the dance to gc as much as possible again. This

time, I also checked in /var/guix/gcroots to make sure I hadn't missed

anything. In fact I had missed some extra manual roots that I had created,

and I cleaned those up as well before running guix gc.

After running guix gc, I rebooted, ran guix pull, followed by a

reconfigure. The first reconfigure failed because of the substitute

networking problem, but when I ran it again, it failed in the same way

building the grub png. After it failed, I ran it again to capture the

following output:

jackhill@kalessin ~$ guix describe

Generation 9 Mar 14 2021 23:24:43 (current)

guix d059485

repository URL: https://git.savannah.gnu.org/git/guix.git

branch: master

commit: d059485257bbe5b4f4d903b357ec99a3af2d4f39

jackhill@kalessin ~$ sudo -E guix system -v3 reconfigure /config.scm

The following derivations will be built:

/gnu/store/xqdm3fslr3n0jyxh6i3nsn237lygjfwf-system.drv

/gnu/store/2p1s41kwh9w7w8cijg3r4zplc9f9i6fw-activate.scm.drv

/gnu/store/jgagsl2m5x5vi63s3hdwg6lb58m8qiz1-activate-service.scm.drv

/gnu/store/dsv31bkl2vwqhqgrqvz59wir009ix3kb-etc.drv

/gnu/store/9f2rvmk0xii50smi8dwn0q9556y7qc94-rottlog.drv

/gnu/store/ky3yw75v55g06ggi4i0xk155i7knn10f-sudoers.drv

/gnu/store/b2h0nkrd03zff082lg7y149aw3j9yfxg-profile.drv

/gnu/store/hlr9ypdb841sz2w949mxi5kqhvv2dd22-boot.drv

/gnu/store/y8s53y9irwbsy1pc07vbczbp7jwsrsw4-shepherd.conf.drv

/gnu/store/6zk7p1iljyayb5hyafgbzik06cq0f00j-shepherd-ssh-daemon-ssh-sshd.go.drv

/gnu/store/p89f6qy78yarsjrmq8mkrjihnk4hpm25-shepherd-ssh-daemon-ssh-sshd.scm.drv

/gnu/store/kscdry7kq4izr7nyzs6gq3kg0hqcjffx-shepherd-guix-daemon.go.drv

/gnu/store/aa4wgjx3625m5k71i5rzb0ywx9z6a0i3-shepherd-guix-daemon.scm.drv

/gnu/store/qy2sl92bqnzahvpzb6imgspp6llpz0cj-shepherd-mcron.go.drv

/gnu/store/xdxd5gfvzk4g0m2idbfcrp3d32gm0vz6-shepherd-mcron.scm.drv

/gnu/store/q8ampzxsdkibl15jhlvq30gic5qgm0wi-mcron-job.drv

/gnu/store/qj9nqyhci6zhkfprpwch90ry5hkhwvbx-mcron-job.drv

/gnu/store/6gx45db5mwraihq1qv8c9vmxhdskjk1a-grub.cfg.drv

/gnu/store/07xw2pp63xin4c4y8ndrcdn3n8z1vmx2-grub-image.png.drv

The following grafts will be made:

/gnu/store/fwwwnlzhckvi4wmw89m9az9y9wb9v6q9-rottlog-0.72.2.drv

/gnu/store/26z2lhnqhzr5b88axv7b38fgqjl3w2h8-usbutils-013.drv

The following profile hooks will be built:

/gnu/store/5c19y82k9pw297w0b5gn8j6p7g7c6h60-ca-certificate-bundle.drv

/gnu/store/j5plp2k4bkjilqx1yw9mkavy37ipp29h-fonts-dir.drv

/gnu/store/lcilg958v3adfl8jljkjwpwihbzsyr6c-info-dir.drv

/gnu/store/z5m7ra9zd3vhqbp5hg4695s2jgsggr6q-manual-database.drv

building /gnu/store/07xw2pp63xin4c4y8ndrcdn3n8z1vmx2-grub-image.png.drv...

Backtrace:

2 (primitive-load "/gnu/store/larqpc2wjhnc6jmj4885k8lynd1?")

In gnu/build/svg.scm:

53:6 1 (svg->png _ "/gnu/store/vmldvxllh07k641wmbnlz3migga29r?" ?)

In unknown file:

0 (rsvg-handle-render-cairo #<rsvg-handle 7ffff5b60150> #)

ERROR: In procedure rsvg-handle-render-cairo:

Wrong type (expecting finalized smob): #<cairo-context 7ffff5b60090>

builder for `/gnu/store/07xw2pp63xin4c4y8ndrcdn3n8z1vmx2-grub-image.png.drv' failed with exit code 1

build of /gnu/store/07xw2pp63xin4c4y8ndrcdn3n8z1vmx2-grub-image.png.drv failed

View build log at '/var/log/guix/drvs/07/xw2pp63xin4c4y8ndrcdn3n8z1vmx2-grub-image.png.drv.bz2'.

cannot build derivation `/gnu/store/6gx45db5mwraihq1qv8c9vmxhdskjk1a-grub.cfg.drv': 1 dependencies couldn't be built

guix system: error: build of `/gnu/store/6gx45db5mwraihq1qv8c9vmxhdskjk1a-grub.cfg.drv' failed

Do you think it is worth creating another VM to see if it's a problem with

the VM configuration?

Best,

Jack

On Sun, 14 Mar 2021, Jack Hill wrote:

Er, I wrote it backwords here, but I ran them in the correct order: delete

roots, reboot, gc, pull, …

severity 47115 important

quit

I was able to reproduce this on a new VM with the same hosting provider

(Ramnode), but in a different data center. Therefore, I conclude that it

is not a fault in the particular hardware the VMs are running on, but that

it could be a general problem with the hardware used by Ramnode or their

virtualization software.

Apologies for not mentioning the provider before. At the time, I didn't

see the need to advertise them, and it didn't seem likely to me that the

problem was particular to them.

The good news is that I now have a VM dedicated to reproducing this

problem, so if anyone would like access to help with investigation, please

let me know (and include your preferred username and ssh public key).

For what it's worth, I noticed that guile-rsvg was substituted from

ci.guix.gnu.org during the failed reconfigure.

Best,

Jack

On Mon, 15 Mar 2021, Jack Hill wrote:

I've now reproduced this problem at a different VM provider (Linode), so I

strongly suspect that it doesn't have anything to do with the hardware or

virtualization configuration. There must be something about my

operating system configuration that triggers this bug.

Best,

Jack

I believe that I have identified the problematic difference in my

operating system config between my working and non-working hosts. After

applying the following patch to my operating system config (good and bad

versions attatched), I was able to successfully reconfigure with guix

8ec0ca8faff62f19426f22aeb1bd59a8950ca05a (I was able to reproduce the

failure with that commit on another VM):

--- bad.scm 2021-03-15 22:36:36.000000001 -0400

+++ good.scm 2021-03-15 22:37:01.000000001 -0400

@@ -79,8 +79,6 @@

(guix-service-type config =>

(guix-configuration

(inherit config)

- (extra-options

- '("--disable-deduplication"))

(authorized-keys

(cons

(local-file "/home/jackhill/alperton-guix-key.pub")

I am forced to conclude that running the guix-daemon with deduplication

disabled causes this build failure. Spooky!

Best,

Jack

retitle 47115 Grafts without deduplication can lead to breakage in Guile

thanks

Hi Jack,

Jack Hill <jackhill@jackhill.us> writes:

Thanks very much for your investigation.

Very interesting!

The only difference deduplication makes is that it (usually) causes

identical files in the store to be hard links to the same inode.

I have a new hypothesis:

Suppose that a reference to the original (ungrafted) version of some

library (e.g. libcairo or librsvg) has survived unchanged by the

grafting process. This could lead to two copies of the same library

being loaded. For example, I guess that libcairo is loaded by both

librsvg and by guile-cairo. One of them might load the original

libcairo, and the other might load the replacement libcairo.

If the library is loaded twice, that could lead to each instance of the

library having its own dynamically-allocated type tags, which could lead

to this kind of error.

Here's the code where the error occurred:

https://git.savannah.gnu.org/cgit/guix.git/tree/gnu/build/svg.scm?id=bc16eacc99e801ac30cbe2aa649a2be3ca5c102a#n40

Guile uses 'cairo-create' (via guile-cairo) to create a cairo-context,

and then passes it to 'rsvg-handle-render-cairo', a 'librsvg' function,

which complains that the context argument has the wrong type.

If 'guile-cairo' was somehow using a different instance of 'libcairo'

than the one that 'librsvg' is linked to, that could explain what we're

seeing, because the two instances of 'libcairo' would have different

ideas of what the cairo-context tag should be.

However, *if* you have deduplication enabled, and *if* the library in

question doesn't contain any references that require rewrites due to

grafts, then these two copies of the library would most likely[*] be

hard links to the same inode. Perhaps in that case, the run-time loader

recognizes that these are in fact the same library, and suppresses the

redundant load.

I don't know if this is what's happening, but it seems plausible.

Thoughts?

Regards,

Mark

retitle 47115 Redundant library grafts leads to breakage

thanks

Hi,

I looked a bit deeper, and now I think I finally know what's going on.

It turns out that the grafting process is creating two redundant

variants of the replacement guile-cairo.

All of the relevant information is in

/gnu/store/07xw2pp63xin4c4y8ndrcdn3n8z1vmx2-grub-image.png.drv and its

dependents, which fails to build if deduplication is disabled.

If you look through the output of "guix size

/gnu/store/07xw2pp63xin4c4y8ndrcdn3n8z1vmx2-grub-image.png.drv", you'll

find three distinct guile-cairo derivations:

(1) /gnu/store/vz4yw7zkm73diy95mdzywgixal3nf2s2-guile-cairo-1.11.2.drv,

=> /gnu/store/5nmzfnxk8kp85xwma2r585fgyz3jfw56-guile-cairo-1.11.2

(the original, ungrafted, cairo)

(2) /gnu/store/rcl324yiq7a56rwkqwgqx097dwc5mgni-guile-cairo-1.11.2.drv,

=> /gnu/store/vjn7ygzzqshvsfzck8hq5lp5pfrr2xp5-guile-cairo-1.11.2

(the first graft)

(3) /gnu/store/9mha4bzbji8iql50prmq9br4j1c51sjn-guile-cairo-1.11.2.drv,

=> /gnu/store/j69k9n0g3h9ppqi7dmqypwy3lrhxvb97-guile-cairo-1.11.2

(the second graft)

In the 'guile-builder' files referenced from the two graft derivations,

we see that they have the same inputs and perform the same rewrites, but

list them in a different order. Graft 1 has this guile-builder:

Graft 2 has this guile-builder:

I think that my last hypothesis was on the right track, but not quite

right:

"libguile-cairo.so" is being loaded twice.

suspect that two different variants of the replacement "guile-cairo"

are being loaded.

duplicated smob tags are being allocated.

However, *if* deduplication is enabled, two redundant replacements

created by grafting _should_ occupy the same inodes, assuming that the

replacement mappings are the same (modulo ordering), and assuming that

/gnu/store/.links doesn't hit a directory size limit (which can happen

on ext3/4, leading to missed deduplication opportunities).

I've known about these redundant replacements in Guix for many years,

but was not aware of any significant practical problems arising from

them until now.

Mark

Hi Mark,

Mark H Weaver <mhw@netris.org> skribis:

Woow, thanks for the investigation! You wouldn’t think that

deduplication can have an effect on this kind of bug.

Do you know why the two guile-cairo grafts differ in this case?

I’m aware of one case that can lead to that: the grafting code can

create grafts for just one output of the original derivation, or for all

of them (commit 482fda2729c3e76999892cb8f9a0391a7bd37119). Maybe that’s

what’s happening here?

Thank you!

Ludo’.

On Sun, 18 Apr 2021, Dr. Arne Babenhauserheide wrote:

[…]

Oh dear. I ran into a similar problem (at least that resulted in the same

Guile error message in #47115) to do with grafts and a store that was not

using the hard links for de-duplication. Essentially the de-duplication

masked an issue with differing files being included in the closure.

Can you provide more information about your system? I'm particularly

interested in whether you're using store de-duplication, but other

specifics about your system might be interesting. Do you know when you

last successfully reconfigured?

Best,

Jack

merge 47115 47853

quit

Hi Guix!

I believe I found another instance of this bug coming back to haunt

unfortunate, wayward souls. (including me! 😭).

--

Take it easy,

Richard Sent

Making my computer weirder one commit at a time.

After updating to the "lisp-team" branch which included an updated sbcl

and more (commit 2d5a7bfed5ccae6ce8adbef3ae1017d6ce8512be), my system broke

down when reading ~/.stumpwm.d/init.lisp with a message like "failed

AVER: (= (ASD (LENGTH KV-VECTOR) -1 ... This is probably a bug in SBCL

itself. ..."

Got a hint to read https://git.sr.ht/~freakingpenguin/rsentwhich

references https://issues.guix.gnu.org/62890.I tried with --no-grafts

as that repo does, and my system works (commit

e32e3d0a03dc17c4c54a91aad053c9036998b601).