#45962 ‘binutils-mesboot0’ includes non-zero timestamps in ar archives

Message #5 received at submit@debbugs.gnu.org (full text, mbox, reply):

From: Ludovic Courtès <ludo@gnu.org>

To: <bug-guix@gnu.org>

Subject: ‘binutils-mesboot0’ includes non-zero timestamps in ar archives

Date: Mon, 18 Jan 2021 18:29:24 +0100

Hi!

On #bootstrappable, mid-kid reported that ‘binutils-mesboot0’ in
commencement.scm lacks ‘--enable-deterministic-archives’.  So I checked
if this had an effect by running:

  guix build -e '(@@ (gnu packages commencement) gcc-core-mesboot0)' \
     --check -K

and yes, it does:

--8<---------------cut here---------------start------------->8---
$ diff -ru /gnu/store/ri28kdl41bb76qjr4cyarylw7kxpvfxy-gcc-core-mesboot0-2.95.3{,-check}
Binary files /gnu/store/ri28kdl41bb76qjr4cyarylw7kxpvfxy-gcc-core-mesboot0-2.95.3/lib/gcc-lib/i686-unknown-linux-gnu/2.95.3/libc.a and /gnu/store/ri28kdl41bb76qjr4cyarylw7kxpvfxy-gcc-core-mesboot0-2.95.3-check/lib/gcc-lib/i686-unknown-linux-gnu/2.95.3/libc.a differ
Binary files /gnu/store/ri28kdl41bb76qjr4cyarylw7kxpvfxy-gcc-core-mesboot0-2.95.3/lib/gcc-lib/i686-unknown-linux-gnu/2.95.3/libgcc.a and /gnu/store/ri28kdl41bb76qjr4cyarylw7kxpvfxy-gcc-core-mesboot0-2.95.3-check/lib/gcc-lib/i686-unknown-linux-gnu/2.95.3/libgcc.a differ
Binary files /gnu/store/ri28kdl41bb76qjr4cyarylw7kxpvfxy-gcc-core-mesboot0-2.95.3/lib/libgcc2.a and /gnu/store/ri28kdl41bb76qjr4cyarylw7kxpvfxy-gcc-core-mesboot0-2.95.3-check/lib/libgcc2.a differ
Binary files /gnu/store/ri28kdl41bb76qjr4cyarylw7kxpvfxy-gcc-core-mesboot0-2.95.3/lib/libiberty.a and /gnu/store/ri28kdl41bb76qjr4cyarylw7kxpvfxy-gcc-core-mesboot0-2.95.3-check/lib/libiberty.a differ
--8<---------------cut here---------------end--------------->8---

Apparently Binutils 2.14 didn’t have ‘--enable-deterministic-archives’
so we’ll have to patch it.

There are a few other Binutils variants in commencement.scm that we’ll
have to check.

Ludo’.

Message #10 received at 45962@debbugs.gnu.org (full text, mbox, reply):

From: Ludovic Courtès <ludo@gnu.org>

To: 45962@debbugs.gnu.org

Cc: Maxim Cournoyer <maxim.cournoyer@gmail.com>, Jan Nieuwenhuizen <janneke@gnu.org>

Subject: Re: bug#45962: ‘binutils-mesboot0’ includes non-zero timestamps in ar archives

Date: Wed, 03 Mar 2021 14:48:44 +0100

[Message part 1 (text/plain, inline)]

Hi,

Ludovic Courtès <ludo@gnu.org> skribis:

> On #bootstrappable, mid-kid reported that ‘binutils-mesboot0’ in
> commencement.scm lacks ‘--enable-deterministic-archives’.  So I checked
> if this had an effect by running:

[...]

> Apparently Binutils 2.14 didn’t have ‘--enable-deterministic-archives’
> so we’ll have to patch it.

Sikonas on #bootstrappable provided a patch for that (thanks!) so I went
ahead and gave it a try on ‘core-updates’ (Guix patch attached).

The binutils source gets patched and repacked, but then decompressing it
fails:

--8<---------------cut here---------------start------------->8---
building /gnu/store/fwz150xjaqbh8n02z6gsmpm9w8lxckak-binutils-mesboot0-2.14.drv...
starting phase `set-SOURCE-DATE-EPOCH'
phase `set-SOURCE-DATE-EPOCH' succeeded after 0.0 seconds
starting phase `set-paths'
environment variable `PATH' set to `/gnu/store/70w6gnl3vzv2rsjhyjg0wsxl3pbnpjb5-bash-mesboot0-2.05b/bin:/gnu/store/2wkjilvkdyb2wrb20ba82dhdpnx2732n-bzip2-mesboot-1.0.8/bin:/gnu/store/4nxicfzj8a390f4scxcgglhdqgbyhlkw-gzip-mesboot-1.2.4/bin:/gnu/store/6fgjdcl2qrrs7fvdznkx1gcbw6wjfrn0-patch-mesboot-2.5.9/bin:/gnu/store/4lpagdav2gm022v1fgzcs8323xrggz4b-sed-mesboot0-1.18/bin:/gnu/store/y70rava2vqf1nilr8smg54qqxdclvrz6-gash-utils-boot-0.1.0/bin:/gnu/store/y9fy4r99q6pqisrw81fn92gk8mzbyzn1-tcc-boot-0.9.27/bin:/gnu/store/krnwfrki71dj6zicl2qwv6bdyqvsxgcg-make-mesboot0-3.80/bin:/gnu/store/xxnxdnjdlav4s8v3lfhg7x7amrqcrv57-gash-boot-0.2.0/bin:/gnu/store/aglbpf6bihv35pwpvdy6chxv318hsafq-bootar-1a/bin:/gnu/store/lgi9x15a0w35mcpd7g1kb9274r6wy4pv-guile-bootstrap-2.0/bin'
environment variable `BASH_LOADABLES_PATH' unset
environment variable `C_INCLUDE_PATH' set to `/gnu/store/2wkjilvkdyb2wrb20ba82dhdpnx2732n-bzip2-mesboot-1.0.8/include:/gnu/store/y9fy4r99q6pqisrw81fn92gk8mzbyzn1-tcc-boot-0.9.27/include'
environment variable `LIBRARY_PATH' set to `/gnu/store/2wkjilvkdyb2wrb20ba82dhdpnx2732n-bzip2-mesboot-1.0.8/lib:/gnu/store/y70rava2vqf1nilr8smg54qqxdclvrz6-gash-utils-boot-0.1.0/lib:/gnu/store/y9fy4r99q6pqisrw81fn92gk8mzbyzn1-tcc-boot-0.9.27/lib:/gnu/store/xxnxdnjdlav4s8v3lfhg7x7amrqcrv57-gash-boot-0.2.0/lib:/gnu/store/aglbpf6bihv35pwpvdy6chxv318hsafq-bootar-1a/lib:/gnu/store/lgi9x15a0w35mcpd7g1kb9274r6wy4pv-guile-bootstrap-2.0/lib'
phase `set-paths' succeeded after 0.0 seconds
starting phase `install-locale'
warning: failed to install 'en_US.utf8' locale: Invalid argument
phase `install-locale' succeeded after 0.0 seconds
starting phase `unpack'
xz: cannot decompress from stdin
Backtrace:
In ice-9/boot-9.scm:
 157: 5 [catch #t #<catch-closure c928e0> ...]
In unknown file:
   ?: 4 [apply-smob/1 #<catch-closure c928e0>]
In ice-9/boot-9.scm:
  63: 3 [call-with-prompt prompt0 ...]
In ice-9/eval.scm:
 432: 2 [eval # #]
In gash/guix-utils.scm:
 138: 1 [call-with-decompressed-port xz ...]
In unknown file:
   ?: 0 [scm-error misc-error #f "~A ~S" ("decompressed-port failure" (7)) #f]

ERROR: In procedure scm-error:
ERROR: decompressed-port failure (7)
error: in phase 'unpack': uncaught exception:
srfi-34 #<condition &invoke-error [program: "tar" arguments: ("xvf" "/gnu/store/ywf5j03423yiawix3z21xa14hyyvd83z-binutils-2.14.tar.xz") exit-status: 1 term-signal: #f stop-signal: #f] 1215400> 
phase `unpack' failed after 0.3 seconds
command "tar" "xvf" "/gnu/store/ywf5j03423yiawix3z21xa14hyyvd83z-binutils-2.14.tar.xz" failed with status 1
builder for `/gnu/store/fwz150xjaqbh8n02z6gsmpm9w8lxckak-binutils-mesboot0-2.14.drv' failed with exit code 1
--8<---------------cut here---------------end--------------->8---

Maxime, does that ring a bell?  Could it be that this bootstrap ‘xz’ is
less capable, or could it be a Gash-Utils bug?

Thanks,
Ludo’.

[Message part 2 (text/x-patch, inline)]

diff --git a/gnu/packages/commencement.scm b/gnu/packages/commencement.scm
index 890d57941f..5c523ae7ad 100644
--- a/gnu/packages/commencement.scm
+++ b/gnu/packages/commencement.scm
@@ -997,13 +997,15 @@ $MES -e '(mescc)' module/mescc.scm -- \"$@\"
     (inherit binutils)
     (name "binutils-mesboot0")
     (version "2.14")
-    (source (origin
-              (method url-fetch)
-              (uri (string-append "mirror://gnu/binutils/binutils-"
-                                  version ".tar.gz"))
-              (sha256
-               (base32
-                "1w8xp7k44bkijr974x9918i4p1sw4g2fcd5mxvspkjpg38m214ds"))))
+    (source (bootstrap-origin
+             (origin
+               (method url-fetch)
+               (uri (string-append "mirror://gnu/binutils/binutils-"
+                                   version ".tar.gz"))
+               (sha256
+                (base32
+                 "1w8xp7k44bkijr974x9918i4p1sw4g2fcd5mxvspkjpg38m214ds"))
+               (patches (search-patches "binutils-2.14-deterministic-ar.patch")))))
     (inputs '())
     (propagated-inputs '())
     (native-inputs (%boot-tcc-inputs))
diff --git a/gnu/packages/patches/binutils-2.14-deterministic-ar.patch b/gnu/packages/patches/binutils-2.14-deterministic-ar.patch
new file mode 100644
index 0000000000..7f9310e994
--- /dev/null
+++ b/gnu/packages/patches/binutils-2.14-deterministic-ar.patch
@@ -0,0 +1,66 @@
+Old binutils do not have support for creating deterministic archives.
+Backported from upstream commit 36e4dce69dd23bea9ea2258dea35f034b6d6351c
+
+Patch by Chris Demetriou <cgd@google.com> (2009), adapted by
+Andrius Štikonas <andrius@stikonas.eu> (2021).
+
+--- a/bfd/archive.c	2021-03-01 00:05:54.888301655 +0000
++++ b/bfd/archive.c	2021-03-02 21:53:51.001617689 +0000
+@@ -1396,10 +1396,6 @@
+     {
+       /* Assume we just "made" the member, and fake it.  */
+       struct bfd_in_memory *bim = (struct bfd_in_memory *) member->iostream;
+-      time (&status.st_mtime);
+-      status.st_uid = getuid ();
+-      status.st_gid = getgid ();
+-      status.st_mode = 0644;
+       status.st_size = bim->size;
+     }
+   else if (stat (filename, &status) != 0)
+@@ -1408,6 +1404,11 @@
+       return NULL;
+     }
+ 
++  status.st_mtime = 0;
++  status.st_uid = 0;
++  status.st_gid = 0;
++  status.st_mode = 0644;
++
+   amt = sizeof (struct ar_hdr) + sizeof (struct areltdata);
+   ared = (struct areltdata *) bfd_zalloc (abfd, amt);
+   if (ared == NULL)
+@@ -2003,13 +2004,11 @@
+   stat (arch->filename, &statbuf);
+   memset ((char *) (&hdr), 0, sizeof (struct ar_hdr));
+   sprintf (hdr.ar_name, RANLIBMAG);
+-  /* Remember the timestamp, to keep it holy.  But fudge it a little.  */
+-  bfd_ardata (arch)->armap_timestamp = statbuf.st_mtime + ARMAP_TIME_OFFSET;
+   bfd_ardata (arch)->armap_datepos = (SARMAG
+ 				      + offsetof (struct ar_hdr, ar_date[0]));
+-  sprintf (hdr.ar_date, "%ld", bfd_ardata (arch)->armap_timestamp);
+-  sprintf (hdr.ar_uid, "%ld", (long) getuid ());
+-  sprintf (hdr.ar_gid, "%ld", (long) getgid ());
++  sprintf (hdr.ar_date, "%ld", 0);
++  sprintf (hdr.ar_uid, "%ld", 0);
++  sprintf (hdr.ar_gid, "%ld", 0);
+   sprintf (hdr.ar_size, "%-10d", (int) mapsize);
+   strncpy (hdr.ar_fmag, ARFMAG, 2);
+   for (i = 0; i < sizeof (struct ar_hdr); i++)
+@@ -2082,6 +2081,8 @@
+   struct ar_hdr hdr;
+   unsigned int i;
+ 
++  return TRUE;
++
+   /* Flush writes, get last-write timestamp from file, and compare it
+      to the timestamp IN the file.  */
+   bfd_flush (arch);
+@@ -2169,7 +2170,7 @@
+   memset ((char *) (&hdr), 0, sizeof (struct ar_hdr));
+   hdr.ar_name[0] = '/';
+   sprintf (hdr.ar_size, "%-10d", (int) mapsize);
+-  sprintf (hdr.ar_date, "%ld", (long) time (NULL));
++  sprintf (hdr.ar_date, "%ld", 0);
+   /* This, at least, is what Intel coff sets the values to.  */
+   sprintf ((hdr.ar_uid), "%d", 0);
+   sprintf ((hdr.ar_gid), "%d", 0);

Message #13 received at 45962@debbugs.gnu.org (full text, mbox, reply):

From: Jan Nieuwenhuizen <janneke@gnu.org>

To: Ludovic Courtès <ludo@gnu.org>

Cc: 45962@debbugs.gnu.org, Maxim Cournoyer <maxim.cournoyer@gmail.com>

Subject: Re: bug#45962: ‘binutils-mesboot0’ includes non-zero timestamps in ar archives

Date: Wed, 03 Mar 2021 18:54:29 +0100

Ludovic Courtès writes:

Hello!

> Ludovic Courtès <ludo@gnu.org> skribis:
>
>> On #bootstrappable, mid-kid reported that ‘binutils-mesboot0’ in
>> commencement.scm lacks ‘--enable-deterministic-archives’.  So I checked
>> if this had an effect by running:

> [...]
>
>> Apparently Binutils 2.14 didn’t have ‘--enable-deterministic-archives’
>> so we’ll have to patch it.
>
> Sikonas on #bootstrappable provided a patch for that (thanks!) so I went
> ahead and gave it a try on ‘core-updates’ (Guix patch attached).

Great!

> The binutils source gets patched and repacked, but then decompressing it
> fails:

[..]

> Maxime, does that ring a bell?  Could it be that this bootstrap ‘xz’ is
> less capable, or could it be a Gash-Utils bug?

Currently, we avoid using non-bootstrapped binaries in the bootstrap,
that includes 'xz'; earlier in the bootstrap that includes also 'patch'.

See also gcc-core-mesboot0: it applies the patch in a manual phase.  So
I'm not sure if we want to start depending on 'xz' an this stage?

Greetings,
Janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Message #16 received at 45962@debbugs.gnu.org (full text, mbox, reply):

From: Maxim Cournoyer <maxim.cournoyer@gmail.com>

To: Ludovic Courtès <ludo@gnu.org>

Cc: 45962@debbugs.gnu.org, Jan Nieuwenhuizen <janneke@gnu.org>

Subject: Re: bug#45962: ‘binutils-mesboot0’ includes non-zero timestamps in ar archives

Date: Thu, 04 Mar 2021 11:14:29 -0500

Hi,

Ludovic Courtès <ludo@gnu.org> writes:

[...]

> ERROR: In procedure scm-error:
> ERROR: decompressed-port failure (7)
> error: in phase 'unpack': uncaught exception:
> srfi-34 #<condition &invoke-error [program: "tar" arguments: ("xvf" "/gnu/store/ywf5j03423yiawix3z21xa14hyyvd83z-binutils-2.14.tar.xz") exit-status: 1 term-signal: #f stop-signal: #f] 1215400> 
> phase `unpack' failed after 0.3 seconds
> command "tar" "xvf" "/gnu/store/ywf5j03423yiawix3z21xa14hyyvd83z-binutils-2.14.tar.xz" failed with status 1
> builder for `/gnu/store/fwz150xjaqbh8n02z6gsmpm9w8lxckak-binutils-mesboot0-2.14.drv' failed with exit code 1
>
> Maxime, does that ring a bell?  Could it be that this bootstrap ‘xz’ is
> less capable, or could it be a Gash-Utils bug?

From my IRC logs:

2020-09-20 22:32:01	apteryx	janneke: haha! the xz-bootstrap supports
--memlimit with % after all, my mistake was really silly... forgetting a
space between the args passed as XZ_DEFAULTS

I recall a similar error I had hit when working on adding multi-core
compression support to xz, but it ended up being just a mistake on my
part; the xz-bootstrap supported the required arguments just fine after
all.

HTH,

Maxim

Message #19 received at 45962@debbugs.gnu.org (full text, mbox, reply):

From: Maxim Cournoyer <maxim.cournoyer@gmail.com>

To: Jan Nieuwenhuizen <janneke@gnu.org>

Cc: Ludovic Courtès <ludo@gnu.org>, 45962@debbugs.gnu.org

Subject: Re: bug#45962: ‘binutils-mesboot0’ includes non-zero timestamps in ar archives

Date: Thu, 04 Mar 2021 11:39:04 -0500

Jan Nieuwenhuizen <janneke@gnu.org> writes:

> Ludovic Courtès writes:
>
> Hello!
>
>> Ludovic Courtès <ludo@gnu.org> skribis:
>>
>>> On #bootstrappable, mid-kid reported that ‘binutils-mesboot0’ in
>>> commencement.scm lacks ‘--enable-deterministic-archives’.  So I checked
>>> if this had an effect by running:
>
>> [...]
>>
>>> Apparently Binutils 2.14 didn’t have ‘--enable-deterministic-archives’
>>> so we’ll have to patch it.
>>
>> Sikonas on #bootstrappable provided a patch for that (thanks!) so I went
>> ahead and gave it a try on ‘core-updates’ (Guix patch attached).
>
> Great!
>
>> The binutils source gets patched and repacked, but then decompressing it
>> fails:
>
> [..]
>
>> Maxime, does that ring a bell?  Could it be that this bootstrap ‘xz’ is
>> less capable, or could it be a Gash-Utils bug?
>
> Currently, we avoid using non-bootstrapped binaries in the bootstrap,
> that includes 'xz'; earlier in the bootstrap that includes also 'patch'.
>
> See also gcc-core-mesboot0: it applies the patch in a manual phase.  So
> I'm not sure if we want to start depending on 'xz' an this stage?

I see; so what Ludovic got surprised by is the fact that when adding
patches or a snippet to an origin it gets repacked as an xz tarball.
That's nothing new (it's how it is on the master branch too); but it can
indeed be surprising.

Maxim

Message #22 received at 45962@debbugs.gnu.org (full text, mbox, reply):

From: Ludovic Courtès <ludo@gnu.org>

To: Jan Nieuwenhuizen <janneke@gnu.org>

Cc: 45962@debbugs.gnu.org, Maxim Cournoyer <maxim.cournoyer@gmail.com>

Subject: Re: bug#45962: ‘binutils-mesboot0’ includes non-zero timestamps in ar archives

Date: Mon, 08 Mar 2021 15:13:04 +0100

Howdy Janneke!

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> Currently, we avoid using non-bootstrapped binaries in the bootstrap,
> that includes 'xz'; earlier in the bootstrap that includes also 'patch'.
>
> See also gcc-core-mesboot0: it applies the patch in a manual phase.  So
> I'm not sure if we want to start depending on 'xz' an this stage?

Probably not, good point!

I can think of two possibilities, then: (1) apply the patch in a phase
rather than via the ‘patches’ field, and (2) arrange so that
‘patch-and-repack’ does not compress the patched code or compresses it
with the bootstrap gzip.

My understanding is that #2 may be doable now thanks to Maxim’s recent
changes.  I’ll take a look!

Thanks,
Ludo’.

Message #25 received at 45962@debbugs.gnu.org (full text, mbox, reply):

From: Jan Nieuwenhuizen <janneke@gnu.org>

To: Ludovic Courtès <ludo@gnu.org>

Cc: 45962@debbugs.gnu.org, Maxim Cournoyer <maxim.cournoyer@gmail.com>

Subject: Re: bug#45962: ‘binutils-mesboot0’ includes non-zero timestamps in ar archives

Date: Wed, 10 Mar 2021 06:52:35 +0100

Ludovic Courtès writes:

Hey Ludo!

> Jan Nieuwenhuizen <janneke@gnu.org> skribis:
>
> I can think of two possibilities, then: (1) apply the patch in a phase
> rather than via the ‘patches’ field, and (2) arrange so that
> ‘patch-and-repack’ does not compress the patched code or compresses it
> with the bootstrap gzip.

Oh, that would be nice: we could remove all these clumsy manual patching
stages.  Also, we may be able to remove XZ from the bootstrap chain, if
no XZ-repackaging happens and only build a final version.

> My understanding is that #2 may be doable now thanks to Maxim’s recent
> changes.  I’ll take a look!

Great!
Greetings,
Janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

Send a report that this bug log contains spam.