[PATCH] packages, scripts, utils: Respect (parallel-job-count) when xz-compressing.

  • Done
  • quality assurance status badge
Details
2 participants
  • Ludovic Courtès
  • Marius Bakke
Owner
unassigned
Submitted by
Marius Bakke
Severity
normal

Debbugs page

M
M
Marius Bakke wrote on 1 Apr 2018 05:54
(address . guix-patches@gnu.org)(name . Marius Bakke)(address . mbakke@fastmail.com)
20180401125431.11996-1-mbakke@fastmail.com
* guix/packages.scm (patch-and-repack): When invoking 'xz', tell it to use the
amount of cores specified by e.g. 'guix build -c'.
* guix/scripts/pack.scm (%compressors, bootstrap-xz): Likewise. While at it,
use the long form '--threads=' for clarity.
* guix/utils.scm (decompressed-port, compressed-port, compressed-output-port):
Likewise.
---
guix/packages.scm | 3 ++-
guix/scripts/pack.scm | 7 +++++--
guix/utils.scm | 18 +++++++++++++++---
3 files changed, 22 insertions(+), 6 deletions(-)

Toggle diff (84 lines)
diff --git a/guix/packages.scm b/guix/packages.scm
index ab4b6278d..cca2c6357 100644
--- a/guix/packages.scm
+++ b/guix/packages.scm
@@ -604,7 +604,8 @@ specifies modules in scope when evaluating SNIPPET."
;; threaded compression (introduced in
;; 5.2.0), but it ignores the extra flag.
(string-append "--use-compress-program="
- #+xz "/bin/xz --threads=0")
+ #+xz "/bin/xz --threads="
+ (number->string (parallel-job-count)))
;; avoid non-determinism in the archive
"--mtime=@0"
"--owner=root:0"
diff --git a/guix/scripts/pack.scm b/guix/scripts/pack.scm
index 488638adc..818de0f32 100644
--- a/guix/scripts/pack.scm
+++ b/guix/scripts/pack.scm
@@ -65,7 +65,9 @@
(compressor "lzip" ".lz"
#~(#+(file-append lzip "/bin/lzip") "-9"))
(compressor "xz" ".xz"
- #~(#+(file-append xz "/bin/xz") "-e -T0"))
+ #~(#+(file-append xz "/bin/xz")
+ (string-append "-e --threads="
+ (number->string (parallel-job-count)))))
(compressor "bzip2" ".bz2"
#~(#+(file-append bzip2 "/bin/bzip2") "-9"))
(compressor "none" "" #f)))
@@ -73,7 +75,8 @@
;; This one is only for use in this module, so don't put it in %compressors.
(define bootstrap-xz
(compressor "bootstrap-xz" ".xz"
- #~(#+(file-append %bootstrap-coreutils&co "/bin/xz") "-e -T0")))
+ #~(#+(file-append %bootstrap-coreutils&co "/bin/xz")
+ (string-append "-e --threads=" (number->string (parallel-job-count))))))
(define (lookup-compressor name)
"Return the compressor object called NAME. Error out if it could not be
diff --git a/guix/utils.scm b/guix/utils.scm
index 92e45de61..043034e99 100644
--- a/guix/utils.scm
+++ b/guix/utils.scm
@@ -172,7 +172,11 @@ a symbol such as 'xz."
(match compression
((or #f 'none) (values input '()))
('bzip2 (filtered-port `(,%bzip2 "-dc") input))
- ('xz (filtered-port `(,%xz "-dc" "-T0") input))
+ ('xz (filtered-port
+ `(,%xz "-dc" (string-append
+ "--threads="
+ (number->string (parallel-job-count))))
+ input))
('gzip (filtered-port `(,%gzip "-dc") input))
(else (error "unsupported compression scheme" compression))))
@@ -182,7 +186,11 @@ a symbol such as 'xz."
(match compression
((or #f 'none) (values input '()))
('bzip2 (filtered-port `(,%bzip2 "-c") input))
- ('xz (filtered-port `(,%xz "-c" "-T0") input))
+ ('xz (filtered-port
+ `(,%xz "-c" (string-append
+ "--threads="
+ (number->string (parallel-job-count))))
+ input))
('gzip (filtered-port `(,%gzip "-c") input))
(else (error "unsupported compression scheme" compression))))
@@ -239,7 +247,11 @@ program--e.g., '(\"--fast\")."
(match compression
((or #f 'none) (values output '()))
('bzip2 (filtered-output-port `(,%bzip2 "-c" ,@options) output))
- ('xz (filtered-output-port `(,%xz "-c" "-T0" ,@options) output))
+ ('xz (filtered-output-port
+ `(,%xz "-c" (string-append
+ "--threads="
+ (number->string (parallel-job-count)))
+ ,@options) output))
('gzip (filtered-output-port `(,%gzip "-c" ,@options) output))
(else (error "unsupported compression scheme" compression))))
--
2.16.3
L
L
Ludovic Courtès wrote on 4 Apr 2018 04:42
(name . Marius Bakke)(address . mbakke@fastmail.com)(address . 31015@debbugs.gnu.org)
877epnfbau.fsf@gnu.org
Hello!

Marius Bakke <mbakke@fastmail.com> skribis:

Toggle quote (7 lines)
> * guix/packages.scm (patch-and-repack): When invoking 'xz', tell it to use the
> amount of cores specified by e.g. 'guix build -c'.
> * guix/scripts/pack.scm (%compressors, bootstrap-xz): Likewise. While at it,
> use the long form '--threads=' for clarity.
> * guix/utils.scm (decompressed-port, compressed-port, compressed-output-port):
> Likewise.

Wasn’t there an issue that parallel xz is non-deterministic? I vaguely
remember something like this when we were producing with glibc tarballs
out of the Git repo.

If so, we should instead use -T1, even if that means getting worse
performance (I hear that lzip “works better” than xz, so we should give
it a try if we want both performance and good compression.)

Besides, the guix/packages.scm part would lead to a full rebuild, so I
think it’s a bit late for ‘core-updates’.

Thanks,
Ludo’.
L
L
Ludovic Courtès wrote on 13 May 2018 13:37
(name . Marius Bakke)(address . mbakke@fastmail.com)(address . 31015@debbugs.gnu.org)
87a7t3l2ah.fsf@gnu.org
Hello Marius,

ludo@gnu.org (Ludovic Courtès) skribis:

Toggle quote (13 lines)
> Marius Bakke <mbakke@fastmail.com> skribis:
>
>> * guix/packages.scm (patch-and-repack): When invoking 'xz', tell it to use the
>> amount of cores specified by e.g. 'guix build -c'.
>> * guix/scripts/pack.scm (%compressors, bootstrap-xz): Likewise. While at it,
>> use the long form '--threads=' for clarity.
>> * guix/utils.scm (decompressed-port, compressed-port, compressed-output-port):
>> Likewise.
>
> Wasn’t there an issue that parallel xz is non-deterministic? I vaguely
> remember something like this when we were producing with glibc tarballs
> out of the Git repo.

That was indeed the case, so I’m closing this issue. Let’s reopen it if
we have more info or new ideas! :-)

Ludo’.
L
L
Ludovic Courtès wrote on 13 May 2018 13:38
control message for bug #31015
(address . control@debbugs.gnu.org)
878t8nl2ab.fsf@gnu.org
tags 31015 wontfix
close 31015
?
Your comment

This issue is archived.

To comment on this conversation send an email to 31015@patchwise.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 31015
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch