Package | Source(s) | Maintainer(s) | |
---|---|---|---|
guix-patches | PTS Buildd Popcon |
Report forwarded
to guix-patches@gnu.org
:
bug#61363
; Package guix-patches
.
(Wed, 08 Feb 2023 07:49:02 GMT) (full text, mbox, link).
Acknowledgement sent
to Christopher Baines <mail@cbaines.net>
:
New bug report received and forwarded. Copy sent to guix-patches@gnu.org
.
(Wed, 08 Feb 2023 07:49:02 GMT) (full text, mbox, link).
Message #5 received at submit@debbugs.gnu.org (full text, mbox, reply):
[Message part 1 (text/plain, inline)]
These patches mean that grafts apply to the outputs of the guix derivation, rather than having grafts apply to the derivation itself. This moves grafting here to work like grafting for packages, where you can think of the grafted outputs as a transformed variant of the ungrafted outputs. I'm looking at this as it'll allow the Guix Data Service to compute the derivations without grafts, and for these to be useful for substitutes regardless of whether users are using grafts. Christopher Baines (2): packages: Add explicit-grafting record type to assist with grafts. self: Apply grafts to the outputs of the guix derivation. build-aux/build-self.scm | 4 ++- guix/packages.scm | 45 +++++++++++++++++++++++++++- guix/self.scm | 65 ++++++++++++++++++++++++++-------------- 3 files changed, 89 insertions(+), 25 deletions(-) -- 2.38.1
[signature.asc (application/pgp-signature, inline)]
Information forwarded
to ludo@gnu.org, guix-patches@gnu.org
:
bug#61363
; Package guix-patches
.
(Wed, 08 Feb 2023 07:55:01 GMT) (full text, mbox, link).
Message #8 received at 61363@debbugs.gnu.org (full text, mbox, reply):
Normally the grafting takes place when lowering packages, but this record assists with applying the same transformation to arbitrary objects/store items. I'm adding this to allow grafting the channel instance derivation outputs. * guix/packages.scm (explicit-grafting, explicit-grafting?, explicit-grafting-obj, explicit-grafting-grafts): New procedures. --- guix/packages.scm | 45 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 44 insertions(+), 1 deletion(-) diff --git a/guix/packages.scm b/guix/packages.scm index 041a872f9d..877bf89522 100644 --- a/guix/packages.scm +++ b/guix/packages.scm @@ -188,7 +188,12 @@ (define-module (guix packages) package-file package->derivation package->cross-derivation - origin->derivation)) + origin->derivation + + explicit-grafting + explicit-grafting? + explicit-grafting-obj + explicit-grafting-grafts)) ;; The 'source-module-closure' procedure ca. 1.2.0 did not recognize ;; #:re-export-and-replace: <https://issues.guix.gnu.org/52694>. @@ -2093,3 +2098,41 @@ (define package-source-derivation ;somewhat deprecated (add-to-store store (basename file) #t "sha256" file)) (_ (lower store source system)))))) + +;; Apply grafts explicitly +(define-immutable-record-type <explicit-grafting> + (%explicit-grafting obj packages) + explicit-grafting? + (obj explicit-grafting-obj) ;obj + (packages explicit-grafting-packages)) ;list of <package>s + +(define (write-explicit-grafting rec port) + (match rec + (($ <explicit-grafting> obj packages) + (format port "#<explicit-grafting ~s ~s>" obj packages)))) + +(define (explicit-grafting obj packages) + (%explicit-grafting obj packages)) + +(define-gexp-compiler (explicit-grafting-compiler (explicit-grafting <explicit-grafting>) + system target) + (match explicit-grafting + (($ <explicit-grafting> obj packages) + (mlet* %store-monad ((drv (without-grafting + (lower-object obj system #:target target))) + (grafts + (mapm %store-monad + (lambda (pkg) + (package-grafts* pkg system #:target target)) + packages))) + (match (delete-duplicates + (concatenate grafts)) + (() + (return drv)) + (grafts + (mlet %store-monad ((guile (package->derivation + (guile-for-grafts) + system #:graft? #f))) + (graft-derivation* drv grafts + #:system system + #:guile guile)))))))) -- 2.38.1
Information forwarded
to ludo@gnu.org, guix-patches@gnu.org
:
bug#61363
; Package guix-patches
.
(Wed, 08 Feb 2023 07:55:01 GMT) (full text, mbox, link).
Message #11 received at 61363@debbugs.gnu.org (full text, mbox, reply):
Rather than having grafts apply to the derivation itself. This moves grafting here to work like grafting for packages, where you can think of the grafted outputs as a transformed variant of the ungrafted outputs. I'm looking at this as it'll allow the Guix Data Service to compute the derivations without grafts, and for these to be useful for substitutes regardless of whether users are using grafts. * guix/self.scm (compiled-guix, guix-derivation): Add a #:graft? keyword argument, to control grafting when computing the guix derivation. * build-aux/build-self.scm (build-program): Call guix-derivation with #:graft? (%graft?) to make the compute-guix-derivation script use or not use grafts as desired. --- build-aux/build-self.scm | 4 ++- guix/self.scm | 65 ++++++++++++++++++++++++++-------------- 2 files changed, 45 insertions(+), 24 deletions(-) diff --git a/build-aux/build-self.scm b/build-aux/build-self.scm index 02822a2ee8..6d0037f20c 100644 --- a/build-aux/build-self.scm +++ b/build-aux/build-self.scm @@ -353,7 +353,9 @@ (define fake-git #:channel-metadata '#$channel-metadata #:pull-version - #$pull-version) + #$pull-version + #:graft? + #$(%graft?)) #:system system)) derivation-file-name)))))) #:module-path (list source)))) diff --git a/guix/self.scm b/guix/self.scm index 93019e1c64..c944dbe9ce 100644 --- a/guix/self.scm +++ b/guix/self.scm @@ -752,7 +752,8 @@ (define* (compiled-guix source #:key (gzip (specification->package "gzip")) (bzip2 (specification->package "bzip2")) (xz (specification->package "xz")) - (guix (specification->package "guix"))) + (guix (specification->package "guix")) + (graft? #t)) "Return a file-like object that contains a compiled Guix." (define guile-avahi (specification->package "guile-avahi")) @@ -802,6 +803,12 @@ (define dependencies guile-json guile-semver guile-ssh guile-sqlite3 guile-lib guile-zlib guile-lzlib guile-zstd))) + (define packages + (cons* gzip + bzip2 + xz + dependencies)) + (define *core-modules* (scheme-node "guix-core" '((guix) @@ -1022,28 +1029,35 @@ (define (built-modules node-subset) guile-lzma dependencies) #:guile guile-for-build - #:guile-version guile-version))) - (whole-package name modules dependencies - #:command command - #:guile guile-for-build - - ;; Include 'guix-daemon'. XXX: Here we inject an - ;; older snapshot of guix-daemon, but that's a good - ;; enough approximation for now. - #:daemon (module-ref (resolve-interface - '(gnu packages - package-management)) - 'guix-daemon) - - #:info (info-manual source) - #:miscellany (miscellaneous-files source) - #:guile-version guile-version))) + #:guile-version guile-version)) + (obj + (whole-package name modules dependencies + #:command command + #:guile guile-for-build + + ;; Include 'guix-daemon'. XXX: Here we inject + ;; an older snapshot of guix-daemon, but + ;; that's a good enough approximation for now. + #:daemon (module-ref (resolve-interface + '(gnu packages + package-management)) + 'guix-daemon) + + #:info (info-manual source) + #:miscellany (miscellaneous-files source) + #:guile-version guile-version))) + (if graft? + (explicit-grafting obj packages) + obj))) ((= 0 pull-version) ;; Legacy 'guix pull': return the .scm and .go files as one ;; directory. - (built-modules (lambda (node) - (list (node-source node) - (node-compiled node))))) + (let ((obj (built-modules (lambda (node) + (list (node-source node) + (node-compiled node)))))) + (if graft? + (explicit-grafting obj packages) + obj))) (else ;; Unsupported 'guix pull' version. #f))) @@ -1273,7 +1287,8 @@ (define (process-directory directory files output) (define* (guix-derivation source version #:optional (guile-version (effective-version)) #:key (pull-version 0) - channel-metadata) + channel-metadata + (graft? #t)) "Return, as a monadic value, the derivation to build the Guix from SOURCE for GUILE-VERSION. Use VERSION as the version string. Use CHANNEL-METADATA as the channel metadata sexp to include in (guix config). @@ -1310,7 +1325,11 @@ (define guile #:pull-version pull-version #:guile-version (if (>= pull-version 1) "3.0" guile-version) - #:guile-for-build guile))) + #:guile-for-build guile + #:graft? graft?))) (if guix - (lower-object guix) + (if graft? + (lower-object guix) + (without-grafting + (lower-object guix))) (return #f))))) -- 2.38.1
Information forwarded
to guix-patches@gnu.org
:
bug#61363
; Package guix-patches
.
(Fri, 10 Feb 2023 09:20:02 GMT) (full text, mbox, link).
Message #14 received at 61363@debbugs.gnu.org (full text, mbox, reply):
[Message part 1 (text/plain, inline)]
The data service comparison is now available for this, and while there are no differences in the packages, you can see some information. This is the channel instances before: https://data.qa.guix.gnu.org/revision/a582d863465990642d331bc05bf073f47fb80908/channel-instances and this is after: https://data.qa.guix.gnu.org/revision/9cfbb22b556d28a0af345824ae5b3e00eb3f4a15/channel-instances
[signature.asc (application/pgp-signature, inline)]
Information forwarded
to guix-patches@gnu.org
:
bug#61363
; Package guix-patches
.
(Wed, 22 Feb 2023 09:17:01 GMT) (full text, mbox, link).
Message #17 received at 61363@debbugs.gnu.org (full text, mbox, reply):
Hi, Christopher Baines <mail@cbaines.net> skribis: > Rather than having grafts apply to the derivation itself. This moves grafting > here to work like grafting for packages, where you can think of the grafted > outputs as a transformed variant of the ungrafted outputs. Hmm. > I'm looking at this as it'll allow the Guix Data Service to compute the > derivations without grafts, and for these to be useful for substitutes > regardless of whether users are using grafts. How does it help exactly? By disabling grafts in that context? > +++ b/guix/self.scm > @@ -752,7 +752,8 @@ (define* (compiled-guix source #:key > (gzip (specification->package "gzip")) > (bzip2 (specification->package "bzip2")) > (xz (specification->package "xz")) > - (guix (specification->package "guix"))) > + (guix (specification->package "guix")) > + (graft? #t)) > "Return a file-like object that contains a compiled Guix." > (define guile-avahi > (specification->package "guile-avahi")) > @@ -802,6 +803,12 @@ (define dependencies > guile-json guile-semver guile-ssh guile-sqlite3 > guile-lib guile-zlib guile-lzlib guile-zstd))) > > + (define packages > + (cons* gzip > + bzip2 > + xz > + dependencies)) > + [...] > + (let ((obj (built-modules (lambda (node) > + (list (node-source node) > + (node-compiled node)))))) > + (if graft? > + (explicit-grafting obj packages) > + obj))) There are two things I’m not comfortable with: 1. Having <explicit-grafting> in (guix packages); it looks misplaced. 2. More importantly, manually listing packages that might require grafting looks like a slippery slope (“oops! we’re not getting the GnuTLS graft for that CVE, too bad”). I designed and implemented several variants to try and delay grafting. One of them consisted in carrying graft information in gexps: https://git.savannah.gnu.org/cgit/guix.git/log?h=wip-gexp-grafts It’s kinda similar to what you’re proposing in that graft information is carried as far as possible. The main difference is that it’s automated. Hmm needs more thought. Ludo’.
Information forwarded
to guix-patches@gnu.org
:
bug#61363
; Package guix-patches
.
(Wed, 22 Feb 2023 15:08:01 GMT) (full text, mbox, link).
Message #20 received at 61363@debbugs.gnu.org (full text, mbox, reply):
[Message part 1 (text/plain, inline)]
Ludovic Courtès <ludo@gnu.org> writes: >> I'm looking at this as it'll allow the Guix Data Service to compute the >> derivations without grafts, and for these to be useful for substitutes >> regardless of whether users are using grafts. > > How does it help exactly? By disabling grafts in that context? So the Guix Data Service is somewhat built on the assumption that it's cheap to compute derivations, at least with grafts disabled. That's always been the case for packages, but for channel instance derivations it's not reliably the case, since currently disabling grafts doesn't apply to the whole process, and even if it did, the derivations you'd get out wouldn't be that useful (since you can't transform the outputs from those derivations to the outputs you'd get if using grafts). With these changes, it's always relatively cheap to compute the channel instance derivations, and it's always possible to compute the derivations for any system without needing to be able to perform builds for that system. You can see this in how the data service has processed Guix before and after these patches. This is the channel instances before: https://data.qa.guix.gnu.org/revision/a582d863465990642d331bc05bf073f47fb80908/channel-instances and this is after: https://data.qa.guix.gnu.org/revision/9cfbb22b556d28a0af345824ae5b3e00eb3f4a15/channel-instances Given data.qa.guix.gnu.org is running on an x86_64-linux system, that and i686-linux isn't generally a problem, but I'm guessing it only managed to compute the powerpc64le-linux and aarch64-linux derivations because it was able to substitute the necessary store items. For other system computing the derivations would have failed. I believe this change will also mean that the build farms will go from performing the grafting for these builds, to being able to not do so, in line with how builds for packages are handled. This isn't a big thing, but I think it makes sense. >> +++ b/guix/self.scm >> @@ -752,7 +752,8 @@ (define* (compiled-guix source #:key >> (gzip (specification->package "gzip")) >> (bzip2 (specification->package "bzip2")) >> (xz (specification->package "xz")) >> - (guix (specification->package "guix"))) >> + (guix (specification->package "guix")) >> + (graft? #t)) >> "Return a file-like object that contains a compiled Guix." >> (define guile-avahi >> (specification->package "guile-avahi")) >> @@ -802,6 +803,12 @@ (define dependencies >> guile-json guile-semver guile-ssh guile-sqlite3 >> guile-lib guile-zlib guile-lzlib guile-zstd))) >> >> + (define packages >> + (cons* gzip >> + bzip2 >> + xz >> + dependencies)) >> + > > [...] > >> + (let ((obj (built-modules (lambda (node) >> + (list (node-source node) >> + (node-compiled node)))))) >> + (if graft? >> + (explicit-grafting obj packages) >> + obj))) > > There are two things I’m not comfortable with: > > 1. Having <explicit-grafting> in (guix packages); it looks misplaced. I didn't put it there at first, but I think it makes sense since grafting is currently specific to packages, as is this additional code. > 2. More importantly, manually listing packages that might require > grafting looks like a slippery slope (“oops! we’re not getting the > GnuTLS graft for that CVE, too bad”). > > I designed and implemented several variants to try and delay grafting. > One of them consisted in carrying graft information in gexps: > > https://git.savannah.gnu.org/cgit/guix.git/log?h=wip-gexp-grafts > > It’s kinda similar to what you’re proposing in that graft information is > carried as far as possible. The main difference is that it’s automated. That's interesting, I think that making grafting not specific to packages, and something where the replacement is handled at a lower level (e.g. gexps) would be an alternative way to handle this. Given that this approach works though, maybe the explicit-grafting functionality could just sit and be used inside of (guix self). Given that module is very explicit about what packages are used, it should be possible to arrange the code so it's very hard to miss a package out, which should address your concern about manually listing packages (maybe specification->package can be tweaked so that it's possible to get all the packages, and that can be the list considered for grafting). I don't know of any other places where this approach would be useful, so while it would be nice to have a more general grafting mechanism eventually, I'd also like to be able to make these changes to channel instance grafts sooner rather than later.
[signature.asc (application/pgp-signature, inline)]
Information forwarded
to guix-patches@gnu.org
:
bug#61363
; Package guix-patches
.
(Tue, 28 Feb 2023 15:48:01 GMT) (full text, mbox, link).
Message #23 received at 61363@debbugs.gnu.org (full text, mbox, reply):
So this can be used in (guix self). * guix/packages.scm (guile-for-grafts): Export. --- guix/packages.scm | 1 + 1 file changed, 1 insertion(+) diff --git a/guix/packages.scm b/guix/packages.scm index 041a872f9d..2f81ad0284 100644 --- a/guix/packages.scm +++ b/guix/packages.scm @@ -183,6 +183,7 @@ (define-module (guix packages) package-closure default-guile + guile-for-grafts default-guile-derivation set-guile-for-build package-file -- 2.39.1
Information forwarded
to guix-patches@gnu.org
:
bug#61363
; Package guix-patches
.
(Tue, 28 Feb 2023 15:48:02 GMT) (full text, mbox, link).
Message #26 received at 61363@debbugs.gnu.org (full text, mbox, reply):
Both for consistency (always use specification->package as defined in this module) and so that all the packages that are used can be accessed (which comes in useful when applying grafts). * guix/self.scm (%packages): New variable. (specification->package): Use %packages. (locale-data, translate-texi-manuals, info-manual, guix-command, compiled-guix): Use specification->package. --- guix/self.scm | 97 +++++++++++++++++++++++++-------------------------- 1 file changed, 48 insertions(+), 49 deletions(-) diff --git a/guix/self.scm b/guix/self.scm index 93019e1c64..c5de3ab8fc 100644 --- a/guix/self.scm +++ b/guix/self.scm @@ -44,34 +44,42 @@ (define-module (guix self) ;;; Dependency handling. ;;; -(define specification->package +(define %packages + (let ((ref (lambda (module variable) + (delay + (module-ref (resolve-interface + `(gnu packages ,module)) + variable))))) + `(("guile" . ,(ref 'guile 'guile-3.0-latest)) + ("guile-avahi" . ,(ref 'guile-xyz 'guile-avahi)) + ("guile-json" . ,(ref 'guile 'guile-json-4)) + ("guile-ssh" . ,(ref 'ssh 'guile-ssh)) + ("guile-git" . ,(ref 'guile 'guile-git)) + ("guile-semver" . ,(ref 'guile-xyz 'guile-semver)) + ("guile-lib" . ,(ref 'guile-xyz 'guile-lib)) + ("guile-sqlite3" . ,(ref 'guile 'guile-sqlite3)) + ("guile-zlib" . ,(ref 'guile 'guile-zlib)) + ("guile-lzlib" . ,(ref 'guile 'guile-lzlib)) + ("guile-zstd" . ,(ref 'guile 'guile-zstd)) + ("guile-gcrypt" . ,(ref 'gnupg 'guile-gcrypt)) + ("guile-gnutls" . ,(ref 'tls 'guile-gnutls)) + ("guix-daemon" . ,(ref 'package-management 'guix-daemon)) + ("disarchive" . ,(ref 'backup 'disarchive)) + ("guile-lzma" . ,(ref 'guile 'guile-lzma)) + ("gzip" . ,(ref 'compression 'gzip)) + ("bzip2" . ,(ref 'compression 'bzip2)) + ("xz" . ,(ref 'compression 'xz)) + ("po4a" . ,(ref 'gettext 'po4a)) + ("gettext-minimal" . ,(ref 'gettext 'gettext-minimal)) + ("gcc-toolchain" . ,(ref 'commencement 'gcc-toolchain)) + ("glibc-utf8-locales" . ,(ref 'base 'glibc-utf8-locales)) + ("graphviz" . ,(ref 'graphviz 'graphviz)) + ("texinfo" . ,(ref 'texinfo 'texinfo))))) + +(define (specification->package name) ;; Use our own variant of that procedure because that of (gnu packages) ;; would traverse all the .scm files, which is wasteful. - (let ((ref (lambda (module variable) - (module-ref (resolve-interface module) variable)))) - (match-lambda - ("guile" (ref '(gnu packages guile) 'guile-3.0-latest)) - ("guile-avahi" (ref '(gnu packages guile-xyz) 'guile-avahi)) - ("guile-json" (ref '(gnu packages guile) 'guile-json-4)) - ("guile-ssh" (ref '(gnu packages ssh) 'guile-ssh)) - ("guile-git" (ref '(gnu packages guile) 'guile-git)) - ("guile-semver" (ref '(gnu packages guile-xyz) 'guile-semver)) - ("guile-lib" (ref '(gnu packages guile-xyz) 'guile-lib)) - ("guile-sqlite3" (ref '(gnu packages guile) 'guile-sqlite3)) - ("guile-zlib" (ref '(gnu packages guile) 'guile-zlib)) - ("guile-lzlib" (ref '(gnu packages guile) 'guile-lzlib)) - ("guile-zstd" (ref '(gnu packages guile) 'guile-zstd)) - ("guile-gcrypt" (ref '(gnu packages gnupg) 'guile-gcrypt)) - ("guile-gnutls" (ref '(gnu packages tls) 'guile-gnutls)) - ("disarchive" (ref '(gnu packages backup) 'disarchive)) - ("guile-lzma" (ref '(gnu packages guile) 'guile-lzma)) - ("gzip" (ref '(gnu packages compression) 'gzip)) - ("bzip2" (ref '(gnu packages compression) 'bzip2)) - ("xz" (ref '(gnu packages compression) 'xz)) - ("po4a" (ref '(gnu packages gettext) 'po4a)) - ("gettext" (ref '(gnu packages gettext) 'gettext-minimal)) - ("gcc-toolchain" (ref '(gnu packages commencement) 'gcc-toolchain)) - (_ #f)))) ;no such package + (and=> (assoc-ref %packages name) force)) ;;; @@ -240,9 +248,8 @@ (define* (locale-data source domain #:optional (directory domain)) "Return the locale data from 'po/DIRECTORY' in SOURCE, corresponding to DOMAIN, a gettext domain." - (define gettext - (module-ref (resolve-interface '(gnu packages gettext)) - 'gettext-minimal)) + (define gettext-minimal + (specification->package "gettext-minimal")) (define build (with-imported-modules '((guix build utils)) @@ -258,7 +265,7 @@ (define (compile language) (let ((gmo (string-append #$output "/" language "/LC_MESSAGES/" #$domain ".mo"))) (mkdir-p (dirname gmo)) - (invoke #+(file-append gettext "/bin/msgfmt") + (invoke #+(file-append gettext-minimal "/bin/msgfmt") "-c" "--statistics" "--verbose" "-o" gmo (string-append po-directory "/" language ".po")))) @@ -280,20 +287,19 @@ (define (translate-texi-manuals source) "Return the translated texinfo manuals built from SOURCE." (define po4a (specification->package "po4a")) - - (define gettext - (specification->package "gettext")) + + (define gettext-minimal + (specification->package "gettext-minimal")) (define glibc-utf8-locales - (module-ref (resolve-interface '(gnu packages base)) - 'glibc-utf8-locales)) + (specification->package "glibc-utf8-locales")) (define documentation (file-append* source "doc")) (define documentation-po (file-append* source "po/doc")) - + (define build (with-imported-modules '((guix build utils) (guix build po)) #~(begin @@ -365,7 +371,7 @@ (define parallel-jobs (setenv "GUIX_LOCPATH" #+(file-append glibc-utf8-locales "/lib/locale")) - (setenv "PATH" #+(file-append gettext "/bin")) + (setenv "PATH" #+(file-append gettext-minimal "/bin")) (setenv "LC_ALL" "en_US.UTF-8") (setlocale LC_ALL "en_US.UTF-8") @@ -394,16 +400,13 @@ (define parallel-jobs (define (info-manual source) "Return the Info manual built from SOURCE." (define texinfo - (module-ref (resolve-interface '(gnu packages texinfo)) - 'texinfo)) + (specification->package "texinfo")) (define graphviz - (module-ref (resolve-interface '(gnu packages graphviz)) - 'graphviz)) + (specification->package "graphviz")) (define glibc-utf8-locales - (module-ref (resolve-interface '(gnu packages base)) - 'glibc-utf8-locales)) + (specification->package "glibc-utf8-locales")) (define documentation (file-append* source "doc")) @@ -586,8 +589,7 @@ (define* (guix-command modules "Return the 'guix' command such that it adds MODULES and DEPENDENCIES in its load path." (define glibc-utf8-locales - (module-ref (resolve-interface '(gnu packages base)) - 'glibc-utf8-locales)) + (specification->package "glibc-utf8-locales")) (define module-directory ;; To minimize the number of 'stat' calls needed to locate a module, @@ -1030,10 +1032,7 @@ (define (built-modules node-subset) ;; Include 'guix-daemon'. XXX: Here we inject an ;; older snapshot of guix-daemon, but that's a good ;; enough approximation for now. - #:daemon (module-ref (resolve-interface - '(gnu packages - package-management)) - 'guix-daemon) + #:daemon (specification->package "guix-daemon") #:info (info-manual source) #:miscellany (miscellaneous-files source) -- 2.39.1
Information forwarded
to guix-patches@gnu.org
:
bug#61363
; Package guix-patches
.
(Tue, 28 Feb 2023 15:48:02 GMT) (full text, mbox, link).
Message #29 received at 61363@debbugs.gnu.org (full text, mbox, reply):
Rather than having grafts apply to the derivation itself. This moves grafting here to work like grafting for packages, where you can think of the grafted outputs as a transformed variant of the ungrafted outputs. I'm looking at this as it'll allow the Guix Data Service to compute the derivations without grafts, and for these to be useful for substitutes regardless of whether users are using grafts. * guix/self.scm (compiled-guix, guix-derivation): Add a #:graft? keyword argument, to control grafting when computing the guix derivation. * build-aux/build-self.scm (build-program): Call guix-derivation with #:graft? (%graft?) to make the compute-guix-derivation script use or not use grafts as desired. --- build-aux/build-self.scm | 4 +- guix/self.scm | 101 +++++++++++++++++++++++++++++++-------- 2 files changed, 84 insertions(+), 21 deletions(-) diff --git a/build-aux/build-self.scm b/build-aux/build-self.scm index 02822a2ee8..6d0037f20c 100644 --- a/build-aux/build-self.scm +++ b/build-aux/build-self.scm @@ -353,7 +353,9 @@ (define fake-git #:channel-metadata '#$channel-metadata #:pull-version - #$pull-version) + #$pull-version + #:graft? + #$(%graft?)) #:system system)) derivation-file-name)))))) #:module-path (list source)))) diff --git a/guix/self.scm b/guix/self.scm index c5de3ab8fc..8842275ff8 100644 --- a/guix/self.scm +++ b/guix/self.scm @@ -22,6 +22,7 @@ (define-module (guix self) #:use-module (guix i18n) #:use-module (guix modules) #:use-module (guix gexp) + #:use-module (guix grafts) #:use-module (guix store) #:use-module (guix monads) #:use-module (guix discovery) @@ -32,6 +33,7 @@ (define-module (guix self) #:use-module ((guix build utils) #:select (find-files)) #:use-module (srfi srfi-1) #:use-module (srfi srfi-9) + #:use-module (srfi srfi-9 gnu) #:use-module (srfi srfi-35) #:use-module (ice-9 match) #:export (make-config.scm @@ -244,6 +246,50 @@ (define* (file-append* item file #:key (recursive? #t)) ;; which isn't great. (file-append item "/" file)))) +(define graft-derivation* + (store-lift graft-derivation)) + +(define package-grafts* + (store-lift package-grafts)) + +;; Apply grafts explicitly +(define-immutable-record-type <explicit-grafting> + (%explicit-grafting obj packages) + explicit-grafting? + (obj explicit-grafting-obj) ;obj + (packages explicit-grafting-packages)) ;list of <package>s + +(define (write-explicit-grafting rec port) + (match rec + (($ <explicit-grafting> obj packages) + (format port "#<explicit-grafting ~s ~s>" obj packages)))) + +(define (explicit-grafting obj packages) + (%explicit-grafting obj packages)) + +(define-gexp-compiler (explicit-grafting-compiler (explicit-grafting <explicit-grafting>) + system target) + (match explicit-grafting + (($ <explicit-grafting> obj packages) + (mlet* %store-monad ((drv (without-grafting + (lower-object obj system #:target target))) + (grafts + (mapm %store-monad + (lambda (pkg) + (package-grafts* pkg system #:target target)) + packages))) + (match (delete-duplicates + (concatenate grafts)) + (() + (return drv)) + (grafts + (mlet %store-monad ((guile (package->derivation + (guile-for-grafts) + system #:graft? #f))) + (graft-derivation* drv grafts + #:system system + #:guile guile)))))))) + (define* (locale-data source domain #:optional (directory domain)) "Return the locale data from 'po/DIRECTORY' in SOURCE, corresponding to @@ -754,7 +800,8 @@ (define* (compiled-guix source #:key (gzip (specification->package "gzip")) (bzip2 (specification->package "bzip2")) (xz (specification->package "xz")) - (guix (specification->package "guix"))) + (guix (specification->package "guix")) + (graft? #t)) "Return a file-like object that contains a compiled Guix." (define guile-avahi (specification->package "guile-avahi")) @@ -1024,25 +1071,34 @@ (define (built-modules node-subset) guile-lzma dependencies) #:guile guile-for-build - #:guile-version guile-version))) - (whole-package name modules dependencies - #:command command - #:guile guile-for-build - - ;; Include 'guix-daemon'. XXX: Here we inject an - ;; older snapshot of guix-daemon, but that's a good - ;; enough approximation for now. - #:daemon (specification->package "guix-daemon") - - #:info (info-manual source) - #:miscellany (miscellaneous-files source) - #:guile-version guile-version))) + #:guile-version guile-version)) + (obj + (whole-package name modules dependencies + #:command command + #:guile guile-for-build + + ;; Include 'guix-daemon'. XXX: Here we inject + ;; an older snapshot of guix-daemon, but + ;; that's a good enough approximation for now. + #:daemon (specification->package "guix-daemon") + + #:info (info-manual source) + #:miscellany (miscellaneous-files source) + #:guile-version guile-version))) + (if graft? + (explicit-grafting obj + (map (compose force cdr) %packages)) + obj))) ((= 0 pull-version) ;; Legacy 'guix pull': return the .scm and .go files as one ;; directory. - (built-modules (lambda (node) - (list (node-source node) - (node-compiled node))))) + (let ((obj (built-modules (lambda (node) + (list (node-source node) + (node-compiled node)))))) + (if graft? + (explicit-grafting obj + (map (compose force cdr) %packages)) + obj))) (else ;; Unsupported 'guix pull' version. #f))) @@ -1272,7 +1328,8 @@ (define (process-directory directory files output) (define* (guix-derivation source version #:optional (guile-version (effective-version)) #:key (pull-version 0) - channel-metadata) + channel-metadata + (graft? #t)) "Return, as a monadic value, the derivation to build the Guix from SOURCE for GUILE-VERSION. Use VERSION as the version string. Use CHANNEL-METADATA as the channel metadata sexp to include in (guix config). @@ -1309,7 +1366,11 @@ (define guile #:pull-version pull-version #:guile-version (if (>= pull-version 1) "3.0" guile-version) - #:guile-for-build guile))) + #:guile-for-build guile + #:graft? graft?))) (if guix - (lower-object guix) + (if graft? + (lower-object guix) + (without-grafting + (lower-object guix))) (return #f))))) -- 2.39.1
Information forwarded
to guix-patches@gnu.org
:
bug#61363
; Package guix-patches
.
(Tue, 28 Feb 2023 15:51:01 GMT) (full text, mbox, link).
Message #32 received at 61363@debbugs.gnu.org (full text, mbox, reply):
[Message part 1 (text/plain, inline)]
Christopher Baines <mail@cbaines.net> writes: > Ludovic Courtès <ludo@gnu.org> writes: > >> 2. More importantly, manually listing packages that might require >> grafting looks like a slippery slope (“oops! we’re not getting the >> GnuTLS graft for that CVE, too bad”). >> >> I designed and implemented several variants to try and delay grafting. >> One of them consisted in carrying graft information in gexps: >> >> https://git.savannah.gnu.org/cgit/guix.git/log?h=wip-gexp-grafts >> >> It’s kinda similar to what you’re proposing in that graft information is >> carried as far as possible. The main difference is that it’s automated. > > That's interesting, I think that making grafting not specific to > packages, and something where the replacement is handled at a lower > level (e.g. gexps) would be an alternative way to handle this. > > Given that this approach works though, maybe the explicit-grafting > functionality could just sit and be used inside of (guix self). Given > that module is very explicit about what packages are used, it should be > possible to arrange the code so it's very hard to miss a package out, > which should address your concern about manually listing packages (maybe > specification->package can be tweaked so that it's possible to get all > the packages, and that can be the list considered for grafting). > > I don't know of any other places where this approach would be useful, so > while it would be nice to have a more general grafting mechanism > eventually, I'd also like to be able to make these changes to channel > instance grafts sooner rather than later. I've sent a v2 series which changes along the above lines. The explicit grafting stuff just sits in (guix self), and (guix self) more rigeriously uses it's own definition of specification->package, which should provide some protection against missing packages out. Obviously it's not quite as rigerous as moving the grafting functionality in to gexps, but hopefully it's rigerous enough for now.
[signature.asc (application/pgp-signature, inline)]
Information forwarded
to guix-patches@gnu.org
:
bug#61363
; Package guix-patches
.
(Mon, 17 Apr 2023 15:00:02 GMT) (full text, mbox, link).
Message #35 received at 61363@debbugs.gnu.org (full text, mbox, reply):
Rather than having grafts apply to the derivation itself. This moves grafting here to work like grafting for packages, where you can think of the grafted outputs as a transformed variant of the ungrafted outputs. I'm looking at this as it'll allow the Guix Data Service to compute the derivations without grafts, and for these to be useful for substitutes regardless of whether users are using grafts. * guix/self.scm (compiled-guix, guix-derivation): Add a #:graft? keyword argument, to control grafting when computing the guix derivation. * build-aux/build-self.scm (build-program): Call guix-derivation with #:graft? (%graft?) to make the compute-guix-derivation script use or not use grafts as desired. Signed-off-by: Christopher Baines <mail@cbaines.net> --- build-aux/build-self.scm | 4 +- guix/self.scm | 101 +++++++++++++++++++++++++++++++-------- 2 files changed, 84 insertions(+), 21 deletions(-) diff --git a/build-aux/build-self.scm b/build-aux/build-self.scm index 02822a2ee8..6d0037f20c 100644 --- a/build-aux/build-self.scm +++ b/build-aux/build-self.scm @@ -353,7 +353,9 @@ (define fake-git #:channel-metadata '#$channel-metadata #:pull-version - #$pull-version) + #$pull-version + #:graft? + #$(%graft?)) #:system system)) derivation-file-name)))))) #:module-path (list source)))) diff --git a/guix/self.scm b/guix/self.scm index 74c953bd50..bbc0beaca8 100644 --- a/guix/self.scm +++ b/guix/self.scm @@ -21,6 +21,7 @@ (define-module (guix self) #:use-module (guix config) #:use-module (guix modules) #:use-module (guix gexp) + #:use-module (guix grafts) #:use-module (guix store) #:use-module (guix monads) #:use-module (guix discovery) @@ -31,6 +32,7 @@ (define-module (guix self) #:use-module ((guix build utils) #:select (find-files)) #:use-module (srfi srfi-1) #:use-module (srfi srfi-9) + #:use-module (srfi srfi-9 gnu) #:use-module (srfi srfi-35) #:use-module (ice-9 match) #:export (make-config.scm @@ -243,6 +245,50 @@ (define* (file-append* item file #:key (recursive? #t)) ;; which isn't great. (file-append item "/" file)))) +(define graft-derivation* + (store-lift graft-derivation)) + +(define package-grafts* + (store-lift package-grafts)) + +;; Apply grafts explicitly +(define-immutable-record-type <explicit-grafting> + (%explicit-grafting obj packages) + explicit-grafting? + (obj explicit-grafting-obj) ;obj + (packages explicit-grafting-packages)) ;list of <package>s + +(define (write-explicit-grafting rec port) + (match rec + (($ <explicit-grafting> obj packages) + (format port "#<explicit-grafting ~s ~s>" obj packages)))) + +(define (explicit-grafting obj packages) + (%explicit-grafting obj packages)) + +(define-gexp-compiler (explicit-grafting-compiler (explicit-grafting <explicit-grafting>) + system target) + (match explicit-grafting + (($ <explicit-grafting> obj packages) + (mlet* %store-monad ((drv (without-grafting + (lower-object obj system #:target target))) + (grafts + (mapm %store-monad + (lambda (pkg) + (package-grafts* pkg system #:target target)) + packages))) + (match (delete-duplicates + (concatenate grafts)) + (() + (return drv)) + (grafts + (mlet %store-monad ((guile (package->derivation + (guile-for-grafts) + system #:graft? #f))) + (graft-derivation* drv grafts + #:system system + #:guile guile)))))))) + (define* (locale-data source domain #:optional (directory domain)) "Return the locale data from 'po/DIRECTORY' in SOURCE, corresponding to @@ -753,7 +799,8 @@ (define* (compiled-guix source #:key (gzip (specification->package "gzip")) (bzip2 (specification->package "bzip2")) (xz (specification->package "xz")) - (guix (specification->package "guix"))) + (guix (specification->package "guix")) + (graft? #t)) "Return a file-like object that contains a compiled Guix." (define guile-avahi (specification->package "guile-avahi")) @@ -1023,25 +1070,34 @@ (define (built-modules node-subset) guile-lzma dependencies) #:guile guile-for-build - #:guile-version guile-version))) - (whole-package name modules dependencies - #:command command - #:guile guile-for-build - - ;; Include 'guix-daemon'. XXX: Here we inject an - ;; older snapshot of guix-daemon, but that's a good - ;; enough approximation for now. - #:daemon (specification->package "guix-daemon") - - #:info (info-manual source) - #:miscellany (miscellaneous-files source) - #:guile-version guile-version))) + #:guile-version guile-version)) + (obj + (whole-package name modules dependencies + #:command command + #:guile guile-for-build + + ;; Include 'guix-daemon'. XXX: Here we inject + ;; an older snapshot of guix-daemon, but + ;; that's a good enough approximation for now. + #:daemon (specification->package "guix-daemon") + + #:info (info-manual source) + #:miscellany (miscellaneous-files source) + #:guile-version guile-version))) + (if graft? + (explicit-grafting obj + (map (compose force cdr) %packages)) + obj))) ((= 0 pull-version) ;; Legacy 'guix pull': return the .scm and .go files as one ;; directory. - (built-modules (lambda (node) - (list (node-source node) - (node-compiled node))))) + (let ((obj (built-modules (lambda (node) + (list (node-source node) + (node-compiled node)))))) + (if graft? + (explicit-grafting obj + (map (compose force cdr) %packages)) + obj))) (else ;; Unsupported 'guix pull' version. #f))) @@ -1271,7 +1327,8 @@ (define (process-directory directory files output) (define* (guix-derivation source version #:optional (guile-version (effective-version)) #:key (pull-version 0) - channel-metadata) + channel-metadata + (graft? #t)) "Return, as a monadic value, the derivation to build the Guix from SOURCE for GUILE-VERSION. Use VERSION as the version string. Use CHANNEL-METADATA as the channel metadata sexp to include in (guix config). @@ -1308,7 +1365,11 @@ (define guile #:pull-version pull-version #:guile-version (if (>= pull-version 1) "3.0" guile-version) - #:guile-for-build guile))) + #:guile-for-build guile + #:graft? graft?))) (if guix - (lower-object guix) + (if graft? + (lower-object guix) + (without-grafting + (lower-object guix))) (return #f))))) -- 2.39.1
Information forwarded
to guix-patches@gnu.org
:
bug#61363
; Package guix-patches
.
(Mon, 17 Apr 2023 15:08:01 GMT) (full text, mbox, link).
Message #38 received at 61363@debbugs.gnu.org (full text, mbox, reply):
[Message part 1 (text/plain, inline)]
Christopher Baines <mail@cbaines.net> writes: > [[PGP Signed Part:Undecided]] > > Christopher Baines <mail@cbaines.net> writes: > >> Ludovic Courtès <ludo@gnu.org> writes: >> >>> 2. More importantly, manually listing packages that might require >>> grafting looks like a slippery slope (“oops! we’re not getting the >>> GnuTLS graft for that CVE, too bad”). >>> >>> I designed and implemented several variants to try and delay grafting. >>> One of them consisted in carrying graft information in gexps: >>> >>> https://git.savannah.gnu.org/cgit/guix.git/log?h=wip-gexp-grafts >>> >>> It’s kinda similar to what you’re proposing in that graft information is >>> carried as far as possible. The main difference is that it’s automated. >> >> That's interesting, I think that making grafting not specific to >> packages, and something where the replacement is handled at a lower >> level (e.g. gexps) would be an alternative way to handle this. >> >> Given that this approach works though, maybe the explicit-grafting >> functionality could just sit and be used inside of (guix self). Given >> that module is very explicit about what packages are used, it should be >> possible to arrange the code so it's very hard to miss a package out, >> which should address your concern about manually listing packages (maybe >> specification->package can be tweaked so that it's possible to get all >> the packages, and that can be the list considered for grafting). >> >> I don't know of any other places where this approach would be useful, so >> while it would be nice to have a more general grafting mechanism >> eventually, I'd also like to be able to make these changes to channel >> instance grafts sooner rather than later. > > I've sent a v2 series which changes along the above lines. The explicit > grafting stuff just sits in (guix self), and (guix self) more > rigeriously uses it's own definition of specification->package, which > should provide some protection against missing packages out. Obviously > it's not quite as rigerous as moving the grafting functionality in to > gexps, but hopefully it's rigerous enough for now. This has stalled a bit, but it would be good to try and get things merged. I've gone ahead and pushed the first two patches in the series I last sent, these just make minor changes to prepare for the functional change here. I've also resent that patch as as v3.
[signature.asc (application/pgp-signature, inline)]
Information forwarded
to guix-patches@gnu.org
:
bug#61363
; Package guix-patches
.
(Tue, 16 May 2023 15:07:02 GMT) (full text, mbox, link).
Message #41 received at 61363@debbugs.gnu.org (full text, mbox, reply):
Hi Chris, I am late to the party and probably do not well understand all that part. Just a quick comment in the same direction as Ludo. On Mon, 17 Apr 2023 at 15:59, Christopher Baines <mail@cbaines.net> wrote: > diff --git a/guix/self.scm b/guix/self.scm > index 74c953bd50..bbc0beaca8 100644 > --- a/guix/self.scm > +++ b/guix/self.scm [...] > + (if graft? > + (explicit-grafting obj > + (map (compose force cdr) %packages)) > + obj))) [...] > + (if graft? > + (explicit-grafting obj > + (map (compose force cdr) %packages)) > + obj))) It means that the grafts are only applied to %packages, right? Other said, defined by: --8<---------------cut here---------------start------------->8--- (define %packages (let ((ref (lambda (module variable) (delay (module-ref (resolve-interface `(gnu packages ,module)) variable))))) `(("guile" . ,(ref 'guile 'guile-3.0-latest)) ("guile-avahi" . ,(ref 'guile-xyz 'guile-avahi)) ("guile-json" . ,(ref 'guile 'guile-json-4)) ("guile-ssh" . ,(ref 'ssh 'guile-ssh)) ("guile-git" . ,(ref 'guile 'guile-git)) ("guile-semver" . ,(ref 'guile-xyz 'guile-semver)) ("guile-lib" . ,(ref 'guile-xyz 'guile-lib)) ("guile-sqlite3" . ,(ref 'guile 'guile-sqlite3)) ("guile-zlib" . ,(ref 'guile 'guile-zlib)) ("guile-lzlib" . ,(ref 'guile 'guile-lzlib)) ("guile-zstd" . ,(ref 'guile 'guile-zstd)) ("guile-gcrypt" . ,(ref 'gnupg 'guile-gcrypt)) ("guile-gnutls" . ,(ref 'tls 'guile-gnutls)) ("guix-daemon" . ,(ref 'package-management 'guix-daemon)) ("disarchive" . ,(ref 'backup 'disarchive)) ("guile-lzma" . ,(ref 'guile 'guile-lzma)) ("gzip" . ,(ref 'compression 'gzip)) ("bzip2" . ,(ref 'compression 'bzip2)) ("xz" . ,(ref 'compression 'xz)) ("po4a" . ,(ref 'gettext 'po4a)) ("gettext-minimal" . ,(ref 'gettext 'gettext-minimal)) ("gcc-toolchain" . ,(ref 'commencement 'gcc-toolchain)) ("glibc-utf8-locales" . ,(ref 'base 'glibc-utf8-locales)) ("graphviz" . ,(ref 'graphviz 'graphviz)) ("texinfo" . ,(ref 'texinfo 'texinfo))))) --8<---------------cut here---------------end--------------->8--- tweaked by e5c33837cbee98d460d9ae09b463501de6f15d97. And there is a slippery slope: the manual addition. These had been added with e5c33837cbee98d460d9ae09b463501de6f15d97: + ("glibc-utf8-locales" . ,(ref 'base 'glibc-utf8-locales)) + ("graphviz" . ,(ref 'graphviz 'graphviz)) + ("guix-daemon" . ,(ref 'package-management 'guix-daemon)) + ("texinfo" . ,(ref 'texinfo 'texinfo))))) Other said, what does it happen if we forget to manually update this list? Cheers, simon
Information forwarded
to guix-patches@gnu.org
:
bug#61363
; Package guix-patches
.
(Sat, 03 Jun 2023 11:45:01 GMT) (full text, mbox, link).
Message #44 received at 61363@debbugs.gnu.org (full text, mbox, reply):
[Message part 1 (text/plain, inline)]
Simon Tournier <zimon.toutoune@gmail.com> writes: > Hi Chris, > > I am late to the party and probably do not well understand all that > part. Just a quick comment in the same direction as Ludo. > > On Mon, 17 Apr 2023 at 15:59, Christopher Baines <mail@cbaines.net> wrote: > >> diff --git a/guix/self.scm b/guix/self.scm >> index 74c953bd50..bbc0beaca8 100644 >> --- a/guix/self.scm >> +++ b/guix/self.scm > > [...] > >> + (if graft? >> + (explicit-grafting obj >> + (map (compose force cdr) %packages)) >> + obj))) > > [...] > >> + (if graft? >> + (explicit-grafting obj >> + (map (compose force cdr) %packages)) >> + obj))) > > It means that the grafts are only applied to %packages, right? > > Other said, defined by: > > (define %packages > (let ((ref (lambda (module variable) > (delay > (module-ref (resolve-interface > `(gnu packages ,module)) > variable))))) > `(("guile" . ,(ref 'guile 'guile-3.0-latest)) > ("guile-avahi" . ,(ref 'guile-xyz 'guile-avahi)) > ("guile-json" . ,(ref 'guile 'guile-json-4)) > ("guile-ssh" . ,(ref 'ssh 'guile-ssh)) > ("guile-git" . ,(ref 'guile 'guile-git)) > ("guile-semver" . ,(ref 'guile-xyz 'guile-semver)) > ("guile-lib" . ,(ref 'guile-xyz 'guile-lib)) > ("guile-sqlite3" . ,(ref 'guile 'guile-sqlite3)) > ("guile-zlib" . ,(ref 'guile 'guile-zlib)) > ("guile-lzlib" . ,(ref 'guile 'guile-lzlib)) > ("guile-zstd" . ,(ref 'guile 'guile-zstd)) > ("guile-gcrypt" . ,(ref 'gnupg 'guile-gcrypt)) > ("guile-gnutls" . ,(ref 'tls 'guile-gnutls)) > ("guix-daemon" . ,(ref 'package-management 'guix-daemon)) > ("disarchive" . ,(ref 'backup 'disarchive)) > ("guile-lzma" . ,(ref 'guile 'guile-lzma)) > ("gzip" . ,(ref 'compression 'gzip)) > ("bzip2" . ,(ref 'compression 'bzip2)) > ("xz" . ,(ref 'compression 'xz)) > ("po4a" . ,(ref 'gettext 'po4a)) > ("gettext-minimal" . ,(ref 'gettext 'gettext-minimal)) > ("gcc-toolchain" . ,(ref 'commencement 'gcc-toolchain)) > ("glibc-utf8-locales" . ,(ref 'base 'glibc-utf8-locales)) > ("graphviz" . ,(ref 'graphviz 'graphviz)) > ("texinfo" . ,(ref 'texinfo 'texinfo))))) > > tweaked by e5c33837cbee98d460d9ae09b463501de6f15d97. And there is a > slippery slope: the manual addition. These had been added with > e5c33837cbee98d460d9ae09b463501de6f15d97: > > + ("glibc-utf8-locales" . ,(ref 'base 'glibc-utf8-locales)) > + ("graphviz" . ,(ref 'graphviz 'graphviz)) > + ("guix-daemon" . ,(ref 'package-management 'guix-daemon)) > + ("texinfo" . ,(ref 'texinfo 'texinfo))))) > > Other said, what does it happen if we forget to manually update this > list? Well, specification->package in (guix self) won't work for the missing packages. It's possible to use packages outside of this list, but that doesn't happen currently.
[signature.asc (application/pgp-signature, inline)]
Added tag(s) moreinfo.
Request was from Christopher Baines <mail@cbaines.net>
to control@debbugs.gnu.org
.
(Sat, 03 Jun 2023 11:45:02 GMT) (full text, mbox, link).
Send a report that this bug log contains spam.
Debbugs is free software and licensed under the terms of the GNU Public License version 2. The current version can be obtained from https://bugs.debian.org/debbugs-source/.
Copyright © 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson, 2005-2017 Don Armstrong, and many other contributors.