Report forwarded
to bug-guix@gnu.org: bug#22138; Package guix.
(Thu, 10 Dec 2015 09:37:02 GMT) (full text, mbox, link).
Acknowledgement sent
to ludo@gnu.org (Ludovic Courtès):
New bug report received and forwarded. Copy sent to bug-guix@gnu.org.
(Thu, 10 Dec 2015 09:37:02 GMT) (full text, mbox, link).
Subject: Search paths of dependencies are not honored
Date: Thu, 10 Dec 2015 10:36:08 +0100
As of 0.9.0+, only the search paths of packages explicitly in the
profile are shown by ‘--search-paths’ and similar. This is a problem
for libraries that have associated environment variables.
For instance, if one installs an application linked against OpenSSL,
they will not know about ‘SSL_CERT_DIR’ and ‘SSL_CERT_FILE’. Similarly
for GStreamer and ‘GST_PLUGIN_PATH’, libc and ‘GUIX_LOCPATH’, and so on.
Fixing it seems tricky. We could pass the profile builder the closure’s
graph (via #:references-graph) and the the set of search paths of all
the direct and indirect dependencies of the profile. The builder would
then need to somehow determine the subset of these search paths that is
actually useful, and use it to build the search path spec in the
manifest as well as etc/profile.
Ludo’.
Severity set to 'important' from 'normal'
Request was from ludo@gnu.org (Ludovic Courtès)
to control@debbugs.gnu.org.
(Sun, 26 Mar 2017 13:19:02 GMT) (full text, mbox, link).
Information forwarded
to bug-guix@gnu.org: bug#22138; Package guix.
(Thu, 01 Aug 2019 20:13:01 GMT) (full text, mbox, link).
Subject: Re: bug#22138: Search paths of dependencies are not honored
Date: Mon, 05 Aug 2019 12:23:55 -0400
Hi Julien,
Julien Lepiller <julien@lepiller.eu> writes:
> Hi, I've been looking at our current code and would like to propose the
> attached patch for that issue.
>
> From cfd2c229087166ab4cc0a9e2bdb72c8b393bcdd5 Mon Sep 17 00:00:00 2001
> From: Julien Lepiller <julien@lepiller.eu>
> Date: Thu, 1 Aug 2019 22:09:38 +0200
> Subject: [PATCH] guix: Recursively honor search paths of dependencies.
>
> * guix/packages.scm (all-transitive-inputs)
> (package-all-transitive-inputs)
> (package-all-transitive-native-search-paths): New procedures.
> * guix/profiles.scm (package->manifest-entry): Use
> package-all-transitive-native-search-paths to generate manifest search
> paths.
As I recall this kind of solution has been proposed in the past and
rejected. It's a reasonable suggestion, but I personally think that it
goes too far, because it would include a great many packages whose code
is nowhere to be found in the resulting profile. For example, it would
include documentation generators used to build man pages, and the
compilers that were used to build those documentation generators, etc,
all the way back to the early bootstrap binaries.
Having said this, I agree that there is a longstanding problem in Guix
with search-paths not including enough packages in its calculation.
We've known about this problem for a long time, but as far as I know we
have not yet found a satisfactory solution.
Our current hacky workaround for problems like this has been to set
certain environment variables unconditionally in /etc/profile. For
example, you'll see that MANPATH, INFOPATH, XDG_DATA_DIRS,
XDG_CONFIG_DIRS, XCURSOR_PATH, DICPATH, and GST_PLUGIN_PATH are all set
there. See 'operating-system-etc-service' in (gnu system) for the
relevant code.
At the very least, I think we should wait for input from Ludovic before
applying anything along these lines.
Anyway, thanks for looking into it and making the proposal.
Best,
Mark
Information forwarded
to bug-guix@gnu.org: bug#22138; Package guix.
(Mon, 05 Aug 2019 16:32:02 GMT) (full text, mbox, link).
Subject: Re: bug#22138: Search paths of dependencies are not honored
Date: Mon, 05 Aug 2019 18:31:15 +0200
Le 5 août 2019 18:23:55 GMT+02:00, Mark H Weaver <mhw@netris.org> a écrit :
>Hi Julien,
>
>Julien Lepiller <julien@lepiller.eu> writes:
>
>> Hi, I've been looking at our current code and would like to propose
>the
>> attached patch for that issue.
>>
>> From cfd2c229087166ab4cc0a9e2bdb72c8b393bcdd5 Mon Sep 17 00:00:00
>2001
>> From: Julien Lepiller <julien@lepiller.eu>
>> Date: Thu, 1 Aug 2019 22:09:38 +0200
>> Subject: [PATCH] guix: Recursively honor search paths of
>dependencies.
>>
>> * guix/packages.scm (all-transitive-inputs)
>> (package-all-transitive-inputs)
>> (package-all-transitive-native-search-paths): New procedures.
>> * guix/profiles.scm (package->manifest-entry): Use
>> package-all-transitive-native-search-paths to generate manifest
>search
>> paths.
>
>As I recall this kind of solution has been proposed in the past and
>rejected. It's a reasonable suggestion, but I personally think that it
>goes too far, because it would include a great many packages whose code
>is nowhere to be found in the resulting profile. For example, it would
>include documentation generators used to build man pages, and the
>compilers that were used to build those documentation generators, etc,
>all the way back to the early bootstrap binaries.
>
>Having said this, I agree that there is a longstanding problem in Guix
>with search-paths not including enough packages in its calculation.
>We've known about this problem for a long time, but as far as I know we
>have not yet found a satisfactory solution.
>
>Our current hacky workaround for problems like this has been to set
>certain environment variables unconditionally in /etc/profile. For
>example, you'll see that MANPATH, INFOPATH, XDG_DATA_DIRS,
>XDG_CONFIG_DIRS, XCURSOR_PATH, DICPATH, and GST_PLUGIN_PATH are all set
>there. See 'operating-system-etc-service' in (gnu system) for the
>relevant code.
>
>At the very least, I think we should wait for input from Ludovic before
>applying anything along these lines.
>
>Anyway, thanks for looking into it and making the proposal.
>
> Best,
> Mark
Tgis patch doesn't add any dependency to any package, since it doesn't record any more stone paths than without. It indeed defines more variables than needed, but isn't that ok?
Information forwarded
to bug-guix@gnu.org: bug#22138; Package guix.
(Fri, 23 Aug 2019 14:43:02 GMT) (full text, mbox, link).
Hi Mark,
Mark H Weaver <mhw@netris.org> skribis:
> Julien Lepiller <julien@lepiller.eu> writes:
>
>> Hi, I've been looking at our current code and would like to propose the
>> attached patch for that issue.
>>
>> From cfd2c229087166ab4cc0a9e2bdb72c8b393bcdd5 Mon Sep 17 00:00:00 2001
>> From: Julien Lepiller <julien@lepiller.eu>
>> Date: Thu, 1 Aug 2019 22:09:38 +0200
>> Subject: [PATCH] guix: Recursively honor search paths of dependencies.
>>
>> * guix/packages.scm (all-transitive-inputs)
>> (package-all-transitive-inputs)
>> (package-all-transitive-native-search-paths): New procedures.
>> * guix/profiles.scm (package->manifest-entry): Use
>> package-all-transitive-native-search-paths to generate manifest search
>> paths.
>
> As I recall this kind of solution has been proposed in the past and
> rejected. It's a reasonable suggestion, but I personally think that it
> goes too far, because it would include a great many packages whose code
> is nowhere to be found in the resulting profile.
I agree.
We need to take into account run-time dependencies (references), not
build-time dependencies, and that’s what makes things more difficult.
The attached procedure computes the search paths of a package based on
its run-time dependencies. ‘profile-derivation’ could use this to
compute its ‘etc/profile’ file and its ‘manifest’ file. Here’s what it
gives for ‘w3m’:
--8<---------------cut here---------------start------------->8---
(("BASH_LOADABLES_PATH"
("lib/bash")
":"
directory
#f)
("GUIX_LOCPATH" ("lib/locale") ":" directory #f)
("SSL_CERT_DIR"
("etc/ssl/certs")
#f
directory
#f)
("SSL_CERT_FILE"
("etc/ssl/certs/ca-certificates.crt")
#f
regular
#f)
("TERMINFO_DIRS"
("share/terminfo")
":"
directory
#f)
("XDG_DATA_DIRS" ("share") ":" directory #f)
("GIO_EXTRA_MODULES"
("lib/gio/modules")
":"
directory
#f)
("PERL5LIB"
("lib/perl5/site_perl")
":"
directory
#f))
--8<---------------cut here---------------end--------------->8---
We get GUIX_LOCPATH and SSL_CERT_DIR, which is exactly what we want.
However, the other search paths that we get look less desirable:
intuitively, it seems that we may not need them, and in the case of
XDG_DATA_DIRS, setting it may be detrimental.
For ‘icecat’ this gives additional things like PYTHONPATH,
GUILE_LOAD_PATH, XML_CATALOG_FILES, and so on.
In practice that’s probably OK though: if you only have ‘icecat’ and
other GUIs in your profile, Guix won’t suggest setting GUILE_LOAD_PATH
or PYTHONPATH.
So I think we can come up with a solution based on the patch below, but
we’ll have to test it on Guix System and on foreign distros to see if
the extra search paths that it brings are warranted.
Thoughts?
Ludo’.
Subject: Re: bug#22138: Search paths of dependencies are not honored
Date: Sat, 24 Aug 2019 15:52:46 +0200
Ludovic Courtès <ludo@gnu.org> skribis:
> The attached procedure computes the search paths of a package based on
> its run-time dependencies. ‘profile-derivation’ could use this to
> compute its ‘etc/profile’ file and its ‘manifest’ file.
One difficulty with this approach is that search paths are currently a
“host-side” feature: there’s a ‘search-paths’ field in <manifest-entry>,
which is populated before the profile is actually built.
What I suggest above would mean turning search paths into a “build-side”
feature, which has some implications.
Needs more thought…
Ludo’.
Information forwarded
to bug-guix@gnu.org: bug#22138; Package guix.
(Tue, 26 Nov 2019 12:02:02 GMT) (full text, mbox, link).
Subject: Search paths of dependencies are not honored
Date: Tue, 26 Nov 2019 13:00:55 +0100
See https://issues.guix.gnu.org/issue/22138
I'm not sure what the implications would be. Your results look good and I think it's close enough to what we need. Now, to get SSL_CERT_DIR to be defined, we simply need a ssl cert dir to be present in the profile, which is what users expect.
I would argue that we could simply add every possible variable to the profile, it wouldn't make much of a difference, but if you feel using only runtime dependencies is better, I won't argue against it. It's fine with me, even if it seems more complicated than it needs to be.
As a user, I have control over the content of my profile. When I have some file in it, I expect it to be "functional" in the sense that it contributes something to the functionalities of the profile. Not setting a variable when we could makes the file non-functional, since it can't be used directly. Currently, one needs to add another non-functional package to the profile (like opennsl), which doesn't contribute anything except metadata.
If I have a package in my profile that adds an unrelated file (eg. a python library), and it is not desirable, I think we should rather fix the package to add a separate output for that library, rather than restricting the set of environment variables.
Information forwarded
to bug-guix@gnu.org: bug#22138; Package guix.
(Fri, 14 Jan 2022 19:50:02 GMT) (full text, mbox, link).
(first section)
> As of 0.9.0+, only the search paths of packages explicitly in the
> profile are shown by ‘--search-paths’ and similar. This is a problem
> for libraries that have associated environment variables.
>
> For instance, if one installs an application linked against OpenSSL,
> they will not know about ‘SSL_CERT_DIR’ and ‘SSL_CERT_FILE’. Similarly
> for GStreamer and ‘GST_PLUGIN_PATH’, libc and ‘GUIX_LOCPATH’, and so on.
> [...]
> [... stuff about references-graph ...]
I want to note that it is possible to use a library in a program without
needing the environment variables. E.g., web servers can use openssl,
but often only for listening so they don't need a certificate bundle.
A game using GStreamer for sounds knows in advance which plugins it will
need, so the package definition would be using 'wrap-program' or the like
to hardcode the set of plugins.
Many (almost all?) packages keep a reference to the "lib" output of "gcc",
but almost no packages need '{,CROSS_}C_INCLUDE_PATH', '{,CROSS_}CPLUS_INCLUDE_PATH' or
'{,CROSS_}LIBRARY_PATH'.
That said, adding these superfluos environment variables is probably
harmless in practice, but it kind of breaks some abstractions:
If I use a package as input that uses a search path and I fix the search path
using 'wrap-program' or the like, then I would expect things to be as if there
wasn't a search path in the first place, but IIUC the proposed patch looking
at references would break things.
Also, the patch seems to only affect profiles and not build environments,
creating another distance between them, which seems suboptimal.
For these reasons, I don't think search paths of dependencies should be honoured.
However, there are things to improve:
(second section)
I believe a better option would be to document search paths better in the manual
(there's almost nothing currently about search paths) and make search paths easier
to use.
E.g., we could move ‘generic’ search paths used by several packages (e.g.
INFOPATH, TERMINFO_DIRS, XDG_DATA_DIRS, GST_PLUGIN_PATH, SSL_CERT_DIR
and SSL_CERT_FILE --- why is SSL_CERT_DIR missing from icecat and guile
anyway?) (counter-example: MINETEST_MOD_PATH, because it is used by only
one package) into a common module next to each other, which some small comments
[...]
;; FWIW I think I suggested something like this elsewhere, in the context
;; of module import loops.
;; for info readers (info-reader, emacs, ...)
(define-public %info-path ...)
;; for programs interacting with terminal emulators,
;; e.g. almost anything using 'ncurses'
(define-public %terminfo-dirs-path ...)
;; for packages using TLS, e.g. web browsers (icecat, ungoogled-chromium, ...),
;; e-mail clients (evolution, icedove ...), most software 'openssl'
;;
;; Not all packages using TLS respect one of these two variables.
;;
;; (TODO check if these examples actually respect SSL_CERT_DIR/PATH)
(define-public %x509-cert-dir-path ...)
(define-public %x509-cert-file-path ...)
[...]
and mention this module in the documentation of 'search-paths',
invite people to look at the source code of the module for examples
people maybe even documenting these search paths in the manual.
Maybe also insert a reminder about search paths in the 'reviewing' section?
I think this (keeping standard search paths together, with some explanation,
with some example packages using these search paths ...) would help discoverability
a lot, reducing the risk that a package is defined without appropriate search paths.
Such a common module would also reduce redunancy (a single definition of %foo-path
instead of sprinkling it among separate packages) and reduce the chance of module import
loops (maybe we can move the python search path there?), although the latter could
be solved with thunking (at some cost).
It's also possible to do both this (manual documentation, common search path module)
and the patch using the reference graph.
This ignores GUIX_LOCPATH, but that's used by almost anything, so it could be
special-cased I think?
Greetings,
Maxime.
Hi,
On second thought, honouring search paths of dependencies
(filtered by looking at the references?) would solve much more problems
than it could create -- a theoretical ‘problem’ is that more
environment variables than strictly needed might be defined (see e.g.
the wrap-program example) but it doesn't seem that it would create
problems in practice.
There's a limitation to keep in mind though: static libraries.
Suppose we have a static library variant curl/static tht links against
openssl:static. Then due to the static linking, curl won't keep a
reference to openssl:static (*), so there won't be a SSL_CERT_DIR/FILE
search path.
(*) Actually, that might wrong, libcrypto.a keeps a reference to
/gnu/store/plr00nij45964cyy7sfcg5rcsi8hks0h-openssl-1.1.1l.
For some examples in the wild where this kind of propagation(*) of
search paths is useful: all packages using ncurses (TERMINFO_DIRS),
openssl (SSL_CERT_DIR/SSL_CERT_FILE), all packages using glibc
(GUIX_LOCPATH) -- basically all software(!).
(*) it's not propagated-inputs but the concept doesn't seem completely
unlike.
As such, I agree with the concept (notwithstanding the previous mail I
sent), with the caveat that I haven't investigated the implementation
or tested it.
Greetings,
Maxime
Debbugs is free software and licensed under the terms of the
GNU Public License version 2. The current version can be
obtained from https://bugs.debian.org/debbugs-source/.