Provide --only-substitutes flag to "guix package --upgrade"

  • Open
  • quality assurance status badge
Details
6 participants
  • Alex Sassmannshausen
  • Christopher Allan Webber
  • Jan Nieuwenhuizen
  • Konrad Hinsen
  • Ludovic Courtès
  • Ricardo Wurmus
Owner
unassigned
Submitted by
Christopher Allan Webber
Severity
important

Debbugs page

C
C
Christopher Allan Webber wrote on 22 Apr 2017 09:03
(address . bug-guix@gnu.org)
874lxg77l4.fsf@dustycloud.org
Sometimes I do an upgrade and I want to get the latest security updates,
but I also am not really available to burn through a lot of cpu time,
(especially on my x200).

I'd love it if thre were a flag so that I could specify "only bother to
upgrade the packages where we only upgrade if a substitute is already
available.

Note that I looked at the source to see if this would be an easy thing
to add; I figured that this would be handled in roughly the same place
as --keep-going or --fallback, but it looks to me like those are
actually just passed over to the build daemon, so I'm not so sure how
easy it would be to patch this in while we're still using Nix's C
daemon? I'm not sure.
L
L
Ludovic Courtès wrote on 22 Apr 2017 16:03
(name . Christopher Allan Webber)(address . cwebber@dustycloud.org)(address . 26608@debbugs.gnu.org)
87d1c4gi3r.fsf@gnu.org
Christopher Allan Webber <cwebber@dustycloud.org> skribis:

Toggle quote (8 lines)
> Sometimes I do an upgrade and I want to get the latest security updates,
> but I also am not really available to burn through a lot of cpu time,
> (especially on my x200).
>
> I'd love it if thre were a flag so that I could specify "only bother to
> upgrade the packages where we only upgrade if a substitute is already
> available.

+1

Toggle quote (7 lines)
> Note that I looked at the source to see if this would be an easy thing
> to add; I figured that this would be handled in roughly the same place
> as --keep-going or --fallback, but it looks to me like those are
> actually just passed over to the build daemon, so I'm not so sure how
> easy it would be to patch this in while we're still using Nix's C
> daemon? I'm not sure.

Clients could check, among the packages that are to be installed, which
ones are substitutable (with the ‘substitute-paths’ RPC or similar) and
filter out those that are not. No C++ involved.

Cheers,
Ludo’.
L
L
Ludovic Courtès wrote on 30 May 2017 08:13
control message for bug #26608
(address . control@debbugs.gnu.org)
87a85u4buf.fsf@gnu.org
severity 26608 important
L
L
Ludovic Courtès wrote on 30 Aug 2018 15:02
Re: bug#22629: “Stable” branch
(name . Alex Sassmannshausen)(address . alex@pompo.co)
878t4nqzqv.fsf@gnu.org
Hi Alex,

(Cc’ing https://bugs.gnu.org/32022 and https://bugs.gnu.org/26608 ,
which are related.)

Alex Sassmannshausen <alex@pompo.co> skribis:

Toggle quote (5 lines)
> I don't know if this is what Konrad desires, but from my perspective, a
> desirable part of the definition of stable would be a that the build
> farms have produced a set of binaries/substitutes for a given Guix
> revision that is "good enough".

I just had a bright idea (yes!): this can be addressed by writing
something like this in ~/.config/guix/channels.scm:

(map latest-commit-with-substitutes-available
%default-channels)

The hypothetical ‘latest-commit-with-substitutes-available’ would use
(git) and (guix ci) to find the latest commit for which substitutes of
interest are available, and would return:

(channel
;; …
(commit "cabbag3")) ;the ideal commit

This has to be done with great care to prevent a downgrade attack and to
make sure the user doesn’t miss out on security updates, but maybe we
could provide a procedure that makes reasonable choices.

Food for thought…

Ludo’.
K
K
Konrad Hinsen wrote on 31 Aug 2018 02:39
m1h8jaq3h0.fsf@fastmail.net
Hi Ludo,

Toggle quote (10 lines)
> I just had a bright idea (yes!): this can be addressed by writing
> something like this in ~/.config/guix/channels.scm:
>
> (map latest-commit-with-substitutes-available
> %default-channels)
>
> The hypothetical ‘latest-commit-with-substitutes-available’ would use
> (git) and (guix ci) to find the latest commit for which substitutes of
> interest are available, and would return:

I really like that idea, but it's a pity to limit it to channels.
Two scenarii I'd like to see covered are:

1) Find the latest commit with all substitutes required by a given
manifest.

2) Find the latest commit with all substitutes required for updating a
given profile.

This is in fact only one problem with two user interfaces.

Konrad.
L
L
Ludovic Courtès wrote on 31 Aug 2018 02:58
(name . Konrad Hinsen)(address . konrad.hinsen@fastmail.net)
874lfarh6w.fsf@gnu.org
Hi Konrad,

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

Toggle quote (12 lines)
>> I just had a bright idea (yes!): this can be addressed by writing
>> something like this in ~/.config/guix/channels.scm:
>>
>> (map latest-commit-with-substitutes-available
>> %default-channels)
>>
>> The hypothetical ‘latest-commit-with-substitutes-available’ would use
>> (git) and (guix ci) to find the latest commit for which substitutes of
>> interest are available, and would return:
>
> I really like that idea, but it's a pity to limit it to channels.

What do you mean by “limit it to channels”? ‘%default-channels’ is an
alias for the official Guix channel (IOW, Guix itself.)

Toggle quote (10 lines)
> Two scenarii I'd like to see covered are:
>
> 1) Find the latest commit with all substitutes required by a given
> manifest.
>
> 2) Find the latest commit with all substitutes required for updating a
> given profile.
>
> This is in fact only one problem with two user interfaces.

Yes, we could do that, and even maybe more sophisticated things (e.g.,
looking at the commit log to determine whether security fixes are
available, and adjusting the strategy accordingly.)

What I find interesting is that we can provide the tools to support such
policies, and then users can choose or implement the policy they want
directly in ~/.config/guix/channels.scm.

Ludo’.
K
K
Konrad Hinsen wrote on 31 Aug 2018 03:33
(name . Ludovic Courtès)(address . ludo@gnu.org)
m1k1o6akr4.fsf@fastmail.net
Hi Ludo,

Toggle quote (3 lines)
> What do you mean by “limit it to channels”? ‘%default-channels’ is an
> alias for the official Guix channel (IOW, Guix itself.)

Fine, but I rarely care about all of Guix, or all of any other channel.
I care about the small subset of packages that I actually use.

Better yet, with a per-manifest/profile approach, I could put my most
critical packages in a special profile and get updates for them more
quickly, while still working only with substitutes.

BTW, just out of curiosity: for how many commits in Guix history all
packages could be built successfully? Is that the rule of the exception?

Toggle quote (4 lines)
> Yes, we could do that, and even maybe more sophisticated things (e.g.,
> looking at the commit log to determine whether security fixes are
> available, and adjusting the strategy accordingly.)

Nice!

Toggle quote (4 lines)
> What I find interesting is that we can provide the tools to support such
> policies, and then users can choose or implement the policy they want
> directly in ~/.config/guix/channels.scm.

I agree, it's nice to give people the tools they need to implement their
own policy.

Konrad.
J
J
Jan Nieuwenhuizen wrote on 31 Aug 2018 04:24
(name . Ludovic Courtès)(address . ludo@gnu.org)
87mut2ok2e.fsf@gnu.org
Ludovic Courtès writes:

Toggle quote (6 lines)
> I just had a bright idea (yes!): this can be addressed by writing
> something like this in ~/.config/guix/channels.scm:
>
> (map latest-commit-with-substitutes-available
> %default-channels)

This is a nice idea and it makes me remember that it would be useful to
provide a way to avoid installing something that is cricitally broken,
like Debian's apt-listbugs package/facility

janneke

--
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com| Avatar® http://AvatarAcademy.com
R
R
Ricardo Wurmus wrote on 31 Aug 2018 04:45
Re: bug#32022: bug#22629: “Stable” branch
(name . Ludovic Courtès)(address . ludo@gnu.org)
87lg8m2206.fsf@elephly.net
Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (18 lines)
> I just had a bright idea (yes!): this can be addressed by writing
> something like this in ~/.config/guix/channels.scm:
>
> (map latest-commit-with-substitutes-available
> %default-channels)
>
> The hypothetical ‘latest-commit-with-substitutes-available’ would use
> (git) and (guix ci) to find the latest commit for which substitutes of
> interest are available, and would return:
>
> (channel
> ;; …
> (commit "cabbag3")) ;the ideal commit
>
> This has to be done with great care to prevent a downgrade attack and to
> make sure the user doesn’t miss out on security updates, but maybe we
> could provide a procedure that makes reasonable choices.

This is a great idea. Any kind of fetch policy could be implemented
with this, including one that considers the contents of a manifest.

This is another of these instances where having a general purpose
programming language underpinning it all really pays off.

--
Ricardo
L
L
Ludovic Courtès wrote on 31 Aug 2018 06:01
Re: bug#22629: “Stable” branch
(name . Konrad Hinsen)(address . konrad.hinsen@fastmail.net)
87k1o6pu4f.fsf@gnu.org
Hello,

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

Toggle quote (10 lines)
>> What do you mean by “limit it to channels”? ‘%default-channels’ is an
>> alias for the official Guix channel (IOW, Guix itself.)
>
> Fine, but I rarely care about all of Guix, or all of any other channel.
> I care about the small subset of packages that I actually use.
>
> Better yet, with a per-manifest/profile approach, I could put my most
> critical packages in a special profile and get updates for them more
> quickly, while still working only with substitutes.

Sure! The hypothetical procedure I gave can perform arbitrary checks;
it could be passed a manifest.

Toggle quote (3 lines)
> BTW, just out of curiosity: for how many commits in Guix history all
> packages could be built successfully? Is that the rule of the exception?

We never have 100% of successful builds. Of course we do our best to
keep the failure rate low, but sometimes there are unpopular packages
that remain broken for some time, or there are packages for which we
forgot to exclude some systems via ‘supported-systems’, and of course
there’s unintended breakage.

Ludo’.
A
A
Alex Sassmannshausen wrote on 3 Sep 2018 07:10
(name . Ludovic Courtès)(address . ludo@gnu.org)
87r2iau0wz.fsf@pompo.co
Hi Ludo,

Ludovic Courtès writes:

Toggle quote (26 lines)
> Hi Alex,
>
> (Cc’ing <https://bugs.gnu.org/32022> and <https://bugs.gnu.org/26608>,
> which are related.)
>
> Alex Sassmannshausen <alex@pompo.co> skribis:
>
>> I don't know if this is what Konrad desires, but from my perspective, a
>> desirable part of the definition of stable would be a that the build
>> farms have produced a set of binaries/substitutes for a given Guix
>> revision that is "good enough".
>
> I just had a bright idea (yes!): this can be addressed by writing
> something like this in ~/.config/guix/channels.scm:
>
> (map latest-commit-with-substitutes-available
> %default-channels)
>
> The hypothetical ‘latest-commit-with-substitutes-available’ would use
> (git) and (guix ci) to find the latest commit for which substitutes of
> interest are available, and would return:
>
> (channel
> ;; …
> (commit "cabbag3")) ;the ideal commit

This sounds incredibly interesting — and it is testament once again to
the power of Guix that this kind of solution could be feasible!

Thinking this through in my head somewhat, I had the following thoughts:
- This procedure is invoked client side, where the channel is defined
- That means the git searching is done client side, on every invocation
of guix (I guess this might be cacheable?)
- So the downside vis-a-vis a maintained "stable branch" would be a
price in performance as experienced by the end user
- The upside of course would be automatic curation of a stable branch
that saves a ton of volunteer effort and work

I have no idea what the performance cost would be. I guess you would
use "guix weather" to turn the set of requested packages into a manifest
which can then be checked with it.

So the cost would be one of the following scenarios:
Option a)
- fetch set of packages in a given commit
- query guix weather for 100% substitutes
- iterate until a match
- then perform the appropriate guix pull

Option b)
- perform a guix pull to the latest commit
- query guix weather for 100% substitutes
- until success, step back one step at a time through guix pull

(because of the cost of guix pull this seems unfeasible)

Option c)
Implement some form of substitute cache set querying on build farms, as
part of guix weather, so the 100% match is done on the build farm
instead of the client.

Dunno. There may be some things that already exist in Guix land that
I'm missing.

It's a super exciting approach for sure.

Toggle quote (4 lines)
> This has to be done with great care to prevent a downgrade attack and to
> make sure the user doesn’t miss out on security updates, but maybe we
> could provide a procedure that makes reasonable choices.

Right — so at the very least it would have to prevent us going "back in
time" from the guix pull commit we are currently at.

The question of security updates is tricky at the moment already — I
would hazard a guess that many people bail out of upgrading when they
can't get substitutes for their entire profile / system right now, which
means they are not getting security upgrades for package (a) when a
substitute for (b) fails.

Thanks for your thoughts — super intriguing!

Alex
L
L
Ludovic Courtès wrote on 3 Sep 2018 12:52
(name . Alex Sassmannshausen)(address . alex@pompo.co)
87zhwywe8v.fsf@gnu.org
Hi Alex,

Alex Sassmannshausen <alex@pompo.co> skribis:

Toggle quote (2 lines)
> Ludovic Courtès writes:

[...]

Toggle quote (17 lines)
>> I just had a bright idea (yes!): this can be addressed by writing
>> something like this in ~/.config/guix/channels.scm:
>>
>> (map latest-commit-with-substitutes-available
>> %default-channels)
>>
>> The hypothetical ‘latest-commit-with-substitutes-available’ would use
>> (git) and (guix ci) to find the latest commit for which substitutes of
>> interest are available, and would return:
>>
>> (channel
>> ;; …
>> (commit "cabbag3")) ;the ideal commit
>
> This sounds incredibly interesting — and it is testament once again to
> the power of Guix that this kind of solution could be feasible!

Just to be clear: I don’t think this would be a substitute for a
“stable” branch; rather, I view as a way to have user-defined policies
such as “pull up to the latest commit for which there’s a substitute for
IceCat.”

Toggle quote (5 lines)
> Thinking this through in my head somewhat, I had the following thoughts:
> - This procedure is invoked client side, where the channel is defined
> - That means the git searching is done client side, on every invocation
> of guix (I guess this might be cacheable?)

On every invocation of ‘guix pull’ only.

Toggle quote (4 lines)
> I have no idea what the performance cost would be. I guess you would
> use "guix weather" to turn the set of requested packages into a manifest
> which can then be checked with it.

As I imagine it, the cost would be a few HTTP queries to the Cuirass
API. I should try to come up with an example to better explain what I
had in mind!

Toggle quote (6 lines)
> The question of security updates is tricky at the moment already — I
> would hazard a guess that many people bail out of upgrading when they
> can't get substitutes for their entire profile / system right now, which
> means they are not getting security upgrades for package (a) when a
> substitute for (b) fails.

That’s probably true, and I agree it’s problematic.

What I typically do is “guix pull && guix package -n -u”. Then I look
at things that would be built; if, say, LibreOffice is among them, I
wait for a little while and try again later, until I can get enough
substitutes. That usually works okay, but it fails if it turns out that
one of the dependencies fails to build: substitutes never become
available in that case.

Ludo’.
L
L
Ludovic Courtès wrote on 3 Sep 2018 13:27
(name . Alex Sassmannshausen)(address . alex@pompo.co)
87h8j6wclu.fsf@gnu.org
Alex Sassmannshausen <alex@pompo.co> skribis:

Toggle quote (2 lines)
> Ludovic Courtès writes:

[...]

Toggle quote (14 lines)
>> I just had a bright idea (yes!): this can be addressed by writing
>> something like this in ~/.config/guix/channels.scm:
>>
>> (map latest-commit-with-substitutes-available
>> %default-channels)
>>
>> The hypothetical ‘latest-commit-with-substitutes-available’ would use
>> (git) and (guix ci) to find the latest commit for which substitutes of
>> interest are available, and would return:
>>
>> (channel
>> ;; …
>> (commit "cabbag3")) ;the ideal commit

The code below is an illustration of that. If you install it as
~/.config/guix/channels.scm, ‘guix pull’ will pull the latest commit
that was fully built on berlin.guixsd.org (see
substitutes for Guix itself should be available, unless ‘guix publish’
hasn’t “baked” them yet.

It takes two GETs and ~1s to do that here.

Ludo’.
(use-modules (guix http-client) (json) (srfi srfi-1) (ice-9 match)) (define (latest-evaluations jobset) "Return the latest evaluations of JOBSET." (filter (lambda (json) (string=? (hash-ref json "specification") jobset)) (json->scm (http-fetch "https://berlin.guixsd.org/api/evaluations?nr=30")))) (define (evaluation-complete? number) "Return true if evaluation NUMBER completed and all its builds were successful." (let ((builds (json->scm (http-fetch (string-append "https://berlin.guixsd.org/api/latestbuilds?nr=30&evaluation=" (number->string number)))))) (every (lambda (build) ;; Zero means build success. (= (hash-ref build "buildstatus") 0)) builds))) (define (latest-commit-successfully-built) "Return the latest commit for which substitutes are (potentially) available." (let* ((evaluations (latest-evaluations "guix-modular-master")) (candidates (filter-map (lambda (json) (match (hash-ref json "checkouts") ((checkout) (cons (hash-ref json "id") (hash-ref checkout "commit"))) (_ #f))) evaluations))) (any (match-lambda ((evaluation . commit) (and (evaluation-complete? evaluation) commit))) candidates))) ;; Pull the latest commit fully built on berlin.guixsd.org. ;; WARNING: This could downgrade your system! (list (channel (name 'guix) (url "https://git.savannah.gnu.org/git/guix.git") (commit (pk 'commit (latest-commit-successfully-built)))))
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 26608@patchwise.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 26608
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch