GNU bug report logs

#28659 Content-addressed mirror is not used upon invalid hash

PackageSource(s)Maintainer(s)
guix PTS Buildd Popcon
Full log

Message #75 received at 28659@debbugs.gnu.org (full text, mbox, reply):

Received: (at 28659) by debbugs.gnu.org; 28 Nov 2017 13:31:21 +0000
From debbugs-submit-bounces@debbugs.gnu.org Tue Nov 28 08:31:20 2017
Received: from localhost ([127.0.0.1]:33613 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces@debbugs.gnu.org>)
	id 1eJfyk-0004mR-E9
	for submit@debbugs.gnu.org; Tue, 28 Nov 2017 08:31:20 -0500
Received: from [141.255.128.1] (port=53718 helo=hera.aquilenet.fr)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@gnu.org>) id 1eJfyh-0004mH-H7
 for 28659@debbugs.gnu.org; Tue, 28 Nov 2017 08:31:13 -0500
Received: from localhost (localhost [127.0.0.1])
 by hera.aquilenet.fr (Postfix) with ESMTP id 927FDEF69;
 Tue, 28 Nov 2017 14:31:12 +0100 (CET)
X-Virus-Scanned: Debian amavisd-new at aquilenet.fr
Received: from hera.aquilenet.fr ([127.0.0.1])
 by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024)
 with ESMTP id DkvuRG0-0kAZ; Tue, 28 Nov 2017 14:31:08 +0100 (CET)
Received: from ribbon (unknown [193.50.110.215])
 by hera.aquilenet.fr (Postfix) with ESMTPSA id A8BACE9D7;
 Tue, 28 Nov 2017 14:31:02 +0100 (CET)
From: ludo@gnu.org (Ludovic Courtès)
To: Leo Famulari <leo@famulari.name>
Subject: Re: bug#28659: v0.13: guix pull fails;
 libgit2-0.26.0 and 0.25.1 content hashes fail
References: <877ewf18d4.fsf@gnu.org> <87o9ppoabw.fsf@gnu.org>
 <20171002182208.GB10773@jasmine.lan> <878tgt721q.fsf@gnu.org>
 <20171020211700.GA32355@jasmine.lan>
Date: Tue, 28 Nov 2017 14:30:59 +0100
In-Reply-To: <20171020211700.GA32355@jasmine.lan> (Leo Famulari's message of
 "Fri, 20 Oct 2017 17:17:00 -0400")
Message-ID: <87d1421qek.fsf@gnu.org>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.3 (gnu/linux)
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
X-Spam-Score: 2.2 (++)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 the administrator of that system for details.
 
 Content preview:  Leo Famulari <leo@famulari.name> skribis: > On Mon, Oct 02,
    2017 at 10:00:33PM +0200, Ludovic Courtès wrote: >> Right. Jan suggested
    checking the content-addressed mirrors *before* >> the real upstream address.
    That would address the problem of upstream >> sources modified in-place,
   but at the cost of privacy/self-sufficiency >> as you note. (Though it’s
    not really making “privacy” any worse in this >> case: it’s gnu.org
    vs. github.com.) > > Yeah, I don't personally think there is a privacy issue
    with fetching > sources from our mirrors at gnu.org, or other domains we
   control. > >> Perhaps we should make content-addressed mirrors configurable
    in a way >> that’s orthogonal to derivations, something similar in spirit
    to >> --substitute-urls? The difficulty is that content-addressed mirrors
    are >> not just URLs; see (guix download). >> >> Thoughts? > > I do think
    we should make it so that users don't suffer from unreliable > upstream sources
    when we know the sources are available on our servers > (or the Nix mirror),
    even with --no-substitutes. [...] 
 
 Content analysis details:   (2.2 points, 10.0 required)
 
  pts rule name              description
 ---- ---------------------- --------------------------------------------------
  1.0 SPF_SOFTFAIL           SPF: sender does not match SPF record (softfail)
  0.0 SPF_HELO_FAIL          SPF: HELO does not match SPF record (fail)
 [SPF failed: Please see http://www.openspf.org/Why?s=helo;id=hera.aquilenet.fr;ip=141.255.128.1;r=debbugs.gnu.org]
  1.3 RDNS_NONE              Delivered to internal network by a host with no rDNS
X-Debbugs-Envelope-To: 28659
Cc: 28659@debbugs.gnu.org, Jan Nieuwenhuizen <janneke@gnu.org>
X-BeenThere: debbugs-submit@debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request@debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit@debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request@debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request@debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces@debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces@debbugs.gnu.org>
X-Spam-Score: 2.2 (++)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
 has NOT identified this incoming email as spam.  The original
 message has been attached to this so you can view it or label
 similar future email.  If you have any questions, see
 the administrator of that system for details.
 
 Content preview:  Leo Famulari <leo@famulari.name> skribis: > On Mon, Oct 02,
    2017 at 10:00:33PM +0200, Ludovic Courtès wrote: >> Right. Jan suggested
    checking the content-addressed mirrors *before* >> the real upstream address.
    That would address the problem of upstream >> sources modified in-place,
   but at the cost of privacy/self-sufficiency >> as you note. (Though it’s
    not really making “privacy” any worse in this >> case: it’s gnu.org
    vs. github.com.) > > Yeah, I don't personally think there is a privacy issue
    with fetching > sources from our mirrors at gnu.org, or other domains we
   control. > >> Perhaps we should make content-addressed mirrors configurable
    in a way >> that’s orthogonal to derivations, something similar in spirit
    to >> --substitute-urls? The difficulty is that content-addressed mirrors
    are >> not just URLs; see (guix download). >> >> Thoughts? > > I do think
    we should make it so that users don't suffer from unreliable > upstream sources
    when we know the sources are available on our servers > (or the Nix mirror),
    even with --no-substitutes. [...] 
 
 Content analysis details:   (2.2 points, 10.0 required)
 
  pts rule name              description
 ---- ---------------------- --------------------------------------------------
  1.0 SPF_SOFTFAIL           SPF: sender does not match SPF record (softfail)
  0.0 SPF_HELO_FAIL          SPF: HELO does not match SPF record (fail)
 [SPF failed: Please see http://www.openspf.org/Why?s=helo;id=hera.aquilenet.fr;ip=141.255.128.1;r=debbugs.gnu.org]
  1.3 RDNS_NONE              Delivered to internal network by a host with no rDNS

[Message part 1 (text/plain, inline)]
Leo Famulari <leo@famulari.name> skribis:

> On Mon, Oct 02, 2017 at 10:00:33PM +0200, Ludovic Courtès wrote:
>> Right.  Jan suggested checking the content-addressed mirrors *before*
>> the real upstream address.  That would address the problem of upstream
>> sources modified in-place, but at the cost of privacy/self-sufficiency
>> as you note.  (Though it’s not really making “privacy” any worse in this
>> case: it’s gnu.org vs. github.com.)
>
> Yeah, I don't personally think there is a privacy issue with fetching
> sources from our mirrors at gnu.org, or other domains we control.
>
>> Perhaps we should make content-addressed mirrors configurable in a way
>> that’s orthogonal to derivations, something similar in spirit to
>> --substitute-urls?  The difficulty is that content-addressed mirrors are
>> not just URLs; see (guix download).
>>
>> Thoughts?
>
> I do think we should make it so that users don't suffer from unreliable
> upstream sources when we know the sources are available on our servers
> (or the Nix mirror), even with --no-substitutes.

The more I think about it, the more I’m inclined to simply move
content-addressed mirrors to the front of the list.  This means that
users, in practice, would be fetching all the source from
mirror.hydra.gnu.org.

The main issue is making it configurable.  Currently the
content-addressed mirror configuration for regular files in (guix
download) looks like this:

--8<---------------cut here---------------start------------->8---
(define %content-addressed-mirrors
  ;; List of content-addressed mirrors.  Each mirror is represented as a
  ;; procedure that takes a file name, an algorithm (symbol) and a hash
  ;; (bytevector), and returns a URL or #f.
  ;; Note: Avoid 'https' to mitigate <http://bugs.gnu.org/22774>.
  ;; TODO: Add more.
  '(list (lambda (file algo hash)
           ;; Files served by 'guix publish' are accessible under a single
           ;; hash algorithm.
           (string-append "http://mirror.hydra.gnu.org/file/"
                          file "/" (symbol->string algo) "/"
                          (bytevector->nix-base32-string hash)))
         (lambda (file algo hash)
           ;; 'tarballs.nixos.org' supports several algorithms.
           (string-append "http://tarballs.nixos.org/"
                          (symbol->string algo) "/"
                          (bytevector->nix-base32-string hash)))))
--8<---------------cut here---------------end--------------->8---

That for VCS checkouts in (guix build download-nar) looks like this:

--8<---------------cut here---------------start------------->8---
(define (urls-for-item item)
  "Return the fallback nar URL for ITEM--e.g.,
\"/gnu/store/cabbag3…-foo-1.2-checkout\"."
  ;; Here we hard-code nar URLs without checking narinfos.  That's probably OK
  ;; though.
  ;; TODO: Use HTTPS?  The downside is the extra dependency.
  (let ((bases '("http://mirror.hydra.gnu.org/guix"
                 "http://berlin.guixsd.org"))
        (item  (basename item)))
    (append (map (cut string-append <> "/nar/gzip/" item) bases)
            (map (cut string-append <> "/nar/" item) bases))))
--8<---------------cut here---------------end--------------->8---

The latter could be expressed by a command-line flag.  In fact it’s the
same as --substitute-urls.

(Time passes…)

Thinking more about it, why not simply always enable substitutes for
fixed-output derivations, like this:


[Message part 2 (text/x-patch, inline)]
diff --git a/nix/libstore/build.cc b/nix/libstore/build.cc
index d68e8b2bc..03a8f5080 100644
--- a/nix/libstore/build.cc
+++ b/nix/libstore/build.cc
@@ -1034,8 +1034,10 @@ void DerivationGoal::haveDerivation()
 
     /* We are first going to try to create the invalid output paths
        through substitutes.  If that doesn't work, we'll build
-       them. */
-    if (settings.useSubstitutes && substitutesAllowed(drv))
+       them.  Always enable substitutes for fixed-output derivations to
+       protect against disappearing files and in-place modifications on
+       upstream sites.  */
+    if ((fixedOutput || settings.useSubstitutes) && substitutesAllowed(drv))
         foreach (PathSet::iterator, i, invalidOutputs)
             addWaitee(worker.makeSubstitutionGoal(*i, buildMode == bmRepair));
 

[Message part 3 (text/plain, inline)]
This solves all our problems and makes download-nar.scm useless.

As an added bonus, it provides a improves the UI since we now always
see:

--8<---------------cut here---------------start------------->8---
0.1 MB will be downloaded:
   /gnu/store/plx9848n6waj6zghn3d54ybx8ihcn23k-guile-git-0.0-4.951a32c-checkout
--8<---------------cut here---------------end--------------->8---

… instead of:

--8<---------------cut here---------------start------------->8---
The following derivation will be built:
   /gnu/store/y86rlb6pdm35im7q02y6479ca84zwylz-guile-git-000.0-4.951a32c-checkout.drv
--8<---------------cut here---------------end--------------->8---

The downside is that it still requires one to authorize the server’s
key, although it’s in theory unnecessary since it’s content addressed.
I’m not sure how to solve that because ‘guix substitute’ doesn’t know
that it’s substituting a fixed-output derivation.  I suppose we’d need
to modify the “protocol” between guix-daemon and ‘guix substitute’.

Thoughts?

Ludo’.

Send a report that this bug log contains spam.

debbugs.gnu.org maintainers <help-debbugs@gnu.org>. Last modified: Mon Sep 8 11:16:34 2025; Machine Name: wallace-server

GNU bug tracking system

Debbugs is free software and licensed under the terms of the GNU Public License version 2. The current version can be obtained from https://bugs.debian.org/debbugs-source/.

Copyright © 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson, 2005-2017 Don Armstrong, and many other contributors.