GNU bug report logs

#70588 Content-addressed mirror is not used upon invalid hash

PackageSource(s)Maintainer(s)
guix PTS Buildd Popcon
Reply or subscribe to this bug. View this bug as an mbox, status mbox, or maintainer mbox

Report forwarded to ludo@gnu.org, bug-guix@gnu.org:
bug#70588; Package guix. (Fri, 26 Apr 2024 08:50:12 GMT) (full text, mbox, link).


Acknowledgement sent to Simon Tournier <zimon.toutoune@gmail.com>:
New bug report received and forwarded. Copy sent to ludo@gnu.org, bug-guix@gnu.org. (Fri, 26 Apr 2024 08:50:12 GMT) (full text, mbox, link).


Message #5 received at submit@debbugs.gnu.org (full text, mbox, reply):

From: Simon Tournier <zimon.toutoune@gmail.com>
To: bug-guix@gnu.org
Subject: hash mismatch error and stop fallback
Date: Thu, 25 Apr 2024 15:28:27 +0200
Hi,

Considering something from the past – well to my knowledge it is still
an issue with the current implementation and such it will be a issue for
the future. ;-)

For instance, the R package MASS from 2019 (Guix revision 711cec):
because the CRAN project did some in-place replacements, we have a hash
mismatch and all the process stops.

--8<---------------cut here---------------start------------->8---
$ guix build /gnu/store/wfq2ji5x9awgyk680i1vv95hq75nx3vv-MASS_7.3-51.4.tar.gz.drv
substitute: updating substitutes from 'https://ci.guix.gnu.org'... 100.0%
substitute: updating substitutes from 'https://bordeaux.guix.gnu.org'... 100.0%
The following derivation will be built:
  /gnu/store/wfq2ji5x9awgyk680i1vv95hq75nx3vv-MASS_7.3-51.4.tar.gz.drv
building /gnu/store/wfq2ji5x9awgyk680i1vv95hq75nx3vv-MASS_7.3-51.4.tar.gz.drv...

Starting download of /gnu/store/gxbam95zqjaxdd6303j97z4djr907yzn-MASS_7.3-51.4.tar.gz
From http://cran.r-project.org/src/contrib/MASS_7.3-51.4.tar.gz...
download failed "http://cran.r-project.org/src/contrib/MASS_7.3-51.4.tar.gz" 404 "Not Found"

Starting download of /gnu/store/gxbam95zqjaxdd6303j97z4djr907yzn-MASS_7.3-51.4.tar.gz
From http://cran.rstudio.com/src/contrib/MASS_7.3-51.4.tar.gz...
download failed "http://cran.rstudio.com/src/contrib/MASS_7.3-51.4.tar.gz" 404 "Not Found"

Starting download of /gnu/store/gxbam95zqjaxdd6303j97z4djr907yzn-MASS_7.3-51.4.tar.gz
From http://cran.univ-lyon1.fr/src/contrib/MASS_7.3-51.4.tar.gz...
following redirection to `https://pbil.univ-lyon1.fr/CRAN/src/contrib/MASS_7.3-51.4.tar.gz'...
download failed "https://pbil.univ-lyon1.fr/CRAN/src/contrib/MASS_7.3-51.4.tar.gz" 404 "Not Found"

Starting download of /gnu/store/gxbam95zqjaxdd6303j97z4djr907yzn-MASS_7.3-51.4.tar.gz
From http://cran.ism.ac.jp/src/contrib/MASS_7.3-51.4.tar.gz...
download failed "http://cran.ism.ac.jp/src/contrib/MASS_7.3-51.4.tar.gz" 404 "Not Found"

Starting download of /gnu/store/gxbam95zqjaxdd6303j97z4djr907yzn-MASS_7.3-51.4.tar.gz
From http://cran.stat.auckland.ac.nz/src/contrib/MASS_7.3-51.4.tar.gz...
download failed "http://cran.stat.auckland.ac.nz/src/contrib/MASS_7.3-51.4.tar.gz" 404 "Not Found"

Starting download of /gnu/store/gxbam95zqjaxdd6303j97z4djr907yzn-MASS_7.3-51.4.tar.gz
From http://cran.mirror.ac.za/src/contrib/MASS_7.3-51.4.tar.gz...
download failed "http://cran.mirror.ac.za/src/contrib/MASS_7.3-51.4.tar.gz" 404 "Not Found"

Starting download of /gnu/store/gxbam95zqjaxdd6303j97z4djr907yzn-MASS_7.3-51.4.tar.gz
From http://cran.csie.ntu.edu.tw/src/contrib/MASS_7.3-51.4.tar.gz...
download failed "http://cran.csie.ntu.edu.tw/src/contrib/MASS_7.3-51.4.tar.gz" 404 "Not Found"

Starting download of /gnu/store/gxbam95zqjaxdd6303j97z4djr907yzn-MASS_7.3-51.4.tar.gz
From http://cran.r-project.org/src/contrib/Archive/MASS/MASS_7.3-51.4.tar.gz...
downloading from http://cran.r-project.org/src/contrib/Archive/MASS/MASS_7.3-51.4.tar.gz ...
 MASS_7.3-51.4.tar.gz  476KiB         2.1MiB/s 00:00 ▕██████████████████▏ 100.0%
sha256 hash mismatch for /gnu/store/gxbam95zqjaxdd6303j97z4djr907yzn-MASS_7.3-51.4.tar.gz:
  expected hash: 0x1ckf51y851g0kvbhsgdxb26vfpmbw3xdbcnh3ck7fjm13da4cr
  actual hash:   1swfd4jnqhsb2imznib629bywrxah7b63yqxhwhd9bhyaji70hl4
hash mismatch for store item '/gnu/store/gxbam95zqjaxdd6303j97z4djr907yzn-MASS_7.3-51.4.tar.gz'
build of /gnu/store/wfq2ji5x9awgyk680i1vv95hq75nx3vv-MASS_7.3-51.4.tar.gz.drv failed
View build log at '/var/log/guix/drvs/wf/q2ji5x9awgyk680i1vv95hq75nx3vv-MASS_7.3-51.4.tar.gz.drv.gz'.
guix build: error: build of `/gnu/store/wfq2ji5x9awgyk680i1vv95hq75nx3vv-MASS_7.3-51.4.tar.gz.drv' failed
--8<---------------cut here---------------end--------------->8---


When the source is correctly available in SWH:

--8<---------------cut here---------------start------------->8---
$ guix build /gnu/store/wfq2ji5x9awgyk680i1vv95hq75nx3vv-MASS_7.3-51.4.tar.gz.drv
The following derivation will be built:
  /gnu/store/wfq2ji5x9awgyk680i1vv95hq75nx3vv-MASS_7.3-51.4.tar.gz.drv
building /gnu/store/wfq2ji5x9awgyk680i1vv95hq75nx3vv-MASS_7.3-51.4.tar.gz.drv...

[...]

Starting download of /gnu/store/gxbam95zqjaxdd6303j97z4djr907yzn-MASS_7.3-51.4.tar.gz
From https://archive.softwareheritage.org/api/1/content/sha256:9911d546a8d29dc906b46cb53ef8aad76d23566f4fc3b52778a1201f8a9b2c74/raw/...
downloading from https://archive.softwareheritage.org/api/1/content/sha256:9911d546a8d29dc906b46cb53ef8aad76d23566f4fc3b52778a1201f8a9b2c74/raw/ ...
 raw/  476KiB                         9.9MiB/s 00:00 ▕██████████████████▏ 100.0%
successfully built /gnu/store/wfq2ji5x9awgyk680i1vv95hq75nx3vv-MASS_7.3-51.4.tar.gz.drv
/gnu/store/gxbam95zqjaxdd6303j97z4djr907yzn-MASS_7.3-51.4.tar.gz

$ guix hash /gnu/store/gxbam95zqjaxdd6303j97z4djr907yzn-MASS_7.3-51.4.tar.gz
0x1ckf51y851g0kvbhsgdxb26vfpmbw3xdbcnh3ck7fjm13da4cr
--8<---------------cut here---------------end--------------->8---

Therefore, the bug is to not consider a hash mismatch as a fullstop and
error out but consider it as a failure of one strategy and thus try the
next strategy.

Other said, hash mismatch should be raised and obviously the content
must be discarded; then the hash mismatch should be considered as “not
found” so let try the next strategy for downloading the content.

Cheers,
simon




Information forwarded to bug-guix@gnu.org:
bug#70588; Package guix. (Wed, 01 May 2024 10:37:03 GMT) (full text, mbox, link).


Message #8 received at 70588@debbugs.gnu.org (full text, mbox, reply):

From: Ludovic Courtès <ludo@gnu.org>
To: Simon Tournier <zimon.toutoune@gmail.com>
Cc: 70588@debbugs.gnu.org
Subject: Re: bug#70588: hash mismatch error and stop fallback
Date: Wed, 01 May 2024 12:36:27 +0200
Hi,

Simon Tournier <zimon.toutoune@gmail.com> skribis:

> Other said, hash mismatch should be raised and obviously the content
> must be discarded; then the hash mismatch should be considered as “not
> found” so let try the next strategy for downloading the content.

Agreed.  I merged it with <https://issues.guix.gnu.org/28659>.

Thanks,
Ludo’.




Severity set to 'important' from 'normal' Request was from Ludovic Courtès <ludo@gnu.org> to control@debbugs.gnu.org. (Wed, 01 May 2024 10:37:03 GMT) (full text, mbox, link).


Merged 28659 70588. Request was from Ludovic Courtès <ludo@gnu.org> to control@debbugs.gnu.org. (Wed, 01 May 2024 10:37:04 GMT) (full text, mbox, link).


Information forwarded to bug-guix@gnu.org:
bug#70588; Package guix. (Fri, 03 May 2024 18:33:02 GMT) (full text, mbox, link).


Message #15 received at 70588@debbugs.gnu.org (full text, mbox, reply):

From: Simon Tournier <zimon.toutoune@gmail.com>
To: 70588@debbugs.gnu.org
Cc: ludo@gnu.org
Subject: Re: bug#70588: hash mismatch error and stop fallback
Date: Fri, 03 May 2024 19:53:39 +0200
Hi,

On lun., 02 oct. 2017 at 17:09, ludo@gnu.org (Ludovic Courtès) wrote:

> The problem is that the hash check is performed by guix-daemon itself,
> not by “guix perform-download”.  So when guix-daemon diagnoses a hash
> mismatch, it’s too late and we cannot try again and use the
> content-addressed mirror.

This quote from [1].  Well, why not run a sequence of “guix download”
then “guix build”?  Ah the store item would be hashed twice.  Let try to
avoid this sub-optimized behaviour.

The only way to avoid hashing twice is to extend the protocol of
guix-daemon.  Because currently it reads,

            /* Check the hash. */
            Hash h2 = recursive ? hashPath(ht, actualPath).first : hashFile(ht, actualPath);
            if (h != h2) {
		if (settings.printBuildTrace)
		    printMsg(lvlError, format("@ hash-mismatch %1% %2% %3% %4%")
			     % path % i->second.hashAlgo
			     % printHash16or32(h) % printHash16or32(h2));
                throw BuildError(format("hash mismatch for store item '%1%'") % path);
	    }

with the call under error catch:

        /* Compute the FS closure of the outputs and register them as
           being valid. */
        registerOutputs();

[...]

    } catch (BuildError & e) {


Thus the number of solutions is very limited. :-)

Well, from my understanding, moving elsewhere in guix-daemon the check
of the hash appears to me too cumbersome.  Therefore, the only way is to
replace this ’BuildError’ by something that is not an error and do not
register the item.

Easier to say than to implement.  Again from my understanding, each
strategy for downloading is seen by the daemon as an element of a list
of independent builds and these elements are tried in sequence.
Therefore, we need to special case in order to be able to distinguish
different kind of failures.

All in all, this bug (missing feature):

        Other said, hash mismatch should be raised and obviously the content
        must be discarded; then the hash mismatch should be considered as “not
        found” so let try the next strategy for downloading the content.

is not straightforward to fix. :-)  Need for more thought…


Cheers,
simon


1: bug#28659: v0.13: guix pull fails; libgit2-0.26.0 and 0.25.1 content hashes fail
ludo@gnu.org (Ludovic Courtès)
Mon, 02 Oct 2017 17:09:39 +0200
id:87o9ppoabw.fsf@gnu.org
https://issues.guix.gnu.org/28659
https://issues.guix.gnu.org/msgid/87o9ppoabw.fsf@gnu.org
https://yhetil.org/guix/87o9ppoabw.fsf@gnu.org




Changed bug title to 'Content-addressed mirror is not used upon invalid hash' from 'hash mismatch error and stop fallback' Request was from Ludovic Courtès <ludo@gnu.org> to control@debbugs.gnu.org. (Mon, 13 May 2024 10:21:02 GMT) (full text, mbox, link).


Send a report that this bug log contains spam.


debbugs.gnu.org maintainers <help-debbugs@gnu.org>. Last modified: Fri Apr 18 12:45:33 2025; Machine Name: wallace-server

GNU bug tracking system

Debbugs is free software and licensed under the terms of the GNU Public License version 2. The current version can be obtained from https://bugs.debian.org/debbugs-source/.

Copyright © 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson, 2005-2017 Don Armstrong, and many other contributors.