(recursive? #t) doesn't seem to be part of the source hash

nomike (they/them) wrote on 30 May 15:56 -0700

Recipients:(address . bug-guix@gnu.org)

Message-ID:331afb8a-fa7f-4935-b990-ebd6f5268b58@nomike.com

Hi!

I'm currently working on a package definition and again stumbled upon an

issue:

I had the flag `(recursive? #t)` added to `source`:

```scheme

(source

(origin

(method git-fetch)

(uri (git-reference

(url "https://github.com/openscad/openscad")

(commit commit)

(recursive? #t)))

(sha256

(base32 "1bkzrjjp0qvfg7pj24j5pa0i6zj0zsqjb5z4w3l6pjdb5q9in0qi"))

(file-name (git-file-name name version))))

```

I then removed the recursive flag and continued on working on my

package, which is based on a commit-ID of the upstream project. Once I

switched to a newer commit, I got strange build errors from cmake. I

switched back the original commit, everything worked again.

It took me a while to remember, that in such a case, guix is not

re-downloading the source as the source hash doesn't change.

IMHO this hash should also contain flags like recursive.

When `git clone foo` is changed to `git clone --recursive foo` the

source has obviously changed (unless the repo doesn't have submodules

perhaps), so it doesn't make sense that the sha256 hash stays the same.

Is this something we can address?

Or is this an issue as it would invalidate all current source hashes at

once?

Thanks

nomike

Rutherther wrote on 31 May 03:13 -0700

Recipients:(address . 78649@debbugs.gnu.org)(name . nomike (they/them))(address . nomike@nomike.com)

Message-ID:87v7phay17.fsf@ditigal.xyz

Hi nomike,

Toggle quote (34 lines)> Hi!
> 
> I'm currently working on a package definition and again stumbled upon an 
> issue:
> 
> I had the flag `(recursive? #t)` added to `source`:
> 
> ```scheme
>  Â Â Â Â Â  (source
>  Â Â Â Â Â Â  (origin
>  Â Â Â Â Â Â Â Â  (method git-fetch)
>  Â Â Â Â Â Â Â Â  (uri (git-reference
>  Â Â Â Â Â Â Â Â Â Â Â Â Â Â  (url "https://github.com/openscad/openscad")
>  Â Â Â Â Â Â Â Â Â Â Â Â Â Â  (commit commit)
>  Â Â Â Â Â Â Â Â Â Â Â Â Â Â  (recursive? #t)))
>  Â Â Â Â Â Â Â Â  (sha256
>  Â Â Â Â Â Â Â Â Â  (base32 "1bkzrjjp0qvfg7pj24j5pa0i6zj0zsqjb5z4w3l6pjdb5q9in0qi"))
>  Â Â Â Â Â Â Â Â  (file-name (git-file-name name version))))
> ```
> 
> I then removed the recursive flag and continued on working on my 
> package, which is based on a commit-ID of the upstream project. Once I 
> switched to a newer commit, I got strange build errors from cmake. I 
> switched back the original commit, everything worked again.
> It took me a while to remember, that in such a case, guix is not 
> re-downloading the source as the source hash doesn't change.
> 
> IMHO this hash should also contain flags like recursive.
> 
> WhenÂ  `git clone foo` is changed to `git clone --recursive foo` the 
> source has obviously changed (unless the repo doesn't have submodules 
> perhaps), so it doesn't make sense that the sha256 hash stays the same.
> 
> Is this something we can address?

I don't think so. For FOD the store path is created from:
- path to gnu store
- hash of the derivation
- name of the package

So when speaking about origin, uri doesn't matter at all. Only file-name
and sha256 does.

That means it is responsibility of the user to change the hash to a new
one when nothing from those changes, but the source is supposed to
change. Ie. if commit is changed to different one without having it in
the file-name, if recursive is changed...

When you change the source, also change the hash to a correct one (or an
invalid one and let it fail to tell you new hash)

Toggle quote (3 lines)

> Or is this an issue as it would invalidate all current source hashes at

> once?

Change like adding the uri to the creation of the hash would definitely
invalidate all source hashes. But it is exactly the point of FOD to not
depend on the uri. If uri is changed, but sha256 stays the same, that
implies the new uri is supposed to fetch the same source => no need for
a redownload.
 

Toggle quote (4 lines)

> Thanks

> nomike

Rutherther

Simon Tournier wrote on 13 Jun 08:46 -0700

Re: bug#78649: (recursive? #t) doesn't seem to be part of the source hash

Recipients:

Message-ID:87sek3myof.fsf@gmail.com

Hi,

On Sat, 31 May 2025 at 00:56, "nomike (they/them)" <nomike@nomike.com> wrote:

Toggle quote (2 lines)

> IMHO this hash should also contain flags like recursive.

The hash contains the recursive flag. :-)

As Rutherther said, the store item hash (path in /gnu/store) of the

source of a package only depends on the checksum hash. It’s named a

fixed-output derivation.

Concretely, compare

Toggle snippet (13 lines)Derive
([("out","/gnu/store/dsngrwg5li7byxfr5bagn8m8yzhalw4d-megacmd-1.5.1-checkout","r:sha256","a8e5f0fca1addc3c9640605efe96ccba34084c2308b47a1bc6e2bed6bd34648b")]
 ,[]
 ,[]
 ,"x86_64-linux","builtin:git-download",[]
 ,[("commit","1.5.1_Linux")
   ,("impureEnvVars","http_proxy https_proxy LC_ALL LC_MESSAGES LANG COLUMNS")
   ,("out","/gnu/store/dsngrwg5li7byxfr5bagn8m8yzhalw4d-megacmd-1.5.1-checkout")
   ,("preferLocalBuild","1")
   ,("recursive?","#t")
   ,("url","\"https://github.com/meganz/MEGAcmd\"")])

and

Toggle snippet (13 lines)Derive
([("out","/gnu/store/dsngrwg5li7byxfr5bagn8m8yzhalw4d-megacmd-1.5.1-checkout","r:sha256","a8e5f0fca1addc3c9640605efe96ccba34084c2308b47a1bc6e2bed6bd34648b")]
 ,[]
 ,[]
 ,"x86_64-linux","builtin:git-download",[]
 ,[("commit","1.5.1_Linux")
   ,("impureEnvVars","http_proxy https_proxy LC_ALL LC_MESSAGES LANG COLUMNS")
   ,("out","/gnu/store/dsngrwg5li7byxfr5bagn8m8yzhalw4d-megacmd-1.5.1-checkout")
   ,("preferLocalBuild","1")
   ,("recursive?","#f")
   ,("url","\"https://github.com/meganz/MEGAcmd\"")])

One have the flag recursive set to #t and the other to #f. But the path

/gnu/store/dsngr…-checkout is the same. That’s because the hash

’dsngr…’ is computed using only the other hash ’a8e5f0fc…’ – which is

another representation (hex format) of the checksum integrity

(nix-base32 format):

(sha256

(base32

"12v46jyxdgp2qqdpmd084d60hd5srjbgwpk082b3rp5dl7yg1rd8"))

One way to see it:

Toggle snippet (7 lines)

$ guix hash -S nar -H sha256 -f hex $(guix build -S megacmd)

a8e5f0fca1addc3c9640605efe96ccba34084c2308b47a1bc6e2bed6bd34648b

$ guix hash -S nar -H sha256 -f nix-base32 $(guix build -S megacmd)

12v46jyxdgp2qqdpmd084d60hd5srjbgwpk082b3rp5dl7yg1rd8

Therefore, if you change the recursive flag without changing the

integrity checksum, then indeed you might have bad surprise.

An example:

Toggle snippet (26 lines)$ ./pre-inst-env guix build -S megacmd 
/gnu/store/dsngrwg5li7byxfr5bagn8m8yzhalw4d-megacmd-1.5.1-checkout

$ git --no-pager diff -- gnu/packages/sync.scm
diff --git a/gnu/packages/sync.scm b/gnu/packages/sync.scm
index e0369f32e13..dd45249cee3 100644
--- a/gnu/packages/sync.scm
+++ b/gnu/packages/sync.scm
@@ -231,7 +231,7 @@ (define-public megacmd
         (uri (git-reference
               (url "https://github.com/meganz/MEGAcmd")
               (commit (string-append version "_Linux"))
-              (recursive? #t)))
+              (recursive? #f)))
         (sha256
          (base32
           "12v46jyxdgp2qqdpmd084d60hd5srjbgwpk082b3rp5dl7yg1rd8"))

$ ./pre-inst-env guix build -S megacmd 
;;; note: source file /home/simon/src/guix/guix/gnu/packages/sync.scm
;;;       newer than compiled /home/simon/src/guix/guix/gnu/packages/sync.go
;;; note: source file /home/simon/src/guix/guix/gnu/packages/sync.scm
;;;       newer than compiled /home/simon/.guix-profile/lib/guile/3.0/site-ccache/gnu/packages/sync.go
/gnu/store/dsngrwg5li7byxfr5bagn8m8yzhalw4d-megacmd-1.5.1-checkout

The recursive flag might appear ineffective but it’s just because Guix

does not recompute all. As Rutherther said: « That means it is

responsibility of the user to change the hash to a new one ». :-)

Well, you see it when using the option --check.

Toggle snippet (59 lines)$ ./pre-inst-env guix build -S megacmd
/gnu/store/dsngrwg5li7byxfr5bagn8m8yzhalw4d-megacmd-1.5.1-checkout

$ ./pre-inst-env guix build -S megacmd --check
The following derivation will be built:
  /gnu/store/58pgh0lqy4mw99hxhwi3wbxsaykqi3hy-megacmd-1.5.1-checkout.drv
building /gnu/store/58pgh0lqy4mw99hxhwi3wbxsaykqi3hy-megacmd-1.5.1-checkout.drv...
Initialized empty Git repository in /gnu/store/1f254hay097h781ildhp498id55mj1pf-megacmd-1.5.1-checkout/.git/
From https://github.com/meganz/MEGAcmd

[...]

HEAD is now at 7886433 Remove unneded Qt plugins
Submodule 'sdk' (https://github.com/meganz/sdk.git) registered for path 'sdk'
Cloning into '/gnu/store/1f254hay097h781ildhp498id55mj1pf-megacmd-1.5.1-checkout/sdk'...
Submodule path 'sdk': checked out 'a1d391d6a9b747892e8033d60ce1f795d181df3c'
warning: rewriting hashes in `/gnu/store/dsngrwg5li7byxfr5bagn8m8yzhalw4d-megacmd-1.5.1-checkout'; cross fingers
successfully built /gnu/store/58pgh0lqy4mw99hxhwi3wbxsaykqi3hy-megacmd-1.5.1-checkout.drv
/gnu/store/dsngrwg5li7byxfr5bagn8m8yzhalw4d-megacmd-1.5.1-checkout

$ git --no-pager diff -- gnu/packages/sync.scm
diff --git a/gnu/packages/sync.scm b/gnu/packages/sync.scm
index e0369f32e13..dd45249cee3 100644
--- a/gnu/packages/sync.scm
+++ b/gnu/packages/sync.scm
@@ -231,7 +231,7 @@ (define-public megacmd
         (uri (git-reference
               (url "https://github.com/meganz/MEGAcmd")
               (commit (string-append version "_Linux"))
-              (recursive? #t)))
+              (recursive? #f)))
         (sha256
          (base32
           "12v46jyxdgp2qqdpmd084d60hd5srjbgwpk082b3rp5dl7yg1rd8"))

$ ./pre-inst-env guix build -S megacmd --check
;;; note: source file /home/simon/src/guix/guix/gnu/packages/sync.scm
;;;       newer than compiled /home/simon/src/guix/guix/gnu/packages/sync.go
;;; note: source file /home/simon/src/guix/guix/gnu/packages/sync.scm
;;;       newer than compiled /home/simon/.guix-profile/lib/guile/3.0/site-ccache/gnu/packages/sync.go
The following derivation will be built:
  /gnu/store/h8kbb614pqm0385mhvwa54bszssx2wid-megacmd-1.5.1-checkout.drv
building /gnu/store/h8kbb614pqm0385mhvwa54bszssx2wid-megacmd-1.5.1-checkout.drv...
Initialized empty Git repository in /gnu/store/1krjpms2pwp1p4269fg3jzdpg3p45h8h-megacmd-1.5.1-checkout/.git/
From https://github.com/meganz/MEGAcmd

[...]

HEAD is now at 7886433 Remove unneded Qt plugins
warning: rewriting hashes in `/gnu/store/dsngrwg5li7byxfr5bagn8m8yzhalw4d-megacmd-1.5.1-checkout'; cross fingers
r:sha256 hash mismatch for /gnu/store/dsngrwg5li7byxfr5bagn8m8yzhalw4d-megacmd-1.5.1-checkout:
  expected hash: 12v46jyxdgp2qqdpmd084d60hd5srjbgwpk082b3rp5dl7yg1rd8
  actual hash:   0in141v0lqgy5ywi00c8bg0c7yisxrnmmfyafc5gam0dl21wybvj
hash mismatch for store item '/gnu/store/dsngrwg5li7byxfr5bagn8m8yzhalw4d-megacmd-1.5.1-checkout'
build of /gnu/store/h8kbb614pqm0385mhvwa54bszssx2wid-megacmd-1.5.1-checkout.drv failed
Could not find build log for '/gnu/store/h8kbb614pqm0385mhvwa54bszssx2wid-megacmd-1.5.1-checkout.drv'.
guix build: error: build of `/gnu/store/h8kbb614pqm0385mhvwa54bszssx2wid-megacmd-1.5.1-checkout.drv' failed

As you see with the line:

    Submodule path 'sdk': checked out 'a1d391d6a9b747892e8033d60ce1f795d181df3c'

it runs “git clone --recurse-submodules”. :-)

For example, if you have another package inheriting ’megacmd’ but
setting the recursive flag to #f

        $ git --no-pager diff -- gnu/packages/sync.scm
        diff --git a/gnu/packages/sync.scm b/gnu/packages/sync.scm
        index e0369f32e13..4b1a8d34074 100644
        --- a/gnu/packages/sync.scm
        +++ b/gnu/packages/sync.scm
        @@ -269,6 +269,23 @@ (define-public megacmd
         distributions.")
             (license (list license:bsd-2 license:gpl3+))))

        +(define-public megacmd-bis
        +  (package
        +    (inherit megacmd)
        +    (name "megacmd-bis")
        +    (version "1.5.1")
        +    (source
        +     (origin
        +       (method git-fetch)
        +       (uri (git-reference
        +             (url "https://github.com/meganz/MEGAcmd")
        +             (commit (string-append version "_Linux"))
        +             (recursive? #f)))
        +       (sha256
        +        (base32
        +         "0in141v0lqgy5ywi00c8bg0c7yisxrnmmfyafc5gam0dl21wybvj"))
        +       (file-name (git-file-name name version))))))

Then, you will see that:

Toggle snippet (9 lines)

$ ls -a $(guix build -S megacmd)/sdk/

. .gitlab CREDITS.md Makefile.win32 bindings contrib include m4 tests

.. .travis.yml LICENSE README.md clean.sh doc libmega.pc.in patches third_party

.gitignore CMakeLists.txt Makefile.am autogen.sh configure.ac examples logo.png src

$ ls -a $(./pre-inst-env guix build -S megacmd-bis)/sdk/

. ..

When using the recursive flag to #t, the submodule is there and the

content is part of the integrity checksum, while the other is not.

HTH.

Cheers,

simon

Maxim Cournoyer wrote on 26 Jul 05:25 -0700

Recipients:(name . Simon Tournier)(address . zimon.toutoune@gmail.com)

Message-ID:87h5yz87g0.fsf@guixotic.coop

Hi,

Thoroughly answered by Simon :-).

Closing.

Thanks,

Maxim

Closed

Your comment

This issue is archived.

To comment on this conversation send an email to 78649@patchwise.org

To respond to this issue using the mumi CLI, first switch to it

mumi current 78649

Then, you may apply the latest patchset in this issue (with sign off)

mumi am -- -s

Or, compose a reply to this issue

mumi compose

Or, send patches to this issue

mumi send-email *.patch

is:open	open issues
is:done	closed issues
submitter:<who>	search issue submitter
author:<who>	search by message author
date:yesterday..now	search by issue date
mdate:3m..2d	search by message date