#78649 (recursive? #t) doesn't seem to be part of the source hash

Message #5 received at submit@debbugs.gnu.org (full text, mbox, reply):

From: "nomike (they/them)" <nomike@nomike.com>

To: bug-guix@gnu.org

Subject: (recursive? #t) doesn't seem to be part of the source hash

Date: Sat, 31 May 2025 00:56:03 +0200

Hi!

I'm currently working on a package definition and again stumbled upon an 
issue:

I had the flag `(recursive? #t)` added to `source`:

```scheme
      (source
       (origin
         (method git-fetch)
         (uri (git-reference
               (url "https://github.com/openscad/openscad")
               (commit commit)
               (recursive? #t)))
         (sha256
          (base32 "1bkzrjjp0qvfg7pj24j5pa0i6zj0zsqjb5z4w3l6pjdb5q9in0qi"))
         (file-name (git-file-name name version))))
```

I then removed the recursive flag and continued on working on my 
package, which is based on a commit-ID of the upstream project. Once I 
switched to a newer commit, I got strange build errors from cmake. I 
switched back the original commit, everything worked again.
It took me a while to remember, that in such a case, guix is not 
re-downloading the source as the source hash doesn't change.

IMHO this hash should also contain flags like recursive.

When  `git clone foo` is changed to `git clone --recursive foo` the 
source has obviously changed (unless the repo doesn't have submodules 
perhaps), so it doesn't make sense that the sha256 hash stays the same.

Is this something we can address?

Or is this an issue as it would invalidate all current source hashes at 
once?

Thanks

nomike

Message #8 received at 78649@debbugs.gnu.org (full text, mbox, reply):

From: Rutherther <rutherther@ditigal.xyz>

To: 78649@debbugs.gnu.org

Cc: "nomike \(they/them\)" <nomike@nomike.com>

Subject: Re: (recursive? #t) doesn't seem to be part of the source hash

Date: Sat, 31 May 2025 12:13:08 +0200

Hi nomike,

> Hi!
> 
> I'm currently working on a package definition and again stumbled upon an 
> issue:
> 
> I had the flag `(recursive? #t)` added to `source`:
> 
> ```scheme
>        (source
>         (origin
>           (method git-fetch)
>           (uri (git-reference
>                 (url "https://github.com/openscad/openscad")
>                 (commit commit)
>                 (recursive? #t)))
>           (sha256
>            (base32 "1bkzrjjp0qvfg7pj24j5pa0i6zj0zsqjb5z4w3l6pjdb5q9in0qi"))
>           (file-name (git-file-name name version))))
> ```
> 
> I then removed the recursive flag and continued on working on my 
> package, which is based on a commit-ID of the upstream project. Once I 
> switched to a newer commit, I got strange build errors from cmake. I 
> switched back the original commit, everything worked again.
> It took me a while to remember, that in such a case, guix is not 
> re-downloading the source as the source hash doesn't change.
> 
> IMHO this hash should also contain flags like recursive.
> 
> When  `git clone foo` is changed to `git clone --recursive foo` the 
> source has obviously changed (unless the repo doesn't have submodules 
> perhaps), so it doesn't make sense that the sha256 hash stays the same.
> 
> Is this something we can address?

I don't think so. For FOD the store path is created from:
- path to gnu store
- hash of the derivation
- name of the package

So when speaking about origin, uri doesn't matter at all. Only file-name
and sha256 does.

That means it is responsibility of the user to change the hash to a new
one when nothing from those changes, but the source is supposed to
change. Ie. if commit is changed to different one without having it in
the file-name, if recursive is changed...

When you change the source, also change the hash to a correct one (or an
invalid one and let it fail to tell you new hash)

> 
> Or is this an issue as it would invalidate all current source hashes at 
> once?

Change like adding the uri to the creation of the hash would definitely
invalidate all source hashes. But it is exactly the point of FOD to not
depend on the uri. If uri is changed, but sha256 stays the same, that
implies the new uri is supposed to fetch the same source => no need for
a redownload.
 
> 
> Thanks
> 
> nomike

Rutherther

Message #11 received at 78649@debbugs.gnu.org (full text, mbox, reply):

From: Simon Tournier <zimon.toutoune@gmail.com>

To: "nomike (they/them)" <nomike@nomike.com>, 78649@debbugs.gnu.org

Subject: Re: bug#78649: (recursive? #t) doesn't seem to be part of the source hash

Date: Fri, 13 Jun 2025 17:46:56 +0200

Hi,

On Sat, 31 May 2025 at 00:56, "nomike (they/them)" <nomike@nomike.com> wrote:

> IMHO this hash should also contain flags like recursive.

The hash contains the recursive flag. :-)

As Rutherther said, the store item hash (path in /gnu/store) of the
source of a package only depends on the checksum hash.  It’s named a
fixed-output derivation.

Concretely, compare

--8<---------------cut here---------------start------------->8---
Derive
([("out","/gnu/store/dsngrwg5li7byxfr5bagn8m8yzhalw4d-megacmd-1.5.1-checkout","r:sha256","a8e5f0fca1addc3c9640605efe96ccba34084c2308b47a1bc6e2bed6bd34648b")]
 ,[]
 ,[]
 ,"x86_64-linux","builtin:git-download",[]
 ,[("commit","1.5.1_Linux")
   ,("impureEnvVars","http_proxy https_proxy LC_ALL LC_MESSAGES LANG COLUMNS")
   ,("out","/gnu/store/dsngrwg5li7byxfr5bagn8m8yzhalw4d-megacmd-1.5.1-checkout")
   ,("preferLocalBuild","1")
   ,("recursive?","#t")
   ,("url","\"https://github.com/meganz/MEGAcmd\"")])
--8<---------------cut here---------------end--------------->8---

and

--8<---------------cut here---------------start------------->8---
Derive
([("out","/gnu/store/dsngrwg5li7byxfr5bagn8m8yzhalw4d-megacmd-1.5.1-checkout","r:sha256","a8e5f0fca1addc3c9640605efe96ccba34084c2308b47a1bc6e2bed6bd34648b")]
 ,[]
 ,[]
 ,"x86_64-linux","builtin:git-download",[]
 ,[("commit","1.5.1_Linux")
   ,("impureEnvVars","http_proxy https_proxy LC_ALL LC_MESSAGES LANG COLUMNS")
   ,("out","/gnu/store/dsngrwg5li7byxfr5bagn8m8yzhalw4d-megacmd-1.5.1-checkout")
   ,("preferLocalBuild","1")
   ,("recursive?","#f")
   ,("url","\"https://github.com/meganz/MEGAcmd\"")])
--8<---------------cut here---------------end--------------->8---

One have the flag recursive set to #t and the other to #f.  But the path
/gnu/store/dsngr…-checkout is the same.  That’s because the hash
’dsngr…’ is computed using only the other hash ’a8e5f0fc…’ – which is
another representation (hex format) of the checksum integrity
(nix-base32 format):

        (sha256
         (base32
          "12v46jyxdgp2qqdpmd084d60hd5srjbgwpk082b3rp5dl7yg1rd8"))

One way to see it:

--8<---------------cut here---------------start------------->8---
$ guix hash -S nar -H sha256 -f hex $(guix build -S megacmd)
a8e5f0fca1addc3c9640605efe96ccba34084c2308b47a1bc6e2bed6bd34648b

$ guix hash -S nar -H sha256 -f nix-base32 $(guix build -S megacmd)
12v46jyxdgp2qqdpmd084d60hd5srjbgwpk082b3rp5dl7yg1rd8
--8<---------------cut here---------------end--------------->8---

Therefore, if you change the recursive flag without changing the
integrity checksum, then indeed you might have bad surprise.

An example:

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix build -S megacmd 
/gnu/store/dsngrwg5li7byxfr5bagn8m8yzhalw4d-megacmd-1.5.1-checkout

$ git --no-pager diff -- gnu/packages/sync.scm
diff --git a/gnu/packages/sync.scm b/gnu/packages/sync.scm
index e0369f32e13..dd45249cee3 100644
--- a/gnu/packages/sync.scm
+++ b/gnu/packages/sync.scm
@@ -231,7 +231,7 @@ (define-public megacmd
         (uri (git-reference
               (url "https://github.com/meganz/MEGAcmd")
               (commit (string-append version "_Linux"))
-              (recursive? #t)))
+              (recursive? #f)))
         (sha256
          (base32
           "12v46jyxdgp2qqdpmd084d60hd5srjbgwpk082b3rp5dl7yg1rd8"))

$ ./pre-inst-env guix build -S megacmd 
;;; note: source file /home/simon/src/guix/guix/gnu/packages/sync.scm
;;;       newer than compiled /home/simon/src/guix/guix/gnu/packages/sync.go
;;; note: source file /home/simon/src/guix/guix/gnu/packages/sync.scm
;;;       newer than compiled /home/simon/.guix-profile/lib/guile/3.0/site-ccache/gnu/packages/sync.go
/gnu/store/dsngrwg5li7byxfr5bagn8m8yzhalw4d-megacmd-1.5.1-checkout
--8<---------------cut here---------------end--------------->8---

The recursive flag might appear ineffective but it’s just because Guix
does not recompute all.  As Rutherther said: « That means it is
responsibility of the user to change the hash to a new one ». :-)

Well, you see it when using the option --check.

--8<---------------cut here---------------start------------->8---
$ ./pre-inst-env guix build -S megacmd
/gnu/store/dsngrwg5li7byxfr5bagn8m8yzhalw4d-megacmd-1.5.1-checkout

$ ./pre-inst-env guix build -S megacmd --check
The following derivation will be built:
  /gnu/store/58pgh0lqy4mw99hxhwi3wbxsaykqi3hy-megacmd-1.5.1-checkout.drv
building /gnu/store/58pgh0lqy4mw99hxhwi3wbxsaykqi3hy-megacmd-1.5.1-checkout.drv...
Initialized empty Git repository in /gnu/store/1f254hay097h781ildhp498id55mj1pf-megacmd-1.5.1-checkout/.git/
From https://github.com/meganz/MEGAcmd

[...]

HEAD is now at 7886433 Remove unneded Qt plugins
Submodule 'sdk' (https://github.com/meganz/sdk.git) registered for path 'sdk'
Cloning into '/gnu/store/1f254hay097h781ildhp498id55mj1pf-megacmd-1.5.1-checkout/sdk'...
Submodule path 'sdk': checked out 'a1d391d6a9b747892e8033d60ce1f795d181df3c'
warning: rewriting hashes in `/gnu/store/dsngrwg5li7byxfr5bagn8m8yzhalw4d-megacmd-1.5.1-checkout'; cross fingers
successfully built /gnu/store/58pgh0lqy4mw99hxhwi3wbxsaykqi3hy-megacmd-1.5.1-checkout.drv
/gnu/store/dsngrwg5li7byxfr5bagn8m8yzhalw4d-megacmd-1.5.1-checkout

$ git --no-pager diff -- gnu/packages/sync.scm
diff --git a/gnu/packages/sync.scm b/gnu/packages/sync.scm
index e0369f32e13..dd45249cee3 100644
--- a/gnu/packages/sync.scm
+++ b/gnu/packages/sync.scm
@@ -231,7 +231,7 @@ (define-public megacmd
         (uri (git-reference
               (url "https://github.com/meganz/MEGAcmd")
               (commit (string-append version "_Linux"))
-              (recursive? #t)))
+              (recursive? #f)))
         (sha256
          (base32
           "12v46jyxdgp2qqdpmd084d60hd5srjbgwpk082b3rp5dl7yg1rd8"))

$ ./pre-inst-env guix build -S megacmd --check
;;; note: source file /home/simon/src/guix/guix/gnu/packages/sync.scm
;;;       newer than compiled /home/simon/src/guix/guix/gnu/packages/sync.go
;;; note: source file /home/simon/src/guix/guix/gnu/packages/sync.scm
;;;       newer than compiled /home/simon/.guix-profile/lib/guile/3.0/site-ccache/gnu/packages/sync.go
The following derivation will be built:
  /gnu/store/h8kbb614pqm0385mhvwa54bszssx2wid-megacmd-1.5.1-checkout.drv
building /gnu/store/h8kbb614pqm0385mhvwa54bszssx2wid-megacmd-1.5.1-checkout.drv...
Initialized empty Git repository in /gnu/store/1krjpms2pwp1p4269fg3jzdpg3p45h8h-megacmd-1.5.1-checkout/.git/
From https://github.com/meganz/MEGAcmd

[...]

HEAD is now at 7886433 Remove unneded Qt plugins
warning: rewriting hashes in `/gnu/store/dsngrwg5li7byxfr5bagn8m8yzhalw4d-megacmd-1.5.1-checkout'; cross fingers
r:sha256 hash mismatch for /gnu/store/dsngrwg5li7byxfr5bagn8m8yzhalw4d-megacmd-1.5.1-checkout:
  expected hash: 12v46jyxdgp2qqdpmd084d60hd5srjbgwpk082b3rp5dl7yg1rd8
  actual hash:   0in141v0lqgy5ywi00c8bg0c7yisxrnmmfyafc5gam0dl21wybvj
hash mismatch for store item '/gnu/store/dsngrwg5li7byxfr5bagn8m8yzhalw4d-megacmd-1.5.1-checkout'
build of /gnu/store/h8kbb614pqm0385mhvwa54bszssx2wid-megacmd-1.5.1-checkout.drv failed
Could not find build log for '/gnu/store/h8kbb614pqm0385mhvwa54bszssx2wid-megacmd-1.5.1-checkout.drv'.
guix build: error: build of `/gnu/store/h8kbb614pqm0385mhvwa54bszssx2wid-megacmd-1.5.1-checkout.drv' failed
--8<---------------cut here---------------end--------------->8---

As you see with the line:

    Submodule path 'sdk': checked out 'a1d391d6a9b747892e8033d60ce1f795d181df3c'

it runs “git clone --recurse-submodules”. :-)

For example, if you have another package inheriting ’megacmd’ but
setting the recursive flag to #f

        $ git --no-pager diff -- gnu/packages/sync.scm
        diff --git a/gnu/packages/sync.scm b/gnu/packages/sync.scm
        index e0369f32e13..4b1a8d34074 100644
        --- a/gnu/packages/sync.scm
        +++ b/gnu/packages/sync.scm
        @@ -269,6 +269,23 @@ (define-public megacmd
         distributions.")
             (license (list license:bsd-2 license:gpl3+))))

        +(define-public megacmd-bis
        +  (package
        +    (inherit megacmd)
        +    (name "megacmd-bis")
        +    (version "1.5.1")
        +    (source
        +     (origin
        +       (method git-fetch)
        +       (uri (git-reference
        +             (url "https://github.com/meganz/MEGAcmd")
        +             (commit (string-append version "_Linux"))
        +             (recursive? #f)))
        +       (sha256
        +        (base32
        +         "0in141v0lqgy5ywi00c8bg0c7yisxrnmmfyafc5gam0dl21wybvj"))
        +       (file-name (git-file-name name version))))))

Then, you will see that:

--8<---------------cut here---------------start------------->8---
$ ls -a $(guix build -S megacmd)/sdk/
.	    .gitlab	    CREDITS.md	 Makefile.win32  bindings      contrib	 include	m4	 tests
..	    .travis.yml     LICENSE	 README.md	 clean.sh      doc	 libmega.pc.in	patches  third_party
.gitignore  CMakeLists.txt  Makefile.am  autogen.sh	 configure.ac  examples  logo.png	src

$ ls -a $(./pre-inst-env guix build -S megacmd-bis)/sdk/
.  ..
--8<---------------cut here---------------end--------------->8---

When using the recursive flag to #t, the submodule is there and the
content is part of the integrity checksum, while the other is not.

HTH.

Cheers,
simon

Message #16 received at 78649-done@debbugs.gnu.org (full text, mbox, reply):

From: Maxim Cournoyer <maxim@guixotic.coop>

To: Simon Tournier <zimon.toutoune@gmail.com>

Cc: 78649-done@debbugs.gnu.org, "nomike \(they/them\)" <nomike@nomike.com>

Subject: Re: bug#78649: (recursive? #t) doesn't seem to be part of the source hash

Date: Sat, 26 Jul 2025 21:25:51 +0900

Hi,

Thoroughly answered by Simon :-).

Closing.

-- 
Thanks,
Maxim

Send a report that this bug log contains spam.