texlive is actually substitutable

Strangely, ‘texlive-texmf’ (the big one) is substitutable:

The info suggests it won’t be substituted, but it’s eventually

substituted. I wonder why, because the .drv has:

("allowSubstitutes","0")

and the daemon has:

bool substitutesAllowed(const Derivation & drv)

{

return get(drv.env, "allowSubstitutes", "1") == "1";

}

and:

if (settings.useSubstitutes && substitutesAllowed(drv))

foreach (PathSet::iterator, i, invalidOutputs)

addWaitee(worker.makeSubstitutionGoal(*i, buildMode == bmRepair));

Thoughts?

Ludo’.

On +2020-05-29 17:15:40 +0200, Ludovic Courtès wrote:

This is the kind of "wonder why" that makes me wonder about trojan horse bug fixes

as described in [1], which is a really interesting and scary read, especially since [1]

could very conceivably be an example of what it itself is talking about (though they

don't sound malicious, so I can hope trusting okular to display it was not giving

them a pdf or image parser to exploit with malice).

Anyway, please note that the "pdf" file starts with these lines:

What looks like the beginning of a normal pdf file starts at line 30 counting from 1 as first line.

okular will display the original as if it were pdf (bug??) though "file" just sees it as "data."

Trim off the first 29 lines and file sees it as pdf, and pdfinfo will find its way too.

Idk, you might want at least to cut out the first 29 lines before looking at it with e.g. okular,

(which I trustingly used to open the file): note that okular got past the 29-line script part, (which

is a bit promiscuous for my taste), and displayed the pdf.

It was really interesting, esp the sections around

Maybe you can view it in a sandbox :) But don't blame me if you don't.

YOU WERE WARNED.

So read it -- and wonder what might come with a mysterious substitute ;-P

[1] https://www.alchemistowl.org/pocorgtfo/pocorgtfo08.pdf

--

Regards,

Bengt Richter

Hi,

Leo Famulari <leo@famulari.name> skribis:

It does, but this texlive-texmf.drv is not a fixed-output derivation.

Ludo’.

severity 41602 important

retitle 41602 texlive is actually substitutable

quit

Hello,

Ludovic Courtès <ludo@gnu.org> writes:

I just verified; this still happens:

--

Thanks,

Maxim

Hi,

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

It's odd, when using --dry-run, it seems the expected situation would

happen:

Only the private texlivetexmf is marked as non-substitutable, so the

above looks correct (though we see the intention of preserving bandwidth

wouldn't be achieved, given the source is as large as the final

package).

When removing the --dry-run, what happens though is that the output of

the texlivetexmf package, which is marked as non-substitutable, is downloaded:

I've had some success as stopping in the daemon right before the downlod

starts, with this sequence:

Start the locally built guix-daemon in gdb, with a command like:

Then, to be able to reach the point in the execution where the downloads

are about to start, we need to break in the first fork (but not the 2nd,

which is the guile substituter), like follows at the GDB prompt:

I haven't investigated where in this execution path the

substitutesAllowed procedure is called, if at all, but this is what I

will try to understand next.

--

Thanks,

Maxim

Hello,

Maxim Cournoyer <maxim.cournoyer@gmail.com> writes:

[...]

This gets called, in the case of texlive, only for the texlive .drv

itself, not for its referenced .drvs, which the daemon attempts to

substitute without checking for their allowSubstitutes variable first,

as this gdb screen grab attempts to show:

But perhaps it is futile that the *client* should enforce this. If we

do not want to distribute something in binary form, we shouldn't bake a

.nar for it and make it available in the first place, as pointed by

Emily in #nix-dev:nixos.org (on their Matrix server).

I'll investigate this idea next.

--

Thanks,

Maxim

Hi,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

Agreed, probably something to fix in ‘guix publish’ & co.

Thanks for addressing this longstanding issue!

Ludo’.

Hi Maxim, Ludo,

I did a bit of work on this, and I think that there might be a problem with limiting the distribution of derivation outputs. The problem is with encoding that a derivation's outputs are non-distributable. Because `#:distributable?` should presumably be an argument to `derivation`, and because `guix publish` does not have access to the underlying derivation object, it seems necessary to store distributability information somehow, like is done with "allowSubstitutes" currently. The problem, then, is that when `guix publish` manages a request, what it has to work with is the path of the output being requested *not the derivation from which it was built*. I could be wrong, but I assume that there is no way to compute, from an output path, the path of the derivation which produced it.

It then seems impossible to effectively prevent distribution! I don't see a sane way to attach metadata to every single object in the store (all of which may, to the best of my understanding, be published), and even if we include information about distributability in the derivation file, there is no way for `guix publish` to locate that derivation file, since it is supplied with the path of the derivation output, not the derivation which produced it.

I would appreciate thoughts on this, but I am not sure if I see how `#:distributable?` is a workable solution. I agree that enforcing non-distribution on the client side is inelegant, but I don't see a way to enforce it server-side.

Best,

Morgan

Hi!

I would rely on #:substitutable? instead of relying on a new

#:distributable? flag that the daemon doesn’t know about anyway.

With that in mind, ‘guix publish’ in ‘render-nar’ could have an extra

check in addition to ‘valid-path?’: ‘substitutable-path?’.

One way to implement ‘substitutable-path?’ is by (1) getting a “deriver”

of the store item with ‘valid-derivers’, and (2) checking whether the

returned derivation (if any) passes ‘substitutable-derivation?’.

One downside is that this extra check could be a bit costly, but it’s a

cost one would only pay when not using ‘--cache’.

Does that make sense?

HTH,

Ludo’.

This commit prevents Guix substitute servers from distributing binaries which are marked non-substitutable. This prevents substitute servers from accidentally committing copyright violations by distributing binaries which are non-substitutable for copyright reasons.

Change-Id: Iaca81f5bdb430a12a3ad41e9b83e0bcc535af607

---

guix/scripts/publish.scm | 8 ++++++--

1 file changed, 6 insertions(+), 2 deletions(-)

This commit prevents Guix substitute servers from distributing binaries which are marked non-substitutable. This prevents substitute servers from accidentally committing copyright violations by distributing binaries which are non-substitutable for copyright reasons.

Change-Id: Iaca81f5bdb430a12a3ad41e9b83e0bcc535af607

---

guix/scripts/publish.scm | 9 +++++++--

1 file changed, 7 insertions(+), 2 deletions(-)

Hey Ludo, Maxim,

Sorry about the double-email. I messed up the first patch by mistake. The fix actually ends up being quite simple! I didn't know about `valid-derivers`, so thanks to Ludo for that.

This patch should fix the issue by restricting substitutions on the server side instead of the client side. The bug in the daemon may remain, but this should at least ensure correct functionality. Besides, it seems that we are in agreement that substitutability isn't something that should really be enforced by the client anyway.

Best,

Morgan

Am Montag, dem 24.02.2025 um 19:10 +0000 schrieb Morgan Arnold:

This should probably be

(and substitutable-store-item? (valid-path? store store-path))

Cheers

This commit prevents Guix substitute servers from distributing binaries which are marked non-substitutable. This prevents substitute servers from accidentally committing copyright violations by distributing binaries which are non-substitutable for copyright reasons.

Change-Id: Iaca81f5bdb430a12a3ad41e9b83e0bcc535af607

---

guix/scripts/publish.scm | 9 +++++++--

1 file changed, 7 insertions(+), 2 deletions(-)

Hi Liliana,

Thanks! What a dumb mistake on my part. I tried to test the changes on my machine, but `guix publish` seems to just crash. I reverted my changes to double-check, and the version in `master` also crashes on my machine, so I'm afraid that I wasn't able to test the changes. I've sent a new version.

Best,

Morgan