cargo build-system should be able to filter out target.cfg(windows) dependencies

  • Open
  • quality assurance status badge
Details
6 participants
  • Hartmut Goebel
  • John Soo
  • Maxime Devos
  • ng0
  • Nicolas Graves
  • zimoun
Owner
unassigned
Submitted by
ng0
Severity
normal

Debbugs page

N
(address . bug-guix@gnu.org)
87zijathke.fsf@wasp.i-did-not-set--mail-host-address--so-tickle-me
The cargo build-system should be able to filter out Windows
dependencies.
Ignore blocks with this in Cargo.toml files:
[target.'cfg(windows)'.dependencies]

until the next [ ] starts.

I also found that most *32-sys packages (currently ALL *32-sys
I've seen) are windows specific.
As the build system is simple enough to not complain, you just
have to learn about this while doing the work.

--
♥Ⓐ ng0
PGP keys and more: https://n0is.noblogs.org/http://ng0.chaosnet.org
Z
Z
zimoun wrote on 18 Dec 2020 11:56
(address . 25327@debbugs.gnu.org)
86pn36vqfe.fsf@gmail.com
Hi,

On Mon, 02 Jan 2017 at 00:46, ng0 <ng0@libertad.pw> wrote:
Toggle quote (12 lines)
> The cargo build-system should be able to filter out Windows
> dependencies.
> Ignore blocks with this in Cargo.toml files:
> [target.'cfg(windows)'.dependencies]
>
> until the next [ ] starts.
>
> I also found that most *32-sys packages (currently ALL *32-sys
> I've seen) are windows specific.
> As the build system is simple enough to not complain, you just
> have to learn about this while doing the work.

This bug is really old and the story about cargo has a bit evolved. Is
it still relevant with the recent additions?

If no, feel free to close it. And without more comment, I will close it
after the usual 2 weeks delay.


All the best,
simon
H
H
Hartmut Goebel wrote on 19 Dec 2020 05:10
(address . 25327@debbugs.gnu.org)
bc4b27b4-5d3a-fac2-f7c2-689147b9c16d@crazy-compilers.com
Am 18.12.20 um 20:56 schrieb zimoun:
Toggle quote (2 lines)
> Is it still relevant with the recent additions?

I just checked this with sequoia 0.20.0: The package "winapi" is still
downloaded and compiled - even if obviously not used sicne on Linux.

--
Regards
Hartmut Goebel

| Hartmut Goebel | h.goebel@crazy-compilers.com |
| www.crazy-compilers.com | compilers which you thought are impossible |
Z
Z
zimoun wrote on 11 Jan 2021 04:41
(name . Hartmut Goebel)(address . h.goebel@crazy-compilers.com)
86lfcztzla.fsf@gmail.com
Hi Hartmut,

On Sat, 19 Dec 2020 at 14:10, Hartmut Goebel <h.goebel@crazy-compilers.com> wrote:
Toggle quote (6 lines)
> Am 18.12.20 um 20:56 schrieb zimoun:
>> Is it still relevant with the recent additions?
>
> I just checked this with sequoia 0.20.0: The package "winapi" is still
> downloaded and compiled - even if obviously not used sicne on Linux.

Should the remove of this “winapi” happen at the build-system level or
at the package per package level?

Other said, are the Windows dependencies normalized by Cargo?

All the best,
simon
J
J
John Soo wrote on 11 Jan 2021 05:33
(name . zimoun)(address . zimon.toutoune@gmail.com)
bf65844a-c17c-4e6c-a5ae-5a88b404ba55@Johns-iPhone
Hi zimoun,

I think that the configuration phase will fail even if conditional compilation dependencies are missing. It’s worth a shot though!

- John
Attachment: file
N
N
Nicolas Graves wrote on 24 Mar 2022 10:18
windows-related dependencies in crates
(address . 25327@debbugs.gnu.org)(name . Maxime Devos)(address . maximedevos@telenet.be)
87h77nyymd.fsf@ngraves.fr
Hi all,

I've delved into the subject the past week and now have a clearer view
on it :

- windows (and mac) dependencies are not needed when compiling a single
package. It is possible to bypass it without changing the
build-system, by a few means :

- either by using the #:cargo-build-flags argument (doesn't
work everytime, seems to work only seldomly with
--target unix)
In this option, one would also need to modify Cargo.toml
file before the package phase, since `cargo package`
vendors all possible dependencies to ensure a package on
all environments.
For instance (1) :
#:phases
(modify-phases %standard-phases
(add-before 'package 'unix-cargo-toml
(lambda _
(delete-file "Cargo.toml")
(substitute* "Cargo.toml.orig"
(("^mac-notification-sys.*") "")
(("^chrono.*") "")
(("^winrt-notification.*") ""))
(rename-file "Cargo.toml.orig" "Cargo.toml")
#t)))

- either by using the environment variable
"CARGO_BUILD_RUSTFLAGS" during the build phase, like that
for instance (2) :
#:phases
(modify-phases %standard-phases
(add-before 'build 'unix-cargo-toml
(setenv
"CARGO_BUILD_RUSTFLAGS"
(string-append
(getenv "CARGO_BUILD_RUSTFLAGS")
"--cfg 'target_os=\"linux\"'"))))
In this case, the package phase would also require the same
change as in (1). Note that this might not be backward
compatible further down the line (seen in the cargo
manual).

For some reason, these two means don't seem to propagate when using such
packages as inputs.

There are also the options to use a snippet or a patch on the source
itself. The snippet on the source could look like (1), or the patch as
its effects, and we would still need to use (2). Haven't experimented
much with that yet.

In any case, all this seems kinda bulky for an edit on an imported base,
and since we have lots of information with `cargo metadata`, I imagine
it would be possible to modify the build-system itself, so that it only
considers `target_os = "linux"` or possibly redox / bsd / dragonfly if
guix considers porting to these platforms.

I have 2 possible ideas :

1) Maybe something like adding a phase after unpack, which would :

- parse `cargo metadata` to isolate only needed dependencies (by maybe
generating and applying a patch ? or maybe there's a cargo command for
generating a Cargo.toml based on modified metadata ?). In any case,
modifying the Cargo.toml to only keep relevant information.

2) Maybe rethinking the package phase entirely instead of relying on
`cargo package --no-verify --no-metadata` which seems to not allow
options without interferring with the Cargo.toml file.

In both cases, we would then need to :
- modify the phase were environment variables are defined to add (2).
- modify the crate import script accordingly (not trivial).

As for the second hypothesis, according to the cargo manual, what `cargo
package` does is the following :

1. Load and check the current workspace, performing some basic checks.
Path dependencies are not allowed unless they have a version key. Cargo
will ignore the path key for dependencies in published
packages. dev-dependencies do not have this restriction.

2. Create the compressed .crate file.
a. The original Cargo.toml file is rewritten and normalized.
b. [patch], [replace], and [workspace] sections are removed from the manifest.
c. Cargo.lock is automatically included if the package contains an
executable binary or example target. cargo-install(1) will use the
packaged lock file if the --locked flag is used.
d. A .cargo_vcs_info.json file is included that contains information about
the current VCS checkout hash if available (not included with
--allow-dirty).

3. Extract the .crate file and build it to verify it can build.
- This will rebuild your package from scratch to ensure that it can be
built from a pristine state. The --no-verify flag can be used to skip
this step.

4. Check that build scripts did not modify any source files.

Step 3 and 4 are actually not performed, since the build-system calls
cargo package with the --no-verify flag. The checks in step 1 doesn't
seem that essential on guix, but I imagine we could replicate
them. Clearly the essential step here is 2, but since we get a crate,
steps a. and b. should actually already be done at the source level, and
step d. is not essential because of origin's field version. Step 2c is
not essential either since Cargo.lock file is actually deleted during
the build system.

I might try to implement these changes, although I do not have enough
time and experience to do that quickly, neither enough horsepower to try
to rebuild everything after that. I included Maxime Devos in the
conversation, would it be possible to fork guix to develop this and try
to rebuild every rust package after that on a guix build farm ? Sorry,
haven't got in touch with guix developers yet, still imposter's
syndrome effect, but implementing this doesn't seem too hard. What would
I need to do to get going on that ?

Cheers,

Nicolas Graves
M
M
Maxime Devos wrote on 24 Mar 2022 12:10
373d643a3526861bd97d3de53b5ec9eb6f21e5f8.camel@telenet.be
Nicolas Graves schreef op do 24-03-2022 om 18:18 [+0100]:
Toggle quote (4 lines)
> I included Maxime Devos in the
> conversation, would it be possible to fork guix to develop this and try
> to rebuild every rust package after that on a guix build farm ?

In the past, for some build system changes, separate branches have been
created that are automatically built by ci.guix.gnu.org. A non-
committer and a committer could then collaborate together (on
guix-devel@gnu.org / guix-patches@gnu.org), though only the committer
can
actually push the changes. If, at some point, the changes appear to be
‘ready’, the build on ci.guix.gnu.org can be started to build the
substitutes. If there are no regressions and the builds completed, the
branch can be merged.

TBC, I'm not a committer, but I'd be willing to review FWIW (I don't
understand cargo).

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYjzCMxccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7m3CAQCfsxiR34W1R5X8j43afQ66FA/l
15iE8IZWUZo5nAAfUgEA9LttOPw2igRrPiHeXcQu8U0CbtRdX7r8fPnyj+ViaAU=
=wERb
-----END PGP SIGNATURE-----


Z
Z
zimoun wrote on 8 Oct 2022 07:15
Re: bug#25327: cargo build-system should be able to filter out target.cfg(windows) dependencies
(name . Nicolas Graves)(address . ngraves@ngraves.fr)
86czb2tn8p.fsf_-_@gmail.com
Hi,

On Thu, 24 Mar 2022 at 18:18, Nicolas Graves <ngraves@ngraves.fr> wrote:

Toggle quote (9 lines)
> I might try to implement these changes, although I do not have enough
> time and experience to do that quickly, neither enough horsepower to try
> to rebuild everything after that. I included Maxime Devos in the
> conversation, would it be possible to fork guix to develop this and try
> to rebuild every rust package after that on a guix build farm ? Sorry,
> haven't got in touch with guix developers yet, still imposter's
> syndrome effect, but implementing this doesn't seem too hard. What would
> I need to do to get going on that ?

Just to know what is the status of this bug#25327 [1]? Well, I have not
followed closely some recent changes about Rust and Guix. Maybe, some
patches are already around?

Cheers,
simon

M
M
Maxime Devos wrote on 8 Oct 2022 09:03
(address . 25327@debbugs.gnu.org)
227baa59-632b-9420-cfdd-c97646991882@telenet.be
On 08-10-2022 16:15, zimoun wrote:
Toggle quote (16 lines)
> Hi,
>
> On Thu, 24 Mar 2022 at 18:18, Nicolas Graves <ngraves@ngraves.fr> wrote:
>
>> I might try to implement these changes, although I do not have enough
>> time and experience to do that quickly, neither enough horsepower to try
>> to rebuild everything after that. I included Maxime Devos in the
>> conversation, would it be possible to fork guix to develop this and try
>> to rebuild every rust package after that on a guix build farm ? Sorry,
>> haven't got in touch with guix developers yet, still imposter's
>> syndrome effect, but implementing this doesn't seem too hard. What would
>> I need to do to get going on that ?
>
> Just to know what is the status of this bug#25327 [1]? Well, I have not
> followed closely some recent changes about Rust and Guix. Maybe, some
> patches are already around?
antioxidant-build-system (*) automatically ignores all dependencies it
can't find (including Windows dependencies), and as such solves this.
(*) close to 100%, the main remaining problem is lack of support for
workspaces; I'm currently looking for packaging a topological sorting
algorithm but it depends on other unpackaged things etc. Other problems
should be rather minor.
Greetings,
Maxime.
Attachment: OpenPGP_signature
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 25327@patchwise.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 25327
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch