[PATCH 1/3] build-system/go: Add subdir parameter to go-version->git-ref.

  • Open
  • quality assurance status badge
Details
2 participants
  • Christina O'Donnell
  • Sharlatan Hellseher
Owner
unassigned
Submitted by
Christina O'Donnell
Severity
normal

Debbugs page

C
C
Christina O'Donnell wrote on 16 Mar 03:26 -0700
(address . guix-patches@gnu.org)(name . Christina O'Donnell)(address . cdo@mutix.org)
bfb6cb47bc6e2a05846653ca0b07470d4ae8ae05.1710584715.git.cdo@mutix.org
* guix/build-system/go.scm (go-version->git-ref): Add subdir keyword
parameter. This is needed because Go can have mutliple modules at different
versions in a single repo. It distinguishes their releases by using tags
with their subdirectory. See https://go.dev/ref/mod#vcs-version.

Change-Id: I68bc9e785e49877bb0b756de8458308549f4c957
---
guix/build-system/go.scm | 13 ++++++++-----
1 file changed, 8 insertions(+), 5 deletions(-)

Toggle diff (36 lines)
diff --git a/guix/build-system/go.scm b/guix/build-system/go.scm
index 0934fded07..94c5439dd1 100644
--- a/guix/build-system/go.scm
+++ b/guix/build-system/go.scm
@@ -56,11 +56,12 @@ (define %go-pseudo-version-rx
"([0-9A-Fa-f]{12})" ;commit hash
"(\\+incompatible)?$"))) ;optional +incompatible tag
-(define (go-version->git-ref version)
+(define* (go-version->git-ref version #:key subdir)
"Parse VERSION, a \"pseudo-version\" as defined at
-<https://golang.org/ref/mod#pseudo-versions>, and extract the commit hash from
-it, defaulting to full VERSION (stripped from the \"+incompatible\" suffix if
-present) if a pseudo-version pattern is not recognized."
+<https://golang.org/ref/mod#pseudo-versions>, and extract the commit hash from it,
+defaulting to full VERSION (stripped from the \"+incompatible\" suffix if present) if
+a pseudo-version pattern is not recognized. If SUBDIR is specified and this is not a
+pseudo-version, then this will prefix SUBDIR/ to the returned tag."
;; A module version like v1.2.3 is introduced by tagging a revision in the
;; underlying source repository. Untagged revisions can be referred to
;; using a "pseudo-version" like v0.0.0-yyyymmddhhmmss-abcdefabcdef, where
@@ -78,7 +79,9 @@ (define (go-version->git-ref version)
(match (regexp-exec %go-pseudo-version-rx version)))
(if match
(match:substring match 2)
- version)))
+ (if subdir
+ (string-append subdir "/" version)
+ version))))
(define (go-pseudo-version? version)
"True if VERSION is a Go pseudo-version, i.e., a version string made of a

base-commit: efc0ee1d7f2b704d3fc0c8aea0ef0ad1ac2972e1
--
2.41.0
C
C
Christina O'Donnell wrote on 16 Mar 03:30 -0700
[PATCH 2/3] import/go: Account for monorepo modules in the Go importer.
(address . 69827@debbugs.gnu.org)(name . Christina O'Donnell)(address . cdo@mutix.org)
e6fa7dee732e4fb0b186296781a321c1de8b143f.1710584715.git.cdo@mutix.org
This change allows for importing of modules situated in repos where they're
not at the root directory.

* guix/import/go.scm (go-module-verstion-string): Delete
(go-module-version-info): New procedure
(module-path->repository-root): Add parameter
(vcs->origin): Add parameter
(path-diff): New procedure

Change-Id: If50f7a951ce8e527e5ea44ed24db10d6a3676ff0
---
guix/import/go.scm | 71 +++++++++++++++++++++++++++++++++-------------
1 file changed, 51 insertions(+), 20 deletions(-)

Toggle diff (170 lines)
diff --git a/guix/import/go.scm b/guix/import/go.scm
index dd9298808d..8276797d9a 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -122,15 +122,14 @@ (define (go-path-escape path)
(define (go.pkg.dev-info name)
(http-fetch* (string-append "https://pkg.go.dev/" name)))
-(define* (go-module-version-string goproxy name #:key version)
- "Fetch the version string of the latest version for NAME from the given
+(define* (go-module-version-info goproxy name #:key version)
+ "Fetch a JSON object encoding about the lastest version for NAME from the given
GOPROXY server, or for VERSION when specified."
(let ((file (if version
(string-append "@v/" version ".info")
"@latest")))
- (assoc-ref (json-fetch* (format #f "~a/~a/~a"
- goproxy (go-path-escape name) file))
- "Version")))
+ (json-fetch* (format #f "~a/~a/~a"
+ goproxy (go-path-escape name) file))))
(define* (go-module-available-versions goproxy name)
"Retrieve the available versions for a given module from the module proxy.
@@ -140,8 +139,12 @@ (define* (go-module-available-versions goproxy name)
(body (http-fetch* url))
(versions (remove string-null? (string-split body #\newline))))
(if (null? versions)
- (list (go-module-version-string goproxy name)) ;latest version
- versions)))
+ (begin
+ ;; If we haven't recieved any versions, look in the version-info json
+ ;; object and return a one-element list if found.
+ (or (and=> (assoc-ref (go-module-version-info goproxy name) "Version")
+ list))))
+ versions))
(define (go-package-licenses name)
"Retrieve the list of licenses that apply to NAME, a Go package or module
@@ -431,7 +434,7 @@ (define known-vcs
(/[A-Za-z0-9_.\\-]+)*$"
'git)))
-(define (module-path->repository-root module-path)
+(define (module-path->repository-root module-path version-info)
"Infer the repository root from a module path. Go modules can be
defined at any level of a repository tree, but querying for the meta tag
usually can only be done from the web page at the root of the repository,
@@ -452,6 +455,17 @@ (define (module-path->repository-root module-path)
(lambda (vcs)
(match:substring (regexp-exec (vcs-root-regex vcs)
module-path) 1)))
+ (and=> (assoc-ref version-info "Origin")
+ (lambda (origin)
+ (and=> (assoc-ref origin "Subdir")
+ (lambda (subdir)
+ ;; If version-info contains a 'subdir' and that is a suffix,
+ ;; then the repo-root can be found by stripping off the
+ ;; suffix.
+ (if (string-suffix? (string-append "/" subdir) module-path)
+ (string-drop-right module-path
+ (+ 1 (string-length subdir)))
+ #f)))))
(vcs-qualified-module-path->root-repo-url module-path)
module-path))
@@ -534,13 +548,21 @@ (define* (git-checkout-hash url reference algorithm)
`(tag-or-commit . ,reference)))))
(file-hash* checkout #:algorithm algorithm #:recursive? #true)))
-(define (vcs->origin vcs-type vcs-repo-url version)
+(define (vcs->origin vcs-type vcs-repo-url version subdir)
"Generate the `origin' block of a package depending on what type of source
control system is being used."
(case vcs-type
((git)
- (let ((plain-version? (string=? version (go-version->git-ref version)))
- (v-prefixed? (string-prefix? "v" version)))
+ (let* ((plain-version? (string=? version (go-version->git-ref version
+ #:subdir subdir)))
+ (v-prefixed? (string-prefix? "v" version))
+ ;; This is done because the version field of the package,
+ ;; which the generated quoted expression refers to, has been
+ ;; stripped of any 'v' prefixed.
+ (version-expr (if (and plain-version? v-prefixed?)
+ '(string-append "v" version)
+ `(go-version->git-ref version
+ ,@(if subdir `(#:subdir ,subdir) '())))))
`(origin
(method git-fetch)
(uri (git-reference
@@ -548,14 +570,13 @@ (define (vcs->origin vcs-type vcs-repo-url version)
;; This is done because the version field of the package,
;; which the generated quoted expression refers to, has been
;; stripped of any 'v' prefixed.
- (commit ,(if (and plain-version? v-prefixed?)
- '(string-append "v" version)
- '(go-version->git-ref version)))))
+ (commit ,version-expr)))
(file-name (git-file-name name version))
(sha256
(base32
,(bytevector->nix-base32-string
- (git-checkout-hash vcs-repo-url (go-version->git-ref version)
+ (git-checkout-hash vcs-repo-url (go-version->git-ref version
+ #:subdir subdir)
(hash-algorithm sha256))))))))
((hg)
`(origin
@@ -612,6 +633,12 @@ (define (validate-version version available-versions module-path)
(map strip-v-prefix
available-versions)))))))))
+(define (path-diff parent child)
+ (if (and (string-prefix? parent child) (not (string=? parent child)))
+ (let ((parent-len (string-length parent)))
+ (string-trim (substring child parent-len) (char-set #\/)))
+ #f))
+
(define* (go-module->guix-package module-path #:key
(goproxy "https://proxy.golang.org")
version
@@ -623,9 +650,11 @@ (define* (go-module->guix-package module-path #:key
(let* ((available-versions (go-module-available-versions goproxy module-path))
(version* (validate-version
(or (and version (ensure-v-prefix version))
- (go-module-version-string goproxy module-path)) ;latest
+ (assoc-ref (go-module-version-info goproxy module-path)
+ "Version")) ;latest
available-versions
module-path))
+ (version-info (go-module-version-info goproxy module-path #:version version*))
(content (fetch-go.mod goproxy module-path version*))
(min-go-version (second (go.mod-go-version (parse-go.mod content))))
(dependencies+versions (go.mod-requirements (parse-go.mod content)))
@@ -634,11 +663,13 @@ (define* (go-module->guix-package module-path #:key
(map car dependencies+versions)))
(module-path-sans-suffix
(match:prefix (string-match "([\\./]v[0-9]+)?$" module-path)))
- (guix-name (go-module->guix-package-name module-path))
- (root-module-path (module-path->repository-root module-path))
+ (guix-name (go-module->guix-package-name module-path-sans-suffix ))
+ (root-module-path (module-path->repository-root module-path-sans-suffix
+ version-info))
;; The VCS type and URL are not included in goproxy information. For
;; this we need to fetch it from the official module page.
(meta-data (fetch-module-meta-data root-module-path))
+ (subdir (path-diff root-module-path module-path-sans-suffix))
(vcs-type (module-meta-vcs meta-data))
(vcs-repo-url (module-meta-data-repo-url meta-data goproxy))
(synopsis (go-package-synopsis module-path))
@@ -649,14 +680,14 @@ (define* (go-module->guix-package module-path #:key
(name ,guix-name)
(version ,(strip-v-prefix version*))
(source
- ,(vcs->origin vcs-type vcs-repo-url version*))
+ ,(vcs->origin vcs-type vcs-repo-url version* subdir))
(build-system go-build-system)
(arguments
(list ,@(if (version>? min-go-version (package-version (go-package)))
`(#:go ,(string->number min-go-version))
'())
#:import-path ,module-path
- ,@(if (string=? module-path-sans-suffix root-module-path)
+ ,@(if (string=? module-path root-module-path)
'()
`(#:unpack-path ,root-module-path))))
,@(maybe-propagated-inputs
--
2.41.0
C
C
Christina O'Donnell wrote on 16 Mar 03:30 -0700
[PATCH 3/3] import/go: Add diagnostics.
(address . 69827@debbugs.gnu.org)(name . Christina O'Donnell)(address . cdo@mutix.org)
e17bb3c54fea676d4c19d9a0ed577fbd26d8bc1e.1710584715.git.cdo@mutix.org
guix/import/go.scm (go-module-available-versions): Add warning when fetching
a list of versions from the proxy fails and raise exception when no version
can be found at all.
(module-path->repository-root): Warn when all attempts to find the repository
root have failed.
(fetch-module-meta-data): Raise exception when no meta element could be found.
(go-module->guix-package): Catch general exceptions and warn that the package
could not be imported.

Change-Id: I6dcdccc71f54bfec7110f6bfc5aeb8855502d1e3
---
guix/import/go.scm | 56 ++++++++++++++++++++++++++++++++++++----------
1 file changed, 44 insertions(+), 12 deletions(-)

Toggle diff (101 lines)
diff --git a/guix/import/go.scm b/guix/import/go.scm
index 8276797d9a..5dd5b3d221 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -140,10 +140,15 @@ (define* (go-module-available-versions goproxy name)
(versions (remove string-null? (string-split body #\newline))))
(if (null? versions)
(begin
+ (warning (G_ "Empty list of versions on proxy ~a for package '~a'. Using latest.~%")
+ goproxy name)
;; If we haven't recieved any versions, look in the version-info json
;; object and return a one-element list if found.
(or (and=> (assoc-ref (go-module-version-info goproxy name) "Version")
- list))))
+ list)
+ (raise (make-compound-condition
+ (formatted-message (G_ "No versions available for '~a' on proxy ~a.")
+ name goproxy))))))
versions))
(define (go-package-licenses name)
@@ -467,7 +472,10 @@ (define (module-path->repository-root module-path version-info)
(+ 1 (string-length subdir)))
#f)))))
(vcs-qualified-module-path->root-repo-url module-path)
- module-path))
+ (begin
+ (warning (G_ "Unable to determine repository root of '~a'. Guessing '~a'.~%")
+ module-path module-path)
+ module-path)))
(define* (go-module->guix-package-name module-path #:optional version)
"Converts a module's path to the canonical Guix format for Go packages.
@@ -512,14 +520,19 @@ (define (fetch-module-meta-data module-path)
(select (sxpath `(// (meta (@ (equal? (name "go-import"))))
// content))))
(match (select (html->sxml meta-data #:strict? #t))
- (() #f) ;nothing selected
+ (() (raise (make-compound-condition
+ (formatted-message (G_ "no <meta/> element in result when accessing module path '~a' using go-get")
+ module-path))))
((('content content-text) ..1)
(or
(find (lambda (meta)
(string-prefix? (module-meta-import-prefix meta) module-path))
(map go-import->module-meta content-text))
;; Fallback to the first meta if no import prefixes match.
- (go-import->module-meta (first content-text)))))))
+ (go-import->module-meta (first content-text))
+ (raise (make-compound-condition
+ (formatted-message (G_ "unable to parse <meta/> when accessing module path '~a' using go-get")
+ module-path))))))))
(define (module-meta-data-repo-url meta-data goproxy)
"Return the URL where the fetcher which will be used can download the
@@ -716,16 +729,35 @@ (define go-module->guix-package*
;; consistently.
(setvbuf (current-error-port) 'none)
(let ((package-name (match args ((name _ ...) name))))
- (guard (c ((http-get-error? c)
- (warning (G_ "Failed to import package ~s.
+ (begin
+ (info (G_ "Importing package ~s...~%") package-name)
+ (guard (c ((http-get-error? c)
+ (warning (G_ "Failed to import package ~s.
reason: ~s could not be fetched: HTTP error ~a (~s).
This package and its dependencies won't be imported.~%")
- package-name
- (uri->string (http-get-error-uri c))
- (http-get-error-code c)
- (http-get-error-reason c))
- (values #f '())))
- (apply go-module->guix-package args)))))
+ package-name
+ (uri->string (http-get-error-uri c))
+ (http-get-error-code c)
+ (http-get-error-reason c))
+
+ (values #f '()))
+ ((formatted-message? c)
+ (warning (G_ "Failed to import package ~s.
+reason: ~a
+This package and its dependencies won't be imported.~%")
+ package-name
+ (apply format #f
+ (formatted-message-string c)
+ (formatted-message-arguments c)))
+ (values #f '()))
+ ((git-error? c)
+ (warning (G_ "Failed to import package ~s.
+reason: ~a
+This package and its dependencies won't be imported.~%")
+ package-name
+ (git-error-message c))
+ (values #f '())))
+ (apply go-module->guix-package args))))))
(define* (go-module-recursive-import package-name
#:key (goproxy "https://proxy.golang.org")
--
2.41.0
C
C
Christina O'Donnell wrote on 16 Mar 03:45 -0700
Fixing the go importer so that it can import modules located in a monorepo
(address . 69827@debbugs.gnu.org)
77a7ac18-b32a-8ba4-b400-5d168114c5db@mutix.org
Hi,

These three patches above allow the go importer to automatically import
modules that are located in a monorepo where the submodules are
versioned independently. See https://go.dev/ref/mod#vcs-version:

> If a module is defined in a subdirectory within the repository, that
is, the module subdirectory portion of the module path is not empty,
then each tag name must be prefixed with the module subdirectory,
followed by a slash. For example, the module golang.org/x/tools/gopls is
defined in the gopls subdirectory of the repository with root path
golang.org/x/tools. The version v0.4.0 of that module must have the tag
named gopls/v0.4.0 in that repository.

After this change, I am able to import github.com/gohugio/hugo and its
400 odd dependencies with much fewer failures.

I'll submit those patches separately once I've tested them.

Kind regards,

Christina

PS. This is my first time sending a patch series to debbugs, so I almost
certainly did something wrong. Feedback on that would be appreciated.
S
S
Sharlatan Hellseher wrote on 16 Apr 12:05 -0700
[PATCH 1/3] build-system/go: Add subdir parameter to go-version->git-ref.
(address . 69827@debbugs.gnu.org)
CAO+9K5rZTV+JCfnv+8906gMk2P8L9J4JH_ACt_kjwXj4dfvctw@mail.gmail.com
Hi,

Just a clarifying question. Do we definitely need to know
anything about the folder structure where go.mod governs the
module definition?

There is an API https://pkg.go.dev/ where the most of go projects
are registered, would it be easier just query it directly?

Thanks,
Oleg
Attachment: file
C
C
Christina O'Donnell wrote on 11 May 05:04 -0700
2b113bb1-8def-657d-1e28-6b793db02c58@mutix.org
Hi Oleg,

Thanks for having a look at this!

I don't believe you copied me in the last email so I didn't see your
email when it went out.

> Just a clarifying question. Do we definitely need to know
> anything about the folder structure where go.mod governs the
> module definition?

> There is an API https://pkg.go.dev/where the most of go projects
> are registered, would it be easier just query it directly?

pkg.go.dev gets its sources from proxy.golang.org, and pkg.go.dev
focuses mainly on building and hosting documentation [1]. It doesn't
expose an API for fetching version info [2].

We're already querying proxy.golang.org to get the latest version, and
that response contains information about the sub-directory and git tag,
so we may as well use it.

Kind regards,

Christina

S
S
Sharlatan Hellseher wrote 3 days ago
[PATCH 1/3] build-system/go: Add subdir parameter to go-version->git-ref.
(address . 69827@debbugs.gnu.org)
CAO+9K5q6oOCgZX2uDVjkRrTjQSncAu74OwJ08ufvLtHKhejC4g@mail.gmail.com
Hi,

It's time to proper review your proposal :-)

I faced with obstacles of monorepo during review process, especially where
nested path (subdir) has more than one level.

which is required for restic update.

I will apply your patches and play with importing/upgrading projects like
restic to check how it feels.

Related issues:
- updating aws-sdk-go-v2 https://github.com/aws/aws-sdk-go-v2

Thanks,
Oleg
Attachment: file
S
S
Sharlatan Hellseher wrote 13 hours ago
(address . 69827@debbugs.gnu.org)
87tterfsi3.fsf@gmail.com
Hi,

Some feed back on the patches: when applied on go-team I could easily
import the full Restic dependencies tree it took about 30min to complete
as Guix missed nearly all packages for google cloud.

Mean while it was handy to use in IPFS/Kubo unbundling process where me
and Artyom in our leisure time try to pack all vendored dependencies.

The import of https://github.com/libp2p/go-libp2precursivly produced
all 26 missing packages which I about to submit soon!

As a side note, it's not the first proposal of that kind, please see:

- <2021-12-07> guix import go error https://issues.guix.gnu.org/52362
by Stephen Webber <montokapro@gmail.com>

- <2023-04-21> Go importer doesn't know MODULE/vX.Y version tags
https://issues.guix.gnu.org/63001 by Timo Wilken guix@twilken.net

- <2023-05-22> [PATCH 0/2] Fix annoyance with "guix import go"
<zimon.toutoune@gmail.com>

- <2023-06-12> [PATH] fix a bug on importing go packages.


May we come to some consensus and apply some median patch including all
proposals and fixing the currently issue with import golang modules with
subdirs?

--
Oleg
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCgAdFiEEmEeB3micIcJkGAhndtcnv/Ys0rUFAmbcYvQACgkQdtcnv/Ys
0rUYRBAAuJ3Cbsb7fsZGwCxy6RRGDbIBN9FkmHqPB2lJAb4gfsyEnI6NUA/8dPK3
e5VSPqmIR0J3HKl9o7AgLyy7zpeNvE4sWSEe/xHc4sbfv2+kgquo+/nF9Lf/XXgQ
OVIycQvad/jhRzby6iKUhFJ92tbR3QaMxyyc+2bxM2ozm1AdRF66BMpKiW0qvz44
6lQadpEfccH0xLMbOanOqRhuPuC13tNG698+SRGOS9aJIVHqAjFEPMa8jrTrrmdH
BPeRSkBaoQa7OFFjCq1+6AyPDajfJjNrtDXWrp54fd1U+3JwaSiDfhDaVRA1INmx
VANDEbzf+AskNjNboaOb5n3AJ+TBWZmdH6XlQGC2+W88Scd+NSMUXt5nXdXcSqlk
Mzhlt4gkCK1LkKorJnTiEibB+7bTjKq84sXxd32RTXDociu0tP23D32VMbbWFLI9
U+ae4Ql2C49K/Qf0R++IH/fslBoMoKIkpANjHQvaSFsnkQcb3p4Xju+c2pRsxLaz
h2+v1uXETmDI4Qhf7qRv1Vc+BAd3J53RJGb0ZZooKwiiqbJwqmwhrUeaqy/lLCK3
QDm35Q0PitpD40NAsGBhbxWsmLXBPuTs+9id9z7qkGBeld4Q92otz077SpjQqu9Q
ozLs6TIj12VQ9iABwVE3ur1I806VvEyh6Q0W2QMJqZhZUwsWgck=
=Edh/
-----END PGP SIGNATURE-----

?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 69827@patchwise.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 69827
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch