[PATCH 0/21] Juliahub import script.

  • Open
  • quality assurance status badge
Details
4 participants
  • Giovanni Biscuolo
  • Ludovic Courtès
  • Nicolas Graves
  • Simon Tournier
Owner
unassigned
Submitted by
Nicolas Graves
Severity
normal

Debbugs page

N
N
Nicolas Graves wrote on 15 Mar 2023 05:47
(address . guix-patches@gnu.org)
871qlq89kz.fsf@ngraves.fr
Hi guix!

Took me quite more time than I would've liked, but I have a usable
juliahub scheme import script!

It seems there's still one edge case that isn't covered and revolves
around when Julia packagers don't properly tag their git repos (I've
only seen the case with SnoopPrecompile). There's the possibility to
rely on tree commit hashes from the General repository (since this is a
valid way to identify/store a git repo), but that needs some major
changes in the way latest-repository-commit works. Otherwise, it needs
to be done by hand. It might also not work for subpackages in
directories that are up-to-date on juliahub but not yet on github, I
haven't met this case yet.

I'm sending a patch series in the coming minutes.

It's detailed since I haven't swauased all commits, for readability, but
I can squash it further if necessary.

--
Best regards,
Nicolas Graves
N
N
Nicolas Graves wrote on 15 Mar 2023 05:51
[PATCH 01/21] import: juliahub: first script draft.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-1-ngraves@ngraves.fr
---
guix/import/go.scm | 6 +-
guix/import/juliahub.scm | 183 +++++++++++++++++++++++++++++++
guix/scripts/import.scm | 2 +-
guix/scripts/import/juliahub.scm | 107 ++++++++++++++++++
4 files changed, 295 insertions(+), 3 deletions(-)
create mode 100644 guix/import/juliahub.scm
create mode 100644 guix/scripts/import/juliahub.scm

Toggle diff (335 lines)
diff --git a/guix/import/go.scm b/guix/import/go.scm
index 69937f8a4d..f264715fbd 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -503,9 +503,11 @@ (define (transform-version version)
'(string-append "v" version)
'(go-version->git-ref version))))
-(define (vcs->origin vcs-type vcs-repo-url version)
+(define* (vcs->origin vcs-type vcs-repo-url version
+ #:optional transform-version)
"Generate the `origin' block of a package depending on what type of source
-control system is being used."
+control system is being used. Optionally use the function TRANSFORM-VERSION
+which takes version as an input."
(case vcs-type
((git)
(git->origin vcs-repo-url `(tag-or-commit . ,version) transform-version))
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
new file mode 100644
index 0000000000..efe6abbb24
--- /dev/null
+++ b/guix/import/juliahub.scm
@@ -0,0 +1,183 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix import juliahub)
+ #:use-module (ice-9 textual-ports)
+ #:use-module (ice-9 regex)
+ #:use-module (ice-9 match)
+ #:use-module (ice-9 string-fun)
+ #:use-module (srfi srfi-9)
+ #:use-module (srfi srfi-1)
+ #:use-module (guix http-client)
+ #:use-module (guix import utils)
+ #:use-module (guix import json)
+ #:use-module (guix base16)
+ #:use-module (guix base32)
+ #:use-module (guix packages)
+ #:use-module (guix upstream)
+ #:use-module (json)
+ #:use-module ((guix licenses) #:prefix license:)
+
+ #:export (juliahub->guix-package))
+
+(define (juliahub-uri name)
+ (let* ((url (string-append "https://docs.juliahub.com/" name "/"))
+ (port (http-fetch url #:text? #t))
+ (_ (get-line port))
+ (meta (get-line port))
+ (regex "url=[a-zA-Z0-9]{5}\\/[0-9\\.]*")
+ (redirect (match:substring (string-match regex meta))))
+ (close-port port)
+ (string-drop redirect 4)))
+
+(define (juliahub-url name)
+ (let* ((url (string-append "https://docs.juliahub.com/" name "/"))
+ (uri (juliahub-uri name)))
+ (string-append url uri "/")))
+
+(define (juliahub-slug-version name)
+ (let* ((uri (juliahub-uri name))
+ (slug (string-take uri 5))
+ (latest-version (string-drop uri 6)))
+ `(,slug ,latest-version)))
+
+(define (json->juliahub-direct-dependencies vector)
+ (if (vector? vector)
+ (filter-map
+ (lambda (el)
+ (let ((dep (json->juliahub-dependency el)))
+ (if (juliahub-dependency-direct? dep)
+ dep
+ #f)))
+ (vector->list vector))))
+
+;; Julia package.
+(define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
+ json->juliahub-package
+ (homepage juliahub-package-homepage) ;string
+ (readme juliahub-package-readme) ;string
+ ;; (slug juliahub-package-slug) ;string
+ (version juliahub-package-version) ;string
+ (description juliahub-package-description) ;string
+ (dependencies
+ juliahub-package-dependencies "deps"
+ json->juliahub-direct-dependencies) ;list of <juliahub-dependency>
+ ;; (lambda (vector)
+ ;; (map json->juliahub-dependency (vector->list vector))))
+ (url juliahub-package-url) ;string
+ (uuid juliahub-package-uuid) ;string
+ (license juliahub-package-license)) ;string
+
+(define-json-mapping <juliahub-dependency>
+ make-juliahub-dependency juliahub-dependency?
+ json->juliahub-dependency
+ (direct? juliahub-dependency-direct? "direct") ;boolean
+ (name juliahub-dependency-name) ;string
+ (uuid juliahub-dependency-uuid) ;string
+ (versions juliahub-dependency-versions "versions" vector->list)) ;list of strings
+ ;; (slug juliahub-dependency-slug) ;string
+
+(define (julia-name->guix-name name)
+ (string-append "julia-" (snake-case name)))
+
+(define* (juliahub-fetch name #:key (version #f))
+ "Return a <juliahub-package> record for package NAME, or #f on failure."
+ (and=> (json-fetch (string-append (juliahub-url name) "pkg.json"))
+ json->juliahub-package))
+
+(define (make-julia-sexp name version uri hash home-page synopsis description
+ dependencies licenses)
+ "Return the `package' s-expression for a Julia package with the given NAME,
+VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DEPENDENCIES, and LICENSES."
+ `(package
+ (name ,(julia-name->guix-name name))
+ (version ,version)
+ (source (origin
+ (method url-fetch)
+ (uri ,uri)
+ (sha256
+ (base32
+ "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5"
+ ;; ,(bytevector->nix-base32-string hash)
+ ))))
+ (build-system julia-build-system)
+ ,@(if (null? dependencies)
+ '()
+ `((inputs
+ (list ,@(map (compose julia-name->guix-name juliahub-dependency-name)
+ dependencies)))))
+ (synopsis ,synopsis)
+ (description ,description)
+ (home-page ,home-page)
+ (license ,(match licenses
+ (() #f)
+ ((license) (license->symbol license))
+ (_ `(list ,@(map license->symbol licenses)))))))
+
+(define* (juliahub->guix-package package-name
+ #:key version #:allow-other-keys)
+ "Fetch the metadata for PACKAGE-NAME from juliahub.org, and return the
+`package' s-expression corresponding to that package, or #f on failure.
+Optionally include a VERSION string to fetch a specific version juliahub."
+ (let ((package (if version
+ (juliahub-fetch package-name version)
+ (juliahub-fetch package-name))))
+ (if package
+ (let* ((dependencies-names
+ (map juliahub-dependency-name
+ (juliahub-package-dependencies package)))
+ (licenses
+ (map spdx-string->license
+ (list (juliahub-package-license package)))))
+ (values (make-julia-sexp
+ package-name
+ (juliahub-package-version package)
+ (juliahub-package-url package)
+ "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5"
+ (juliahub-package-homepage package)
+ (juliahub-package-description package)
+ (beautify-description (juliahub-package-readme package))
+ (juliahub-package-dependencies package)
+ licenses)
+ dependencies-names))
+ (values #f '()))))
+
+(define* (import-release package #:key (version #f))
+ "Return an <upstream-source> for the latest release of PACKAGE."
+ (let* ((package-name (guix-package->juliahub-name package))
+ (package (juliahub-fetch package-name))
+ (version (or version (juliahub-version gem)))
+ (url (rubyjuliahubs-uri gem-name version)))
+ (upstream-source
+ (package (package-name package))
+ (version version)
+ (urls (list url)))))
+
+(define %juliahub-updater
+ (upstream-updater
+ (name 'juliahub)
+ (description "Updater for Juliahub packages")
+ (pred juliahub-package?)
+ (import import-release)))
+
+(define* (juliahub-recursive-import package-name #:optional version)
+ (recursive-import package-name
+ #:repo '()
+ #:repo->guix-package juliahub->guix-package
+ #:guix-name ruby-package-name
+ #:version version))
diff --git a/guix/scripts/import.scm b/guix/scripts/import.scm
index f84a964a53..ef4e0b9cc6 100644
--- a/guix/scripts/import.scm
+++ b/guix/scripts/import.scm
@@ -47,7 +47,7 @@ (define %standard-import-options '())
(define importers '("gnu" "pypi" "cpan" "hackage" "stackage" "egg" "elpa"
"gem" "go" "cran" "crate" "texlive" "json" "opam"
- "minetest" "elm" "hexpm"))
+ "minetest" "elm" "hexpm" "juliahub"))
(define (resolve-importer name)
(let ((module (resolve-interface
diff --git a/guix/scripts/import/juliahub.scm b/guix/scripts/import/juliahub.scm
new file mode 100644
index 0000000000..1317c67aa3
--- /dev/null
+++ b/guix/scripts/import/juliahub.scm
@@ -0,0 +1,107 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix scripts import juliahub)
+ #:use-module (guix ui)
+ #:use-module (guix utils)
+ #:use-module (guix scripts)
+ #:use-module (guix import juliahub)
+ #:use-module (guix scripts import)
+ #:use-module (srfi srfi-1)
+ #:use-module (srfi srfi-11)
+ #:use-module (srfi srfi-37)
+ #:use-module (ice-9 match)
+ #:use-module (ice-9 format)
+ #:use-module (ice-9 receive)
+ #:export (guix-import-juliahub))
+
+
+;;;
+;;; Command-line options.
+;;;
+
+(define %default-options
+ '())
+
+(define (show-help)
+ (display (G_ "Usage: guix import juliahub PACKAGE-NAME[@VERSION] Import and
+convert the Julia package for PACKAGE-NAME. Optionally, a version can be
+specified after the at-sign (@) character.\n"))
+ (display (G_ "
+ -h, --help display this help and exit"))
+ (display (G_ "
+ -V, --version display version information and exit"))
+ (display (G_ "
+ -r, --recursive generate package expressions for all Gem packages\
+ that are not yet in Guix"))
+ (newline)
+ (show-bug-report-information))
+
+(define %options
+ ;; Specification of the command-line options.
+ (cons* (option '(#\h "help") #f #f
+ (lambda args
+ (show-help)
+ (exit 0)))
+ (option '(#\V "version") #f #f
+ (lambda args
+ (show-version-and-exit "guix import gem")))
+ (option '(#\r "recursive") #f #f
+ (lambda (opt name arg result)
+ (alist-cons 'recursive #t result)))
+ %standard-import-options))
+
+
+;;;
+;;; Entry point.
+;;;
+
+(define (guix-import-juliahub . args)
+ (define (parse-options)
+ ;; Return the alist of option values.
+ (parse-command-line args %options (list %default-options)
+ #:build-options? #f))
+
+ (let* ((opts (parse-options))
+ (args (filter-map (match-lambda
+ (('argument . value)
+ value)
+ (_ #f))
+ (reverse opts))))
+ (match args
+ ((spec)
+ (receive (package-name package-version)
+ (package-name->name+version spec)
+ (let ((code (if (assoc-ref opts 'recursive)
+ (map (match-lambda
+ ((and ('package ('name name) . rest) pkg)
+ `(define-public ,(string->symbol name)
+ ,pkg))
+ (_ #f))
+ (juliahub-recursive-import package-name package-version))
+ (let ((sexp (juliahub->guix-package package-name #:version package-version)))
+ (if sexp sexp #f)))))
+ (match code
+ ((or #f '(#f))
+ (leave (G_ "failed to download meta-data for package '~a'~%")
+ package-name))
+ (_ code)))))
+ (()
+ (leave (G_ "too few arguments~%")))
+ ((many ...)
+ (leave (G_ "too many arguments~%"))))))
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 2023 05:51
[PATCH 02/21] import: utils: Change git->origin function to git->origin+version.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-2-ngraves@ngraves.fr
---
guix/import/go.scm | 2 +-
guix/import/utils.scm | 50 +++++++++++++++++++++++++++----------------
2 files changed, 33 insertions(+), 19 deletions(-)

Toggle diff (90 lines)
diff --git a/guix/import/go.scm b/guix/import/go.scm
index f264715fbd..3be82ed164 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -510,7 +510,7 @@ (define* (vcs->origin vcs-type vcs-repo-url version
which takes version as an input."
(case vcs-type
((git)
- (git->origin vcs-repo-url `(tag-or-commit . ,version) transform-version))
+ (git->origin vcs-repo-url `(tag-or-commit . ,version) transform-version))
((hg)
`(origin
(method hg-fetch)
diff --git a/guix/import/utils.scm b/guix/import/utils.scm
index 529d7f11f8..171dca54e8 100644
--- a/guix/import/utils.scm
+++ b/guix/import/utils.scm
@@ -65,6 +65,7 @@ (define-module (guix import utils)
url-fetch
guix-hash-url
+ git->origin+dir
git->origin
package-names->package-inputs
@@ -156,9 +157,12 @@ (define (guix-hash-url filename)
"Return the hash of FILENAME in nix-base32 format."
(bytevector->nix-base32-string (file-sha256 filename)))
-(define* (git->origin repo-url ref #:optional ref->commit)
- "Generate the `origin' block of a package depending on the git source
-control system. REPO-URL or REF can be null."
+(define* (git->origin+dir repo-url ref #:optional ref->commit)
+ "Returns a generated `origin' block of a package depending on the git source
+control system, and the directory in the store where the package has been
+downloaded, in case further processing is necessary. REPO-URL or REF can be
+null. REF->COMMIT can be a function or #t, in which case the commit matching
+ref is used. If REF->COMMIT is not used, the value inside REF is used."
(let-values (((directory commit)
(with-store store
(latest-repository-commit store repo-url #:ref ref))))
@@ -172,21 +176,31 @@ (define* (git->origin repo-url ref #:optional ref->commit)
version)
(_
(ref->commit version)))))
- `(origin
- (method git-fetch)
- (uri (git-reference
- (url ,(and (not (eq? repo-url 'null)) repo-url))
- (commit ,vcommit)))
- (file-name (git-file-name name version))
- (sha256
- (base32
- ,(if (pair? ref)
- (bytevector->nix-base32-string
- (file-hash* directory
- ;; 'git-fetch' already filtered out the '.git' directory.
- #:select? (const #true)
- #:recursive? #true))
- #f)))))))
+ (values
+ `(origin
+ (method git-fetch)
+ (uri (git-reference
+ (url ,(and (not (eq? repo-url 'null)) repo-url))
+ (commit ,vcommit)))
+ (file-name (git-file-name name version))
+ (sha256
+ (base32
+ ,(if (pair? ref)
+ (bytevector->nix-base32-string
+ (file-hash* directory
+ ;; 'git-fetch' already filtered out the '.git' directory.
+ #:select? (const #true)
+ #:recursive? #true))
+ #f))))
+ directory))))
+
+(define* (git->origin repo-url ref #:optional ref->commit)
+ "Returns a generated `origin' block of a package depending on the git source
+control system. REPO-URL or REF can be null. REF->COMMIT can be a function or
+#t, in which case the commit matching ref is used. If REF->COMMIT is not used,
+the value inside REF is used."
+ (let-values (((origin _) (git->origin+dir repo-url ref ref->commit)))
+ origin))
(define %spdx-license-identifiers
;; https://spdx.org/licenses/
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 2023 05:51
[PATCH 03/21] import: juliahub: Add support for native-inputs.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-3-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 105 ++++++++++++++++++++++++---------------
1 file changed, 64 insertions(+), 41 deletions(-)

Toggle diff (167 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index efe6abbb24..4544dee980 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -20,13 +20,14 @@ (define-module (guix import juliahub)
#:use-module (ice-9 textual-ports)
#:use-module (ice-9 regex)
#:use-module (ice-9 match)
+ #:use-module (ice-9 streams)
#:use-module (ice-9 string-fun)
- #:use-module (srfi srfi-9)
#:use-module (srfi srfi-1)
+ #:use-module (srfi srfi-9)
+ #:use-module (srfi srfi-11)
#:use-module (guix http-client)
#:use-module (guix import utils)
#:use-module (guix import json)
- #:use-module (guix base16)
#:use-module (guix base32)
#:use-module (guix packages)
#:use-module (guix upstream)
@@ -66,19 +67,42 @@ (define (json->juliahub-direct-dependencies vector)
#f)))
(vector->list vector))))
+(define (ini-list->extra-dependencies lst)
+ (match lst
+ (('(extras) ooo ...)
+ (extra-list->extra-dependencies ooo))
+ (((tag) ooo ...)
+ (ini-list->extra-dependencies ooo))
+ ((attribute '= value ooo ...)
+ (ini-list->extra-dependencies ooo))
+ ('()
+ '())))
+
+(define (extra-list->extra-dependencies lst)
+ (match lst
+ ((attribute '= value ooo ...)
+ `(,(symbol->string attribute) ,@(extra-list->extra-dependencies ooo)))
+ (((tag) ooo ...)
+ '())
+ ('()
+ '())))
+
+(define (parse-extra-dependencies directory)
+ (let* ((port (open-input-file (string-append directory "/Project.toml")))
+ (ini-list (stream->list (port->stream port read))))
+ (close-port port)
+ (ini-list->extra-dependencies ini-list)))
+
;; Julia package.
(define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
json->juliahub-package
(homepage juliahub-package-homepage) ;string
(readme juliahub-package-readme) ;string
- ;; (slug juliahub-package-slug) ;string
(version juliahub-package-version) ;string
(description juliahub-package-description) ;string
- (dependencies
- juliahub-package-dependencies "deps"
+ (direct-dependencies
+ juliahub-package-direct-dependencies "deps"
json->juliahub-direct-dependencies) ;list of <juliahub-dependency>
- ;; (lambda (vector)
- ;; (map json->juliahub-dependency (vector->list vector))))
(url juliahub-package-url) ;string
(uuid juliahub-package-uuid) ;string
(license juliahub-package-license)) ;string
@@ -90,7 +114,6 @@ (define-json-mapping <juliahub-dependency>
(name juliahub-dependency-name) ;string
(uuid juliahub-dependency-uuid) ;string
(versions juliahub-dependency-versions "versions" vector->list)) ;list of strings
- ;; (slug juliahub-dependency-slug) ;string
(define (julia-name->guix-name name)
(string-append "julia-" (snake-case name)))
@@ -100,27 +123,25 @@ (define* (juliahub-fetch name #:key (version #f))
(and=> (json-fetch (string-append (juliahub-url name) "pkg.json"))
json->juliahub-package))
-(define (make-julia-sexp name version uri hash home-page synopsis description
- dependencies licenses)
+(define (make-julia-sexp name source home-page synopsis description
+ direct-dependencies test-dependencies-names licenses)
"Return the `package' s-expression for a Julia package with the given NAME,
-VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DEPENDENCIES, and LICENSES."
+VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DEPENDENCIES,
+TEST-DEPENDENCIES-NAMES and LICENSES."
`(package
(name ,(julia-name->guix-name name))
(version ,version)
- (source (origin
- (method url-fetch)
- (uri ,uri)
- (sha256
- (base32
- "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5"
- ;; ,(bytevector->nix-base32-string hash)
- ))))
+ (source ,source)
(build-system julia-build-system)
- ,@(if (null? dependencies)
+ ,@(if (null? direct-dependencies)
'()
- `((inputs
+ `((propagated-inputs
(list ,@(map (compose julia-name->guix-name juliahub-dependency-name)
- dependencies)))))
+ direct-dependencies)))))
+ ,@(if (null? test-dependencies-names)
+ '()
+ `((native-inputs
+ (list ,@(map julia-name->guix-name test-dependencies-names)))))
(synopsis ,synopsis)
(description ,description)
(home-page ,home-page)
@@ -135,26 +156,28 @@ (define* (juliahub->guix-package package-name
`package' s-expression corresponding to that package, or #f on failure.
Optionally include a VERSION string to fetch a specific version juliahub."
(let ((package (if version
- (juliahub-fetch package-name version)
- (juliahub-fetch package-name))))
+ (juliahub-fetch package-name version)
+ (juliahub-fetch package-name))))
(if package
- (let* ((dependencies-names
- (map juliahub-dependency-name
- (juliahub-package-dependencies package)))
- (licenses
- (map spdx-string->license
- (list (juliahub-package-license package)))))
- (values (make-julia-sexp
- package-name
- (juliahub-package-version package)
- (juliahub-package-url package)
- "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5"
- (juliahub-package-homepage package)
- (juliahub-package-description package)
- (beautify-description (juliahub-package-readme package))
- (juliahub-package-dependencies package)
- licenses)
- dependencies-names))
+ (let-values (((source directory)
+ (git->origin+dir url `(tag-or-commit . ,package-version))))
+ (let* ((dependencies-names
+ (map juliahub-dependency-name
+ (juliahub-package-direct-dependencies package)))
+ (licenses
+ (map spdx-string->license
+ (list (juliahub-package-license package))))
+ (test-dependencies-names (parse-extra-dependencies directory)))
+ (values (make-julia-sexp
+ package-name
+ source
+ (juliahub-package-homepage package)
+ (juliahub-package-description package)
+ (beautify-description (juliahub-package-readme package))
+ (juliahub-package-direct-dependencies package)
+ test-dependencies-names
+ licenses)
+ (append dependencies-names test-dependencies))))
(values #f '()))))
(define* (import-release package #:key (version #f))
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 2023 05:51
[PATCH 04/21] import: juliahub: Correct source parsing.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-4-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)

Toggle diff (19 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 4544dee980..4c3ceed109 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -160,7 +160,11 @@ (define* (juliahub->guix-package package-name
(juliahub-fetch package-name))))
(if package
(let-values (((source directory)
- (git->origin+dir url `(tag-or-commit . ,package-version))))
+ (git->origin+dir
+ (juliahub-package-url package)
+ `(tag-or-commit
+ . ,(string-append
+ "v" (juliahub-package-version package))))))
(let* ((dependencies-names
(map juliahub-dependency-name
(juliahub-package-direct-dependencies package)))
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 2023 05:51
[PATCH 05/21] import: juliahub: Add indirect dependencies.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-5-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 15 ++++++++++++++-
1 file changed, 14 insertions(+), 1 deletion(-)

Toggle diff (42 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 4c3ceed109..fb361a0044 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -67,6 +67,16 @@ (define (json->juliahub-direct-dependencies vector)
#f)))
(vector->list vector))))
+(define (json->juliahub-indirect-dependencies vector)
+ (if (vector? vector)
+ (filter-map
+ (lambda (el)
+ (let ((dep (json->juliahub-dependency el)))
+ (if (not (juliahub-dependency-direct? dep))
+ dep
+ #f)))
+ (vector->list vector))))
+
(define (ini-list->extra-dependencies lst)
(match lst
(('(extras) ooo ...)
@@ -103,6 +113,9 @@ (define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
(direct-dependencies
juliahub-package-direct-dependencies "deps"
json->juliahub-direct-dependencies) ;list of <juliahub-dependency>
+ (indirect-dependencies
+ juliahub-package-indirect-dependencies "deps"
+ json->juliahub-indirect-dependencies) ;list of <juliahub-dependency>
(url juliahub-package-url) ;string
(uuid juliahub-package-uuid) ;string
(license juliahub-package-license)) ;string
@@ -181,7 +194,7 @@ (define* (juliahub->guix-package package-name
(juliahub-package-direct-dependencies package)
test-dependencies-names
licenses)
- (append dependencies-names test-dependencies))))
+ (append dependencies-names test-dependencies-names))))
(values #f '()))))
(define* (import-release package #:key (version #f))
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 2023 05:51
[PATCH 06/21] import: juliahub: Add updater and recursive-importer.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-6-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 22 +++++++++++++++++-----
1 file changed, 17 insertions(+), 5 deletions(-)

Toggle diff (54 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index fb361a0044..c38c830caa 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -34,7 +34,9 @@ (define-module (guix import juliahub)
#:use-module (json)
#:use-module ((guix licenses) #:prefix license:)
- #:export (juliahub->guix-package))
+ #:export (juliahub->guix-package
+ %juliahub-updater
+ juliahub-recursive-import))
(define (juliahub-uri name)
(let* ((url (string-append "https://docs.juliahub.com/" name "/"))
@@ -197,16 +199,26 @@ (define* (juliahub->guix-package package-name
(append dependencies-names test-dependencies-names))))
(values #f '()))))
+(define (guix-package->juliahub-name package)
+ (let* ((url (juliahub-package-url package))
+ (git-name (car (last-pair (string-split url #\/))))
+ (ungitted-name (if (string-suffix? ".git" git-name)
+ (string-drop-right git-name 4)
+ git-name))
+ (package-name (if (string-suffix? ".jl" ungitted-name)
+ (string-drop-right ungitted-name 4)
+ ungitted-name)))
+ package-name))
+
(define* (import-release package #:key (version #f))
"Return an <upstream-source> for the latest release of PACKAGE."
(let* ((package-name (guix-package->juliahub-name package))
(package (juliahub-fetch package-name))
- (version (or version (juliahub-version gem)))
- (url (rubyjuliahubs-uri gem-name version)))
+ (version (or version (juliahub-package-version package))))
(upstream-source
(package (package-name package))
(version version)
- (urls (list url)))))
+ (urls (list (juliahub-package-url package))))))
(define %juliahub-updater
(upstream-updater
@@ -219,5 +231,5 @@ (define* (juliahub-recursive-import package-name #:optional version)
(recursive-import package-name
#:repo '()
#:repo->guix-package juliahub->guix-package
- #:guix-name ruby-package-name
+ #:guix-name julia-name->guix-name
#:version version))
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 2023 05:51
[PATCH 07/21] import: juliahub: Filter out julia stdlibs.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-7-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 49 ++++++++++++++++++++++++++++++++++++++--
1 file changed, 47 insertions(+), 2 deletions(-)

Toggle diff (76 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index c38c830caa..af08f3d698 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -38,6 +38,47 @@ (define-module (guix import juliahub)
%juliahub-updater
juliahub-recursive-import))
+;; To update, see file sysimg.jl
+(define %julia-stdlibs
+ (list "julia"
+ "ArgTools"
+ "Artifacts"
+ "Base64"
+ "CRC32c"
+ "FileWatching"
+ "Libdl"
+ "Logging"
+ "Mmap"
+ "NetworkOptions"
+ "SHA"
+ "Serialization"
+ "Sockets"
+ "Unicode"
+ "DelimitedFiles"
+ "LinearAlgebra"
+ "Markdown"
+ "Printf"
+ "Random"
+ "Tar"
+ "Dates"
+ "Distributed"
+ "Future"
+ "InteractiveUtils"
+ "LibGit2"
+ "Profile"
+ "SparseArrays"
+ "UUIDs"
+ "REPL"
+ "SharedArrays"
+ "Statistics"
+ "SuiteSparse"
+ "TOML"
+ "Test"
+ "LibCURL"
+ "Downloads"
+ "Pkg"
+ "LazyArtifacts"))
+
(define (juliahub-uri name)
(let* ((url (string-append "https://docs.juliahub.com/" name "/"))
(port (http-fetch url #:text? #t))
@@ -64,7 +105,9 @@ (define (json->juliahub-direct-dependencies vector)
(filter-map
(lambda (el)
(let ((dep (json->juliahub-dependency el)))
- (if (juliahub-dependency-direct? dep)
+ (if (and (juliahub-dependency-direct? dep)
+ (not (member (juliahub-dependency-name dep)
+ %julia-stdlibs)))
dep
#f)))
(vector->list vector))))
@@ -74,7 +117,9 @@ (define (json->juliahub-indirect-dependencies vector)
(filter-map
(lambda (el)
(let ((dep (json->juliahub-dependency el)))
- (if (not (juliahub-dependency-direct? dep))
+ (if (and (not (juliahub-dependency-direct? dep))
+ (not (member (juliahub-dependency-name dep)
+ %julia-stdlibs)))
dep
#f)))
(vector->list vector))))
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 2023 05:51
[PATCH 08/21] import: juliahub: Simplify juliahub dependency management.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-8-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 42 ++++++++++++++--------------------------
1 file changed, 14 insertions(+), 28 deletions(-)

Toggle diff (80 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index af08f3d698..b1eeb736a8 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -100,26 +100,13 @@ (define (juliahub-slug-version name)
(latest-version (string-drop uri 6)))
`(,slug ,latest-version)))
-(define (json->juliahub-direct-dependencies vector)
+(define (json->juliahub-dependencies vector)
(if (vector? vector)
(filter-map
(lambda (el)
(let ((dep (json->juliahub-dependency el)))
- (if (and (juliahub-dependency-direct? dep)
- (not (member (juliahub-dependency-name dep)
- %julia-stdlibs)))
- dep
- #f)))
- (vector->list vector))))
-
-(define (json->juliahub-indirect-dependencies vector)
- (if (vector? vector)
- (filter-map
- (lambda (el)
- (let ((dep (json->juliahub-dependency el)))
- (if (and (not (juliahub-dependency-direct? dep))
- (not (member (juliahub-dependency-name dep)
- %julia-stdlibs)))
+ (if (not (member (juliahub-dependency-name dep)
+ %julia-stdlibs))
dep
#f)))
(vector->list vector))))
@@ -157,12 +144,9 @@ (define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
(readme juliahub-package-readme) ;string
(version juliahub-package-version) ;string
(description juliahub-package-description) ;string
- (direct-dependencies
- juliahub-package-direct-dependencies "deps"
- json->juliahub-direct-dependencies) ;list of <juliahub-dependency>
- (indirect-dependencies
- juliahub-package-indirect-dependencies "deps"
- json->juliahub-indirect-dependencies) ;list of <juliahub-dependency>
+ (dependencies
+ juliahub-package-dependencies "deps"
+ json->juliahub-dependencies) ;list of <juliahub-dependency>
(url juliahub-package-url) ;string
(uuid juliahub-package-uuid) ;string
(license juliahub-package-license)) ;string
@@ -184,7 +168,7 @@ (define* (juliahub-fetch name #:key (version #f))
json->juliahub-package))
(define (make-julia-sexp name source home-page synopsis description
- direct-dependencies test-dependencies-names licenses)
+ dependencies test-dependencies-names licenses)
"Return the `package' s-expression for a Julia package with the given NAME,
VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DEPENDENCIES,
TEST-DEPENDENCIES-NAMES and LICENSES."
@@ -193,11 +177,13 @@ (define (make-julia-sexp name source home-page synopsis description
(version ,version)
(source ,source)
(build-system julia-build-system)
- ,@(if (null? direct-dependencies)
- '()
- `((propagated-inputs
- (list ,@(map (compose julia-name->guix-name juliahub-dependency-name)
- direct-dependencies)))))
+ ,@(let ((direct-dependencies
+ (filter julia-dependency-direct? dependencies)))
+ (if (null? direct-dependencies)
+ '()
+ `((propagated-inputs
+ (list ,@(map (compose julia-name->guix-name juliahub-dependency-name)
+ direct-dependencies))))))
,@(if (null? test-dependencies-names)
'()
`((native-inputs
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 2023 05:51
[PATCH 09/21] import: juliahub: Improve dependency management.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-9-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 25 +++++++++++++------------
1 file changed, 13 insertions(+), 12 deletions(-)

Toggle diff (60 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index b1eeb736a8..fc25ba1d42 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -168,22 +168,20 @@ (define* (juliahub-fetch name #:key (version #f))
json->juliahub-package))
(define (make-julia-sexp name source home-page synopsis description
- dependencies test-dependencies-names licenses)
+ direct-dependencies test-dependencies-names licenses)
"Return the `package' s-expression for a Julia package with the given NAME,
-VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DEPENDENCIES,
+VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DIRECT-DEPENDENCIES,
TEST-DEPENDENCIES-NAMES and LICENSES."
`(package
(name ,(julia-name->guix-name name))
(version ,version)
(source ,source)
(build-system julia-build-system)
- ,@(let ((direct-dependencies
- (filter julia-dependency-direct? dependencies)))
- (if (null? direct-dependencies)
- '()
- `((propagated-inputs
- (list ,@(map (compose julia-name->guix-name juliahub-dependency-name)
- direct-dependencies))))))
+ ,@(if (null? direct-dependencies)
+ '()
+ `((propagated-inputs
+ (list ,@(map (compose julia-name->guix-name juliahub-dependency-name)
+ direct-dependencies)))))
,@(if (null? test-dependencies-names)
'()
`((native-inputs
@@ -211,9 +209,12 @@ (define* (juliahub->guix-package package-name
`(tag-or-commit
. ,(string-append
"v" (juliahub-package-version package))))))
- (let* ((dependencies-names
+ (let* ((direct-dependencies
+ (filter juliahub-dependency-direct?
+ (juliahub-package-dependencies package)))
+ (dependencies-names
(map juliahub-dependency-name
- (juliahub-package-direct-dependencies package)))
+ direct-dependencies))
(licenses
(map spdx-string->license
(list (juliahub-package-license package))))
@@ -224,7 +225,7 @@ (define* (juliahub->guix-package package-name
(juliahub-package-homepage package)
(juliahub-package-description package)
(beautify-description (juliahub-package-readme package))
- (juliahub-package-direct-dependencies package)
+ direct-dependencies
test-dependencies-names
licenses)
(append dependencies-names test-dependencies-names))))
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 2023 05:51
[PATCH 11/21] import: juliahub: Improve test dependencies parsing.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-11-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 30 +++++++++++++-----------------
1 file changed, 13 insertions(+), 17 deletions(-)

Toggle diff (59 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 5327e92325..2ea461b72a 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -154,31 +154,27 @@ (define (json->juliahub-dependencies vector)
#f)))
(vector->list vector))))
-(define (ini-list->extra-dependencies lst)
+(define (ini-list->test-dependencies lst)
(match lst
- (('(extras) ooo ...)
- (extra-list->extra-dependencies ooo))
- (((tag) ooo ...)
- (ini-list->extra-dependencies ooo))
- ((attribute '= value ooo ...)
- (ini-list->extra-dependencies ooo))
+ (('test '= ooo ...)
+ `(,(caar ooo) ,@(test-list->test-dependencies (cdar ooo))))
+ ((value ooo ...)
+ (ini-list->test-dependencies ooo))
('()
- '())))
+ '())))
-(define (extra-list->extra-dependencies lst)
+(define (test-list->test-dependencies lst)
(match lst
- ((attribute '= value ooo ...)
- `(,(symbol->string attribute) ,@(extra-list->extra-dependencies ooo)))
- (((tag) ooo ...)
- '())
+ ((('unquote value) ooo ...)
+ `(,value ,@(test-list->test-dependencies ooo)))
('()
- '())))
+ '())))
-(define (parse-extra-dependencies directory)
+(define (parse-test-dependencies directory)
(let* ((port (open-input-file (string-append directory "/Project.toml")))
(ini-list (stream->list (port->stream port read))))
(close-port port)
- (ini-list->extra-dependencies ini-list)))
+ (ini-list->test-dependencies ini-list)))
;; Julia package.
(define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
@@ -268,7 +264,7 @@ (define* (juliahub->guix-package package-name
(licenses
(map spdx-string->license
(list (juliahub-package-license package))))
- (test-dependencies-names (parse-extra-dependencies directory)))
+ (test-dependencies-names (parse-test-dependencies directory)))
(values (make-julia-sexp
package-name
source
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 2023 05:51
[PATCH 10/21] import: juliahub: Add functions to parse the git repo for a git tag.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-10-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 62 ++++++++++++++++++++++++++++++++++++----
1 file changed, 56 insertions(+), 6 deletions(-)

Toggle diff (105 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index fc25ba1d42..5327e92325 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -26,6 +26,7 @@ (define-module (guix import juliahub)
#:use-module (srfi srfi-9)
#:use-module (srfi srfi-11)
#:use-module (guix http-client)
+ #:use-module (guix git)
#:use-module (guix import utils)
#:use-module (guix import json)
#:use-module (guix base32)
@@ -38,6 +39,48 @@ (define-module (guix import juliahub)
%juliahub-updater
juliahub-recursive-import))
+;; Juliahub may be more up-to-date than the General registry or the actual git
+;; tag (it seems around 6 hours pass between the time a commit is supplied to
+;; JuliaRegistrator as a release, and the time Julia TagBot Github Action makes
+;; the git tag). We have no simple way to get the commit of the latest-version.
+;; Thus the best simple thing we can do is get the latest-git-version, and
+;; import this version instead. We do this by parsing Package.toml in the General
+;; registry, and then getting the refs of the git repo supplied by this
+;; file. Parsing this file is also necessary if the package is in a subdir of a
+;; git repository, because the information isn't present in Juliahub.
+;; There's a last case where some Julia packages are not based on a particular
+;; git tag, and where looking for the commit is tedious and artificial. In these
+;; cases, we introduce the tree-commit which is available in the Versions.toml
+;; file in the General repository. This is equivalent to a commit, since we have
+;; a unique hash of the listing of files and directories, thus it can be used to
+;; identify the state of a repository.
+
+(define %general-base-url
+ "https://raw.githubusercontent.com/JuliaRegistries/General/master/")
+
+(define (general-url package-name file)
+ (let ((folder (string-capitalize (string-take package-name 1))))
+ (string-append
+ %general-base-url folder "/" package-name "/" file)))
+
+(define (ini-list->alist lst)
+ (match lst
+ ((attribute '= value ooo ...)
+ `((,attribute . ,value) ,@(ini-list->alist ooo)))
+ ('()
+ '())))
+
+(define (ini-fetch url)
+ (let* ((port (http-fetch url #:text? #t))
+ (ini-list (stream->list (port->stream port read))))
+ (close-port port)
+ (ini-list->alist ini-list)))
+
+(define (latest-git-tag repo)
+ (let* ((last-ref (last (remote-refs repo #:tags? #t)))
+ (last-git-tag (last (string-split last-ref #\/))))
+ (string-drop last-git-tag 1)))
+
;; To update, see file sysimg.jl
(define %julia-stdlibs
(list "julia"
@@ -194,14 +237,21 @@ (define (make-julia-sexp name source home-page synopsis description
((license) (license->symbol license))
(_ `(list ,@(map license->symbol licenses)))))))
+;; TODO handle subdir case properly.
+
(define* (juliahub->guix-package package-name
#:key version #:allow-other-keys)
"Fetch the metadata for PACKAGE-NAME from juliahub.org, and return the
`package' s-expression corresponding to that package, or #f on failure.
Optionally include a VERSION string to fetch a specific version juliahub."
- (let ((package (if version
- (juliahub-fetch package-name version)
- (juliahub-fetch package-name))))
+ (let* ((package-toml (ini-fetch (general-url name "Package.toml")))
+ (subdir (assoc-ref package-toml 'subdir))
+ (tag (latest-git-tag (assoc-ref package-toml 'repo)))
+ (package (if version
+ (juliahub-fetch package-name version)
+ (if tag
+ (juliahub-fetch package-name tag)
+ (juliahub-fetch package-name)))))
(if package
(let-values (((source directory)
(git->origin+dir
@@ -233,13 +283,13 @@ (define* (juliahub->guix-package package-name
(define (guix-package->juliahub-name package)
(let* ((url (juliahub-package-url package))
- (git-name (car (last-pair (string-split url #\/))))
+ (git-name (last (string-split url #\/)))
(ungitted-name (if (string-suffix? ".git" git-name)
(string-drop-right git-name 4)
git-name))
(package-name (if (string-suffix? ".jl" ungitted-name)
- (string-drop-right ungitted-name 4)
- ungitted-name)))
+ (string-drop-right ungitted-name 4)
+ ungitted-name)))
package-name))
(define* (import-release package #:key (version #f))
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 2023 05:51
[PATCH 12/21] import: juliahub: Handle the case where we have a subdirectory.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-12-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)

Toggle diff (35 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 2ea461b72a..b646f93295 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -233,14 +233,12 @@ (define (make-julia-sexp name source home-page synopsis description
((license) (license->symbol license))
(_ `(list ,@(map license->symbol licenses)))))))
-;; TODO handle subdir case properly.
-
(define* (juliahub->guix-package package-name
#:key version #:allow-other-keys)
"Fetch the metadata for PACKAGE-NAME from juliahub.org, and return the
`package' s-expression corresponding to that package, or #f on failure.
Optionally include a VERSION string to fetch a specific version juliahub."
- (let* ((package-toml (ini-fetch (general-url name "Package.toml")))
+ (let* ((package-toml (ini-fetch (general-url package-name "Package.toml")))
(subdir (assoc-ref package-toml 'subdir))
(tag (latest-git-tag (assoc-ref package-toml 'repo)))
(package (if version
@@ -264,7 +262,11 @@ (define* (juliahub->guix-package package-name
(licenses
(map spdx-string->license
(list (juliahub-package-license package))))
- (test-dependencies-names (parse-test-dependencies directory)))
+ (test-dependencies-names
+ (if subdir
+ (parse-test-dependencies
+ (string-append subdir "/" directory))
+ (parse-test-dependencies directory))))
(values (make-julia-sexp
package-name
source
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 2023 05:51
[PATCH 13/21] import: juliahub: Add support for versions for juliahub-fetch.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-13-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 20 ++++++++++++--------
1 file changed, 12 insertions(+), 8 deletions(-)

Toggle diff (57 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index b646f93295..6ce0487dba 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -122,7 +122,7 @@ (define %julia-stdlibs
"Pkg"
"LazyArtifacts"))
-(define (juliahub-uri name)
+(define (juliahub-redirect-uri name)
(let* ((url (string-append "https://docs.juliahub.com/" name "/"))
(port (http-fetch url #:text? #t))
(_ (get-line port))
@@ -134,11 +134,11 @@ (define (juliahub-uri name)
(define (juliahub-url name)
(let* ((url (string-append "https://docs.juliahub.com/" name "/"))
- (uri (juliahub-uri name)))
+ (uri (juliahub-redirect-uri name)))
(string-append url uri "/")))
-(define (juliahub-slug-version name)
- (let* ((uri (juliahub-uri name))
+(define (juliahub-slug+version name)
+ (let* ((uri (juliahub-redirect-uri name))
(slug (string-take uri 5))
(latest-version (string-drop uri 6)))
`(,slug ,latest-version)))
@@ -203,8 +203,12 @@ (define (julia-name->guix-name name)
(define* (juliahub-fetch name #:key (version #f))
"Return a <juliahub-package> record for package NAME, or #f on failure."
- (and=> (json-fetch (string-append (juliahub-url name) "pkg.json"))
- json->juliahub-package))
+ (let ((url (if version
+ (string-append "https://docs.juliahub.com/" name "/"
+ (car (juliahub-slug+version name)) "/"
+ version "/pkg.json")
+ (string-append (juliahub-url name) "pkg.json"))))
+ (and=> (json-fetch url) json->juliahub-package)))
(define (make-julia-sexp name source home-page synopsis description
direct-dependencies test-dependencies-names licenses)
@@ -242,9 +246,9 @@ (define* (juliahub->guix-package package-name
(subdir (assoc-ref package-toml 'subdir))
(tag (latest-git-tag (assoc-ref package-toml 'repo)))
(package (if version
- (juliahub-fetch package-name version)
+ (juliahub-fetch package-name #:version version)
(if tag
- (juliahub-fetch package-name tag)
+ (juliahub-fetch package-name #:version tag)
(juliahub-fetch package-name)))))
(if package
(let-values (((source directory)
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 2023 05:51
[PATCH 14/21] import: juliahub: Filter out stdlibs from test-dependencies.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-14-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)

Toggle diff (16 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 6ce0487dba..1c7b029296 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -174,7 +174,8 @@ (define (parse-test-dependencies directory)
(let* ((port (open-input-file (string-append directory "/Project.toml")))
(ini-list (stream->list (port->stream port read))))
(close-port port)
- (ini-list->test-dependencies ini-list)))
+ (filter (lambda (x) (not (member x %julia-stdlibs)))
+ (ini-list->test-dependencies ini-list))))
;; Julia package.
(define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 2023 05:51
[PATCH 15/21] import: juliahub: More robust toml regex parser.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-15-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 29 ++++++++---------------------
1 file changed, 8 insertions(+), 21 deletions(-)

Toggle diff (53 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 1c7b029296..3985d8d0be 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -154,28 +154,15 @@ (define (json->juliahub-dependencies vector)
#f)))
(vector->list vector))))
-(define (ini-list->test-dependencies lst)
- (match lst
- (('test '= ooo ...)
- `(,(caar ooo) ,@(test-list->test-dependencies (cdar ooo))))
- ((value ooo ...)
- (ini-list->test-dependencies ooo))
- ('()
- '())))
-
-(define (test-list->test-dependencies lst)
- (match lst
- ((('unquote value) ooo ...)
- `(,value ,@(test-list->test-dependencies ooo)))
- ('()
- '())))
-
(define (parse-test-dependencies directory)
(let* ((port (open-input-file (string-append directory "/Project.toml")))
- (ini-list (stream->list (port->stream port read))))
+ (project.toml (get-string-all port))
+ (regex "\ntest = \\[.*\\]")
+ (deps (match:substring (string-match regex project.toml)))
+ (pure (string-delete (list->char-set (list #\" #\ )) deps)))
(close-port port)
(filter (lambda (x) (not (member x %julia-stdlibs)))
- (ini-list->test-dependencies ini-list))))
+ (string-split (string-drop (string-drop-right pure 1) 7) #\,))))
;; Julia package.
(define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
@@ -243,9 +230,9 @@ (define* (juliahub->guix-package package-name
"Fetch the metadata for PACKAGE-NAME from juliahub.org, and return the
`package' s-expression corresponding to that package, or #f on failure.
Optionally include a VERSION string to fetch a specific version juliahub."
- (let* ((package-toml (ini-fetch (general-url package-name "Package.toml")))
- (subdir (assoc-ref package-toml 'subdir))
- (tag (latest-git-tag (assoc-ref package-toml 'repo)))
+ (let* ((package.toml (ini-fetch (general-url package-name "Package.toml")))
+ (subdir (assoc-ref package.toml 'subdir))
+ (tag (latest-git-tag (assoc-ref package.toml 'repo)))
(package (if version
(juliahub-fetch package-name #:version version)
(if tag
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 2023 05:51
[PATCH 16/21] import: juliahub: Beautify description.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-16-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 60 +++++++++++++++++++++++++++++++++++++---
1 file changed, 56 insertions(+), 4 deletions(-)

Toggle diff (95 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 3985d8d0be..338f042441 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -32,8 +32,10 @@ (define-module (guix import juliahub)
#:use-module (guix base32)
#:use-module (guix packages)
#:use-module (guix upstream)
- #:use-module (json)
#:use-module ((guix licenses) #:prefix license:)
+ #:use-module (json)
+ #:use-module (htmlprag)
+ #:use-module (sxml transform)
#:export (juliahub->guix-package
%juliahub-updater
@@ -164,6 +166,53 @@ (define (parse-test-dependencies directory)
(filter (lambda (x) (not (member x %julia-stdlibs)))
(string-split (string-drop (string-drop-right pure 1) 7) #\,))))
+(define %juliahub-beautify-description-rules
+ `((h1 *preorder* . ,(lambda args #f))
+ (h2 *preorder* . ,(lambda args #f))
+ (h3 *preorder* . ,(lambda args #f))
+ (h4 *preorder* . ,(lambda args #f))
+ (hr *preorder* . ,(lambda args #f))
+ (span *preorder* . ,(lambda args #f))
+ (img *preorder* . ,(lambda args #f))
+ (pre *preorder* . ,(lambda args #f))
+ (div *preorder* . ,(lambda args #f))
+ (table *preorder* . ,(lambda args #f))
+ (imgalt *preorder* . ,(lambda args #f))
+ (@ *preorder* . ,(lambda args #f))
+ (*TOP* . ,(lambda args (cdr args)))
+ (p . ,(lambda args (cdr args)))
+ (em . ,(lambda args (cdr args)))
+ (strong . ,(lambda args (cdr args)))
+ (a . ,(lambda args
+ (match args
+ ((tag link ref)
+ (if ref ref #f))
+ (_ #f))))
+ (ul . ,(lambda args
+ `("@itemize" ,@(cdr args) "\n@end itemize")))
+ (ol . ,(lambda args
+ `("@enumerate" ,@(cdr args) "@end enumerate")))
+ (blockquote . ,(lambda args
+ `("@quotation" ,@(cdr args) "@end quotation")))
+ (li . ,(lambda args
+ `("\n@item" ,@(cdr args))))
+ (code . ,(lambda args
+ (string-append
+ "@code{"
+ (string-join (cdr args) " ")
+ "}")))
+ (*text* . ,(lambda (tag x) x))
+ (*default* . ,(lambda (tag . body)
+ (cons tag body)))))
+
+(define (juliahub-beautify-description description)
+ (string-join
+ (filter (lambda (x) (if (equal? x " ") #f x))
+ (flatten
+ (pre-post-order (html->sxml description)
+ %juliahub-beautify-description-rules)))
+ " "))
+
;; Julia package.
(define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
json->juliahub-package
@@ -211,8 +260,9 @@ (define (make-julia-sexp name source home-page synopsis description
,@(if (null? direct-dependencies)
'()
`((propagated-inputs
- (list ,@(map (compose julia-name->guix-name juliahub-dependency-name)
- direct-dependencies)))))
+ (list
+ ,@(map (compose julia-name->guix-name juliahub-dependency-name)
+ direct-dependencies)))))
,@(if (null? test-dependencies-names)
'()
`((native-inputs
@@ -264,7 +314,9 @@ (define* (juliahub->guix-package package-name
source
(juliahub-package-homepage package)
(juliahub-package-description package)
- (beautify-description (juliahub-package-readme package))
+ ((compose beautify-description
+ juliahub-beautify-description)
+ (juliahub-package-readme package))
direct-dependencies
test-dependencies-names
licenses)
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 2023 05:51
[PATCH 17/21] import: juliahub: Fix license management.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-17-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 15 +++++----------
1 file changed, 5 insertions(+), 10 deletions(-)

Toggle diff (51 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 338f042441..e4b26bea34 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -248,10 +248,10 @@ (define* (juliahub-fetch name #:key (version #f))
(and=> (json-fetch url) json->juliahub-package)))
(define (make-julia-sexp name source home-page synopsis description
- direct-dependencies test-dependencies-names licenses)
+ direct-dependencies test-dependencies-names license)
"Return the `package' s-expression for a Julia package with the given NAME,
VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DIRECT-DEPENDENCIES,
-TEST-DEPENDENCIES-NAMES and LICENSES."
+TEST-DEPENDENCIES-NAMES and LICENSE."
`(package
(name ,(julia-name->guix-name name))
(version ,version)
@@ -270,10 +270,8 @@ (define (make-julia-sexp name source home-page synopsis description
(synopsis ,synopsis)
(description ,description)
(home-page ,home-page)
- (license ,(match licenses
- (() #f)
- ((license) (license->symbol license))
- (_ `(list ,@(map license->symbol licenses)))))))
+ (license
+ ,(if license (spdx-string->license license) #f))))
(define* (juliahub->guix-package package-name
#:key version #:allow-other-keys)
@@ -301,9 +299,6 @@ (define* (juliahub->guix-package package-name
(dependencies-names
(map juliahub-dependency-name
direct-dependencies))
- (licenses
- (map spdx-string->license
- (list (juliahub-package-license package))))
(test-dependencies-names
(if subdir
(parse-test-dependencies
@@ -319,7 +314,7 @@ (define* (juliahub->guix-package package-name
(juliahub-package-readme package))
direct-dependencies
test-dependencies-names
- licenses)
+ (juliahub-package-license package))
(append dependencies-names test-dependencies-names))))
(values #f '()))))
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 2023 05:51
[PATCH 18/21] import: juliahub: Fix version management.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-18-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)

Toggle diff (27 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index e4b26bea34..94d4ae8233 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -247,10 +247,10 @@ (define* (juliahub-fetch name #:key (version #f))
(string-append (juliahub-url name) "pkg.json"))))
(and=> (json-fetch url) json->juliahub-package)))
-(define (make-julia-sexp name source home-page synopsis description
+(define (make-julia-sexp name version source home-page synopsis description
direct-dependencies test-dependencies-names license)
"Return the `package' s-expression for a Julia package with the given NAME,
-VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DIRECT-DEPENDENCIES,
+VERSION, SOURCE, HOME-PAGE, DESCRIPTION, DIRECT-DEPENDENCIES,
TEST-DEPENDENCIES-NAMES and LICENSE."
`(package
(name ,(julia-name->guix-name name))
@@ -306,6 +306,7 @@ (define* (juliahub->guix-package package-name
(parse-test-dependencies directory))))
(values (make-julia-sexp
package-name
+ (juliahub-package-version package)
source
(juliahub-package-homepage package)
(juliahub-package-description package)
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 2023 05:51
[PATCH 19/21] import: juliahub: Fix undefined homepages.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-19-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)

Toggle diff (25 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 94d4ae8233..06574db724 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -303,12 +303,16 @@ (define* (juliahub->guix-package package-name
(if subdir
(parse-test-dependencies
(string-append subdir "/" directory))
- (parse-test-dependencies directory))))
+ (parse-test-dependencies directory)))
+ (homepage (juliahub-package-homepage package)))
(values (make-julia-sexp
package-name
(juliahub-package-version package)
source
- (juliahub-package-homepage package)
+ (match homepage
+ ("" (juliahub-package-url package))
+ ((? string?) homepage)
+ (_ (juliahub-package-url package)))
(juliahub-package-description package)
((compose beautify-description
juliahub-beautify-description)
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 2023 05:51
[PATCH 20/21] import: utils: Rule out texinfo common syntax from @ escape.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-20-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 5 +----
guix/import/utils.scm | 8 +++++++-
2 files changed, 8 insertions(+), 5 deletions(-)

Toggle diff (37 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 06574db724..0d3f89ad61 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -197,10 +197,7 @@ (define %juliahub-beautify-description-rules
(li . ,(lambda args
`("\n@item" ,@(cdr args))))
(code . ,(lambda args
- (string-append
- "@code{"
- (string-join (cdr args) " ")
- "}")))
+ `("@code{" ,@(cdr args) "}")))
(*text* . ,(lambda (tag x) x))
(*default* . ,(lambda (tag . body)
(cons tag body)))))
diff --git a/guix/import/utils.scm b/guix/import/utils.scm
index 171dca54e8..5ed1dfd815 100644
--- a/guix/import/utils.scm
+++ b/guix/import/utils.scm
@@ -376,7 +376,13 @@ (define* (beautify-description description #:optional (length 80))
(cut string-trim-both <> #\')
;; Escape single @ to prevent it from being understood as
;; invalid Texinfo syntax.
- (cut regexp-substitute/global #f "@" <> 'pre "@@" 'post)))))
+ (lambda (word)
+ (if
+ (member word '("@itemize" "@item" "@end" "@quotation"
+ "@enumerate" "@code" "@code{"))
+ word
+ ((cut regexp-substitute/global
+ #f "@" <> 'pre "@@" 'post) word)))))))
(words
(string-tokenize (string-trim-both description)
(char-set-complement
--
2.39.2
N
N
Nicolas Graves wrote on 15 Mar 2023 05:51
[PATCH 21/21] import: juliahub: output package names as symbols.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20230315125130.23041-21-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)

Toggle diff (24 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 0d3f89ad61..3e5735b950 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -258,12 +258,15 @@ (define (make-julia-sexp name version source home-page synopsis description
'()
`((propagated-inputs
(list
- ,@(map (compose julia-name->guix-name juliahub-dependency-name)
+ ,@(map (compose string->symbol
+ julia-name->guix-name
+ juliahub-dependency-name)
direct-dependencies)))))
,@(if (null? test-dependencies-names)
'()
`((native-inputs
- (list ,@(map julia-name->guix-name test-dependencies-names)))))
+ (list ,@(map (compose string->symbol julia-name->guix-name)
+ test-dependencies-names)))))
(synopsis ,synopsis)
(description ,description)
(home-page ,home-page)
--
2.39.2
S
S
Simon Tournier wrote on 7 Apr 2023 09:14
Re: [bug#62202] [PATCH 0/21] Juliahub import script.
87v8i7k6uv.fsf@gmail.com
Hi Nicolas,

Sorry for the delay. (I was in holidays \o/ :-))

I am not able to apply the series. Could you rebase it or provide here
by email the commit against which the series apply. Thanks in advance.


On mer., 15 mars 2023 at 13:47, Nicolas Graves via Guix-patches via <guix-patches@gnu.org> wrote:

Toggle quote (3 lines)
> Took me quite more time than I would've liked, but I have a usable
> juliahub scheme import script!

Really cool! Thank you!


Toggle quote (12 lines)
> It seems there's still one edge case that isn't covered and revolves
> around when Julia packagers don't properly tag their git repos (I've
> only seen the case with SnoopPrecompile). There's the possibility to
> rely on tree commit hashes from the General repository (since this is a
> valid way to identify/store a git repo), but that needs some major
> changes in the way latest-repository-commit works. Otherwise, it needs
> to be done by hand. It might also not work for subpackages in
> directories that are up-to-date on juliahub but not yet on github, I
> haven't met this case yet.
>
> I'm sending a patch series in the coming minutes.

Well, I have not read all series. :-)

Cheers,
simon
L
L
Ludovic Courtès wrote on 8 Apr 2023 15:07
Re: bug#62202: [PATCH 0/21] Juliahub import script.
(name . Nicolas Graves)(address . ngraves@ngraves.fr)(address . 62202@debbugs.gnu.org)
87ile69geh.fsf@gnu.org
Hi!

Nicolas Graves <ngraves@ngraves.fr> skribis:

Toggle quote (18 lines)
> Took me quite more time than I would've liked, but I have a usable
> juliahub scheme import script!
>
> It seems there's still one edge case that isn't covered and revolves
> around when Julia packagers don't properly tag their git repos (I've
> only seen the case with SnoopPrecompile). There's the possibility to
> rely on tree commit hashes from the General repository (since this is a
> valid way to identify/store a git repo), but that needs some major
> changes in the way latest-repository-commit works. Otherwise, it needs
> to be done by hand. It might also not work for subpackages in
> directories that are up-to-date on juliahub but not yet on github, I
> haven't met this case yet.
>
> I'm sending a patch series in the coming minutes.
>
> It's detailed since I haven't swauased all commits, for readability, but
> I can squash it further if necessary.

I’ll let Simon comment on the actual code since I’m not a Julia person.
:-)

Some more general comments:

• Please make sure to document it in ‘doc/guix.texi’ under “Invoking
guix import”, following the same template as the others there.

• Please write ‘tests/juliahub.scm’. I recommend the same strategy as
‘tests/cpan.scm’, which is to mock the upstream HTTP server.

• Prefer (srfi srfi-41) over (ice-9 streams) (see rationale at

• Prefer (srfi srfi-71) over (srfi srfi-11) for multiple-value
bindings.

Thanks,
Ludo’.
L
L
Ludovic Courtès wrote on 14 Jun 2023 08:50
control message for bug #62202
(address . control@debbugs.gnu.org)
87ttvarqko.fsf@gnu.org
tags 62202 + moreinfo
quit
L
L
Ludovic Courtès wrote on 8 Aug 2023 08:24
Re: bug#62202: [PATCH 0/21] Juliahub import script.
(name . Nicolas Graves)(address . ngraves@ngraves.fr)
87cyzxr1zd.fsf_-_@gnu.org
Hello Nicolas & Simon!

What should we do about this importer? Looks like useful code to me!

Thanks,
Ludo’.

Ludovic Courtès <ludo@gnu.org> skribis:

Toggle quote (41 lines)
> Hi!
>
> Nicolas Graves <ngraves@ngraves.fr> skribis:
>
>> Took me quite more time than I would've liked, but I have a usable
>> juliahub scheme import script!
>>
>> It seems there's still one edge case that isn't covered and revolves
>> around when Julia packagers don't properly tag their git repos (I've
>> only seen the case with SnoopPrecompile). There's the possibility to
>> rely on tree commit hashes from the General repository (since this is a
>> valid way to identify/store a git repo), but that needs some major
>> changes in the way latest-repository-commit works. Otherwise, it needs
>> to be done by hand. It might also not work for subpackages in
>> directories that are up-to-date on juliahub but not yet on github, I
>> haven't met this case yet.
>>
>> I'm sending a patch series in the coming minutes.
>>
>> It's detailed since I haven't swauased all commits, for readability, but
>> I can squash it further if necessary.
>
> I’ll let Simon comment on the actual code since I’m not a Julia person.
> :-)
>
> Some more general comments:
>
> • Please make sure to document it in ‘doc/guix.texi’ under “Invoking
> guix import”, following the same template as the others there.
>
> • Please write ‘tests/juliahub.scm’. I recommend the same strategy as
> ‘tests/cpan.scm’, which is to mock the upstream HTTP server.
>
> • Prefer (srfi srfi-41) over (ice-9 streams) (see rationale at
> <https://srfi.schemers.org/srfi-41/srfi-41.html>).
>
> • Prefer (srfi srfi-71) over (srfi srfi-11) for multiple-value
> bindings.
>
> Thanks,
> Ludo’.
S
S
Simon Tournier wrote on 16 Aug 2023 08:43
Re: [bug#62202] [PATCH 0/21] Juliahub import script.
(address . 62202@debbugs.gnu.org)
87fs4jf0w6.fsf@gmail.com
Hi Ludo,

Sorry for the delay, I was again in holidays. :-)

On Tue, 08 Aug 2023 at 17:24, Ludovic Courtès <ludo@gnu.org> wrote:

Toggle quote (2 lines)
> What should we do about this importer? Looks like useful code to me!

Yeah, it looks nice. As I said [1], “I am not able to apply the
series.” And since the series is 21 patches, I have been lazy.




Cheers,
simon
G
G
Giovanni Biscuolo wrote on 15 Sep 2023 02:45
(name . Nicolas Graves)(address . ngraves@ngraves.fr)
87bke393cn.fsf@xelera.eu
Hello Nicolas,

Simon Tournier <zimon.toutoune@gmail.com> writes:

[...]

Toggle quote (3 lines)
> Yeah, it looks nice. As I said [1], “I am not able to apply the
> series.” And since the series is 21 patches, I have been lazy.

I just want to add that on 2023-09-13 Simon explained failed attempts to
apply this patch series in a message to guix-devel [1] (starting from
the string "I am not speaking on the vacuum of an hypothetical
problem").

I quote some output Simon got:

Toggle quote (13 lines)
>> Using the usual “git am -3s” from Emacs Debbugs, then I get:
>>
>> --8<---------------cut here---------------start------------->8---
>> Applying: import: juliahub: first script draft.
>> error: sha1 information is lacking or useless (guix/import/go.scm).
>> error: could not build fake ancestor
>> hint: Use 'git am --show-current-patch=diff' to see the failed patch
>> Patch failed at 0001 import: juliahub: first script draft.
>> When you have resolved this problem, run "git am --continue".
>> If you prefer to skip this patch, run "git am --skip" instead.
>> To restore the original branch and stop patching, run "git am --abort".
>> --8<---------------cut here---------------end--------------->8---

Please see the original message [1] for details.

Sorry I can't help resolving this problem right now and/or send an
updated patch series.

Nicolas could you please try to apply it yourself to see if you are
succeful and eventually send a V2 patch series?

Thanks! Gio'



--
Giovanni Biscuolo

Xelera IT Infrastructures
-----BEGIN PGP SIGNATURE-----

iQJABAEBCgAqFiEERcxjuFJYydVfNLI5030Op87MORIFAmUEJ8gMHGdAeGVsZXJh
LmV1AAoJENN9DqfOzDkSKf8P/3Zc1nmBjDbRLxGEOFBXFCrVEtc8iHkLIV+71ZkX
Tys4erk17kobBvU6/0UCLnHzj2S1MgK14KRizeKYKJXTs5vbZEaqpMTfZKZo/3aO
RViaysRN12KNBCuNamJxMQUpQJwdylOBdHxa6/dkUP6CAtMGnhREEJHVIiPcO6IX
HoYSfc+q5Zcx7lwGvOvgQs5fhwgON6XQsaFu8EteS7kIJbrHnoG2NZH08u9rSlMG
k1wWAxw1G5w2vkfgbQjwdUtzZ+Cj4wzvJePPgnBIGOkqmXk+8v4DxNDC6J2e3Z8c
n4epu3pj3XnChWeM7ok0rYVe4mVqB/5U4Wgcx9nmtLx2kTdVHIWnhlviuH3Zk6Nz
R4BUtofgo4MDHkjlcaEWAZCc3ZhHxTFdllf2+4eHDarr6i05ON6h6tq/T90UsVau
nP7ApWNtlmPzJ9F8C6McKCO49e+FJYx/V/s6LhOJBCG5Xk6fb3fioD/Mk4szLxcv
mqifzF5F+ahQbADr6X0CsNBneKartP5IodnUua9Jk07/bTOIdzZw+jpUm/L08R5M
1v3eKpnvVXUqTrGeYAoi/dvk+SsfQ+A4WW72fYyOhmwGEjKv9qbQWY6kZ4EfnlBX
BRJXvWYkDTPGv5mLYqmKtDqNbNa3jghI4SHvJsxm4y6q0LMes0M/dmhPjzG7f9e1
QvKF
=OEPM
-----END PGP SIGNATURE-----

N
N
Nicolas Graves wrote on 15 Sep 2023 06:32
(name . Giovanni Biscuolo)(address . g@xelera.eu)
87pm2j4l5d.fsf@ngraves.fr
On 2023-09-15 11:45, Giovanni Biscuolo wrote:

Toggle quote (37 lines)
> Hello Nicolas,
>
> Simon Tournier <zimon.toutoune@gmail.com> writes:
>
> [...]
>
>> Yeah, it looks nice. As I said [1], “I am not able to apply the
>> series.” And since the series is 21 patches, I have been lazy.
>
> I just want to add that on 2023-09-13 Simon explained failed attempts to
> apply this patch series in a message to guix-devel [1] (starting from
> the string "I am not speaking on the vacuum of an hypothetical
> problem").
>
> I quote some output Simon got:
>
>>> Using the usual “git am -3s” from Emacs Debbugs, then I get:
>>>
>>> --8<---------------cut here---------------start------------->8---
>>> Applying: import: juliahub: first script draft.
>>> error: sha1 information is lacking or useless (guix/import/go.scm).
>>> error: could not build fake ancestor
>>> hint: Use 'git am --show-current-patch=diff' to see the failed patch
>>> Patch failed at 0001 import: juliahub: first script draft.
>>> When you have resolved this problem, run "git am --continue".
>>> If you prefer to skip this patch, run "git am --skip" instead.
>>> To restore the original branch and stop patching, run "git am --abort".
>>> --8<---------------cut here---------------end--------------->8---
>
> Please see the original message [1] for details.
>
> Sorry I can't help resolving this problem right now and/or send an
> updated patch series.
>
> Nicolas could you please try to apply it yourself to see if you are
> succeful and eventually send a V2 patch series?

Hi Giovanni,

Sorry for the lack of work to get this merged. I must still have this
branch locally, I'll try to output a V2.

The issue I had with the use is that my factorization of some functions
broke the Go importer, so either I have to fix that or drop the
factorization and duplicate the code in the juliahub importer.

I'll try to work on that this weekend, thanks for asking.

Nicolas
Toggle quote (6 lines)
>
> Thanks! Gio'
>
>
> [1] id:874jjzfhx0.fsf@gmail.com https://yhetil.org/guix/874jjzfhx0.fsf@gmail.com/

--
Best regards,
Nicolas Graves
S
S
Simon Tournier wrote on 15 Sep 2023 07:01
(name . Nicolas Graves)(address . ngraves@ngraves.fr)
CAJ3okZ3Nok7UtC-i=nBq+whc_HxWz1UFLBr+L+ydftgPXxbhwA@mail.gmail.com
Hi Nicolas,

On Fri, 15 Sept 2023 at 15:32, Nicolas Graves <ngraves@ngraves.fr> wrote:

Toggle quote (3 lines)
> Sorry for the lack of work to get this merged. I must still have this
> branch locally, I'll try to output a V2.

Oh, thank you. I am really sorry for the burden.

Toggle quote (2 lines)
> I'll try to work on that this weekend, thanks for asking.

Previously, I have asked too and you have probably missed it. :-) And
I was in the mood to go via some boring manual work this week. I am
very happy if it can be avoided and you are able to find some v2.

Have a nice week-end.

Cheers,
simon
N
N
Nicolas Graves wrote on 18 Sep 2023 02:31
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)
87wmwnj095.fsf@ngraves.fr
On 2023-09-15 16:01, Simon Tournier wrote:

Toggle quote (17 lines)
> Hi Nicolas,
>
> On Fri, 15 Sept 2023 at 15:32, Nicolas Graves <ngraves@ngraves.fr> wrote:
>
>> Sorry for the lack of work to get this merged. I must still have this
>> branch locally, I'll try to output a V2.
>
> Oh, thank you. I am really sorry for the burden.
>
>> I'll try to work on that this weekend, thanks for asking.
>
> Previously, I have asked too and you have probably missed it. :-) And
> I was in the mood to go via some boring manual work this week. I am
> very happy if it can be avoided and you are able to find some v2.
>
> Have a nice week-end.

Couldn't find time this wkend, but it's on my backlog, I'll try to do
that soon.
Toggle quote (4 lines)
>
> Cheers,
> simon

--
Best regards,
Nicolas Graves
S
S
Simon Tournier wrote on 18 Sep 2023 11:03
[PATCH v2 01/23] DRAFT guix: import: go: Add optional transform-version to vcs->origin.
(address . 62202@debbugs.gnu.org)(name . Simon Tournier)(address . zimon.toutoune@gmail.com)
c50c63bce62fc13aca5b51aaddf4158d2108c8ee.1695060058.git.zimon.toutoune@gmail.com
---
guix/import/go.scm | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)

Toggle diff (18 lines)
diff --git a/guix/import/go.scm b/guix/import/go.scm
index 0357e6a1ebe7..d1c6f68d9c9a 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -513,7 +513,8 @@ (define* (git-checkout-hash url reference algorithm)
`(tag-or-commit . ,reference)))))
(file-hash* checkout #:algorithm algorithm #:recursive? #true)))
-(define (vcs->origin vcs-type vcs-repo-url version)
+(define* (vcs->origin vcs-type vcs-repo-url version
+ #:optional transform-version)
"Generate the `origin' block of a package depending on what type of source
control system is being used."
(case vcs-type

base-commit: 3d9ebc7b2ed24312fd6a0916c203f7b86d57753d
--
2.38.1
S
S
Simon Tournier wrote on 18 Sep 2023 11:03
[PATCH v2 02/23] DRAFT TODO guix: import: utils: Add git->origin+dir.
(address . 62202@debbugs.gnu.org)(name . Simon Tournier)(address . zimon.toutoune@gmail.com)
119a9c3cf93b5b5df0daf917bdddde9863051d09.1695060058.git.zimon.toutoune@gmail.com
---
guix/import/utils.scm | 2 ++
1 file changed, 2 insertions(+)

Toggle diff (15 lines)
diff --git a/guix/import/utils.scm b/guix/import/utils.scm
index 0cf52cdbde73..189facfcf823 100644
--- a/guix/import/utils.scm
+++ b/guix/import/utils.scm
@@ -63,6 +63,8 @@ (define-module (guix import utils)
url-fetch
guix-hash-url
+ git->origin+dir
+ ;; git->origin
package-names->package-inputs
maybe-inputs
--
2.38.1
S
S
Simon Tournier wrote on 18 Sep 2023 11:03
[PATCH v2 03/23] DRAFT TODO: guix: import: utils: Fix corner cases beautify-descritption.
(address . 62202@debbugs.gnu.org)(name . Simon Tournier)(address . zimon.toutoune@gmail.com)
58ed3520a130805887d797300094637ba16ef0a7.1695060058.git.zimon.toutoune@gmail.com
* guix/import/utils.scm (beautify-descritption): Fix.
---
guix/import/utils.scm | 10 ++++++++++
1 file changed, 10 insertions(+)

Toggle diff (23 lines)
diff --git a/guix/import/utils.scm b/guix/import/utils.scm
index 189facfcf823..01d9861f279c 100644
--- a/guix/import/utils.scm
+++ b/guix/import/utils.scm
@@ -339,6 +339,16 @@ (define* (beautify-description description #:optional (length 80))
;; Escape single @ to prevent it from being understood as
;; invalid Texinfo syntax.
(cut regexp-substitute/global #f "@" <> 'pre "@@" 'post)
+
+ ;; Remove cut above
+ ;; (lambda (word)
+ ;; (if
+ ;; (member word '("@itemize" "@item" "@end" "@quotation"
+ ;; "@enumerate" "@code" "@code{"))
+ ;; word
+ ;; ((cut regexp-substitute/global
+ ;; #f "@" <> 'pre "@@" 'post) word)))))))
+
;; Wrap camelCase or PascalCase words in @code{...}.
(lambda (word)
(let ((pattern (make-regexp "([A-Z][a-z]+[A-Z]|[a-z]+[A-Z])")))
--
2.38.1
S
S
Simon Tournier wrote on 18 Sep 2023 11:03
[PATCH v2 05/23] DRAFT import: juliahub: Add support for native-inputs.
(address . 62202@debbugs.gnu.org)
57feddb288135a1d5953340fcc5bfe9e3159e372.1695060058.git.zimon.toutoune@gmail.com
From: Nicolas Graves via Guix-patches via <guix-patches@gnu.org>

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
guix/import/juliahub.scm | 105 ++++++++++++++++++++++++---------------
1 file changed, 64 insertions(+), 41 deletions(-)

Toggle diff (167 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index efe6abbb2481..4544dee98016 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -20,13 +20,14 @@ (define-module (guix import juliahub)
#:use-module (ice-9 textual-ports)
#:use-module (ice-9 regex)
#:use-module (ice-9 match)
+ #:use-module (ice-9 streams)
#:use-module (ice-9 string-fun)
- #:use-module (srfi srfi-9)
#:use-module (srfi srfi-1)
+ #:use-module (srfi srfi-9)
+ #:use-module (srfi srfi-11)
#:use-module (guix http-client)
#:use-module (guix import utils)
#:use-module (guix import json)
- #:use-module (guix base16)
#:use-module (guix base32)
#:use-module (guix packages)
#:use-module (guix upstream)
@@ -66,19 +67,42 @@ (define (json->juliahub-direct-dependencies vector)
#f)))
(vector->list vector))))
+(define (ini-list->extra-dependencies lst)
+ (match lst
+ (('(extras) ooo ...)
+ (extra-list->extra-dependencies ooo))
+ (((tag) ooo ...)
+ (ini-list->extra-dependencies ooo))
+ ((attribute '= value ooo ...)
+ (ini-list->extra-dependencies ooo))
+ ('()
+ '())))
+
+(define (extra-list->extra-dependencies lst)
+ (match lst
+ ((attribute '= value ooo ...)
+ `(,(symbol->string attribute) ,@(extra-list->extra-dependencies ooo)))
+ (((tag) ooo ...)
+ '())
+ ('()
+ '())))
+
+(define (parse-extra-dependencies directory)
+ (let* ((port (open-input-file (string-append directory "/Project.toml")))
+ (ini-list (stream->list (port->stream port read))))
+ (close-port port)
+ (ini-list->extra-dependencies ini-list)))
+
;; Julia package.
(define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
json->juliahub-package
(homepage juliahub-package-homepage) ;string
(readme juliahub-package-readme) ;string
- ;; (slug juliahub-package-slug) ;string
(version juliahub-package-version) ;string
(description juliahub-package-description) ;string
- (dependencies
- juliahub-package-dependencies "deps"
+ (direct-dependencies
+ juliahub-package-direct-dependencies "deps"
json->juliahub-direct-dependencies) ;list of <juliahub-dependency>
- ;; (lambda (vector)
- ;; (map json->juliahub-dependency (vector->list vector))))
(url juliahub-package-url) ;string
(uuid juliahub-package-uuid) ;string
(license juliahub-package-license)) ;string
@@ -90,7 +114,6 @@ (define-json-mapping <juliahub-dependency>
(name juliahub-dependency-name) ;string
(uuid juliahub-dependency-uuid) ;string
(versions juliahub-dependency-versions "versions" vector->list)) ;list of strings
- ;; (slug juliahub-dependency-slug) ;string
(define (julia-name->guix-name name)
(string-append "julia-" (snake-case name)))
@@ -100,27 +123,25 @@ (define* (juliahub-fetch name #:key (version #f))
(and=> (json-fetch (string-append (juliahub-url name) "pkg.json"))
json->juliahub-package))
-(define (make-julia-sexp name version uri hash home-page synopsis description
- dependencies licenses)
+(define (make-julia-sexp name source home-page synopsis description
+ direct-dependencies test-dependencies-names licenses)
"Return the `package' s-expression for a Julia package with the given NAME,
-VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DEPENDENCIES, and LICENSES."
+VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DEPENDENCIES,
+TEST-DEPENDENCIES-NAMES and LICENSES."
`(package
(name ,(julia-name->guix-name name))
(version ,version)
- (source (origin
- (method url-fetch)
- (uri ,uri)
- (sha256
- (base32
- "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5"
- ;; ,(bytevector->nix-base32-string hash)
- ))))
+ (source ,source)
(build-system julia-build-system)
- ,@(if (null? dependencies)
+ ,@(if (null? direct-dependencies)
'()
- `((inputs
+ `((propagated-inputs
(list ,@(map (compose julia-name->guix-name juliahub-dependency-name)
- dependencies)))))
+ direct-dependencies)))))
+ ,@(if (null? test-dependencies-names)
+ '()
+ `((native-inputs
+ (list ,@(map julia-name->guix-name test-dependencies-names)))))
(synopsis ,synopsis)
(description ,description)
(home-page ,home-page)
@@ -135,26 +156,28 @@ (define* (juliahub->guix-package package-name
`package' s-expression corresponding to that package, or #f on failure.
Optionally include a VERSION string to fetch a specific version juliahub."
(let ((package (if version
- (juliahub-fetch package-name version)
- (juliahub-fetch package-name))))
+ (juliahub-fetch package-name version)
+ (juliahub-fetch package-name))))
(if package
- (let* ((dependencies-names
- (map juliahub-dependency-name
- (juliahub-package-dependencies package)))
- (licenses
- (map spdx-string->license
- (list (juliahub-package-license package)))))
- (values (make-julia-sexp
- package-name
- (juliahub-package-version package)
- (juliahub-package-url package)
- "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5"
- (juliahub-package-homepage package)
- (juliahub-package-description package)
- (beautify-description (juliahub-package-readme package))
- (juliahub-package-dependencies package)
- licenses)
- dependencies-names))
+ (let-values (((source directory)
+ (git->origin+dir url `(tag-or-commit . ,package-version))))
+ (let* ((dependencies-names
+ (map juliahub-dependency-name
+ (juliahub-package-direct-dependencies package)))
+ (licenses
+ (map spdx-string->license
+ (list (juliahub-package-license package))))
+ (test-dependencies-names (parse-extra-dependencies directory)))
+ (values (make-julia-sexp
+ package-name
+ source
+ (juliahub-package-homepage package)
+ (juliahub-package-description package)
+ (beautify-description (juliahub-package-readme package))
+ (juliahub-package-direct-dependencies package)
+ test-dependencies-names
+ licenses)
+ (append dependencies-names test-dependencies))))
(values #f '()))))
(define* (import-release package #:key (version #f))
--
2.38.1
S
S
Simon Tournier wrote on 18 Sep 2023 11:03
[PATCH v2 06/23] DRAFT import: juliahub: Correct source parsing.
(address . 62202@debbugs.gnu.org)
14e997a8b389bb18d62e00178a68fc5434869da2.1695060058.git.zimon.toutoune@gmail.com
From: Nicolas Graves via Guix-patches via <guix-patches@gnu.org>

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
guix/import/juliahub.scm | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)

Toggle diff (19 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 4544dee98016..4c3ceed10904 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -160,7 +160,11 @@ (define* (juliahub->guix-package package-name
(juliahub-fetch package-name))))
(if package
(let-values (((source directory)
- (git->origin+dir url `(tag-or-commit . ,package-version))))
+ (git->origin+dir
+ (juliahub-package-url package)
+ `(tag-or-commit
+ . ,(string-append
+ "v" (juliahub-package-version package))))))
(let* ((dependencies-names
(map juliahub-dependency-name
(juliahub-package-direct-dependencies package)))
--
2.38.1
S
S
Simon Tournier wrote on 18 Sep 2023 11:03
[PATCH v2 07/23] DRAFT import: juliahub: Add indirect dependencies.
(address . 62202@debbugs.gnu.org)
9582ebba5df585ec5c71e008e71aaefafc036753.1695060058.git.zimon.toutoune@gmail.com
From: Nicolas Graves via Guix-patches via <guix-patches@gnu.org>

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
guix/import/juliahub.scm | 15 ++++++++++++++-
1 file changed, 14 insertions(+), 1 deletion(-)

Toggle diff (42 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 4c3ceed10904..fb361a004435 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -67,6 +67,16 @@ (define (json->juliahub-direct-dependencies vector)
#f)))
(vector->list vector))))
+(define (json->juliahub-indirect-dependencies vector)
+ (if (vector? vector)
+ (filter-map
+ (lambda (el)
+ (let ((dep (json->juliahub-dependency el)))
+ (if (not (juliahub-dependency-direct? dep))
+ dep
+ #f)))
+ (vector->list vector))))
+
(define (ini-list->extra-dependencies lst)
(match lst
(('(extras) ooo ...)
@@ -103,6 +113,9 @@ (define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
(direct-dependencies
juliahub-package-direct-dependencies "deps"
json->juliahub-direct-dependencies) ;list of <juliahub-dependency>
+ (indirect-dependencies
+ juliahub-package-indirect-dependencies "deps"
+ json->juliahub-indirect-dependencies) ;list of <juliahub-dependency>
(url juliahub-package-url) ;string
(uuid juliahub-package-uuid) ;string
(license juliahub-package-license)) ;string
@@ -181,7 +194,7 @@ (define* (juliahub->guix-package package-name
(juliahub-package-direct-dependencies package)
test-dependencies-names
licenses)
- (append dependencies-names test-dependencies))))
+ (append dependencies-names test-dependencies-names))))
(values #f '()))))
(define* (import-release package #:key (version #f))
--
2.38.1
S
S
Simon Tournier wrote on 18 Sep 2023 11:03
[PATCH v2 08/23] DRAFT import: juliahub: Add updater and recursive-importer.
(address . 62202@debbugs.gnu.org)
26c9a5ca41ad1697225de11b5513fcfe04d26f71.1695060058.git.zimon.toutoune@gmail.com
From: Nicolas Graves via Guix-patches via <guix-patches@gnu.org>

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
guix/import/juliahub.scm | 22 +++++++++++++++++-----
1 file changed, 17 insertions(+), 5 deletions(-)

Toggle diff (54 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index fb361a004435..c38c830caaa0 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -34,7 +34,9 @@ (define-module (guix import juliahub)
#:use-module (json)
#:use-module ((guix licenses) #:prefix license:)
- #:export (juliahub->guix-package))
+ #:export (juliahub->guix-package
+ %juliahub-updater
+ juliahub-recursive-import))
(define (juliahub-uri name)
(let* ((url (string-append "https://docs.juliahub.com/" name "/"))
@@ -197,16 +199,26 @@ (define* (juliahub->guix-package package-name
(append dependencies-names test-dependencies-names))))
(values #f '()))))
+(define (guix-package->juliahub-name package)
+ (let* ((url (juliahub-package-url package))
+ (git-name (car (last-pair (string-split url #\/))))
+ (ungitted-name (if (string-suffix? ".git" git-name)
+ (string-drop-right git-name 4)
+ git-name))
+ (package-name (if (string-suffix? ".jl" ungitted-name)
+ (string-drop-right ungitted-name 4)
+ ungitted-name)))
+ package-name))
+
(define* (import-release package #:key (version #f))
"Return an <upstream-source> for the latest release of PACKAGE."
(let* ((package-name (guix-package->juliahub-name package))
(package (juliahub-fetch package-name))
- (version (or version (juliahub-version gem)))
- (url (rubyjuliahubs-uri gem-name version)))
+ (version (or version (juliahub-package-version package))))
(upstream-source
(package (package-name package))
(version version)
- (urls (list url)))))
+ (urls (list (juliahub-package-url package))))))
(define %juliahub-updater
(upstream-updater
@@ -219,5 +231,5 @@ (define* (juliahub-recursive-import package-name #:optional version)
(recursive-import package-name
#:repo '()
#:repo->guix-package juliahub->guix-package
- #:guix-name ruby-package-name
+ #:guix-name julia-name->guix-name
#:version version))
--
2.38.1
S
S
Simon Tournier wrote on 18 Sep 2023 11:03
[PATCH v2 09/23] DRAFT import: juliahub: Filter out julia stdlibs.
(address . 62202@debbugs.gnu.org)
be0f04de20abad380175b8f9509fed461e15690f.1695060058.git.zimon.toutoune@gmail.com
From: Nicolas Graves via Guix-patches via <guix-patches@gnu.org>

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
guix/import/juliahub.scm | 49 ++++++++++++++++++++++++++++++++++++++--
1 file changed, 47 insertions(+), 2 deletions(-)

Toggle diff (76 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index c38c830caaa0..af08f3d698d0 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -38,6 +38,47 @@ (define-module (guix import juliahub)
%juliahub-updater
juliahub-recursive-import))
+;; To update, see file sysimg.jl
+(define %julia-stdlibs
+ (list "julia"
+ "ArgTools"
+ "Artifacts"
+ "Base64"
+ "CRC32c"
+ "FileWatching"
+ "Libdl"
+ "Logging"
+ "Mmap"
+ "NetworkOptions"
+ "SHA"
+ "Serialization"
+ "Sockets"
+ "Unicode"
+ "DelimitedFiles"
+ "LinearAlgebra"
+ "Markdown"
+ "Printf"
+ "Random"
+ "Tar"
+ "Dates"
+ "Distributed"
+ "Future"
+ "InteractiveUtils"
+ "LibGit2"
+ "Profile"
+ "SparseArrays"
+ "UUIDs"
+ "REPL"
+ "SharedArrays"
+ "Statistics"
+ "SuiteSparse"
+ "TOML"
+ "Test"
+ "LibCURL"
+ "Downloads"
+ "Pkg"
+ "LazyArtifacts"))
+
(define (juliahub-uri name)
(let* ((url (string-append "https://docs.juliahub.com/" name "/"))
(port (http-fetch url #:text? #t))
@@ -64,7 +105,9 @@ (define (json->juliahub-direct-dependencies vector)
(filter-map
(lambda (el)
(let ((dep (json->juliahub-dependency el)))
- (if (juliahub-dependency-direct? dep)
+ (if (and (juliahub-dependency-direct? dep)
+ (not (member (juliahub-dependency-name dep)
+ %julia-stdlibs)))
dep
#f)))
(vector->list vector))))
@@ -74,7 +117,9 @@ (define (json->juliahub-indirect-dependencies vector)
(filter-map
(lambda (el)
(let ((dep (json->juliahub-dependency el)))
- (if (not (juliahub-dependency-direct? dep))
+ (if (and (not (juliahub-dependency-direct? dep))
+ (not (member (juliahub-dependency-name dep)
+ %julia-stdlibs)))
dep
#f)))
(vector->list vector))))
--
2.38.1
S
S
Simon Tournier wrote on 18 Sep 2023 11:03
[PATCH v2 04/23] DRAFT import: Add julia.
(address . 62202@debbugs.gnu.org)(name . Simon Tournier)(address . zimon.toutoune@gmail.com)
cc098ba63817ae293b4c0ba5d72bc61c11057f22.1695060058.git.zimon.toutoune@gmail.com
* guix/scripts/import.scm (importers): Add juliahub.
* guix/import/juliahub.scm: New file.
* guix/scripts/import/juliahub.scm: New file.
* Makefile.am: Add them.
---
Makefile.am | 2 +
guix/import/juliahub.scm | 183 +++++++++++++++++++++++++++++++
guix/scripts/import.scm | 2 +-
guix/scripts/import/juliahub.scm | 107 ++++++++++++++++++
4 files changed, 293 insertions(+), 1 deletion(-)
create mode 100644 guix/import/juliahub.scm
create mode 100644 guix/scripts/import/juliahub.scm

Toggle diff (337 lines)
diff --git a/Makefile.am b/Makefile.am
index 922913355c45..70b06c1a793f 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -289,6 +289,7 @@ MODULES = \
guix/import/hackage.scm \
guix/import/hexpm.scm \
guix/import/json.scm \
+ guix/import/juliahub.scm \
guix/import/kde.scm \
guix/import/launchpad.scm \
guix/import/minetest.scm \
@@ -344,6 +345,7 @@ MODULES = \
guix/scripts/import/hackage.scm \
guix/scripts/import/hexpm.scm \
guix/scripts/import/json.scm \
+ guix/scripts/import/juliahub.scm \
guix/scripts/import/minetest.scm \
guix/scripts/import/opam.scm \
guix/scripts/import/pypi.scm \
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
new file mode 100644
index 000000000000..efe6abbb2481
--- /dev/null
+++ b/guix/import/juliahub.scm
@@ -0,0 +1,183 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix import juliahub)
+ #:use-module (ice-9 textual-ports)
+ #:use-module (ice-9 regex)
+ #:use-module (ice-9 match)
+ #:use-module (ice-9 string-fun)
+ #:use-module (srfi srfi-9)
+ #:use-module (srfi srfi-1)
+ #:use-module (guix http-client)
+ #:use-module (guix import utils)
+ #:use-module (guix import json)
+ #:use-module (guix base16)
+ #:use-module (guix base32)
+ #:use-module (guix packages)
+ #:use-module (guix upstream)
+ #:use-module (json)
+ #:use-module ((guix licenses) #:prefix license:)
+
+ #:export (juliahub->guix-package))
+
+(define (juliahub-uri name)
+ (let* ((url (string-append "https://docs.juliahub.com/" name "/"))
+ (port (http-fetch url #:text? #t))
+ (_ (get-line port))
+ (meta (get-line port))
+ (regex "url=[a-zA-Z0-9]{5}\\/[0-9\\.]*")
+ (redirect (match:substring (string-match regex meta))))
+ (close-port port)
+ (string-drop redirect 4)))
+
+(define (juliahub-url name)
+ (let* ((url (string-append "https://docs.juliahub.com/" name "/"))
+ (uri (juliahub-uri name)))
+ (string-append url uri "/")))
+
+(define (juliahub-slug-version name)
+ (let* ((uri (juliahub-uri name))
+ (slug (string-take uri 5))
+ (latest-version (string-drop uri 6)))
+ `(,slug ,latest-version)))
+
+(define (json->juliahub-direct-dependencies vector)
+ (if (vector? vector)
+ (filter-map
+ (lambda (el)
+ (let ((dep (json->juliahub-dependency el)))
+ (if (juliahub-dependency-direct? dep)
+ dep
+ #f)))
+ (vector->list vector))))
+
+;; Julia package.
+(define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
+ json->juliahub-package
+ (homepage juliahub-package-homepage) ;string
+ (readme juliahub-package-readme) ;string
+ ;; (slug juliahub-package-slug) ;string
+ (version juliahub-package-version) ;string
+ (description juliahub-package-description) ;string
+ (dependencies
+ juliahub-package-dependencies "deps"
+ json->juliahub-direct-dependencies) ;list of <juliahub-dependency>
+ ;; (lambda (vector)
+ ;; (map json->juliahub-dependency (vector->list vector))))
+ (url juliahub-package-url) ;string
+ (uuid juliahub-package-uuid) ;string
+ (license juliahub-package-license)) ;string
+
+(define-json-mapping <juliahub-dependency>
+ make-juliahub-dependency juliahub-dependency?
+ json->juliahub-dependency
+ (direct? juliahub-dependency-direct? "direct") ;boolean
+ (name juliahub-dependency-name) ;string
+ (uuid juliahub-dependency-uuid) ;string
+ (versions juliahub-dependency-versions "versions" vector->list)) ;list of strings
+ ;; (slug juliahub-dependency-slug) ;string
+
+(define (julia-name->guix-name name)
+ (string-append "julia-" (snake-case name)))
+
+(define* (juliahub-fetch name #:key (version #f))
+ "Return a <juliahub-package> record for package NAME, or #f on failure."
+ (and=> (json-fetch (string-append (juliahub-url name) "pkg.json"))
+ json->juliahub-package))
+
+(define (make-julia-sexp name version uri hash home-page synopsis description
+ dependencies licenses)
+ "Return the `package' s-expression for a Julia package with the given NAME,
+VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DEPENDENCIES, and LICENSES."
+ `(package
+ (name ,(julia-name->guix-name name))
+ (version ,version)
+ (source (origin
+ (method url-fetch)
+ (uri ,uri)
+ (sha256
+ (base32
+ "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5"
+ ;; ,(bytevector->nix-base32-string hash)
+ ))))
+ (build-system julia-build-system)
+ ,@(if (null? dependencies)
+ '()
+ `((inputs
+ (list ,@(map (compose julia-name->guix-name juliahub-dependency-name)
+ dependencies)))))
+ (synopsis ,synopsis)
+ (description ,description)
+ (home-page ,home-page)
+ (license ,(match licenses
+ (() #f)
+ ((license) (license->symbol license))
+ (_ `(list ,@(map license->symbol licenses)))))))
+
+(define* (juliahub->guix-package package-name
+ #:key version #:allow-other-keys)
+ "Fetch the metadata for PACKAGE-NAME from juliahub.org, and return the
+`package' s-expression corresponding to that package, or #f on failure.
+Optionally include a VERSION string to fetch a specific version juliahub."
+ (let ((package (if version
+ (juliahub-fetch package-name version)
+ (juliahub-fetch package-name))))
+ (if package
+ (let* ((dependencies-names
+ (map juliahub-dependency-name
+ (juliahub-package-dependencies package)))
+ (licenses
+ (map spdx-string->license
+ (list (juliahub-package-license package)))))
+ (values (make-julia-sexp
+ package-name
+ (juliahub-package-version package)
+ (juliahub-package-url package)
+ "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5"
+ (juliahub-package-homepage package)
+ (juliahub-package-description package)
+ (beautify-description (juliahub-package-readme package))
+ (juliahub-package-dependencies package)
+ licenses)
+ dependencies-names))
+ (values #f '()))))
+
+(define* (import-release package #:key (version #f))
+ "Return an <upstream-source> for the latest release of PACKAGE."
+ (let* ((package-name (guix-package->juliahub-name package))
+ (package (juliahub-fetch package-name))
+ (version (or version (juliahub-version gem)))
+ (url (rubyjuliahubs-uri gem-name version)))
+ (upstream-source
+ (package (package-name package))
+ (version version)
+ (urls (list url)))))
+
+(define %juliahub-updater
+ (upstream-updater
+ (name 'juliahub)
+ (description "Updater for Juliahub packages")
+ (pred juliahub-package?)
+ (import import-release)))
+
+(define* (juliahub-recursive-import package-name #:optional version)
+ (recursive-import package-name
+ #:repo '()
+ #:repo->guix-package juliahub->guix-package
+ #:guix-name ruby-package-name
+ #:version version))
diff --git a/guix/scripts/import.scm b/guix/scripts/import.scm
index 4ddd8d46a121..74c26e347335 100644
--- a/guix/scripts/import.scm
+++ b/guix/scripts/import.scm
@@ -47,7 +47,7 @@ (define %standard-import-options '())
(define importers '("gnu" "pypi" "cpan" "hackage" "stackage" "egg" "elpa"
"gem" "go" "cran" "crate" "texlive" "json" "opam"
- "minetest" "elm" "hexpm"))
+ "minetest" "elm" "hexpm" "juliahub"))
(define (resolve-importer name)
(let ((module (resolve-interface
diff --git a/guix/scripts/import/juliahub.scm b/guix/scripts/import/juliahub.scm
new file mode 100644
index 000000000000..1317c67aa342
--- /dev/null
+++ b/guix/scripts/import/juliahub.scm
@@ -0,0 +1,107 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix scripts import juliahub)
+ #:use-module (guix ui)
+ #:use-module (guix utils)
+ #:use-module (guix scripts)
+ #:use-module (guix import juliahub)
+ #:use-module (guix scripts import)
+ #:use-module (srfi srfi-1)
+ #:use-module (srfi srfi-11)
+ #:use-module (srfi srfi-37)
+ #:use-module (ice-9 match)
+ #:use-module (ice-9 format)
+ #:use-module (ice-9 receive)
+ #:export (guix-import-juliahub))
+
+
+;;;
+;;; Command-line options.
+;;;
+
+(define %default-options
+ '())
+
+(define (show-help)
+ (display (G_ "Usage: guix import juliahub PACKAGE-NAME[@VERSION] Import and
+convert the Julia package for PACKAGE-NAME. Optionally, a version can be
+specified after the at-sign (@) character.\n"))
+ (display (G_ "
+ -h, --help display this help and exit"))
+ (display (G_ "
+ -V, --version display version information and exit"))
+ (display (G_ "
+ -r, --recursive generate package expressions for all Gem packages\
+ that are not yet in Guix"))
+ (newline)
+ (show-bug-report-information))
+
+(define %options
+ ;; Specification of the command-line options.
+ (cons* (option '(#\h "help") #f #f
+ (lambda args
+ (show-help)
+ (exit 0)))
+ (option '(#\V "version") #f #f
+ (lambda args
+ (show-version-and-exit "guix import gem")))
+ (option '(#\r "recursive") #f #f
+ (lambda (opt name arg result)
+ (alist-cons 'recursive #t result)))
+ %standard-import-options))
+
+
+;;;
+;;; Entry point.
+;;;
+
+(define (guix-import-juliahub . args)
+ (define (parse-options)
+ ;; Return the alist of option values.
+ (parse-command-line args %options (list %default-options)
+ #:build-options? #f))
+
+ (let* ((opts (parse-options))
+ (args (filter-map (match-lambda
+ (('argument . value)
+ value)
+ (_ #f))
+ (reverse opts))))
+ (match args
+ ((spec)
+ (receive (package-name package-version)
+ (package-name->name+version spec)
+ (let ((code (if (assoc-ref opts 'recursive)
+ (map (match-lambda
+ ((and ('package ('name name) . rest) pkg)
+ `(define-public ,(string->symbol name)
+ ,pkg))
+ (_ #f))
+ (juliahub-recursive-import package-name package-version))
+ (let ((sexp (juliahub->guix-package package-name #:version package-version)))
+ (if sexp sexp #f)))))
+ (match code
+ ((or #f '(#f))
+ (leave (G_ "failed to download meta-data for package '~a'~%")
+ package-name))
+ (_ code)))))
+ (()
+ (leave (G_ "too few arguments~%")))
+ ((many ...)
+ (leave (G_ "too many arguments~%"))))))
--
2.38.1
S
S
Simon Tournier wrote on 18 Sep 2023 11:03
[PATCH v2 10/23] DRAFT import: juliahub: Simplify juliahub dependency management.
(address . 62202@debbugs.gnu.org)
d21d6314221dd876b598a86a655b658821c3334f.1695060058.git.zimon.toutoune@gmail.com
From: Nicolas Graves via Guix-patches via <guix-patches@gnu.org>

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
guix/import/juliahub.scm | 42 ++++++++++++++--------------------------
1 file changed, 14 insertions(+), 28 deletions(-)

Toggle diff (80 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index af08f3d698d0..b1eeb736a824 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -100,26 +100,13 @@ (define (juliahub-slug-version name)
(latest-version (string-drop uri 6)))
`(,slug ,latest-version)))
-(define (json->juliahub-direct-dependencies vector)
+(define (json->juliahub-dependencies vector)
(if (vector? vector)
(filter-map
(lambda (el)
(let ((dep (json->juliahub-dependency el)))
- (if (and (juliahub-dependency-direct? dep)
- (not (member (juliahub-dependency-name dep)
- %julia-stdlibs)))
- dep
- #f)))
- (vector->list vector))))
-
-(define (json->juliahub-indirect-dependencies vector)
- (if (vector? vector)
- (filter-map
- (lambda (el)
- (let ((dep (json->juliahub-dependency el)))
- (if (and (not (juliahub-dependency-direct? dep))
- (not (member (juliahub-dependency-name dep)
- %julia-stdlibs)))
+ (if (not (member (juliahub-dependency-name dep)
+ %julia-stdlibs))
dep
#f)))
(vector->list vector))))
@@ -157,12 +144,9 @@ (define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
(readme juliahub-package-readme) ;string
(version juliahub-package-version) ;string
(description juliahub-package-description) ;string
- (direct-dependencies
- juliahub-package-direct-dependencies "deps"
- json->juliahub-direct-dependencies) ;list of <juliahub-dependency>
- (indirect-dependencies
- juliahub-package-indirect-dependencies "deps"
- json->juliahub-indirect-dependencies) ;list of <juliahub-dependency>
+ (dependencies
+ juliahub-package-dependencies "deps"
+ json->juliahub-dependencies) ;list of <juliahub-dependency>
(url juliahub-package-url) ;string
(uuid juliahub-package-uuid) ;string
(license juliahub-package-license)) ;string
@@ -184,7 +168,7 @@ (define* (juliahub-fetch name #:key (version #f))
json->juliahub-package))
(define (make-julia-sexp name source home-page synopsis description
- direct-dependencies test-dependencies-names licenses)
+ dependencies test-dependencies-names licenses)
"Return the `package' s-expression for a Julia package with the given NAME,
VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DEPENDENCIES,
TEST-DEPENDENCIES-NAMES and LICENSES."
@@ -193,11 +177,13 @@ (define (make-julia-sexp name source home-page synopsis description
(version ,version)
(source ,source)
(build-system julia-build-system)
- ,@(if (null? direct-dependencies)
- '()
- `((propagated-inputs
- (list ,@(map (compose julia-name->guix-name juliahub-dependency-name)
- direct-dependencies)))))
+ ,@(let ((direct-dependencies
+ (filter julia-dependency-direct? dependencies)))
+ (if (null? direct-dependencies)
+ '()
+ `((propagated-inputs
+ (list ,@(map (compose julia-name->guix-name juliahub-dependency-name)
+ direct-dependencies))))))
,@(if (null? test-dependencies-names)
'()
`((native-inputs
--
2.38.1
S
S
Simon Tournier wrote on 18 Sep 2023 11:03
[PATCH v2 11/23] DRAFT import: juliahub: Improve dependency management.
(address . 62202@debbugs.gnu.org)
9c97220836376a2e6de87f52b4a7b842581225ed.1695060058.git.zimon.toutoune@gmail.com
From: Nicolas Graves via Guix-patches via <guix-patches@gnu.org>

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
guix/import/juliahub.scm | 25 +++++++++++++------------
1 file changed, 13 insertions(+), 12 deletions(-)

Toggle diff (60 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index b1eeb736a824..fc25ba1d4220 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -168,22 +168,20 @@ (define* (juliahub-fetch name #:key (version #f))
json->juliahub-package))
(define (make-julia-sexp name source home-page synopsis description
- dependencies test-dependencies-names licenses)
+ direct-dependencies test-dependencies-names licenses)
"Return the `package' s-expression for a Julia package with the given NAME,
-VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DEPENDENCIES,
+VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DIRECT-DEPENDENCIES,
TEST-DEPENDENCIES-NAMES and LICENSES."
`(package
(name ,(julia-name->guix-name name))
(version ,version)
(source ,source)
(build-system julia-build-system)
- ,@(let ((direct-dependencies
- (filter julia-dependency-direct? dependencies)))
- (if (null? direct-dependencies)
- '()
- `((propagated-inputs
- (list ,@(map (compose julia-name->guix-name juliahub-dependency-name)
- direct-dependencies))))))
+ ,@(if (null? direct-dependencies)
+ '()
+ `((propagated-inputs
+ (list ,@(map (compose julia-name->guix-name juliahub-dependency-name)
+ direct-dependencies)))))
,@(if (null? test-dependencies-names)
'()
`((native-inputs
@@ -211,9 +209,12 @@ (define* (juliahub->guix-package package-name
`(tag-or-commit
. ,(string-append
"v" (juliahub-package-version package))))))
- (let* ((dependencies-names
+ (let* ((direct-dependencies
+ (filter juliahub-dependency-direct?
+ (juliahub-package-dependencies package)))
+ (dependencies-names
(map juliahub-dependency-name
- (juliahub-package-direct-dependencies package)))
+ direct-dependencies))
(licenses
(map spdx-string->license
(list (juliahub-package-license package))))
@@ -224,7 +225,7 @@ (define* (juliahub->guix-package package-name
(juliahub-package-homepage package)
(juliahub-package-description package)
(beautify-description (juliahub-package-readme package))
- (juliahub-package-direct-dependencies package)
+ direct-dependencies
test-dependencies-names
licenses)
(append dependencies-names test-dependencies-names))))
--
2.38.1
S
S
Simon Tournier wrote on 18 Sep 2023 11:03
[PATCH v2 14/23] DRAFT import: juliahub: Handle the case where we have a subdirectory.
(address . 62202@debbugs.gnu.org)
cfb939c1da2ab7c3997297e91dd69712d715d2af.1695060058.git.zimon.toutoune@gmail.com
From: Nicolas Graves via Guix-patches via <guix-patches@gnu.org>

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
guix/import/juliahub.scm | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)

Toggle diff (35 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 2ea461b72aba..b646f9329562 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -233,14 +233,12 @@ (define (make-julia-sexp name source home-page synopsis description
((license) (license->symbol license))
(_ `(list ,@(map license->symbol licenses)))))))
-;; TODO handle subdir case properly.
-
(define* (juliahub->guix-package package-name
#:key version #:allow-other-keys)
"Fetch the metadata for PACKAGE-NAME from juliahub.org, and return the
`package' s-expression corresponding to that package, or #f on failure.
Optionally include a VERSION string to fetch a specific version juliahub."
- (let* ((package-toml (ini-fetch (general-url name "Package.toml")))
+ (let* ((package-toml (ini-fetch (general-url package-name "Package.toml")))
(subdir (assoc-ref package-toml 'subdir))
(tag (latest-git-tag (assoc-ref package-toml 'repo)))
(package (if version
@@ -264,7 +262,11 @@ (define* (juliahub->guix-package package-name
(licenses
(map spdx-string->license
(list (juliahub-package-license package))))
- (test-dependencies-names (parse-test-dependencies directory)))
+ (test-dependencies-names
+ (if subdir
+ (parse-test-dependencies
+ (string-append subdir "/" directory))
+ (parse-test-dependencies directory))))
(values (make-julia-sexp
package-name
source
--
2.38.1
S
S
Simon Tournier wrote on 18 Sep 2023 11:03
[PATCH v2 13/23] DRAFT import: juliahub: Improve test dependencies parsing.
(address . 62202@debbugs.gnu.org)
f48f3d3027eacc1834b627f2b4e6a5903d53314c.1695060058.git.zimon.toutoune@gmail.com
From: Nicolas Graves via Guix-patches via <guix-patches@gnu.org>

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
guix/import/juliahub.scm | 30 +++++++++++++-----------------
1 file changed, 13 insertions(+), 17 deletions(-)

Toggle diff (59 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 5327e923251d..2ea461b72aba 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -154,31 +154,27 @@ (define (json->juliahub-dependencies vector)
#f)))
(vector->list vector))))
-(define (ini-list->extra-dependencies lst)
+(define (ini-list->test-dependencies lst)
(match lst
- (('(extras) ooo ...)
- (extra-list->extra-dependencies ooo))
- (((tag) ooo ...)
- (ini-list->extra-dependencies ooo))
- ((attribute '= value ooo ...)
- (ini-list->extra-dependencies ooo))
+ (('test '= ooo ...)
+ `(,(caar ooo) ,@(test-list->test-dependencies (cdar ooo))))
+ ((value ooo ...)
+ (ini-list->test-dependencies ooo))
('()
- '())))
+ '())))
-(define (extra-list->extra-dependencies lst)
+(define (test-list->test-dependencies lst)
(match lst
- ((attribute '= value ooo ...)
- `(,(symbol->string attribute) ,@(extra-list->extra-dependencies ooo)))
- (((tag) ooo ...)
- '())
+ ((('unquote value) ooo ...)
+ `(,value ,@(test-list->test-dependencies ooo)))
('()
- '())))
+ '())))
-(define (parse-extra-dependencies directory)
+(define (parse-test-dependencies directory)
(let* ((port (open-input-file (string-append directory "/Project.toml")))
(ini-list (stream->list (port->stream port read))))
(close-port port)
- (ini-list->extra-dependencies ini-list)))
+ (ini-list->test-dependencies ini-list)))
;; Julia package.
(define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
@@ -268,7 +264,7 @@ (define* (juliahub->guix-package package-name
(licenses
(map spdx-string->license
(list (juliahub-package-license package))))
- (test-dependencies-names (parse-extra-dependencies directory)))
+ (test-dependencies-names (parse-test-dependencies directory)))
(values (make-julia-sexp
package-name
source
--
2.38.1
S
S
Simon Tournier wrote on 18 Sep 2023 11:03
[PATCH v2 16/23] DRAFT import: juliahub: Filter out stdlibs from test-dependencies.
(address . 62202@debbugs.gnu.org)
a01184089d2ac0d70c56bf0642a3f0c18efdb6f5.1695060058.git.zimon.toutoune@gmail.com
From: Nicolas Graves via Guix-patches via <guix-patches@gnu.org>

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
guix/import/juliahub.scm | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)

Toggle diff (16 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 6ce0487dba38..1c7b02929634 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -174,7 +174,8 @@ (define (parse-test-dependencies directory)
(let* ((port (open-input-file (string-append directory "/Project.toml")))
(ini-list (stream->list (port->stream port read))))
(close-port port)
- (ini-list->test-dependencies ini-list)))
+ (filter (lambda (x) (not (member x %julia-stdlibs)))
+ (ini-list->test-dependencies ini-list))))
;; Julia package.
(define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
--
2.38.1
S
S
Simon Tournier wrote on 18 Sep 2023 11:03
[PATCH v2 12/23] DRAFT import: juliahub: Add functions to parse the git repo for a git tag.
(address . 62202@debbugs.gnu.org)
e42032f4db1c8b76ff5ba94805deba47992a7c39.1695060058.git.zimon.toutoune@gmail.com
From: Nicolas Graves via Guix-patches via <guix-patches@gnu.org>

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
guix/import/juliahub.scm | 62 ++++++++++++++++++++++++++++++++++++----
1 file changed, 56 insertions(+), 6 deletions(-)

Toggle diff (105 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index fc25ba1d4220..5327e923251d 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -26,6 +26,7 @@ (define-module (guix import juliahub)
#:use-module (srfi srfi-9)
#:use-module (srfi srfi-11)
#:use-module (guix http-client)
+ #:use-module (guix git)
#:use-module (guix import utils)
#:use-module (guix import json)
#:use-module (guix base32)
@@ -38,6 +39,48 @@ (define-module (guix import juliahub)
%juliahub-updater
juliahub-recursive-import))
+;; Juliahub may be more up-to-date than the General registry or the actual git
+;; tag (it seems around 6 hours pass between the time a commit is supplied to
+;; JuliaRegistrator as a release, and the time Julia TagBot Github Action makes
+;; the git tag). We have no simple way to get the commit of the latest-version.
+;; Thus the best simple thing we can do is get the latest-git-version, and
+;; import this version instead. We do this by parsing Package.toml in the General
+;; registry, and then getting the refs of the git repo supplied by this
+;; file. Parsing this file is also necessary if the package is in a subdir of a
+;; git repository, because the information isn't present in Juliahub.
+;; There's a last case where some Julia packages are not based on a particular
+;; git tag, and where looking for the commit is tedious and artificial. In these
+;; cases, we introduce the tree-commit which is available in the Versions.toml
+;; file in the General repository. This is equivalent to a commit, since we have
+;; a unique hash of the listing of files and directories, thus it can be used to
+;; identify the state of a repository.
+
+(define %general-base-url
+ "https://raw.githubusercontent.com/JuliaRegistries/General/master/")
+
+(define (general-url package-name file)
+ (let ((folder (string-capitalize (string-take package-name 1))))
+ (string-append
+ %general-base-url folder "/" package-name "/" file)))
+
+(define (ini-list->alist lst)
+ (match lst
+ ((attribute '= value ooo ...)
+ `((,attribute . ,value) ,@(ini-list->alist ooo)))
+ ('()
+ '())))
+
+(define (ini-fetch url)
+ (let* ((port (http-fetch url #:text? #t))
+ (ini-list (stream->list (port->stream port read))))
+ (close-port port)
+ (ini-list->alist ini-list)))
+
+(define (latest-git-tag repo)
+ (let* ((last-ref (last (remote-refs repo #:tags? #t)))
+ (last-git-tag (last (string-split last-ref #\/))))
+ (string-drop last-git-tag 1)))
+
;; To update, see file sysimg.jl
(define %julia-stdlibs
(list "julia"
@@ -194,14 +237,21 @@ (define (make-julia-sexp name source home-page synopsis description
((license) (license->symbol license))
(_ `(list ,@(map license->symbol licenses)))))))
+;; TODO handle subdir case properly.
+
(define* (juliahub->guix-package package-name
#:key version #:allow-other-keys)
"Fetch the metadata for PACKAGE-NAME from juliahub.org, and return the
`package' s-expression corresponding to that package, or #f on failure.
Optionally include a VERSION string to fetch a specific version juliahub."
- (let ((package (if version
- (juliahub-fetch package-name version)
- (juliahub-fetch package-name))))
+ (let* ((package-toml (ini-fetch (general-url name "Package.toml")))
+ (subdir (assoc-ref package-toml 'subdir))
+ (tag (latest-git-tag (assoc-ref package-toml 'repo)))
+ (package (if version
+ (juliahub-fetch package-name version)
+ (if tag
+ (juliahub-fetch package-name tag)
+ (juliahub-fetch package-name)))))
(if package
(let-values (((source directory)
(git->origin+dir
@@ -233,13 +283,13 @@ (define* (juliahub->guix-package package-name
(define (guix-package->juliahub-name package)
(let* ((url (juliahub-package-url package))
- (git-name (car (last-pair (string-split url #\/))))
+ (git-name (last (string-split url #\/)))
(ungitted-name (if (string-suffix? ".git" git-name)
(string-drop-right git-name 4)
git-name))
(package-name (if (string-suffix? ".jl" ungitted-name)
- (string-drop-right ungitted-name 4)
- ungitted-name)))
+ (string-drop-right ungitted-name 4)
+ ungitted-name)))
package-name))
(define* (import-release package #:key (version #f))
--
2.38.1
S
S
Simon Tournier wrote on 18 Sep 2023 11:03
[PATCH v2 15/23] DRAFT import: juliahub: Add support for versions for juliahub-fetch.
(address . 62202@debbugs.gnu.org)
1fef7a18856fd9670be3746e6f3579a8be39163e.1695060058.git.zimon.toutoune@gmail.com
From: Nicolas Graves via Guix-patches via <guix-patches@gnu.org>

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
guix/import/juliahub.scm | 20 ++++++++++++--------
1 file changed, 12 insertions(+), 8 deletions(-)

Toggle diff (57 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index b646f9329562..6ce0487dba38 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -122,7 +122,7 @@ (define %julia-stdlibs
"Pkg"
"LazyArtifacts"))
-(define (juliahub-uri name)
+(define (juliahub-redirect-uri name)
(let* ((url (string-append "https://docs.juliahub.com/" name "/"))
(port (http-fetch url #:text? #t))
(_ (get-line port))
@@ -134,11 +134,11 @@ (define (juliahub-uri name)
(define (juliahub-url name)
(let* ((url (string-append "https://docs.juliahub.com/" name "/"))
- (uri (juliahub-uri name)))
+ (uri (juliahub-redirect-uri name)))
(string-append url uri "/")))
-(define (juliahub-slug-version name)
- (let* ((uri (juliahub-uri name))
+(define (juliahub-slug+version name)
+ (let* ((uri (juliahub-redirect-uri name))
(slug (string-take uri 5))
(latest-version (string-drop uri 6)))
`(,slug ,latest-version)))
@@ -203,8 +203,12 @@ (define (julia-name->guix-name name)
(define* (juliahub-fetch name #:key (version #f))
"Return a <juliahub-package> record for package NAME, or #f on failure."
- (and=> (json-fetch (string-append (juliahub-url name) "pkg.json"))
- json->juliahub-package))
+ (let ((url (if version
+ (string-append "https://docs.juliahub.com/" name "/"
+ (car (juliahub-slug+version name)) "/"
+ version "/pkg.json")
+ (string-append (juliahub-url name) "pkg.json"))))
+ (and=> (json-fetch url) json->juliahub-package)))
(define (make-julia-sexp name source home-page synopsis description
direct-dependencies test-dependencies-names licenses)
@@ -242,9 +246,9 @@ (define* (juliahub->guix-package package-name
(subdir (assoc-ref package-toml 'subdir))
(tag (latest-git-tag (assoc-ref package-toml 'repo)))
(package (if version
- (juliahub-fetch package-name version)
+ (juliahub-fetch package-name #:version version)
(if tag
- (juliahub-fetch package-name tag)
+ (juliahub-fetch package-name #:version tag)
(juliahub-fetch package-name)))))
(if package
(let-values (((source directory)
--
2.38.1
S
S
Simon Tournier wrote on 18 Sep 2023 11:03
[PATCH v2 17/23] DRAFT import: juliahub: More robust toml regex parser.
(address . 62202@debbugs.gnu.org)
2423ea8dafa29faa49e75da40a480873531e5126.1695060058.git.zimon.toutoune@gmail.com
From: Nicolas Graves via Guix-patches via <guix-patches@gnu.org>

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
guix/import/juliahub.scm | 29 ++++++++---------------------
1 file changed, 8 insertions(+), 21 deletions(-)

Toggle diff (53 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 1c7b02929634..3985d8d0be44 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -154,28 +154,15 @@ (define (json->juliahub-dependencies vector)
#f)))
(vector->list vector))))
-(define (ini-list->test-dependencies lst)
- (match lst
- (('test '= ooo ...)
- `(,(caar ooo) ,@(test-list->test-dependencies (cdar ooo))))
- ((value ooo ...)
- (ini-list->test-dependencies ooo))
- ('()
- '())))
-
-(define (test-list->test-dependencies lst)
- (match lst
- ((('unquote value) ooo ...)
- `(,value ,@(test-list->test-dependencies ooo)))
- ('()
- '())))
-
(define (parse-test-dependencies directory)
(let* ((port (open-input-file (string-append directory "/Project.toml")))
- (ini-list (stream->list (port->stream port read))))
+ (project.toml (get-string-all port))
+ (regex "\ntest = \\[.*\\]")
+ (deps (match:substring (string-match regex project.toml)))
+ (pure (string-delete (list->char-set (list #\" #\ )) deps)))
(close-port port)
(filter (lambda (x) (not (member x %julia-stdlibs)))
- (ini-list->test-dependencies ini-list))))
+ (string-split (string-drop (string-drop-right pure 1) 7) #\,))))
;; Julia package.
(define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
@@ -243,9 +230,9 @@ (define* (juliahub->guix-package package-name
"Fetch the metadata for PACKAGE-NAME from juliahub.org, and return the
`package' s-expression corresponding to that package, or #f on failure.
Optionally include a VERSION string to fetch a specific version juliahub."
- (let* ((package-toml (ini-fetch (general-url package-name "Package.toml")))
- (subdir (assoc-ref package-toml 'subdir))
- (tag (latest-git-tag (assoc-ref package-toml 'repo)))
+ (let* ((package.toml (ini-fetch (general-url package-name "Package.toml")))
+ (subdir (assoc-ref package.toml 'subdir))
+ (tag (latest-git-tag (assoc-ref package.toml 'repo)))
(package (if version
(juliahub-fetch package-name #:version version)
(if tag
--
2.38.1
S
S
Simon Tournier wrote on 18 Sep 2023 11:03
[PATCH v2 20/23] DRAFT import: juliahub: Fix version management.
(address . 62202@debbugs.gnu.org)
d842837030aa42b7e6b2094e4639a458a63be3b6.1695060058.git.zimon.toutoune@gmail.com
From: Nicolas Graves via Guix-patches via <guix-patches@gnu.org>

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
guix/import/juliahub.scm | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)

Toggle diff (27 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index e4b26bea340a..94d4ae823334 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -247,10 +247,10 @@ (define* (juliahub-fetch name #:key (version #f))
(string-append (juliahub-url name) "pkg.json"))))
(and=> (json-fetch url) json->juliahub-package)))
-(define (make-julia-sexp name source home-page synopsis description
+(define (make-julia-sexp name version source home-page synopsis description
direct-dependencies test-dependencies-names license)
"Return the `package' s-expression for a Julia package with the given NAME,
-VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DIRECT-DEPENDENCIES,
+VERSION, SOURCE, HOME-PAGE, DESCRIPTION, DIRECT-DEPENDENCIES,
TEST-DEPENDENCIES-NAMES and LICENSE."
`(package
(name ,(julia-name->guix-name name))
@@ -306,6 +306,7 @@ (define* (juliahub->guix-package package-name
(parse-test-dependencies directory))))
(values (make-julia-sexp
package-name
+ (juliahub-package-version package)
source
(juliahub-package-homepage package)
(juliahub-package-description package)
--
2.38.1
S
S
Simon Tournier wrote on 18 Sep 2023 11:03
[PATCH v2 21/23] DRAFT import: juliahub: Fix undefined homepages.
(address . 62202@debbugs.gnu.org)
a45fc1b1d0c059678123d67cfc221f60f6c15afd.1695060058.git.zimon.toutoune@gmail.com
From: Nicolas Graves via Guix-patches via <guix-patches@gnu.org>

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
guix/import/juliahub.scm | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)

Toggle diff (25 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 94d4ae823334..06574db724fe 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -303,12 +303,16 @@ (define* (juliahub->guix-package package-name
(if subdir
(parse-test-dependencies
(string-append subdir "/" directory))
- (parse-test-dependencies directory))))
+ (parse-test-dependencies directory)))
+ (homepage (juliahub-package-homepage package)))
(values (make-julia-sexp
package-name
(juliahub-package-version package)
source
- (juliahub-package-homepage package)
+ (match homepage
+ ("" (juliahub-package-url package))
+ ((? string?) homepage)
+ (_ (juliahub-package-url package)))
(juliahub-package-description package)
((compose beautify-description
juliahub-beautify-description)
--
2.38.1
S
S
Simon Tournier wrote on 18 Sep 2023 11:03
[PATCH v2 18/23] DRAFT import: juliahub: Beautify description.
(address . 62202@debbugs.gnu.org)
378987b3ca7040ffbc2f8f665651e8efee4238ae.1695060058.git.zimon.toutoune@gmail.com
From: Nicolas Graves via Guix-patches via <guix-patches@gnu.org>

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
guix/import/juliahub.scm | 60 +++++++++++++++++++++++++++++++++++++---
1 file changed, 56 insertions(+), 4 deletions(-)

Toggle diff (95 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 3985d8d0be44..338f0424414c 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -32,8 +32,10 @@ (define-module (guix import juliahub)
#:use-module (guix base32)
#:use-module (guix packages)
#:use-module (guix upstream)
- #:use-module (json)
#:use-module ((guix licenses) #:prefix license:)
+ #:use-module (json)
+ #:use-module (htmlprag)
+ #:use-module (sxml transform)
#:export (juliahub->guix-package
%juliahub-updater
@@ -164,6 +166,53 @@ (define (parse-test-dependencies directory)
(filter (lambda (x) (not (member x %julia-stdlibs)))
(string-split (string-drop (string-drop-right pure 1) 7) #\,))))
+(define %juliahub-beautify-description-rules
+ `((h1 *preorder* . ,(lambda args #f))
+ (h2 *preorder* . ,(lambda args #f))
+ (h3 *preorder* . ,(lambda args #f))
+ (h4 *preorder* . ,(lambda args #f))
+ (hr *preorder* . ,(lambda args #f))
+ (span *preorder* . ,(lambda args #f))
+ (img *preorder* . ,(lambda args #f))
+ (pre *preorder* . ,(lambda args #f))
+ (div *preorder* . ,(lambda args #f))
+ (table *preorder* . ,(lambda args #f))
+ (imgalt *preorder* . ,(lambda args #f))
+ (@ *preorder* . ,(lambda args #f))
+ (*TOP* . ,(lambda args (cdr args)))
+ (p . ,(lambda args (cdr args)))
+ (em . ,(lambda args (cdr args)))
+ (strong . ,(lambda args (cdr args)))
+ (a . ,(lambda args
+ (match args
+ ((tag link ref)
+ (if ref ref #f))
+ (_ #f))))
+ (ul . ,(lambda args
+ `("@itemize" ,@(cdr args) "\n@end itemize")))
+ (ol . ,(lambda args
+ `("@enumerate" ,@(cdr args) "@end enumerate")))
+ (blockquote . ,(lambda args
+ `("@quotation" ,@(cdr args) "@end quotation")))
+ (li . ,(lambda args
+ `("\n@item" ,@(cdr args))))
+ (code . ,(lambda args
+ (string-append
+ "@code{"
+ (string-join (cdr args) " ")
+ "}")))
+ (*text* . ,(lambda (tag x) x))
+ (*default* . ,(lambda (tag . body)
+ (cons tag body)))))
+
+(define (juliahub-beautify-description description)
+ (string-join
+ (filter (lambda (x) (if (equal? x " ") #f x))
+ (flatten
+ (pre-post-order (html->sxml description)
+ %juliahub-beautify-description-rules)))
+ " "))
+
;; Julia package.
(define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
json->juliahub-package
@@ -211,8 +260,9 @@ (define (make-julia-sexp name source home-page synopsis description
,@(if (null? direct-dependencies)
'()
`((propagated-inputs
- (list ,@(map (compose julia-name->guix-name juliahub-dependency-name)
- direct-dependencies)))))
+ (list
+ ,@(map (compose julia-name->guix-name juliahub-dependency-name)
+ direct-dependencies)))))
,@(if (null? test-dependencies-names)
'()
`((native-inputs
@@ -264,7 +314,9 @@ (define* (juliahub->guix-package package-name
source
(juliahub-package-homepage package)
(juliahub-package-description package)
- (beautify-description (juliahub-package-readme package))
+ ((compose beautify-description
+ juliahub-beautify-description)
+ (juliahub-package-readme package))
direct-dependencies
test-dependencies-names
licenses)
--
2.38.1
S
S
Simon Tournier wrote on 18 Sep 2023 11:03
[PATCH v2 22/23] DRAFT import: juliahub: output package names as symbols.
(address . 62202@debbugs.gnu.org)
11807ccb1ff39f04a76a62e37556c9c5727b36a2.1695060058.git.zimon.toutoune@gmail.com
From: Nicolas Graves via Guix-patches via <guix-patches@gnu.org>

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
guix/import/juliahub.scm | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)

Toggle diff (24 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 06574db724fe..be874b193b28 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -261,12 +261,15 @@ (define (make-julia-sexp name version source home-page synopsis description
'()
`((propagated-inputs
(list
- ,@(map (compose julia-name->guix-name juliahub-dependency-name)
+ ,@(map (compose string->symbol
+ julia-name->guix-name
+ juliahub-dependency-name)
direct-dependencies)))))
,@(if (null? test-dependencies-names)
'()
`((native-inputs
- (list ,@(map julia-name->guix-name test-dependencies-names)))))
+ (list ,@(map (compose string->symbol julia-name->guix-name)
+ test-dependencies-names)))))
(synopsis ,synopsis)
(description ,description)
(home-page ,home-page)
--
2.38.1
S
S
Simon Tournier wrote on 18 Sep 2023 11:03
[PATCH v2 23/23] DRAFT guix: import: julia: Fix beautify.
(address . 62202@debbugs.gnu.org)(name . Simon Tournier)(address . zimon.toutoune@gmail.com)
a88ae8ef5cf22189cc72ac10aa25b839270be3f8.1695060058.git.zimon.toutoune@gmail.com
---
guix/import/juliahub.scm | 5 +----
1 file changed, 1 insertion(+), 4 deletions(-)

Toggle diff (18 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index be874b193b28..3e5735b9503f 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -197,10 +197,7 @@ (define %juliahub-beautify-description-rules
(li . ,(lambda args
`("\n@item" ,@(cdr args))))
(code . ,(lambda args
- (string-append
- "@code{"
- (string-join (cdr args) " ")
- "}")))
+ `("@code{" ,@(cdr args) "}")))
(*text* . ,(lambda (tag x) x))
(*default* . ,(lambda (tag . body)
(cons tag body)))))
--
2.38.1
S
S
Simon Tournier wrote on 18 Sep 2023 11:03
[PATCH v2 19/23] DRAFT import: juliahub: Fix license management.
(address . 62202@debbugs.gnu.org)
80498439f6fb5e349b5e9fc643e6538a361983fd.1695060058.git.zimon.toutoune@gmail.com
From: Nicolas Graves via Guix-patches via <guix-patches@gnu.org>

Signed-off-by: Simon Tournier <zimon.toutoune@gmail.com>
---
guix/import/juliahub.scm | 15 +++++----------
1 file changed, 5 insertions(+), 10 deletions(-)

Toggle diff (51 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index 338f0424414c..e4b26bea340a 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -248,10 +248,10 @@ (define* (juliahub-fetch name #:key (version #f))
(and=> (json-fetch url) json->juliahub-package)))
(define (make-julia-sexp name source home-page synopsis description
- direct-dependencies test-dependencies-names licenses)
+ direct-dependencies test-dependencies-names license)
"Return the `package' s-expression for a Julia package with the given NAME,
VERSION, URI, HASH, HOME-PAGE, DESCRIPTION, DIRECT-DEPENDENCIES,
-TEST-DEPENDENCIES-NAMES and LICENSES."
+TEST-DEPENDENCIES-NAMES and LICENSE."
`(package
(name ,(julia-name->guix-name name))
(version ,version)
@@ -270,10 +270,8 @@ (define (make-julia-sexp name source home-page synopsis description
(synopsis ,synopsis)
(description ,description)
(home-page ,home-page)
- (license ,(match licenses
- (() #f)
- ((license) (license->symbol license))
- (_ `(list ,@(map license->symbol licenses)))))))
+ (license
+ ,(if license (spdx-string->license license) #f))))
(define* (juliahub->guix-package package-name
#:key version #:allow-other-keys)
@@ -301,9 +299,6 @@ (define* (juliahub->guix-package package-name
(dependencies-names
(map juliahub-dependency-name
direct-dependencies))
- (licenses
- (map spdx-string->license
- (list (juliahub-package-license package))))
(test-dependencies-names
(if subdir
(parse-test-dependencies
@@ -319,7 +314,7 @@ (define* (juliahub->guix-package package-name
(juliahub-package-readme package))
direct-dependencies
test-dependencies-names
- licenses)
+ (juliahub-package-license package))
(append dependencies-names test-dependencies-names))))
(values #f '()))))
--
2.38.1
S
S
Simon Tournier wrote on 18 Sep 2023 11:06
Re: [bug#62202] [PATCH 0/21] Juliahub import script.
(name . Nicolas Graves)(address . ngraves@ngraves.fr)
87led3qrts.fsf@gmail.com
Hi Nicolas,

On Mon, 18 Sep 2023 at 11:31, Nicolas Graves via Guix-patches via <guix-patches@gnu.org> wrote:

Toggle quote (3 lines)
> Couldn't find time this wkend, but it's on my backlog, I'll try to do
> that soon.

No worry. I did it manually. :-)

I have sent v2 which is just your patches applied the top of
3d9ebc7b2ed24312fd6a0916c203f7b86d57753d.

My plan is to give a look to the series this week.

Cheers,
simon
G
G
Giovanni Biscuolo wrote on 18 Sep 2023 23:23
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)
87v8c64r72.fsf@xelera.eu
Hi Simon,

Simon Tournier <zimon.toutoune@gmail.com> writes:

[...]

Toggle quote (2 lines)
> No worry. I did it manually. :-)

out of curiosity, what do you mean by "manually" please? I mean: ho did
you got to apply the patches?

I'm asking because it could be helpful to others

Toggle quote (5 lines)
> I have sent v2 which is just your patches applied the top of
> 3d9ebc7b2ed24312fd6a0916c203f7b86d57753d.
>
> My plan is to give a look to the series this week.

thank you!

Happy hacking! Gio'

--
Giovanni Biscuolo

Xelera IT Infrastructures
-----BEGIN PGP SIGNATURE-----

iQJABAEBCgAqFiEERcxjuFJYydVfNLI5030Op87MORIFAmUJPlEMHGdAeGVsZXJh
LmV1AAoJENN9DqfOzDkSYPQQALZLeQUhGIShoxyTtAk6eX/GGcDWoB0y9Ccz131b
LahCl0JX4e9PJ7UdnVarBBVO2l8JM6UwOCooXa1P+d5TJE7TOFkAdYOvzNsy5PGt
efYksTZgeVTV6cpWZkT5Ptq8ITMnyvK5YxFxoa8+29031VA/qWTbF6WRIBkOifVc
rZQqABO8GQd0IiUK2fah+xnzAE4mxzvxQSvkw5XSWd8IoXSbNs47OwH9UOSBUFOD
kZo6KtyZjwWYOSp2KDw5KucB8xnuBSPecqqeVWChgJcHFA5AW1rY2bO8e58nIMuY
izRmcPShuWcXyky1ZeQyVd34Jmm5qq4GWGfWO6587OKeSvxiaef0xmTju1H4Lb3L
BLit4+7ObFe+4CO0vL6tZL7CvRS4udJkpxUKueIheVi9Y5utNHgyz8NEIPM0oC1s
rFiM9IVe5SXFbHHVnlhHW7mQWT/WSECHbjcFkOqUmLl2MmcG2lOV1UJiz6jVU6aI
CB6PNLPuWE9RAMMjjtxMsWw2bLWAU5R+qRwY19LkmJQTwhVRhlbniVMz9tsdkXVF
umIUCmw+o22hKFo4TFXSieQUYUS1k/0hzyySO/S1CBNfZkkdu/98xD0Mf+eBMlNr
FHxvboKccDD7OkIw+ceP+ZWPVdrVvW7K+farvpX8ZYk7X6vbaLbI4iyyedUXaB3O
nPJ6
=P6PQ
-----END PGP SIGNATURE-----

N
N
Nicolas Graves wrote on 18 Sep 2023 23:51
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)
875y46bqqo.fsf@ngraves.fr
On 2023-09-18 20:06, Simon Tournier wrote:

Toggle quote (14 lines)
> Hi Nicolas,
>
> On Mon, 18 Sep 2023 at 11:31, Nicolas Graves via Guix-patches via <guix-patches@gnu.org> wrote:
>
>> Couldn't find time this wkend, but it's on my backlog, I'll try to do
>> that soon.
>
> No worry. I did it manually. :-)
>
> I have sent v2 which is just your patches applied the top of
> 3d9ebc7b2ed24312fd6a0916c203f7b86d57753d.
>
> My plan is to give a look to the series this week.

Just note that in my experience, the juliahub importer worked quite
well (there was sometimes errors, mainly annoying with recursive imports
when the git repository didn't have any tag IIRC, not sure if I had
fixed that or not). The other annoying thing was that the factorization
broke the go importer IIRC.

But these fixes shouldn't be too hard compared to the initial effort to
make the importer.

Thanks a lot and good luck, sorry, quite hard to find time recently.

Toggle quote (3 lines)
> Cheers,
> simon

--
Best regards,
Nicolas Graves
S
S
Simon Tournier wrote on 18 Sep 2023 23:37
(name . Giovanni Biscuolo)(address . g@xelera.eu)
864jjqmzx7.fsf@gmail.com
Hi,

On Tue, 19 Sep 2023 at 08:23, Giovanni Biscuolo <g@xelera.eu> wrote:

Toggle quote (3 lines)
> out of curiosity, what do you mean by "manually" please? I mean: ho did
> you got to apply the patches?

I mean literally. ;-)

Open the patch, read it and then open the file, modify it, commit.
Repeat.

Else, I just pipe the message with “git am -3s”.

Cheers,
simon
S
S
Simon Tournier wrote on 2 Oct 2023 02:34
(name . Nicolas Graves)(address . ngraves@ngraves.fr)
868r8l9xlr.fsf@gmail.com
Hi,

On Tue, 19 Sep 2023 at 08:51, Nicolas Graves via Guix-patches via <guix-patches@gnu.org> wrote:

Toggle quote (2 lines)
>> My plan is to give a look to the series this week.

[...]

Toggle quote (2 lines)
> Thanks a lot and good luck, sorry, quite hard to find time recently.

I pause a bit. If someone is motivated, feel free to pick it. Else I
will resume when I will be able to dedicate some time for that.

Cheers,
simon
N
N
Nicolas Graves wrote on 19 Dec 2023 10:28
(name . Simon Tournier)(address . zimon.toutoune@gmail.com)
875y0ukpwm.fsf@ngraves.fr
On 2023-10-02 11:34, Simon Tournier wrote:

Toggle quote (13 lines)
> Hi,
>
> On Tue, 19 Sep 2023 at 08:51, Nicolas Graves via Guix-patches via <guix-patches@gnu.org> wrote:
>
>>> My plan is to give a look to the series this week.
>
> [...]
>
>> Thanks a lot and good luck, sorry, quite hard to find time recently.
>
> I pause a bit. If someone is motivated, feel free to pick it. Else I
> will resume when I will be able to dedicate some time for that.

Will probably try to give it a go, either this evening or during winter
vacation. Could you point out if some changes other than properly
rebasing have been done at the point of the DRAFT V2 ? Thanks!


Toggle quote (7 lines)
>
> Cheers,
> simon
>
>
>

--
Best regards,
Nicolas Graves
N
N
Nicolas Graves wrote on 21 Dec 2023 06:01
[PATCH v3 1/4] import: utils: Add function git->origin.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20231221140142.16523-1-ngraves@ngraves.fr
* guix/import/utils.scm: (git->origin): Add function.

* guix/import/elpa.scm
(download-git-repository): Remove function download-git-repository.
(git-repository->origin): Remove function git-repository->origin.
(ref): Add function ref.
(melpa-recipe->origin): Use functions git->origin and ref.

* guix/import/go.scm
(git-checkout-hash): Remove function git-checkout-hash.
(transform-version): Add function transform-version.
(vcs->origin): Use functions git->origin and transform-version. Add
optional argument transform-version.

* tests/import/go.scm
(go-module->guix-package): Adapt test case to changes in guix/import/go.scm.

* guix/import/minetest.scm
(download-git-repository): Remove function download-git-repository.
(make-minetest-sexp): Use function git->origin.

* tests/minetest.scm
(make-package-sexp): Use function git->origin.
(example-package): Adapt test-case to git->origin.

* guix/import/composer.scm
(make-php-sexp): Use function git->origin.

Change-Id: Ied05a63bdd60fbafe26fbbb4e115ff6f0bb9db3c
---
guix/import/composer.scm | 86 ++++++++++++++--------------------------
guix/import/elpa.scm | 43 ++++++--------------
guix/import/go.scm | 57 ++++++++------------------
guix/import/minetest.scm | 28 ++-----------
guix/import/utils.scm | 39 ++++++++++++++++++
tests/go.scm | 29 ++++++++++----
tests/minetest.scm | 15 ++-----
7 files changed, 127 insertions(+), 170 deletions(-)

Toggle diff (435 lines)
diff --git a/guix/import/composer.scm b/guix/import/composer.scm
index 1ad608964b..dabc5423ed 100644
--- a/guix/import/composer.scm
+++ b/guix/import/composer.scm
@@ -19,22 +19,17 @@
(define-module (guix import composer)
#:use-module (ice-9 match)
#:use-module (json)
- #:use-module (guix hash)
- #:use-module (guix base32)
- #:use-module (guix build git)
- #:use-module (guix build utils)
- #:use-module (guix build-system)
#:use-module (guix build-system composer)
+ #:use-module ((guix download) #:select (download-to-store))
#:use-module (guix import json)
#:use-module (guix import utils)
#:use-module ((guix licenses) #:prefix license:)
#:use-module (guix memoization)
#:use-module (guix packages)
- #:use-module (guix serialization)
+ #:use-module (guix store)
#:use-module (guix upstream)
#:use-module (guix utils)
#:use-module (srfi srfi-1)
- #:use-module (srfi srfi-11)
#:use-module (srfi srfi-26)
#:export (composer->guix-package
%composer-updater
@@ -141,55 +136,34 @@ (define (make-php-sexp composer-package)
(dependencies (map php-package-name
(composer-package-require composer-package)))
(dev-dependencies (map php-package-name
- (composer-package-dev-require composer-package)))
- (git? (equal? (composer-source-type source) "git")))
- ((if git? call-with-temporary-directory call-with-temporary-output-file)
- (lambda* (temp #:optional port)
- (and (if git?
- (begin
- (mkdir-p temp)
- (git-fetch (composer-source-url source)
- (composer-source-reference source)
- temp))
- (url-fetch (composer-source-url source) temp))
- `(package
- (name ,(composer-package-name composer-package))
- (version ,(composer-package-version composer-package))
- (source
- (origin
- ,@(if git?
- `((method git-fetch)
- (uri (git-reference
- (url ,(if (string-suffix?
- ".git"
- (composer-source-url source))
- (string-drop-right
- (composer-source-url source)
- (string-length ".git"))
- (composer-source-url source)))
- (commit ,(composer-source-reference source))))
- (file-name (git-file-name name version))
- (sha256
- (base32
- ,(bytevector->nix-base32-string
- (file-hash* temp)))))
- `((method url-fetch)
- (uri ,(composer-source-url source))
- (sha256 (base32 ,(guix-hash-url temp)))))))
- (build-system composer-build-system)
- ,@(if (null? dependencies)
- '()
- `((inputs
- (list ,@(map string->symbol dependencies)))))
- ,@(if (null? dev-dependencies)
- '()
- `((native-inputs
- (list ,@(map string->symbol dev-dependencies)))))
- (synopsis "")
- (description ,(composer-package-description composer-package))
- (home-page ,(composer-package-homepage composer-package))
- (license ,(or (composer-package-license composer-package)
- 'unknown-license!))))))))
+ (composer-package-dev-require composer-package))))
+ `(package
+ (name ,(composer-package-name composer-package))
+ (version ,(composer-package-version composer-package))
+ (source
+ ,(if (string= (composer-source-type source) "git")
+ (git->origin (composer-source-url source)
+ `(tag-or-commit . ,(composer-source-reference source)))
+ (let* ((source (composer-source-url source))
+ (tarball (with-store store (download-to-store store source))))
+ `(origin
+ (method url-fetch)
+ (uri ,source)
+ (sha256 (base32 ,(guix-hash-url tarball)))))))
+ (build-system composer-build-system)
+ ,@(if (null? dependencies)
+ '()
+ `((inputs
+ (list ,@(map string->symbol dependencies)))))
+ ,@(if (null? dev-dependencies)
+ '()
+ `((native-inputs
+ (list ,@(map string->symbol dev-dependencies)))))
+ (synopsis "")
+ (description ,(composer-package-description composer-package))
+ (home-page ,(composer-package-homepage composer-package))
+ (license ,(or (composer-package-license composer-package)
+ 'unknown-license!)))))
(define composer->guix-package
(memoize
diff --git a/guix/import/elpa.scm b/guix/import/elpa.scm
index d1855b3698..a755387242 100644
--- a/guix/import/elpa.scm
+++ b/guix/import/elpa.scm
@@ -8,6 +8,7 @@
;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev>
;;; Copyright © 2021 Simon Tournier <zimon.toutoune@gmail.com>
;;; Copyright © 2022 Hartmut Goebel <h.goebel@crazy-compilers.com>
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -208,11 +209,6 @@ (define* (fetch-elpa-package name #:optional (repo 'gnu))
url)))
(_ #f))))
-(define* (download-git-repository url ref)
- "Fetch the given REF from the Git repository at URL."
- (with-store store
- (latest-repository-commit store url #:ref ref)))
-
(define (package-name->melpa-recipe package-name)
"Fetch the MELPA recipe for PACKAGE-NAME, represented as an alist from
keywords to values."
@@ -232,28 +228,15 @@ (define (data->recipe data)
(close-port port)
(data->recipe (cons ':name data))))
-(define (git-repository->origin recipe url)
- "Fetch origin details from the Git repository at URL for the provided MELPA
-RECIPE."
- (define ref
- (cond
- ((assoc-ref recipe #:branch)
- => (lambda (branch) (cons 'branch branch)))
- ((assoc-ref recipe #:commit)
- => (lambda (commit) (cons 'commit commit)))
- (else
- '())))
-
- (let-values (((directory commit) (download-git-repository url ref)))
- `(origin
- (method git-fetch)
- (uri (git-reference
- (url ,url)
- (commit ,commit)))
- (sha256
- (base32
- ,(bytevector->nix-base32-string
- (file-hash* directory #:recursive? #true)))))))
+(define (ref recipe)
+ "Create REF from MELPA RECIPE."
+ (cond
+ ((assoc-ref recipe #:branch)
+ => (lambda (branch) (cons 'branch branch)))
+ ((assoc-ref recipe #:commit)
+ => (lambda (commit) (cons 'commit commit)))
+ (else
+ '())))
(define* (melpa-recipe->origin recipe)
"Fetch origin details from the MELPA recipe and associated repository for
@@ -264,9 +247,9 @@ (define (gitlab-repo->url repo)
(string-append "https://gitlab.com/" repo ".git"))
(match (assq-ref recipe ':fetcher)
- ('github (git-repository->origin recipe (github-repo->url (assq-ref recipe ':repo))))
- ('gitlab (git-repository->origin recipe (gitlab-repo->url (assq-ref recipe ':repo))))
- ('git (git-repository->origin recipe (assq-ref recipe ':url)))
+ ('github (git->origin (github-repo->url (assq-ref recipe ':repo)) (ref recipe)))
+ ('gitlab (git->origin (gitlab-repo->url (assq-ref recipe ':repo)) (ref recipe)))
+ ('git (git->origin (assq-ref recipe ':url) (ref recipe)))
(#f #f) ; if we're not using melpa then this stops us printing a warning
(_ (warning (G_ "unsupported MELPA fetcher: ~a, falling back to unstable MELPA source~%")
(assq-ref recipe ':fetcher))
diff --git a/guix/import/go.scm b/guix/import/go.scm
index dd9298808d..6e2ce2ed00 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -8,6 +8,7 @@
;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev>
;;; Copyright © 2021 Simon Tournier <zimon.toutoune@gmail.com>
;;; Copyright © 2023 Efraim Flashner <efraim@flashner.co.il>
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -514,49 +515,24 @@ (define (module-meta-data-repo-url meta-data goproxy)
goproxy
(module-meta-repo-root meta-data)))
-(define* (git-checkout-hash url reference algorithm)
- "Return the ALGORITHM hash of the checkout of URL at REFERENCE, a commit or
-tag."
- (define cache
- (string-append (or (getenv "TMPDIR") "/tmp")
- "/guix-import-go-"
- (passwd:name (getpwuid (getuid)))))
+;; This is done because the version field of the package, which the generated
+;; quoted expression refers to, has been stripped of any 'v' prefixed.
+(define (transform-version version)
+ (let ((plain-version? (string=? version (go-version->git-ref version)))
+ (v-prefixed? (string-prefix? "v" version)))
+ (if (and plain-version? v-prefixed?)
+ '(string-append "v" version)
+ '(go-version->git-ref version))))
- ;; Use a custom cache to avoid cluttering the default one under
- ;; ~/.cache/guix, but choose one under /tmp so that it's persistent across
- ;; subsequent "guix import" invocations.
- (mkdir-p cache)
- (chmod cache #o700)
- (let-values (((checkout commit _)
- (parameterize ((%repository-cache-directory cache))
- (update-cached-checkout url
- #:ref
- `(tag-or-commit . ,reference)))))
- (file-hash* checkout #:algorithm algorithm #:recursive? #true)))
-
-(define (vcs->origin vcs-type vcs-repo-url version)
+(define* (vcs->origin vcs-type vcs-repo-url version
+ #:key (transform-version #f))
"Generate the `origin' block of a package depending on what type of source
-control system is being used."
+control system is being used. Optionally use the function TRANSFORM-VERSION
+which takes version as an input."
(case vcs-type
((git)
- (let ((plain-version? (string=? version (go-version->git-ref version)))
- (v-prefixed? (string-prefix? "v" version)))
- `(origin
- (method git-fetch)
- (uri (git-reference
- (url ,vcs-repo-url)
- ;; This is done because the version field of the package,
- ;; which the generated quoted expression refers to, has been
- ;; stripped of any 'v' prefixed.
- (commit ,(if (and plain-version? v-prefixed?)
- '(string-append "v" version)
- '(go-version->git-ref version)))))
- (file-name (git-file-name name version))
- (sha256
- (base32
- ,(bytevector->nix-base32-string
- (git-checkout-hash vcs-repo-url (go-version->git-ref version)
- (hash-algorithm sha256))))))))
+ (git->origin vcs-repo-url `(tag-or-commit . ,version)
+ #:ref->commit transform-version))
((hg)
`(origin
(method hg-fetch)
@@ -649,7 +625,8 @@ (define* (go-module->guix-package module-path #:key
(name ,guix-name)
(version ,(strip-v-prefix version*))
(source
- ,(vcs->origin vcs-type vcs-repo-url version*))
+ ,(vcs->origin vcs-type vcs-repo-url version*
+ #:transform-version transform-version))
(build-system go-build-system)
(arguments
(list ,@(if (version>? min-go-version (package-version (go-package)))
diff --git a/guix/import/minetest.scm b/guix/import/minetest.scm
index 5ea6e023ce..65ef242431 100644
--- a/guix/import/minetest.scm
+++ b/guix/import/minetest.scm
@@ -1,6 +1,7 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2021, 2022 Maxime Devos <maximedevos@telenet.be>
;;; Copyright © 2022 Hartmut Goebel <h.goebel@crazy-compilers.com>
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -32,7 +33,6 @@ (define-module (guix import minetest)
#:use-module (guix import utils)
#:use-module (guix import json)
#:use-module (json)
- #:use-module (guix base32)
#:use-module (guix git)
#:use-module ((guix git-download) #:prefix download:)
#:use-module (guix hash)
@@ -277,12 +277,6 @@ (define url (string-append (%contentdb-api) "packages/?type=" type
-;; XXX copied from (guix import elpa)
-(define* (download-git-repository url ref)
- "Fetch the given REF from the Git repository at URL."
- (with-store store
- (latest-repository-commit store url #:ref ref)))
-
(define (make-minetest-sexp author/name version repository commit
inputs home-page synopsis
description media-license license)
@@ -293,24 +287,8 @@ (define (make-minetest-sexp author/name version repository commit
(name ,(contentdb->package-name author/name))
(version ,version)
(source
- (origin
- (method git-fetch)
- (uri (git-reference
- (url ,repository)
- (commit ,commit)))
- (sha256
- (base32
- ;; The git commit is not always available.
- ,(and commit
- (bytevector->nix-base32-string
- (file-hash*
- (download-git-repository repository
- `(commit . ,commit))
- ;; 'download-git-repository' already filtered out the '.git'
- ;; directory.
- #:select? (const #true)
- #:recursive? #true)))))
- (file-name (git-file-name name version))))
+ ,(git->origin
+ repository `(tag-or-commit . ,commit) #:ref->commit #t))
(build-system minetest-mod-build-system)
,@(maybe-propagated-inputs (map contentdb->package-name inputs))
(home-page ,home-page)
diff --git a/guix/import/utils.scm b/guix/import/utils.scm
index 0cf52cdbde..47254539a1 100644
--- a/guix/import/utils.scm
+++ b/guix/import/utils.scm
@@ -13,6 +13,7 @@
;;; Copyright © 2022 Alice Brenon <alice.brenon@ens-lyon.fr>
;;; Copyright © 2022 Kyle Meyer <kyle@kyleam.com>
;;; Copyright © 2022 Philip McGrath <philip@philipmcgrath.com>
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -39,6 +40,8 @@ (define-module (guix import utils)
#:use-module (guix packages)
#:use-module (guix discovery)
#:use-module (guix build-system)
+ #:use-module (guix git)
+ #:use-module (guix hash)
#:use-module ((guix i18n) #:select (G_))
#:use-module (guix store)
#:use-module (guix download)
@@ -63,6 +66,7 @@ (define-module (guix import utils)
url-fetch
guix-hash-url
+ git->origin
package-names->package-inputs
maybe-inputs
@@ -161,6 +165,41 @@ (define (guix-hash-url filename)
"Return the hash of FILENAME in nix-base32 format."
(bytevector->nix-base32-string (file-sha256 filename)))
+(define* (git->origin repo-url ref #:key (ref->commit #f))
+ "Returns a generated `origin' block of a package depending on the git source
+control system, and the directory in the store where the package has been
+downloaded, in case further processing is necessary. REPO-URL or REF can be
+null. REF->COMMIT can be a function or #t, in which case the commit matching
+ref is used. If REF->COMMIT is not used, the value inside REF is used."
+ (let* ((version (and (pair? ref) (cdr ref)))
+ (directory commit
+ (if version
+ (with-store store
+ (latest-repository-commit store repo-url
+ #:ref (if version ref '())))
+ (values #f #f)))
+ (vcommit (match ref->commit
+ (#t commit)
+ (#f version)
+ ((? procedure?) (ref->commit version))
+ (_ #f))))
+ (values
+ `(origin
+ (method git-fetch)
+ (uri (git-reference
+ (url ,(and (not (eq? repo-url 'null)) repo-url))
+ (commit ,vcommit)))
+ (file-name (git-file-name name version))
+ (sha256
+ (base32
+ ,(and version ; Version or commit is not always available.
+ (bytevector->nix-base32-string
+ (file-hash* directory
+ ;; 'git-fetch' already filtered out '.git'.
+ #:select? (const #true)
+ #:recursive? #true))))))
+ directory)))
+
(define %spdx-license-identifiers
;; https://spdx.org/licenses/
;; The gfl1.0, nmap, repoze
diff --git a/tests/go.scm b/tests/go.scm
index d2e8846b30..4644e1bd1c 100644
--- a/tests/go.scm
+++ b/tests/go.scm
@@ -1,6 +1,7 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2021 François Joulaud <francois.joulaud@radiofrance.com>
;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev>
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -24,7 +25,6 @@ (define-module (tests-import-go)
#:use-module (guix base32)
#:use-module (guix build-system go)
#:use-module (guix import go)
- #:use-module (guix base32)
#:use-module ((guix utils) #:select (call-with-temporary-directory))
#:use-module (guix tests)
#:use-module (ice-9 match)
@@ -403,13 +403,26 @@ (define (mock-http-get testcase)
(mock-http-get fixtures-go-check-test))
(mock ((guix http-client) http-fetch
(mock-http-fetch fixtures-go-check-test))
- (mock ((guix git) update-cached-checkout
- (lambda* (url #:key ref)
- ;; Return an empty directory and its hash.
- (values checkout
- (nix-base32-string->bytevector
-
This message was truncated. Download the full message here.
N
N
Nicolas Graves wrote on 21 Dec 2023 06:01
[PATCH v3 3/4] import: juliahub: Beautify description.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20231221140142.16523-3-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 53 ++++++++++++++++++++++++++++++++++++++--
1 file changed, 51 insertions(+), 2 deletions(-)

Toggle diff (81 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index ab838b6035..e4540de06d 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -30,8 +30,10 @@ (define-module (guix import juliahub)
#:use-module (guix base32)
#:use-module (guix packages)
#:use-module (guix upstream)
- #:use-module (json)
#:use-module ((guix licenses) #:prefix license:)
+ #:use-module (json)
+ #:use-module (htmlprag)
+ #:use-module (sxml transform)
#:export (juliahub->guix-package
%juliahub-updater
@@ -76,6 +78,53 @@ (define (ini-fetch url)
(close-port port)
(map ini-line->alist lines)))
+;; Beautify description.
+(define %juliahub-beautify-description-rules
+ `((h1 *preorder* . ,(lambda args #f))
+ (h2 *preorder* . ,(lambda args #f))
+ (h3 *preorder* . ,(lambda args #f))
+ (h4 *preorder* . ,(lambda args #f))
+ (h5 *preorder* . ,(lambda args #f))
+ (h6 *preorder* . ,(lambda args #f))
+ (hr *preorder* . ,(lambda args #f))
+ (span *preorder* . ,(lambda args #f))
+ (img *preorder* . ,(lambda args #f))
+ (pre *preorder* . ,(lambda args #f))
+ (div *preorder* . ,(lambda args #f))
+ (table *preorder* . ,(lambda args #f))
+ (imgalt *preorder* . ,(lambda args #f))
+ (@ *preorder* . ,(lambda args #f))
+ (*TOP* . ,(lambda args (cdr args)))
+ (p . ,(lambda args (cdr args)))
+ (em . ,(lambda args (cdr args)))
+ (strong . ,(lambda args (cdr args)))
+ (a . ,(lambda args
+ (match args
+ ((tag link ref)
+ (if ref ref #f))
+ (_ #f))))
+ (ul . ,(lambda args
+ `("@itemize" ,@(cdr args) "\n@end itemize")))
+ (ol . ,(lambda args
+ `("@enumerate" ,@(cdr args) "@end enumerate")))
+ (blockquote . ,(lambda args
+ `("@quotation" ,@(cdr args) "@end quotation")))
+ (li . ,(lambda args
+ `("\n@item" ,@(cdr args))))
+ (code . ,(lambda args
+ `("@code{" ,@(cdr args) "}")))
+ (*text* . ,(lambda (tag x) x))
+ (*default* . ,(lambda (tag . body)
+ (cons tag body)))))
+
+(define (juliahub-beautify-description description)
+ (string-join
+ (filter (lambda (x) (if (equal? x " ") #f x))
+ (flatten
+ (pre-post-order (html->sxml description)
+ %juliahub-beautify-description-rules)))
+ " "))
+
;; Filtering out julia-stdlibs.
;; To update them, see file sysimg.jl.
(define %julia-stdlibs
@@ -264,7 +313,7 @@ (define* (juliahub->guix-package package-name
((? string?) homepage)
(_ (juliahub-package-url package)))
(juliahub-package-description package)
- (beautify-description
+ ((compose beautify-description juliahub-beautify-description)
(juliahub-package-readme package))
direct-dependencies
test-dependencies-names
--
2.41.0
N
N
Nicolas Graves wrote on 21 Dec 2023 06:01
[PATCH v3 2/4] import: Add juliahub importer.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20231221140142.16523-2-ngraves@ngraves.fr
---
doc/guix.texi | 27 +++
guix/import/juliahub.scm | 309 +++++++++++++++++++++++++++++++
guix/scripts/import.scm | 2 +-
guix/scripts/import/juliahub.scm | 107 +++++++++++
4 files changed, 444 insertions(+), 1 deletion(-)
create mode 100644 guix/import/juliahub.scm
create mode 100644 guix/scripts/import/juliahub.scm

Toggle diff (481 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index b742a3d5b2..f50bb3f328 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -14678,6 +14678,33 @@ guix import hexpm cf@@0.3.0
Additional options include:
+@table @code
+@item --recursive
+@itemx -r
+Traverse the dependency graph of the given upstream package recursively
+and generate package expressions for all those packages that are not yet
+in Guix.
+@end table
+
+@item juliahub
+@cindex juliahub
+Import metadata from both the General
+@uref{https://github.com/JuliaRegistries/General} and Juliahub
+@uref{https://juliahub.com} Julia package repositories, as in this
+example:
+
+@example
+guix import juliahub Cthulhu@@2.8.9
+@end example
+
+The supplied package name must have the same case as in the
+aforementioned package repositories, and the version used must be an
+exact version (e.g. @code{2.8.9} instead of @code{2.8}). The command
+will also fail in the case of a Julia package that doesn't use a git
+tag.
+
+Additional options include:
+
@table @code
@item --recursive
@itemx -r
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
new file mode 100644
index 0000000000..ab838b6035
--- /dev/null
+++ b/guix/import/juliahub.scm
@@ -0,0 +1,309 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix import juliahub)
+ #:use-module (ice-9 textual-ports)
+ #:use-module (ice-9 regex)
+ #:use-module (ice-9 match)
+ #:use-module (srfi srfi-1)
+ #:use-module (srfi srfi-9)
+ #:use-module (srfi srfi-71)
+ #:use-module (guix http-client)
+ #:use-module (guix git)
+ #:use-module (guix import utils)
+ #:use-module (guix import json)
+ #:use-module (guix base32)
+ #:use-module (guix packages)
+ #:use-module (guix upstream)
+ #:use-module (json)
+ #:use-module ((guix licenses) #:prefix license:)
+
+ #:export (juliahub->guix-package
+ %juliahub-updater
+ juliahub-recursive-import))
+
+
+;; JuliaHub API.
+(define (juliahub-redirect-uri name)
+ (let* ((url (string-append "https://docs.juliahub.com/" name "/"))
+ (port (http-fetch url #:text? #t))
+ (_ (get-line port))
+ (meta (get-line port))
+ (regex "url=[a-zA-Z0-9]{5}\\/[0-9\\.]*")
+ (redirect (match:substring (string-match regex meta))))
+ (close-port port)
+ (string-drop redirect 4)))
+
+(define (juliahub-url name)
+ (let* ((url (string-append "https://docs.juliahub.com/" name "/"))
+ (uri (juliahub-redirect-uri name)))
+ (string-append url uri "/")))
+
+;; General package repository.
+(define %general-base-url
+ "https://raw.githubusercontent.com/JuliaRegistries/General/master/")
+
+(define (general-url package-name file)
+ (let ((folder (string-capitalize (string-take package-name 1))))
+ (string-append
+ %general-base-url folder "/" package-name "/" file)))
+
+(define (ini-line->alist line)
+ (let* ((l (string-split line #\=))
+ (attribute (string->symbol (string-drop-right (car l) 1)))
+ (value (string-drop (string-drop-right (cadr l) 1) 2)))
+ `(,attribute . ,value)))
+
+(define (ini-fetch url)
+ (let* ((port (http-fetch url #:text? #t))
+ (raw (get-string-all port))
+ (lines (drop-right (string-split raw #\newline) 1)))
+ (close-port port)
+ (map ini-line->alist lines)))
+
+;; Filtering out julia-stdlibs.
+;; To update them, see file sysimg.jl.
+(define %julia-stdlibs
+ (list "julia"
+ "ArgTools"
+ "Artifacts"
+ "Base64"
+ "CRC32c"
+ "FileWatching"
+ "Libdl"
+ "Logging"
+ "Mmap"
+ "NetworkOptions"
+ "SHA"
+ "Serialization"
+ "Sockets"
+ "Unicode"
+ "DelimitedFiles"
+ "LinearAlgebra"
+ "Markdown"
+ "Printf"
+ "Random"
+ "Tar"
+ "Dates"
+ "Distributed"
+ "Future"
+ "InteractiveUtils"
+ "LibGit2"
+ "Profile"
+ "SparseArrays"
+ "UUIDs"
+ "REPL"
+ "SharedArrays"
+ "Statistics"
+ "SuiteSparse"
+ "TOML"
+ "Test"
+ "LibCURL"
+ "Downloads"
+ "Pkg"
+ "LazyArtifacts"))
+
+;; Julia package.
+(define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
+ json->juliahub-package
+ (homepage juliahub-package-homepage) ;string
+ (readme juliahub-package-readme) ;string
+ (version juliahub-package-version) ;string
+ (description juliahub-package-description) ;string
+ (dependencies
+ juliahub-package-dependencies "deps"
+ json->juliahub-dependencies) ;list of <juliahub-dependency>
+ (url juliahub-package-url) ;string
+ (uuid juliahub-package-uuid) ;string
+ (license juliahub-package-license)) ;string
+
+(define-json-mapping <juliahub-dependency>
+ make-juliahub-dependency juliahub-dependency?
+ json->juliahub-dependency
+ (direct? juliahub-dependency-direct? "direct") ;boolean
+ (name juliahub-dependency-name) ;string
+ (uuid juliahub-dependency-uuid) ;string
+ (versions juliahub-dependency-versions "versions" vector->list)) ;list of strings
+
+(define (julia-name->guix-name name)
+ (string-append "julia-" (snake-case name)))
+
+(define* (juliahub-fetch name #:key (version #f))
+ "Return a <juliahub-package> record for package NAME, or #f on failure."
+ (let* ((uri (juliahub-redirect-uri name))
+ (slug (string-take uri 5))
+ (url (if version
+ (string-append "https://docs.juliahub.com/" name "/"
+ slug "/" version "/pkg.json")
+ (string-append (juliahub-url name) "pkg.json"))))
+ (and=> (json-fetch url) json->juliahub-package)))
+
+(define (make-julia-sexp name version source home-page synopsis description
+ direct-dependencies test-dependencies-names license)
+ "Return the `package' s-expression for a Julia package with the given NAME,
+VERSION, SOURCE, HOME-PAGE, DESCRIPTION, DIRECT-DEPENDENCIES,
+TEST-DEPENDENCIES-NAMES and LICENSE."
+ `(package
+ (name ,(julia-name->guix-name name))
+ (version ,version)
+ (source ,source)
+ (build-system julia-build-system)
+ ,@(if (null? direct-dependencies)
+ '()
+ `((propagated-inputs
+ (list ,@(map (compose string->symbol
+ julia-name->guix-name
+ juliahub-dependency-name)
+ direct-dependencies)))))
+ ,@(if (null? test-dependencies-names)
+ '()
+ `((native-inputs
+ (list ,@(map (compose string->symbol julia-name->guix-name)
+ test-dependencies-names)))))
+ (synopsis ,synopsis)
+ (description ,description)
+ (home-page ,home-page)
+ (license ,(if license (spdx-string->license license) #f))))
+
+;; Dependencies helpers.
+(define (json->juliahub-dependencies vector)
+ (if (vector? vector)
+ (filter-map
+ (lambda (el)
+ (let ((dep (json->juliahub-dependency el)))
+ (if (and (juliahub-dependency-direct? dep)
+ (not (member (juliahub-dependency-name dep)
+ %julia-stdlibs)))
+ dep
+ #f)))
+ (vector->list vector))))
+
+(define (parse-test-dependencies directory)
+ (let* ((port (open-input-file (string-append directory "/Project.toml")))
+ (project.toml (get-string-all port))
+ (regex "\ntest = \\[.*\\]")
+ (deps (match:substring (string-match regex project.toml)))
+ (pure (string-delete (list->char-set (list #\" #\ )) deps)))
+ (close-port port)
+ (filter (lambda (x) (not (member x %julia-stdlibs)))
+ (string-split (string-drop (string-drop-right pure 1) 7) #\,))))
+
+;; Juliahub may be more up-to-date than the General registry or the actual git
+;; tag (it seems around 6 hours pass between the time a commit is supplied to
+;; JuliaRegistrator as a release, and the time Julia TagBot Github Action makes
+;; the git tag). We have no simple way to get the commit of the latest-version.
+;; Thus the best simple thing we can do is get the latest-git-tag, and import
+;; this version instead. We do this by parsing Package.toml in the General
+;; registry, and then getting the refs of the git repo supplied by this
+;; file. Parsing this file is also necessary if the package is in a subdir of a
+;; git repository, because the information isn't present in Juliahub.
+
+;; There's a last case where some Julia packages are not based on a particular
+;; git tag. In this case, the script fails, but it seems quite rare. We could
+;; introduce the tree-commit which is available in the Versions.toml file in the
+;; General repository. This can be used to identify the state of a repository,
+;; since we have a unique hash of the listing of files and directories.
+
+(define (latest-git-tag repo)
+ (let* ((last-ref (last (remote-refs repo #:tags? #t)))
+ (last-git-tag (last (string-split last-ref #\/))))
+ (string-drop last-git-tag 1)))
+
+(define* (juliahub->guix-package package-name
+ #:key version #:allow-other-keys)
+ "Fetch the metadata for PACKAGE-NAME from juliahub.org, and return the
+`package' s-expression corresponding to that package, or #f on failure.
+Optionally include a VERSION string to fetch a specific version juliahub."
+ (let* ((package-toml (ini-fetch (general-url package-name "Package.toml")))
+ (subdir (assoc-ref package-toml 'subdir))
+ (tag (latest-git-tag (assoc-ref package-toml 'repo)))
+ (package (if version
+ (juliahub-fetch package-name #:version version)
+ (if tag
+ (juliahub-fetch package-name #:version tag)
+ (juliahub-fetch package-name)))))
+ (if package
+ (let* ((source directory
+ (git->origin
+ (juliahub-package-url package)
+ `(tag-or-commit
+ . ,(string-append
+ "v" (juliahub-package-version package)))))
+ (direct-dependencies
+ (filter juliahub-dependency-direct?
+ (juliahub-package-dependencies package)))
+ (dependencies-names (map juliahub-dependency-name
+ direct-dependencies))
+ (test-dependencies-names
+ (if subdir
+ (parse-test-dependencies
+ (string-append subdir "/" directory))
+ (parse-test-dependencies directory)))
+ (homepage (juliahub-package-homepage package)))
+ (values (make-julia-sexp
+ package-name
+ (juliahub-package-version package)
+ source
+ (match homepage
+ ("" (juliahub-package-url package))
+ ((? string?) homepage)
+ (_ (juliahub-package-url package)))
+ (juliahub-package-description package)
+ (beautify-description
+ (juliahub-package-readme package))
+ direct-dependencies
+ test-dependencies-names
+ (juliahub-package-license package))
+ (append dependencies-names test-dependencies-names)))
+ (values #f '()))))
+
+;; We must use the url to get a name with the true case of juliahub/general.
+(define (guix-package->juliahub-name package)
+ (let* ((url (juliahub-package-url package))
+ (git-name (last (string-split url #\/)))
+ (ungitted-name (if (string-suffix? ".git" git-name)
+ (string-drop-right git-name 4)
+ git-name))
+ (package-name (if (string-suffix? ".jl" ungitted-name)
+ (string-drop-right ungitted-name 4)
+ ungitted-name)))
+ package-name))
+
+(define* (import-release package #:key (version #f))
+ "Return an <upstream-source> for the latest release of PACKAGE."
+ (let* ((package-name (guix-package->juliahub-name package))
+ (package (juliahub-fetch package-name))
+ (version (or version (juliahub-package-version package))))
+ (upstream-source
+ (package (package-name package))
+ (version version)
+ (urls (list (juliahub-package-url package))))))
+
+(define %juliahub-updater
+ (upstream-updater
+ (name 'juliahub)
+ (description "Updater for Juliahub packages")
+ (pred juliahub-package?)
+ (import import-release)))
+
+(define* (juliahub-recursive-import package-name #:optional version)
+ (recursive-import package-name
+ #:repo '()
+ #:repo->guix-package juliahub->guix-package
+ #:guix-name julia-name->guix-name
+ #:version version))
diff --git a/guix/scripts/import.scm b/guix/scripts/import.scm
index d2a1cee56e..8926c9610f 100644
--- a/guix/scripts/import.scm
+++ b/guix/scripts/import.scm
@@ -47,7 +47,7 @@ (define %standard-import-options '())
(define importers '("gnu" "pypi" "cpan" "hackage" "stackage" "egg" "elpa"
"gem" "go" "cran" "crate" "texlive" "json" "opam"
- "minetest" "elm" "hexpm" "composer"))
+ "minetest" "elm" "hexpm" "composer" "juliahub"))
(define (resolve-importer name)
(let ((module (resolve-interface
diff --git a/guix/scripts/import/juliahub.scm b/guix/scripts/import/juliahub.scm
new file mode 100644
index 0000000000..1317c67aa3
--- /dev/null
+++ b/guix/scripts/import/juliahub.scm
@@ -0,0 +1,107 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix scripts import juliahub)
+ #:use-module (guix ui)
+ #:use-module (guix utils)
+ #:use-module (guix scripts)
+ #:use-module (guix import juliahub)
+ #:use-module (guix scripts import)
+ #:use-module (srfi srfi-1)
+ #:use-module (srfi srfi-11)
+ #:use-module (srfi srfi-37)
+ #:use-module (ice-9 match)
+ #:use-module (ice-9 format)
+ #:use-module (ice-9 receive)
+ #:export (guix-import-juliahub))
+
+
+;;;
+;;; Command-line options.
+;;;
+
+(define %default-options
+ '())
+
+(define (show-help)
+ (display (G_ "Usage: guix import juliahub PACKAGE-NAME[@VERSION] Import and
+convert the Julia package for PACKAGE-NAME. Optionally, a version can be
+specified after the at-sign (@) character.\n"))
+ (display (G_ "
+ -h, --help display this help and exit"))
+ (display (G_ "
+ -V, --version display version information and exit"))
+ (display (G_ "
+ -r, --recursive generate package expressions for all Gem packages\
+ that are not yet in Guix"))
+ (newline)
+ (show-bug-report-information))
+
+(define %options
+ ;; Specification of the command-line options.
+ (cons* (option '(#\h "help") #f #f
+ (lambda args
+ (show-help)
+ (exit 0)))
+ (option '(#\V "version") #f #f
+ (lambda args
+ (show-version-and-exit "guix import gem")))
+ (option '(#\r "recursive") #f #f
+ (lambda (opt name arg result)
+ (alist-cons 'recursive #t result)))
+ %standard-import-options))
+
+
+;;;
+;;; Entry point.
+;;;
+
+(define (guix-import-juliahub . args)
+ (define (parse-options)
+ ;; Return the alist of option values.
+ (parse-command-line args %options (list %default-options)
+ #:build-options? #f))
+
+ (let* ((opts (parse-options))
+ (args (filter-map (match-lambda
+ (('argument . value)
+ value)
+ (_ #f))
+ (reverse opts))))
+ (match args
+ ((spec)
+ (receive (package-name package-version)
+ (package-name->name+version spec)
+ (let ((code (if (assoc-ref opts 'recursive)
+ (map (match-lambda
+ ((and ('package ('name name) . rest) pkg)
+ `(define-public ,(string->symbol name)
+ ,pkg))
+ (_ #f))
+ (juliahub-recursive-import package-name package-version))
+ (let ((sexp (juliahub->guix-package package-name #:version package-version)))
+ (if sexp sexp #f)))))
+ (match code
+ ((or #f '(#f))
+ (leave (G_ "failed to download meta-data for package '~a'~%")
+ package-name))
+ (_ code)))))
+ (()
+ (leave (G_ "too few arguments~%")))
+ ((many ...)
+ (leave (G_ "too many arguments~%"))))))
--
2.41.0
N
N
Nicolas Graves wrote on 21 Dec 2023 06:01
[PATCH v3 4/4] import: utils: Rule out texinfo common syntax from @ escape.
(address . 62202@debbugs.gnu.org)(address . ngraves@ngraves.fr)
20231221140142.16523-4-ngraves@ngraves.fr
---
guix/import/utils.scm | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)

Toggle diff (21 lines)
diff --git a/guix/import/utils.scm b/guix/import/utils.scm
index 47254539a1..57e4ec0ce7 100644
--- a/guix/import/utils.scm
+++ b/guix/import/utils.scm
@@ -375,7 +375,13 @@ (define* (beautify-description description #:optional (length 80))
(cut string-trim-both <> #\')
;; Escape single @ to prevent it from being understood as
;; invalid Texinfo syntax.
- (cut regexp-substitute/global #f "@" <> 'pre "@@" 'post)
+ (lambda (word)
+ (if ; Rule out some valid Texinfo syntax.
+ (member word '("@itemize" "@item" "@end" "@quotation"
+ "@enumerate" "@code" "@code{"))
+ word
+ ((cut regexp-substitute/global
+ #f "@" <> 'pre "@@" 'post) word)))
;; Wrap camelCase or PascalCase words in @code{...}.
(lambda (word)
(let ((pattern (make-regexp "([A-Z][a-z]+[A-Z]|[a-z]+[A-Z])")))
--
2.41.0
N
N
Nicolas Graves wrote on 3 Feb 15:07 -0800
[PATCH v4 1/6] import: utils: Add function git->origin.
(address . 62202@debbugs.gnu.org)
20240203230807.25751-1-ngraves@ngraves.fr
* guix/import/utils.scm: (git->origin): Add function.

* guix/import/elpa.scm
(download-git-repository): Remove function download-git-repository.
(git-repository->origin): Remove function git-repository->origin.
(ref): Add function ref.
(melpa-recipe->origin): Use functions git->origin and ref.

* guix/import/go.scm
(git-checkout-hash): Remove function git-checkout-hash.
(transform-version): Add function transform-version.
(vcs->origin): Use functions git->origin and transform-version. Add
optional argument transform-version.

* tests/import/go.scm
(go-module->guix-package): Adapt test case to changes in guix/import/go.scm.

* guix/import/minetest.scm
(download-git-repository): Remove function download-git-repository.
(make-minetest-sexp): Use function git->origin.

* tests/minetest.scm
(make-package-sexp): Use function git->origin.
(example-package): Adapt test-case to git->origin.

* guix/import/composer.scm
(make-php-sexp): Use function git->origin.

Change-Id: Ied05a63bdd60fbafe26fbbb4e115ff6f0bb9db3c
---
guix/import/composer.scm | 86 ++++++++++++++--------------------------
guix/import/elpa.scm | 43 ++++++--------------
guix/import/go.scm | 57 ++++++++------------------
guix/import/minetest.scm | 28 ++-----------
guix/import/utils.scm | 39 ++++++++++++++++++
tests/go.scm | 29 ++++++++++----
tests/minetest.scm | 15 ++-----
7 files changed, 127 insertions(+), 170 deletions(-)

Toggle diff (435 lines)
diff --git a/guix/import/composer.scm b/guix/import/composer.scm
index 1ad608964b..dabc5423ed 100644
--- a/guix/import/composer.scm
+++ b/guix/import/composer.scm
@@ -19,22 +19,17 @@
(define-module (guix import composer)
#:use-module (ice-9 match)
#:use-module (json)
- #:use-module (guix hash)
- #:use-module (guix base32)
- #:use-module (guix build git)
- #:use-module (guix build utils)
- #:use-module (guix build-system)
#:use-module (guix build-system composer)
+ #:use-module ((guix download) #:select (download-to-store))
#:use-module (guix import json)
#:use-module (guix import utils)
#:use-module ((guix licenses) #:prefix license:)
#:use-module (guix memoization)
#:use-module (guix packages)
- #:use-module (guix serialization)
+ #:use-module (guix store)
#:use-module (guix upstream)
#:use-module (guix utils)
#:use-module (srfi srfi-1)
- #:use-module (srfi srfi-11)
#:use-module (srfi srfi-26)
#:export (composer->guix-package
%composer-updater
@@ -141,55 +136,34 @@ (define (make-php-sexp composer-package)
(dependencies (map php-package-name
(composer-package-require composer-package)))
(dev-dependencies (map php-package-name
- (composer-package-dev-require composer-package)))
- (git? (equal? (composer-source-type source) "git")))
- ((if git? call-with-temporary-directory call-with-temporary-output-file)
- (lambda* (temp #:optional port)
- (and (if git?
- (begin
- (mkdir-p temp)
- (git-fetch (composer-source-url source)
- (composer-source-reference source)
- temp))
- (url-fetch (composer-source-url source) temp))
- `(package
- (name ,(composer-package-name composer-package))
- (version ,(composer-package-version composer-package))
- (source
- (origin
- ,@(if git?
- `((method git-fetch)
- (uri (git-reference
- (url ,(if (string-suffix?
- ".git"
- (composer-source-url source))
- (string-drop-right
- (composer-source-url source)
- (string-length ".git"))
- (composer-source-url source)))
- (commit ,(composer-source-reference source))))
- (file-name (git-file-name name version))
- (sha256
- (base32
- ,(bytevector->nix-base32-string
- (file-hash* temp)))))
- `((method url-fetch)
- (uri ,(composer-source-url source))
- (sha256 (base32 ,(guix-hash-url temp)))))))
- (build-system composer-build-system)
- ,@(if (null? dependencies)
- '()
- `((inputs
- (list ,@(map string->symbol dependencies)))))
- ,@(if (null? dev-dependencies)
- '()
- `((native-inputs
- (list ,@(map string->symbol dev-dependencies)))))
- (synopsis "")
- (description ,(composer-package-description composer-package))
- (home-page ,(composer-package-homepage composer-package))
- (license ,(or (composer-package-license composer-package)
- 'unknown-license!))))))))
+ (composer-package-dev-require composer-package))))
+ `(package
+ (name ,(composer-package-name composer-package))
+ (version ,(composer-package-version composer-package))
+ (source
+ ,(if (string= (composer-source-type source) "git")
+ (git->origin (composer-source-url source)
+ `(tag-or-commit . ,(composer-source-reference source)))
+ (let* ((source (composer-source-url source))
+ (tarball (with-store store (download-to-store store source))))
+ `(origin
+ (method url-fetch)
+ (uri ,source)
+ (sha256 (base32 ,(guix-hash-url tarball)))))))
+ (build-system composer-build-system)
+ ,@(if (null? dependencies)
+ '()
+ `((inputs
+ (list ,@(map string->symbol dependencies)))))
+ ,@(if (null? dev-dependencies)
+ '()
+ `((native-inputs
+ (list ,@(map string->symbol dev-dependencies)))))
+ (synopsis "")
+ (description ,(composer-package-description composer-package))
+ (home-page ,(composer-package-homepage composer-package))
+ (license ,(or (composer-package-license composer-package)
+ 'unknown-license!)))))
(define composer->guix-package
(memoize
diff --git a/guix/import/elpa.scm b/guix/import/elpa.scm
index d1855b3698..a755387242 100644
--- a/guix/import/elpa.scm
+++ b/guix/import/elpa.scm
@@ -8,6 +8,7 @@
;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev>
;;; Copyright © 2021 Simon Tournier <zimon.toutoune@gmail.com>
;;; Copyright © 2022 Hartmut Goebel <h.goebel@crazy-compilers.com>
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -208,11 +209,6 @@ (define* (fetch-elpa-package name #:optional (repo 'gnu))
url)))
(_ #f))))
-(define* (download-git-repository url ref)
- "Fetch the given REF from the Git repository at URL."
- (with-store store
- (latest-repository-commit store url #:ref ref)))
-
(define (package-name->melpa-recipe package-name)
"Fetch the MELPA recipe for PACKAGE-NAME, represented as an alist from
keywords to values."
@@ -232,28 +228,15 @@ (define (data->recipe data)
(close-port port)
(data->recipe (cons ':name data))))
-(define (git-repository->origin recipe url)
- "Fetch origin details from the Git repository at URL for the provided MELPA
-RECIPE."
- (define ref
- (cond
- ((assoc-ref recipe #:branch)
- => (lambda (branch) (cons 'branch branch)))
- ((assoc-ref recipe #:commit)
- => (lambda (commit) (cons 'commit commit)))
- (else
- '())))
-
- (let-values (((directory commit) (download-git-repository url ref)))
- `(origin
- (method git-fetch)
- (uri (git-reference
- (url ,url)
- (commit ,commit)))
- (sha256
- (base32
- ,(bytevector->nix-base32-string
- (file-hash* directory #:recursive? #true)))))))
+(define (ref recipe)
+ "Create REF from MELPA RECIPE."
+ (cond
+ ((assoc-ref recipe #:branch)
+ => (lambda (branch) (cons 'branch branch)))
+ ((assoc-ref recipe #:commit)
+ => (lambda (commit) (cons 'commit commit)))
+ (else
+ '())))
(define* (melpa-recipe->origin recipe)
"Fetch origin details from the MELPA recipe and associated repository for
@@ -264,9 +247,9 @@ (define (gitlab-repo->url repo)
(string-append "https://gitlab.com/" repo ".git"))
(match (assq-ref recipe ':fetcher)
- ('github (git-repository->origin recipe (github-repo->url (assq-ref recipe ':repo))))
- ('gitlab (git-repository->origin recipe (gitlab-repo->url (assq-ref recipe ':repo))))
- ('git (git-repository->origin recipe (assq-ref recipe ':url)))
+ ('github (git->origin (github-repo->url (assq-ref recipe ':repo)) (ref recipe)))
+ ('gitlab (git->origin (gitlab-repo->url (assq-ref recipe ':repo)) (ref recipe)))
+ ('git (git->origin (assq-ref recipe ':url) (ref recipe)))
(#f #f) ; if we're not using melpa then this stops us printing a warning
(_ (warning (G_ "unsupported MELPA fetcher: ~a, falling back to unstable MELPA source~%")
(assq-ref recipe ':fetcher))
diff --git a/guix/import/go.scm b/guix/import/go.scm
index dd9298808d..6e2ce2ed00 100644
--- a/guix/import/go.scm
+++ b/guix/import/go.scm
@@ -8,6 +8,7 @@
;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev>
;;; Copyright © 2021 Simon Tournier <zimon.toutoune@gmail.com>
;;; Copyright © 2023 Efraim Flashner <efraim@flashner.co.il>
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -514,49 +515,24 @@ (define (module-meta-data-repo-url meta-data goproxy)
goproxy
(module-meta-repo-root meta-data)))
-(define* (git-checkout-hash url reference algorithm)
- "Return the ALGORITHM hash of the checkout of URL at REFERENCE, a commit or
-tag."
- (define cache
- (string-append (or (getenv "TMPDIR") "/tmp")
- "/guix-import-go-"
- (passwd:name (getpwuid (getuid)))))
+;; This is done because the version field of the package, which the generated
+;; quoted expression refers to, has been stripped of any 'v' prefixed.
+(define (transform-version version)
+ (let ((plain-version? (string=? version (go-version->git-ref version)))
+ (v-prefixed? (string-prefix? "v" version)))
+ (if (and plain-version? v-prefixed?)
+ '(string-append "v" version)
+ '(go-version->git-ref version))))
- ;; Use a custom cache to avoid cluttering the default one under
- ;; ~/.cache/guix, but choose one under /tmp so that it's persistent across
- ;; subsequent "guix import" invocations.
- (mkdir-p cache)
- (chmod cache #o700)
- (let-values (((checkout commit _)
- (parameterize ((%repository-cache-directory cache))
- (update-cached-checkout url
- #:ref
- `(tag-or-commit . ,reference)))))
- (file-hash* checkout #:algorithm algorithm #:recursive? #true)))
-
-(define (vcs->origin vcs-type vcs-repo-url version)
+(define* (vcs->origin vcs-type vcs-repo-url version
+ #:key (transform-version #f))
"Generate the `origin' block of a package depending on what type of source
-control system is being used."
+control system is being used. Optionally use the function TRANSFORM-VERSION
+which takes version as an input."
(case vcs-type
((git)
- (let ((plain-version? (string=? version (go-version->git-ref version)))
- (v-prefixed? (string-prefix? "v" version)))
- `(origin
- (method git-fetch)
- (uri (git-reference
- (url ,vcs-repo-url)
- ;; This is done because the version field of the package,
- ;; which the generated quoted expression refers to, has been
- ;; stripped of any 'v' prefixed.
- (commit ,(if (and plain-version? v-prefixed?)
- '(string-append "v" version)
- '(go-version->git-ref version)))))
- (file-name (git-file-name name version))
- (sha256
- (base32
- ,(bytevector->nix-base32-string
- (git-checkout-hash vcs-repo-url (go-version->git-ref version)
- (hash-algorithm sha256))))))))
+ (git->origin vcs-repo-url `(tag-or-commit . ,version)
+ #:ref->commit transform-version))
((hg)
`(origin
(method hg-fetch)
@@ -649,7 +625,8 @@ (define* (go-module->guix-package module-path #:key
(name ,guix-name)
(version ,(strip-v-prefix version*))
(source
- ,(vcs->origin vcs-type vcs-repo-url version*))
+ ,(vcs->origin vcs-type vcs-repo-url version*
+ #:transform-version transform-version))
(build-system go-build-system)
(arguments
(list ,@(if (version>? min-go-version (package-version (go-package)))
diff --git a/guix/import/minetest.scm b/guix/import/minetest.scm
index 5ea6e023ce..65ef242431 100644
--- a/guix/import/minetest.scm
+++ b/guix/import/minetest.scm
@@ -1,6 +1,7 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2021, 2022 Maxime Devos <maximedevos@telenet.be>
;;; Copyright © 2022 Hartmut Goebel <h.goebel@crazy-compilers.com>
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -32,7 +33,6 @@ (define-module (guix import minetest)
#:use-module (guix import utils)
#:use-module (guix import json)
#:use-module (json)
- #:use-module (guix base32)
#:use-module (guix git)
#:use-module ((guix git-download) #:prefix download:)
#:use-module (guix hash)
@@ -277,12 +277,6 @@ (define url (string-append (%contentdb-api) "packages/?type=" type
-;; XXX copied from (guix import elpa)
-(define* (download-git-repository url ref)
- "Fetch the given REF from the Git repository at URL."
- (with-store store
- (latest-repository-commit store url #:ref ref)))
-
(define (make-minetest-sexp author/name version repository commit
inputs home-page synopsis
description media-license license)
@@ -293,24 +287,8 @@ (define (make-minetest-sexp author/name version repository commit
(name ,(contentdb->package-name author/name))
(version ,version)
(source
- (origin
- (method git-fetch)
- (uri (git-reference
- (url ,repository)
- (commit ,commit)))
- (sha256
- (base32
- ;; The git commit is not always available.
- ,(and commit
- (bytevector->nix-base32-string
- (file-hash*
- (download-git-repository repository
- `(commit . ,commit))
- ;; 'download-git-repository' already filtered out the '.git'
- ;; directory.
- #:select? (const #true)
- #:recursive? #true)))))
- (file-name (git-file-name name version))))
+ ,(git->origin
+ repository `(tag-or-commit . ,commit) #:ref->commit #t))
(build-system minetest-mod-build-system)
,@(maybe-propagated-inputs (map contentdb->package-name inputs))
(home-page ,home-page)
diff --git a/guix/import/utils.scm b/guix/import/utils.scm
index 0cf52cdbde..47254539a1 100644
--- a/guix/import/utils.scm
+++ b/guix/import/utils.scm
@@ -13,6 +13,7 @@
;;; Copyright © 2022 Alice Brenon <alice.brenon@ens-lyon.fr>
;;; Copyright © 2022 Kyle Meyer <kyle@kyleam.com>
;;; Copyright © 2022 Philip McGrath <philip@philipmcgrath.com>
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -39,6 +40,8 @@ (define-module (guix import utils)
#:use-module (guix packages)
#:use-module (guix discovery)
#:use-module (guix build-system)
+ #:use-module (guix git)
+ #:use-module (guix hash)
#:use-module ((guix i18n) #:select (G_))
#:use-module (guix store)
#:use-module (guix download)
@@ -63,6 +66,7 @@ (define-module (guix import utils)
url-fetch
guix-hash-url
+ git->origin
package-names->package-inputs
maybe-inputs
@@ -161,6 +165,41 @@ (define (guix-hash-url filename)
"Return the hash of FILENAME in nix-base32 format."
(bytevector->nix-base32-string (file-sha256 filename)))
+(define* (git->origin repo-url ref #:key (ref->commit #f))
+ "Returns a generated `origin' block of a package depending on the git source
+control system, and the directory in the store where the package has been
+downloaded, in case further processing is necessary. REPO-URL or REF can be
+null. REF->COMMIT can be a function or #t, in which case the commit matching
+ref is used. If REF->COMMIT is not used, the value inside REF is used."
+ (let* ((version (and (pair? ref) (cdr ref)))
+ (directory commit
+ (if version
+ (with-store store
+ (latest-repository-commit store repo-url
+ #:ref (if version ref '())))
+ (values #f #f)))
+ (vcommit (match ref->commit
+ (#t commit)
+ (#f version)
+ ((? procedure?) (ref->commit version))
+ (_ #f))))
+ (values
+ `(origin
+ (method git-fetch)
+ (uri (git-reference
+ (url ,(and (not (eq? repo-url 'null)) repo-url))
+ (commit ,vcommit)))
+ (file-name (git-file-name name version))
+ (sha256
+ (base32
+ ,(and version ; Version or commit is not always available.
+ (bytevector->nix-base32-string
+ (file-hash* directory
+ ;; 'git-fetch' already filtered out '.git'.
+ #:select? (const #true)
+ #:recursive? #true))))))
+ directory)))
+
(define %spdx-license-identifiers
;; https://spdx.org/licenses/
;; The gfl1.0, nmap, repoze
diff --git a/tests/go.scm b/tests/go.scm
index d2e8846b30..4644e1bd1c 100644
--- a/tests/go.scm
+++ b/tests/go.scm
@@ -1,6 +1,7 @@
;;; GNU Guix --- Functional package management for GNU
;;; Copyright © 2021 François Joulaud <francois.joulaud@radiofrance.com>
;;; Copyright © 2021 Sarah Morgensen <iskarian@mgsn.dev>
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
;;;
;;; This file is part of GNU Guix.
;;;
@@ -24,7 +25,6 @@ (define-module (tests-import-go)
#:use-module (guix base32)
#:use-module (guix build-system go)
#:use-module (guix import go)
- #:use-module (guix base32)
#:use-module ((guix utils) #:select (call-with-temporary-directory))
#:use-module (guix tests)
#:use-module (ice-9 match)
@@ -403,13 +403,26 @@ (define (mock-http-get testcase)
(mock-http-get fixtures-go-check-test))
(mock ((guix http-client) http-fetch
(mock-http-fetch fixtures-go-check-test))
- (mock ((guix git) update-cached-checkout
- (lambda* (url #:key ref)
- ;; Return an empty directory and its hash.
- (values checkout
- (nix-base32-string->bytevector
-
This message was truncated. Download the full message here.
N
N
Nicolas Graves wrote on 3 Feb 15:07 -0800
[PATCH v4 2/6] import: Add juliahub importer.
(address . 62202@debbugs.gnu.org)
20240203230807.25751-2-ngraves@ngraves.fr
---
doc/guix.texi | 27 +++
guix/import/juliahub.scm | 309 +++++++++++++++++++++++++++++++
guix/scripts/import.scm | 2 +-
guix/scripts/import/juliahub.scm | 107 +++++++++++
4 files changed, 444 insertions(+), 1 deletion(-)
create mode 100644 guix/import/juliahub.scm
create mode 100644 guix/scripts/import/juliahub.scm

Toggle diff (481 lines)
diff --git a/doc/guix.texi b/doc/guix.texi
index c71d7e94cf..6baa726517 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -14782,6 +14782,33 @@ guix import hexpm cf@@0.3.0
Additional options include:
+@table @code
+@item --recursive
+@itemx -r
+Traverse the dependency graph of the given upstream package recursively
+and generate package expressions for all those packages that are not yet
+in Guix.
+@end table
+
+@item juliahub
+@cindex juliahub
+Import metadata from both the General
+@uref{https://github.com/JuliaRegistries/General} and Juliahub
+@uref{https://juliahub.com} Julia package repositories, as in this
+example:
+
+@example
+guix import juliahub Cthulhu@@2.8.9
+@end example
+
+The supplied package name must have the same case as in the
+aforementioned package repositories, and the version used must be an
+exact version (e.g. @code{2.8.9} instead of @code{2.8}). The command
+will also fail in the case of a Julia package that doesn't use a git
+tag.
+
+Additional options include:
+
@table @code
@item --recursive
@itemx -r
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
new file mode 100644
index 0000000000..ab838b6035
--- /dev/null
+++ b/guix/import/juliahub.scm
@@ -0,0 +1,309 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix import juliahub)
+ #:use-module (ice-9 textual-ports)
+ #:use-module (ice-9 regex)
+ #:use-module (ice-9 match)
+ #:use-module (srfi srfi-1)
+ #:use-module (srfi srfi-9)
+ #:use-module (srfi srfi-71)
+ #:use-module (guix http-client)
+ #:use-module (guix git)
+ #:use-module (guix import utils)
+ #:use-module (guix import json)
+ #:use-module (guix base32)
+ #:use-module (guix packages)
+ #:use-module (guix upstream)
+ #:use-module (json)
+ #:use-module ((guix licenses) #:prefix license:)
+
+ #:export (juliahub->guix-package
+ %juliahub-updater
+ juliahub-recursive-import))
+
+
+;; JuliaHub API.
+(define (juliahub-redirect-uri name)
+ (let* ((url (string-append "https://docs.juliahub.com/" name "/"))
+ (port (http-fetch url #:text? #t))
+ (_ (get-line port))
+ (meta (get-line port))
+ (regex "url=[a-zA-Z0-9]{5}\\/[0-9\\.]*")
+ (redirect (match:substring (string-match regex meta))))
+ (close-port port)
+ (string-drop redirect 4)))
+
+(define (juliahub-url name)
+ (let* ((url (string-append "https://docs.juliahub.com/" name "/"))
+ (uri (juliahub-redirect-uri name)))
+ (string-append url uri "/")))
+
+;; General package repository.
+(define %general-base-url
+ "https://raw.githubusercontent.com/JuliaRegistries/General/master/")
+
+(define (general-url package-name file)
+ (let ((folder (string-capitalize (string-take package-name 1))))
+ (string-append
+ %general-base-url folder "/" package-name "/" file)))
+
+(define (ini-line->alist line)
+ (let* ((l (string-split line #\=))
+ (attribute (string->symbol (string-drop-right (car l) 1)))
+ (value (string-drop (string-drop-right (cadr l) 1) 2)))
+ `(,attribute . ,value)))
+
+(define (ini-fetch url)
+ (let* ((port (http-fetch url #:text? #t))
+ (raw (get-string-all port))
+ (lines (drop-right (string-split raw #\newline) 1)))
+ (close-port port)
+ (map ini-line->alist lines)))
+
+;; Filtering out julia-stdlibs.
+;; To update them, see file sysimg.jl.
+(define %julia-stdlibs
+ (list "julia"
+ "ArgTools"
+ "Artifacts"
+ "Base64"
+ "CRC32c"
+ "FileWatching"
+ "Libdl"
+ "Logging"
+ "Mmap"
+ "NetworkOptions"
+ "SHA"
+ "Serialization"
+ "Sockets"
+ "Unicode"
+ "DelimitedFiles"
+ "LinearAlgebra"
+ "Markdown"
+ "Printf"
+ "Random"
+ "Tar"
+ "Dates"
+ "Distributed"
+ "Future"
+ "InteractiveUtils"
+ "LibGit2"
+ "Profile"
+ "SparseArrays"
+ "UUIDs"
+ "REPL"
+ "SharedArrays"
+ "Statistics"
+ "SuiteSparse"
+ "TOML"
+ "Test"
+ "LibCURL"
+ "Downloads"
+ "Pkg"
+ "LazyArtifacts"))
+
+;; Julia package.
+(define-json-mapping <juliahub-package> make-juliahub-package juliahub-package?
+ json->juliahub-package
+ (homepage juliahub-package-homepage) ;string
+ (readme juliahub-package-readme) ;string
+ (version juliahub-package-version) ;string
+ (description juliahub-package-description) ;string
+ (dependencies
+ juliahub-package-dependencies "deps"
+ json->juliahub-dependencies) ;list of <juliahub-dependency>
+ (url juliahub-package-url) ;string
+ (uuid juliahub-package-uuid) ;string
+ (license juliahub-package-license)) ;string
+
+(define-json-mapping <juliahub-dependency>
+ make-juliahub-dependency juliahub-dependency?
+ json->juliahub-dependency
+ (direct? juliahub-dependency-direct? "direct") ;boolean
+ (name juliahub-dependency-name) ;string
+ (uuid juliahub-dependency-uuid) ;string
+ (versions juliahub-dependency-versions "versions" vector->list)) ;list of strings
+
+(define (julia-name->guix-name name)
+ (string-append "julia-" (snake-case name)))
+
+(define* (juliahub-fetch name #:key (version #f))
+ "Return a <juliahub-package> record for package NAME, or #f on failure."
+ (let* ((uri (juliahub-redirect-uri name))
+ (slug (string-take uri 5))
+ (url (if version
+ (string-append "https://docs.juliahub.com/" name "/"
+ slug "/" version "/pkg.json")
+ (string-append (juliahub-url name) "pkg.json"))))
+ (and=> (json-fetch url) json->juliahub-package)))
+
+(define (make-julia-sexp name version source home-page synopsis description
+ direct-dependencies test-dependencies-names license)
+ "Return the `package' s-expression for a Julia package with the given NAME,
+VERSION, SOURCE, HOME-PAGE, DESCRIPTION, DIRECT-DEPENDENCIES,
+TEST-DEPENDENCIES-NAMES and LICENSE."
+ `(package
+ (name ,(julia-name->guix-name name))
+ (version ,version)
+ (source ,source)
+ (build-system julia-build-system)
+ ,@(if (null? direct-dependencies)
+ '()
+ `((propagated-inputs
+ (list ,@(map (compose string->symbol
+ julia-name->guix-name
+ juliahub-dependency-name)
+ direct-dependencies)))))
+ ,@(if (null? test-dependencies-names)
+ '()
+ `((native-inputs
+ (list ,@(map (compose string->symbol julia-name->guix-name)
+ test-dependencies-names)))))
+ (synopsis ,synopsis)
+ (description ,description)
+ (home-page ,home-page)
+ (license ,(if license (spdx-string->license license) #f))))
+
+;; Dependencies helpers.
+(define (json->juliahub-dependencies vector)
+ (if (vector? vector)
+ (filter-map
+ (lambda (el)
+ (let ((dep (json->juliahub-dependency el)))
+ (if (and (juliahub-dependency-direct? dep)
+ (not (member (juliahub-dependency-name dep)
+ %julia-stdlibs)))
+ dep
+ #f)))
+ (vector->list vector))))
+
+(define (parse-test-dependencies directory)
+ (let* ((port (open-input-file (string-append directory "/Project.toml")))
+ (project.toml (get-string-all port))
+ (regex "\ntest = \\[.*\\]")
+ (deps (match:substring (string-match regex project.toml)))
+ (pure (string-delete (list->char-set (list #\" #\ )) deps)))
+ (close-port port)
+ (filter (lambda (x) (not (member x %julia-stdlibs)))
+ (string-split (string-drop (string-drop-right pure 1) 7) #\,))))
+
+;; Juliahub may be more up-to-date than the General registry or the actual git
+;; tag (it seems around 6 hours pass between the time a commit is supplied to
+;; JuliaRegistrator as a release, and the time Julia TagBot Github Action makes
+;; the git tag). We have no simple way to get the commit of the latest-version.
+;; Thus the best simple thing we can do is get the latest-git-tag, and import
+;; this version instead. We do this by parsing Package.toml in the General
+;; registry, and then getting the refs of the git repo supplied by this
+;; file. Parsing this file is also necessary if the package is in a subdir of a
+;; git repository, because the information isn't present in Juliahub.
+
+;; There's a last case where some Julia packages are not based on a particular
+;; git tag. In this case, the script fails, but it seems quite rare. We could
+;; introduce the tree-commit which is available in the Versions.toml file in the
+;; General repository. This can be used to identify the state of a repository,
+;; since we have a unique hash of the listing of files and directories.
+
+(define (latest-git-tag repo)
+ (let* ((last-ref (last (remote-refs repo #:tags? #t)))
+ (last-git-tag (last (string-split last-ref #\/))))
+ (string-drop last-git-tag 1)))
+
+(define* (juliahub->guix-package package-name
+ #:key version #:allow-other-keys)
+ "Fetch the metadata for PACKAGE-NAME from juliahub.org, and return the
+`package' s-expression corresponding to that package, or #f on failure.
+Optionally include a VERSION string to fetch a specific version juliahub."
+ (let* ((package-toml (ini-fetch (general-url package-name "Package.toml")))
+ (subdir (assoc-ref package-toml 'subdir))
+ (tag (latest-git-tag (assoc-ref package-toml 'repo)))
+ (package (if version
+ (juliahub-fetch package-name #:version version)
+ (if tag
+ (juliahub-fetch package-name #:version tag)
+ (juliahub-fetch package-name)))))
+ (if package
+ (let* ((source directory
+ (git->origin
+ (juliahub-package-url package)
+ `(tag-or-commit
+ . ,(string-append
+ "v" (juliahub-package-version package)))))
+ (direct-dependencies
+ (filter juliahub-dependency-direct?
+ (juliahub-package-dependencies package)))
+ (dependencies-names (map juliahub-dependency-name
+ direct-dependencies))
+ (test-dependencies-names
+ (if subdir
+ (parse-test-dependencies
+ (string-append subdir "/" directory))
+ (parse-test-dependencies directory)))
+ (homepage (juliahub-package-homepage package)))
+ (values (make-julia-sexp
+ package-name
+ (juliahub-package-version package)
+ source
+ (match homepage
+ ("" (juliahub-package-url package))
+ ((? string?) homepage)
+ (_ (juliahub-package-url package)))
+ (juliahub-package-description package)
+ (beautify-description
+ (juliahub-package-readme package))
+ direct-dependencies
+ test-dependencies-names
+ (juliahub-package-license package))
+ (append dependencies-names test-dependencies-names)))
+ (values #f '()))))
+
+;; We must use the url to get a name with the true case of juliahub/general.
+(define (guix-package->juliahub-name package)
+ (let* ((url (juliahub-package-url package))
+ (git-name (last (string-split url #\/)))
+ (ungitted-name (if (string-suffix? ".git" git-name)
+ (string-drop-right git-name 4)
+ git-name))
+ (package-name (if (string-suffix? ".jl" ungitted-name)
+ (string-drop-right ungitted-name 4)
+ ungitted-name)))
+ package-name))
+
+(define* (import-release package #:key (version #f))
+ "Return an <upstream-source> for the latest release of PACKAGE."
+ (let* ((package-name (guix-package->juliahub-name package))
+ (package (juliahub-fetch package-name))
+ (version (or version (juliahub-package-version package))))
+ (upstream-source
+ (package (package-name package))
+ (version version)
+ (urls (list (juliahub-package-url package))))))
+
+(define %juliahub-updater
+ (upstream-updater
+ (name 'juliahub)
+ (description "Updater for Juliahub packages")
+ (pred juliahub-package?)
+ (import import-release)))
+
+(define* (juliahub-recursive-import package-name #:optional version)
+ (recursive-import package-name
+ #:repo '()
+ #:repo->guix-package juliahub->guix-package
+ #:guix-name julia-name->guix-name
+ #:version version))
diff --git a/guix/scripts/import.scm b/guix/scripts/import.scm
index d2a1cee56e..8926c9610f 100644
--- a/guix/scripts/import.scm
+++ b/guix/scripts/import.scm
@@ -47,7 +47,7 @@ (define %standard-import-options '())
(define importers '("gnu" "pypi" "cpan" "hackage" "stackage" "egg" "elpa"
"gem" "go" "cran" "crate" "texlive" "json" "opam"
- "minetest" "elm" "hexpm" "composer"))
+ "minetest" "elm" "hexpm" "composer" "juliahub"))
(define (resolve-importer name)
(let ((module (resolve-interface
diff --git a/guix/scripts/import/juliahub.scm b/guix/scripts/import/juliahub.scm
new file mode 100644
index 0000000000..1317c67aa3
--- /dev/null
+++ b/guix/scripts/import/juliahub.scm
@@ -0,0 +1,107 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2023 Nicolas Graves <ngraves@ngraves.fr>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix scripts import juliahub)
+ #:use-module (guix ui)
+ #:use-module (guix utils)
+ #:use-module (guix scripts)
+ #:use-module (guix import juliahub)
+ #:use-module (guix scripts import)
+ #:use-module (srfi srfi-1)
+ #:use-module (srfi srfi-11)
+ #:use-module (srfi srfi-37)
+ #:use-module (ice-9 match)
+ #:use-module (ice-9 format)
+ #:use-module (ice-9 receive)
+ #:export (guix-import-juliahub))
+
+
+;;;
+;;; Command-line options.
+;;;
+
+(define %default-options
+ '())
+
+(define (show-help)
+ (display (G_ "Usage: guix import juliahub PACKAGE-NAME[@VERSION] Import and
+convert the Julia package for PACKAGE-NAME. Optionally, a version can be
+specified after the at-sign (@) character.\n"))
+ (display (G_ "
+ -h, --help display this help and exit"))
+ (display (G_ "
+ -V, --version display version information and exit"))
+ (display (G_ "
+ -r, --recursive generate package expressions for all Gem packages\
+ that are not yet in Guix"))
+ (newline)
+ (show-bug-report-information))
+
+(define %options
+ ;; Specification of the command-line options.
+ (cons* (option '(#\h "help") #f #f
+ (lambda args
+ (show-help)
+ (exit 0)))
+ (option '(#\V "version") #f #f
+ (lambda args
+ (show-version-and-exit "guix import gem")))
+ (option '(#\r "recursive") #f #f
+ (lambda (opt name arg result)
+ (alist-cons 'recursive #t result)))
+ %standard-import-options))
+
+
+;;;
+;;; Entry point.
+;;;
+
+(define (guix-import-juliahub . args)
+ (define (parse-options)
+ ;; Return the alist of option values.
+ (parse-command-line args %options (list %default-options)
+ #:build-options? #f))
+
+ (let* ((opts (parse-options))
+ (args (filter-map (match-lambda
+ (('argument . value)
+ value)
+ (_ #f))
+ (reverse opts))))
+ (match args
+ ((spec)
+ (receive (package-name package-version)
+ (package-name->name+version spec)
+ (let ((code (if (assoc-ref opts 'recursive)
+ (map (match-lambda
+ ((and ('package ('name name) . rest) pkg)
+ `(define-public ,(string->symbol name)
+ ,pkg))
+ (_ #f))
+ (juliahub-recursive-import package-name package-version))
+ (let ((sexp (juliahub->guix-package package-name #:version package-version)))
+ (if sexp sexp #f)))))
+ (match code
+ ((or #f '(#f))
+ (leave (G_ "failed to download meta-data for package '~a'~%")
+ package-name))
+ (_ code)))))
+ (()
+ (leave (G_ "too few arguments~%")))
+ ((many ...)
+ (leave (G_ "too many arguments~%"))))))
--
2.41.0
N
N
Nicolas Graves wrote on 3 Feb 15:07 -0800
[PATCH v4 3/6] import: juliahub: Beautify description.
(address . 62202@debbugs.gnu.org)
20240203230807.25751-3-ngraves@ngraves.fr
---
guix/import/juliahub.scm | 53 ++++++++++++++++++++++++++++++++++++++--
1 file changed, 51 insertions(+), 2 deletions(-)

Toggle diff (81 lines)
diff --git a/guix/import/juliahub.scm b/guix/import/juliahub.scm
index ab838b6035..e4540de06d 100644
--- a/guix/import/juliahub.scm
+++ b/guix/import/juliahub.scm
@@ -30,8 +30,10 @@ (define-module (guix import juliahub)
#:use-module (guix base32)
#:use-module (guix packages)
#:use-module (guix upstream)
- #:use-module (json)
#:use-module ((guix licenses) #:prefix license:)
+ #:use-module (json)
+ #:use-module (htmlprag)
+ #:use-module (sxml transform)
#:export (juliahub->guix-package
%juliahub-updater
@@ -76,6 +78,53 @@ (define (ini-fetch url)
(close-port port)
(map ini-line->alist lines)))
+;; Beautify description.
+(define %juliahub-beautify-description-rules
+ `((h1 *preorder* . ,(lambda args #f))
+ (h2 *preorder* . ,(lambda args #f))
+ (h3 *preorder* . ,(lambda args #f))
+ (h4 *preorder* . ,(lambda args #f))
+ (h5 *preorder* . ,(lambda args #f))
+ (h6 *preorder* . ,(lambda args #f))
+ (hr *preorder* . ,(lambda args #f))
+ (span *preorder* . ,(lambda args #f))
+ (img *preorder* . ,(lambda args #f))
+ (pre *preorder* . ,(lambda args #f))
+ (div *preorder* . ,(lambda args #f))
+ (table *preorder* . ,(lambda args #f))
+ (imgalt *preorder* . ,(lambda args #f))
+ (@ *preorder* . ,(lambda args #f))
+ (*TOP* . ,(lambda args (cdr args)))
+ (p . ,(lambda args (cdr args)))
+ (em . ,(lambda args (cdr args)))
+ (strong . ,(lambda args (cdr args)))
+ (a . ,(lambda args
+ (match args
+ ((tag link ref)
+ (if ref ref #f))
+ (_ #f))))
+ (ul . ,(lambda args
+ `("@itemize" ,@(cdr args) "\n@end itemize")))
+ (ol . ,(lambda args
+ `("@enumerate" ,@(cdr args) "@end enumerate")))
+ (blockquote . ,(lambda args
+ `("@quotation" ,@(cdr args) "@end quotation")))
+ (li . ,(lambda args
+ `("\n@item" ,@(cdr args))))
+ (code . ,(lambda args
+ `("@code{" ,@(cdr args) "}")))
+ (*text* . ,(lambda (tag x) x))
+ (*default* . ,(lambda (tag . body)
+ (cons tag body)))))
+
+(define (juliahub-beautify-description description)
+ (string-join
+ (filter (lambda (x) (if (equal? x " ") #f x))
+ (flatten
+ (pre-post-order (html->sxml description)
+ %juliahub-beautify-description-rules)))
+ " "))
+
;; Filtering out julia-stdlibs.
;; To update them, see file sysimg.jl.
(define %julia-stdlibs
@@ -264,7 +313,7 @@ (define* (juliahub->guix-package package-name
((? string?) homepage)
(_ (juliahub-package-url package)))
(juliahub-package-description package)
- (beautify-description
+ ((compose beautify-description juliahub-beautify-description)
(juliahub-package-readme package))
direct-dependencies
test-dependencies-names
--
2.41.0
N
N
Nicolas Graves wrote on 3 Feb 15:07 -0800
[PATCH v4 4/6] import: utils: Rule out texinfo common syntax from @ escape.
(address . 62202@debbugs.gnu.org)
20240203230807.25751-4-ngraves@ngraves.fr
---
guix/import/utils.scm | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)

Toggle diff (21 lines)
diff --git a/guix/import/utils.scm b/guix/import/utils.scm
index 47254539a1..57e4ec0ce7 100644
--- a/guix/import/utils.scm
+++ b/guix/import/utils.scm
@@ -375,7 +375,13 @@ (define* (beautify-description description #:optional (length 80))
(cut string-trim-both <> #\')
;; Escape single @ to prevent it from being understood as
;; invalid Texinfo syntax.
- (cut regexp-substitute/global #f "@" <> 'pre "@@" 'post)
+ (lambda (word)
+ (if ; Rule out some valid Texinfo syntax.
+ (member word '("@itemize" "@item" "@end" "@quotation"
+ "@enumerate" "@code" "@code{"))
+ word
+ ((cut regexp-substitute/global
+ #f "@" <> 'pre "@@" 'post) word)))
;; Wrap camelCase or PascalCase words in @code{...}.
(lambda (word)
(let ((pattern (make-regexp "([A-Z][a-z]+[A-Z]|[a-z]+[A-Z])")))
--
2.41.0
N
N
Nicolas Graves wrote on 3 Feb 15:07 -0800
[PATCH v4 5/6] tests: go: Add mock-git->origin function.
(address . 62202@debbugs.gnu.org)
20240203230807.25751-5-ngraves@ngraves.fr
* tests/go.scm
(mock-git->origin): Add function.
(go-module->guix-package test): Use mock-git->origin.

Change-Id: I9f9de814fc6c9dcf9772507c9e5bf32c16d81784
---
tests/go.scm | 41 +++++++++++++++++++++--------------------
1 file changed, 21 insertions(+), 20 deletions(-)

Toggle diff (61 lines)
diff --git a/tests/go.scm b/tests/go.scm
index 4644e1bd1c..cca27506e6 100644
--- a/tests/go.scm
+++ b/tests/go.scm
@@ -371,6 +371,26 @@ (define (mock-http-get testcase)
(values response-header body)
(error "mocked http-get Unexpected URL: " url)))))
+;; Mock an empty directory by replacing hash.
+(define* (mock-git->origin repo-url ref #:key (ref->commit #f))
+ (let* ((version (if (pair? ref)
+ (cdr ref)
+ #f))
+ (vcommit (match ref->commit
+ (#t commit)
+ (#f version)
+ ((? procedure?) (ref->commit version))
+ (_ #f))))
+ `(origin
+ (method git-fetch)
+ (uri (git-reference
+ (url ,(and (not (eq? repo-url 'null)) repo-url))
+ (commit ,vcommit)))
+ (file-name (git-file-name name version))
+ (sha256
+ (base32
+ "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5")))))
+
(test-equal "go-module->guix-package"
'(package
(name "go-github-com-go-check-check")
@@ -403,26 +423,7 @@ (define (mock-http-get testcase)
(mock-http-get fixtures-go-check-test))
(mock ((guix http-client) http-fetch
(mock-http-fetch fixtures-go-check-test))
- (mock ((guix import utils) git->origin
- ;; Mock an empty directory by replacing hash.
- (lambda* (repo-url ref #:key (ref->commit #f))
- (let* ((version (if (pair? ref)
- (cdr ref)
- #f))
- (vcommit (match ref->commit
- (#t commit)
- (#f version)
- ((? procedure?) (ref->commit version))
- (_ #f))))
- `(origin
- (method git-fetch)
- (uri (git-reference
- (url ,(and (not (eq? repo-url 'null)) repo-url))
- (commit ,vcommit)))
- (file-name (git-file-name name version))
- (sha256
- (base32
- "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5"))))))
+ (mock ((guix import utils) git->origin mock-git->origin)
(go-module->guix-package* "github.com/go-check/check")))))))
(test-end "go")
--
2.41.0
N
N
Nicolas Graves wrote on 3 Feb 15:07 -0800
[PATCH v4 6/6] tests: juliahub: Add unit tests for (guix import juliahub).
(address . 62202@debbugs.gnu.org)
20240203230807.25751-6-ngraves@ngraves.fr
* tests/juliahub.scm : Add unit tests juliahub-redirect,
julia-general-registry-parsing, juliahub-fetch.

Change-Id: Ief5133b49a15d0bfc72bb357321126d296ba2802
---
tests/juliahub.scm | 185 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 185 insertions(+)
create mode 100644 tests/juliahub.scm

Toggle diff (193 lines)
diff --git a/tests/juliahub.scm b/tests/juliahub.scm
new file mode 100644
index 0000000000..21fdc3a4eb
--- /dev/null
+++ b/tests/juliahub.scm
@@ -0,0 +1,185 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2024 Nicolas Graves <ngraves@ngraves.fr>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+;;; Summary
+;; Tests for guix/import/juliahub.scm
+
+(define-module (tests-import-juliahub)
+ #:use-module (guix base32)
+ #:use-module (json)
+ #:use-module (guix import juliahub)
+ #:use-module (guix import utils)
+ #:use-module ((guix utils) #:select (call-with-temporary-directory))
+ #:use-module (guix tests)
+ #:use-module (ice-9 match)
+ #:use-module (srfi srfi-19)
+ #:use-module (srfi srfi-64)
+ #:use-module (srfi srfi-71)
+ #:use-module (web response))
+
+;; XXX: Copied from tests/go.scm
+
+(define (mock-http-fetch testcase)
+ (lambda (url . rest)
+ (let ((body (assoc-ref testcase url)))
+ (if body
+ (open-input-string body)
+ (error "mocked http-fetch Unexpected URL: " url)))))
+
+(define (mock-http-get testcase)
+ (lambda (url . rest)
+ (let ((body (assoc-ref testcase url))
+ (response-header
+ (build-response
+ #:version '(1 . 1)
+ #:code 200
+ #:reason-phrase "Ok"
+ #:headers `(
+ (content-type text/html (charset . "utf-8"))
+ (date . ,(make-date 0 10 58 12 6 3 2021 0))
+ (transfer-encoding (chunked)))
+ #:port #f
+ #:validate-headers? #t)))
+ (if body
+ (values response-header body)
+ (error "mocked http-get Unexpected URL: " url)))))
+
+;; Mock an empty directory by replacing hash.
+(define* (mock-git->origin repo-url ref #:key (ref->commit #f))
+ (let* ((version (if (pair? ref)
+ (cdr ref)
+ #f))
+ (vcommit (match ref->commit
+ (#t commit)
+ (#f version)
+ ((? procedure?) (ref->commit version))
+ (_ #f))))
+ `(origin
+ (method git-fetch)
+ (uri (git-reference
+ (url ,(and (not (eq? repo-url 'null)) repo-url))
+ (commit ,vcommit)))
+ (file-name (git-file-name name version))
+ (sha256
+ (base32
+ "0sjjj9z1dhilhpc8pq4154czrb79z9cm044jvn75kxcjv6v5l2m5")))))
+
+;; Fixtures.
+
+(define fixture-pkg.json
+ (scm->json-string
+ `(("hosted_uri" . "https://git.savannah.gnu.org/cgit/MyPackage.git")
+ ("installable" . "missing")
+ ("license" . "GPL-3.0")
+ ("tags" . #("my-tag"))
+ ("uuid" . "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa")
+ ("url" . "https://git.savannah.gnu.org/cgit/MyPackage.git")
+ ("contributors" . #((("url" . "https://github.com/me") ("type" . "User") ("contributions" . 1) ("name" . "me"))))
+ ("success" . #t)
+ ("reversedeps" . #())
+ ("pkgeval"
+ ("report_url" . "https://github.com/JuliaCI/NanosoldierReports/blob/master/pkgeval/by_date/2024-02/01/report.md")
+ ("version" ("minor" . 18) ("patch" . 1) ("build" . #()) ("prerelease" . #()) ("major" . 1))
+ ("reason" . "time_limit")
+ ("log_url" . "https://s3.amazonaws.com/julialang-reports/nanosoldier/pkgeval/by_date/2024-02/01/MyPackage.jl.primary.log")
+ ("status" . "fail")
+ ("duration" . 0))
+ ("stargazers_count" . 0)
+ ("deps" . #((("slug" . "3FQLY")
+ ("registry" . "")
+ ("versions" . #("0.0.0" "1.6.0-1"))
+ ("uuid" . "de0858da-6303-5e67-8744-51eddeeeb8d7")
+ ("name" . "Printf")
+ ("direct" . #t))
+ (("slug" . "SIw1t")
+ ("registry" . "")
+ ("versions" . #("*" "0.0.0" "1"))
+ ("uuid" . "cf7118a7-6976-5b1a-9a39-7adc72f591a4")
+ ("name" . "UUIDs")
+ ("direct" . #f))
+ (("slug" . "THl1k")
+ ("registry" . "General")
+ ("versions" . #("0.7" "1" "1-1.10" "1.0-1.5" "1.3.0-1" "1.6.0-1" "1.9.0-1" "1.9.4-1"))
+ ("uuid" . "1222c4b2-2114-5bfd-aeef-88e4692bbb3e")
+ ("name" . "julia")
+ ("direct" . #f))
+ (("slug" . "THl1k")
+ ("registry" . "General")
+ ("versions" . #("1.6.0-1"))
+ ("uuid" . "1222c4b2-2114-5bfd-aeef-88e4692bbb3e")
+ ("name" . "julia")
+ ("direct" . #t))))
+ ("description" . "My description")
+ ("version" . "1.0.0")
+ ("documenter_errored" . "missing")
+ ("slug" . "MySlg")
+ ("owner" . "me")
+ ("release_date" . "Feb 2024")
+ ("name" . "MyPackage.jl")
+ ("readme" . "My readme")
+ ("homepage" . "https://git.savannah.gnu.org/cgit/MyPackage.git")
+ ("doctype" . "hosted")
+ ("license_url" . "/docs/MyPackage/MySlg/1.0.0/_packagesource/LICENSE.md"))))
+
+(define fixtures-juliahub-check-test
+ `(("https://docs.juliahub.com/MyPackage/" . "\
+<head>
+<meta http-equiv=\"refresh\" content=\"0; url=MySlg/1.0.0\" />
+</head>")
+ ("https://raw.githubusercontent.com/JuliaRegistries/General/master\
+/M/MyPackage/Package.toml" . "\
+name = \"MyPackage\"
+uuid = \"aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa\"
+repo = \"https://git.savannah.gnu.org/cgit/MyPackage.git\"
+")
+ ("https://docs.juliahub.com/MyPackage/MySlg/1.0.0/pkg.json" . ,fixture-pkg.json)))
+
+(test-begin "juliahub")
+
+;;; Unit tests for (guix import juliahub)
+
+(test-equal "juliahub-redirect"
+ "https://docs.juliahub.com/MyPackage/MySlg/1.0.0/"
+ (mock ((web client) http-get
+ (mock-http-get fixtures-juliahub-check-test))
+ (mock ((guix http-client) http-fetch
+ (mock-http-fetch fixtures-juliahub-check-test))
+ ((@@ (guix import juliahub) juliahub-url) "MyPackage"))))
+
+(test-equal "julia-general-registry-parsing"
+ "https://git.savannah.gnu.org/cgit/MyPackage.git"
+ (mock ((web client) http-get
+ (mock-http-get fixtures-juliahub-check-test))
+ (mock ((guix http-client) http-fetch
+ (mock-http-fetch fixtures-juliahub-check-test))
+ (assoc-ref
+ ((@@ (guix import juliahub) ini-fetch)
+ ((@@ (guix import juliahub) general-url) "MyPackage" "Package.toml"))
+ 'repo))))
+
+(test-equal "juliahub-fetch"
+ #t
+ (mock ((web client) http-get
+ (mock-http-get fixtures-juliahub-check-test))
+ (mock ((guix http-client) http-fetch
+ (mock-http-fetch fixtures-juliahub-check-test))
+ (mock ((guix import utils) git->origin mock-git->origin)
+ ((@@ (guix import juliahub) juliahub-package?)
+ ((@@ (guix import juliahub) juliahub-fetch) "MyPackage"))))))
+
+(test-end "juliahub")
--
2.41.0
L
L
Ludovic Courtès wrote on 1 Apr 13:50 -0700
(name . Nicolas Graves)(address . ngraves@ngraves.fr)
87plv8bye8.fsf@gnu.org
Hi,

As part of this v4, I would recommend merging patches 2, 3, and 6, such
that there’s a single self-contained patch adding ‘guix import
juliahub’. (That’s what we usually do and I find it clearer because we
immediately see what goes together.)

Nicolas Graves <ngraves@ngraves.fr> skribis:

Toggle quote (3 lines)
> * tests/juliahub.scm : Add unit tests juliahub-redirect,
> julia-general-registry-parsing, juliahub-fetch.

Just “New file.”

Some of the other files lack a commit log; we can add it for you, but
it’d be great if you could do it upfront.

Toggle quote (3 lines)
> ---
> tests/juliahub.scm | 185 +++++++++++++++++++++++++++++++++++++++++++++

Please add it to ‘Makefile.am’.

[...]

Toggle quote (14 lines)
> +(define (mock-http-fetch testcase)
> + (lambda (url . rest)
> + (let ((body (assoc-ref testcase url)))
> + (if body
> + (open-input-string body)
> + (error "mocked http-fetch Unexpected URL: " url)))))
> +
> +(define (mock-http-get testcase)
> + (lambda (url . rest)
> + (let ((body (assoc-ref testcase url))
> + (response-header
> + (build-response
> + #:version '(1 . 1)

I strongly encourage using ‘with-http-server’ using the same strategy
that’s used in ‘tests/pypi.scm’ and others instead of mocking. (‘mock’
is very sensitive to inlining, plus you sorta have to make assumptions
about the code path to be able to mock the right things.)

Toggle quote (10 lines)
> +(test-equal "juliahub-fetch"
> + #t
> + (mock ((web client) http-get
> + (mock-http-get fixtures-juliahub-check-test))
> + (mock ((guix http-client) http-fetch
> + (mock-http-fetch fixtures-juliahub-check-test))
> + (mock ((guix import utils) git->origin mock-git->origin)
> + ((@@ (guix import juliahub) juliahub-package?)
> + ((@@ (guix import juliahub) juliahub-fetch) "MyPackage"))))))

Checking for ‘juliahub-package?’ doesn’t tell us much; what about
checking the whole package, similar to what is done in other importer
tests?

Thanks,
Ludo’.
N
N
Nicolas Graves wrote on 2 Apr 04:52 -0700
(name . Ludovic Courtès)(address . ludo@gnu.org)
87edbovv5q.fsf@ngraves.fr
On 2024-04-01 22:50, Ludovic Courtès wrote:

Toggle quote (7 lines)
> Hi,
>
> As part of this v4, I would recommend merging patches 2, 3, and 6, such
> that there’s a single self-contained patch adding ‘guix import
> juliahub’. (That’s what we usually do and I find it clearer because we
> immediately see what goes together.)

Will do.
Toggle quote (11 lines)
>
> Nicolas Graves <ngraves@ngraves.fr> skribis:
>
>> * tests/juliahub.scm : Add unit tests juliahub-redirect,
>> julia-general-registry-parsing, juliahub-fetch.
>
> Just “New file.”
>
> Some of the other files lack a commit log; we can add it for you, but
> it’d be great if you could do it upfront.

Sorry for these past contribution, I do it now.
Toggle quote (6 lines)
>
>> ---
>> tests/juliahub.scm | 185 +++++++++++++++++++++++++++++++++++++++++++++
>
> Please add it to ‘Makefile.am’.

Will do.
Toggle quote (36 lines)
>
> [...]
>
>> +(define (mock-http-fetch testcase)
>> + (lambda (url . rest)
>> + (let ((body (assoc-ref testcase url)))
>> + (if body
>> + (open-input-string body)
>> + (error "mocked http-fetch Unexpected URL: " url)))))
>> +
>> +(define (mock-http-get testcase)
>> + (lambda (url . rest)
>> + (let ((body (assoc-ref testcase url))
>> + (response-header
>> + (build-response
>> + #:version '(1 . 1)
>
> I strongly encourage using ‘with-http-server’ using the same strategy
> that’s used in ‘tests/pypi.scm’ and others instead of mocking. (‘mock’
> is very sensitive to inlining, plus you sorta have to make assumptions
> about the code path to be able to mock the right things.)
>
>> +(test-equal "juliahub-fetch"
>> + #t
>> + (mock ((web client) http-get
>> + (mock-http-get fixtures-juliahub-check-test))
>> + (mock ((guix http-client) http-fetch
>> + (mock-http-fetch fixtures-juliahub-check-test))
>> + (mock ((guix import utils) git->origin mock-git->origin)
>> + ((@@ (guix import juliahub) juliahub-package?)
>> + ((@@ (guix import juliahub) juliahub-fetch) "MyPackage"))))))
>
> Checking for ‘juliahub-package?’ doesn’t tell us much; what about
> checking the whole package, similar to what is done in other importer
> tests?

Couldn't manage to get it to work, don't remember why exactly but it was
related to gexps. Will retry it when moving to 'with-http-server' as
advised.

Toggle quote (4 lines)
>
> Thanks,
> Ludo’.

--
Best regards,
Nicolas Graves
N
N
Nicolas Graves wrote on 9 Apr 00:29 -0700
(name . Ludovic Courtès)(address . ludo@gnu.org)
87bk6jdmdk.fsf@ngraves.fr
On 2024-04-01 22:50, Ludovic Courtès wrote:

Toggle quote (43 lines)
> Hi,
>
> As part of this v4, I would recommend merging patches 2, 3, and 6, such
> that there’s a single self-contained patch adding ‘guix import
> juliahub’. (That’s what we usually do and I find it clearer because we
> immediately see what goes together.)
>
> Nicolas Graves <ngraves@ngraves.fr> skribis:
>
>> * tests/juliahub.scm : Add unit tests juliahub-redirect,
>> julia-general-registry-parsing, juliahub-fetch.
>
> Just “New file.”
>
> Some of the other files lack a commit log; we can add it for you, but
> it’d be great if you could do it upfront.
>
>> ---
>> tests/juliahub.scm | 185 +++++++++++++++++++++++++++++++++++++++++++++
>
> Please add it to ‘Makefile.am’.
>
> [...]
>
>> +(define (mock-http-fetch testcase)
>> + (lambda (url . rest)
>> + (let ((body (assoc-ref testcase url)))
>> + (if body
>> + (open-input-string body)
>> + (error "mocked http-fetch Unexpected URL: " url)))))
>> +
>> +(define (mock-http-get testcase)
>> + (lambda (url . rest)
>> + (let ((body (assoc-ref testcase url))
>> + (response-header
>> + (build-response
>> + #:version '(1 . 1)
>
> I strongly encourage using ‘with-http-server’ using the same strategy
> that’s used in ‘tests/pypi.scm’ and others instead of mocking. (‘mock’
> is very sensitive to inlining, plus you sorta have to make assumptions
> about the code path to be able to mock the right things.)

I can't however mock a git server, right? I still must mock at least the
git repo instead of getting it through a custom server, or is there a
better solution here?

Toggle quote (18 lines)
>
>> +(test-equal "juliahub-fetch"
>> + #t
>> + (mock ((web client) http-get
>> + (mock-http-get fixtures-juliahub-check-test))
>> + (mock ((guix http-client) http-fetch
>> + (mock-http-fetch fixtures-juliahub-check-test))
>> + (mock ((guix import utils) git->origin mock-git->origin)
>> + ((@@ (guix import juliahub) juliahub-package?)
>> + ((@@ (guix import juliahub) juliahub-fetch) "MyPackage"))))))
>
> Checking for ‘juliahub-package?’ doesn’t tell us much; what about
> checking the whole package, similar to what is done in other importer
> tests?
>
> Thanks,
> Ludo’.

--
Best regards,
Nicolas Graves
N
N
Nicolas Graves wrote on 11 Apr 03:56 -0700
(name . Ludovic Courtès)(address . ludo@gnu.org)
874jc8np51.fsf@ngraves.fr
On 2024-04-09 09:29, Nicolas Graves wrote:

Toggle quote (49 lines)
> On 2024-04-01 22:50, Ludovic Courtès wrote:
>
>> Hi,
>>
>> As part of this v4, I would recommend merging patches 2, 3, and 6, such
>> that there’s a single self-contained patch adding ‘guix import
>> juliahub’. (That’s what we usually do and I find it clearer because we
>> immediately see what goes together.)
>>
>> Nicolas Graves <ngraves@ngraves.fr> skribis:
>>
>>> * tests/juliahub.scm : Add unit tests juliahub-redirect,
>>> julia-general-registry-parsing, juliahub-fetch.
>>
>> Just “New file.”
>>
>> Some of the other files lack a commit log; we can add it for you, but
>> it’d be great if you could do it upfront.
>>
>>> ---
>>> tests/juliahub.scm | 185 +++++++++++++++++++++++++++++++++++++++++++++
>>
>> Please add it to ‘Makefile.am’.
>>
>> [...]
>>
>>> +(define (mock-http-fetch testcase)
>>> + (lambda (url . rest)
>>> + (let ((body (assoc-ref testcase url)))
>>> + (if body
>>> + (open-input-string body)
>>> + (error "mocked http-fetch Unexpected URL: " url)))))
>>> +
>>> +(define (mock-http-get testcase)
>>> + (lambda (url . rest)
>>> + (let ((body (assoc-ref testcase url))
>>> + (response-header
>>> + (build-response
>>> + #:version '(1 . 1)
>>
>> I strongly encourage using ‘with-http-server’ using the same strategy
>> that’s used in ‘tests/pypi.scm’ and others instead of mocking. (‘mock’
>> is very sensitive to inlining, plus you sorta have to make assumptions
>> about the code path to be able to mock the right things.)
>
> I can't however mock a git server, right? I still must mock at least the
> git repo instead of getting it through a custom server, or is there a
> better solution here?

It's actually simpler than I thought, but there's an impediment in guile
http server implementation that doesn't allow me to push this effort to
the end.


I'm currently writing it, it'll result in a handy helper for tests, such
as :

(with-git-forge ; spawns a dumb but functional git server
'(("MyPackage" . ((add "a.txt" "A")
(commit "First commit")
(tag "v1.0.0" "Release 1.0"))))
(with-julia-test-servers
`(("/juliahub/MyPackage/" 200 ,juliahub-redirect.html)
("/juliahub/MyPackage/" 200 ,juliahub-redirect.html)
("/juliahub/MyPackage/MySlg/1.0.0/pkg.json" 200
,(lambda (port) (display (fixture-pkg.json) port)))
("/general/M/MyPackage/Package.toml" 200
,(lambda (port) (display (pk 'd (general-Package.toml)) port))))
(juliahub->guix-package "MyPackage")))


However, for that I'll need the http server to be able to respond with a
(content-type . (application/x-git-upload-pack-advertisement))
header to git. But in guile's web server implementation, this is not
possible because of sanitize-response's charset addition, which is not
configurable.

That's outside my field, how can we progress further ? We do indeed need
such a server to properly test juliahub since we go get the tag from the
actual repo (this is justified in the patch series).

_____________________________________________________________________________
;;; Git Forge = Git HTTP Server with Dump transfer protocol and repositories

(define (call-with-temporary-git-repositories names+directives proc)
"Call PROC with populated git temporary directories as per NAMES+DIRECTIVES;
close the directories and delete them when leaving the dynamic extent of this
call."
(call-with-temporary-directory
(lambda (directory)
(for-each
(match-lambda
((name . directives)
(populate-git-repository
(string-append directory "/" name ".git") directives)))
names+directives)
(proc directory))))

(define %git-forge-port
;; TCP port to use for the dumb git server.
;; If 0, the OS will automatically choose
;; a port.
(make-parameter 0))

(define (binary-file-dump file)
"Return a procedure that dumps binary FILE to the given port."
(lambda (output)
(call-with-input-file file
(lambda (input)
(put-bytevector output (get-bytevector-all input)))
#:binary #t)))

(define (serialize-git-ref ref oid)
(format #f "~a ~a\n" oid ref))

(define (refs->alist repo refs)
(let ((repository (repository-open repo)))
(map
(lambda (ref)
(cons ref (oid->string (reference-name->oid repository ref))))
refs)))

(define* (call-with-git-forge repositories+directives thunk)
"Call THUNK with a running GIT test forge, i.e. an HTTP server implementing
the git dumb protocol (see
This server behaves like a GIT forge with the repositories constructed from
REPOSITORIES+DIRECTIVES. Each element of REPOSITORIES+DIRECTIVES must be a
tuple containing a repository name and a list of DIRECTIVES.

%git-forge-port will be set to the port listened at
The port listened at will be set for the dynamic extent of THUNK."
(call-with-temporary-git-repositories
repositories+directives
(lambda (dir-with-repos)
(define responses+data
(let ((repos (scandir dir-with-repos
(lambda (name)
(not (member name '("." "..")))))))
(append-map
(lambda (relative-repo)
(let* ((name (string-drop-right relative-repo (string-length ".git")))
(repo (string-append dir-with-repos "/" relative-repo)))
`((,(string-append "/" name ".git/info/refs")
200
((content-type . (application/x-git-upload-pack-advertisement)))
,((@ (gnu services configuration) generic-serialize-alist)
string-append
serialize-git-ref
(refs->alist repo (remote-refs repo))))
(,(string-append "/" name ".git/HEAD")
200
"ref: refs/heads/master")
,@(map
(lambda (object)
`(,(string-append "/" name ".git/objects/"
(string-take-right object 41))
200
,(binary-file-dump
(string-append repo "/.git/objects/" object))))
(find-files (string-append repo "/.git/objects")))
(,(string-append "/" name ".git/objects/info/http-alternates")
200
"")
(,(string-append "/" name ".git/objects/info/packs")
200 ""))))
repos)))

(parameterize ((%http-server-port (%git-forge-port)))
(call-with-http-server (pk 'responses+data responses+data) thunk)))))

(define-syntax with-git-forge
(syntax-rules ()
((_ repositories+directives body ...)
(call-with-git-forge repositories+directives (lambda () body ...)))))
__________________________________________________________________________________

Toggle quote (18 lines)
>>
>>> +(test-equal "juliahub-fetch"
>>> + #t
>>> + (mock ((web client) http-get
>>> + (mock-http-get fixtures-juliahub-check-test))
>>> + (mock ((guix http-client) http-fetch
>>> + (mock-http-fetch fixtures-juliahub-check-test))
>>> + (mock ((guix import utils) git->origin mock-git->origin)
>>> + ((@@ (guix import juliahub) juliahub-package?)
>>> + ((@@ (guix import juliahub) juliahub-fetch) "MyPackage"))))))
>>
>> Checking for ‘juliahub-package?’ doesn’t tell us much; what about
>> checking the whole package, similar to what is done in other importer
>> tests?
>>
>> Thanks,
>> Ludo’.

--
Best regards,
Nicolas Graves
L
L
Ludovic Courtès wrote on 16 Apr 09:52 -0700
(name . Nicolas Graves)(address . ngraves@ngraves.fr)
875xwhmerp.fsf@gnu.org
Hi,

Nicolas Graves <ngraves@ngraves.fr> skribis:

Toggle quote (16 lines)
> I'm currently writing it, it'll result in a handy helper for tests, such
> as :
>
> (with-git-forge ; spawns a dumb but functional git server
> '(("MyPackage" . ((add "a.txt" "A")
> (commit "First commit")
> (tag "v1.0.0" "Release 1.0"))))
> (with-julia-test-servers
> `(("/juliahub/MyPackage/" 200 ,juliahub-redirect.html)
> ("/juliahub/MyPackage/" 200 ,juliahub-redirect.html)
> ("/juliahub/MyPackage/MySlg/1.0.0/pkg.json" 200
> ,(lambda (port) (display (fixture-pkg.json) port)))
> ("/general/M/MyPackage/Package.toml" 200
> ,(lambda (port) (display (pk 'd (general-Package.toml)) port))))
> (juliahub->guix-package "MyPackage")))

Nice!

Toggle quote (6 lines)
> However, for that I'll need the http server to be able to respond with a
> (content-type . (application/x-git-upload-pack-advertisement))
> header to git. But in guile's web server implementation, this is not
> possible because of sanitize-response's charset addition, which is not
> configurable.

D’oh. I’m not sure what this is about; it’s definitely possible to add
pretty much any ‘content-type’ header (we do that in ‘guix publish’ for
instance), but if it’s going too far, then maybe you should go back to
procedure mocks.

Toggle quote (7 lines)
> That's outside my field, how can we progress further ? We do indeed need
> such a server to properly test juliahub since we go get the tag from the
> actual repo (this is justified in the patch series).
>
> _____________________________________________________________________________
> ;;; Git Forge = Git HTTP Server with Dump transfer protocol and repositories

This work’s not lost: we can definitely switch back to it when the
limitations you encountered has been fixed on the Guile side, and/or in
other, simpler cases.

Thanks!

Ludo’.
N
N
Nicolas Graves wrote on 16 Apr 12:11 -0700
(name . Ludovic Courtès)(address . ludo@gnu.org)
874jc1nmve.fsf@ngraves.fr
On 2024-04-16 18:52, Ludovic Courtès wrote:

Toggle quote (33 lines)
> Hi,
>
> Nicolas Graves <ngraves@ngraves.fr> skribis:
>
>> I'm currently writing it, it'll result in a handy helper for tests, such
>> as :
>>
>> (with-git-forge ; spawns a dumb but functional git server
>> '(("MyPackage" . ((add "a.txt" "A")
>> (commit "First commit")
>> (tag "v1.0.0" "Release 1.0"))))
>> (with-julia-test-servers
>> `(("/juliahub/MyPackage/" 200 ,juliahub-redirect.html)
>> ("/juliahub/MyPackage/" 200 ,juliahub-redirect.html)
>> ("/juliahub/MyPackage/MySlg/1.0.0/pkg.json" 200
>> ,(lambda (port) (display (fixture-pkg.json) port)))
>> ("/general/M/MyPackage/Package.toml" 200
>> ,(lambda (port) (display (pk 'd (general-Package.toml)) port))))
>> (juliahub->guix-package "MyPackage")))
>
> Nice!
>
>> However, for that I'll need the http server to be able to respond with a
>> (content-type . (application/x-git-upload-pack-advertisement))
>> header to git. But in guile's web server implementation, this is not
>> possible because of sanitize-response's charset addition, which is not
>> configurable.
>
> D’oh. I’m not sure what this is about; it’s definitely possible to add
> pretty much any ‘content-type’ header (we do that in ‘guix publish’ for
> instance), but if it’s going too far, then maybe you should go back to
> procedure mocks.

Just to write that in a clearer way:

libgit2, which is behind guile-git, expects this exact header or fails,
with no additional charset field. The sanitize-response function in the
guile web server implementation on its side ensures that a charset field
is added no matter what.

So indeed we can set

(content-type . (application/x-git-upload-pack-advertisement))
but in reality guile-git // libgit2 will read

(content-type . (application/x-git-upload-pack-advertisement
(charset . "utf-8"))
and will fail in this case.

Since it's exterior (expected by libgit2), I thought the more relevant
would be to allow more flexibility on the guile web server side. But if
you think it's way better to ask on the libgit2 or guile-git side, I'll
be happy to. WDYT ?

Toggle quote (16 lines)
>
>> That's outside my field, how can we progress further ? We do indeed need
>> such a server to properly test juliahub since we go get the tag from the
>> actual repo (this is justified in the patch series).
>>
>> _____________________________________________________________________________
>> ;;; Git Forge = Git HTTP Server with Dump transfer protocol and repositories
>
> This work’s not lost: we can definitely switch back to it when the
> limitations you encountered has been fixed on the Guile side, and/or in
> other, simpler cases.
>
> Thanks!
>
> Ludo’.

--
Best regards,
Nicolas Graves
L
L
Ludovic Courtès wrote on 17 Apr 01:51 -0700
(name . Nicolas Graves)(address . ngraves@ngraves.fr)
87mspsjrt1.fsf@gnu.org
Nicolas Graves <ngraves@ngraves.fr> skribis:

Toggle quote (16 lines)
> libgit2, which is behind guile-git, expects this exact header or fails,
> with no additional charset field. The sanitize-response function in the
> guile web server implementation on its side ensures that a charset field
> is added no matter what.
>
> So indeed we can set
>
> (content-type . (application/x-git-upload-pack-advertisement))
>
> but in reality guile-git // libgit2 will read
>
> (content-type . (application/x-git-upload-pack-advertisement
> (charset . "utf-8"))
>
> and will fail in this case.

Oh I see, I had misunderstood that.

Note that ‘sanitize-response’ does not add a ‘charset’ header when BODY
is #f or a bytevector. Maybe we could do that?

Thanks,
Ludo’.
N
N
Nicolas Graves wrote on 21 Apr 09:08 -0700
(name . Ludovic Courtès)(address . ludo@gnu.org)
87mspmsnoz.fsf@ngraves.fr
On 2024-04-17 10:51, Ludovic Courtès wrote:

Toggle quote (23 lines)
> Nicolas Graves <ngraves@ngraves.fr> skribis:
>
>> libgit2, which is behind guile-git, expects this exact header or fails,
>> with no additional charset field. The sanitize-response function in the
>> guile web server implementation on its side ensures that a charset field
>> is added no matter what.
>>
>> So indeed we can set
>>
>> (content-type . (application/x-git-upload-pack-advertisement))
>>
>> but in reality guile-git // libgit2 will read
>>
>> (content-type . (application/x-git-upload-pack-advertisement
>> (charset . "utf-8"))
>>
>> and will fail in this case.
>
> Oh I see, I had misunderstood that.
>
> Note that ‘sanitize-response’ does not add a ‘charset’ header when BODY
> is #f or a bytevector. Maybe we could do that?

I've just taken a look at your suggestion. That's possible although not
elegant.

I stumbled accross a bigger issue however: the dumb protocol is
implemented for git but not for libgit2, thus is not present in
guile-git.


The smart protocol is however way too complex / unadapted for such a
light use in tests, and requires more that a simple http server easily
configured.

That's sad, such a light implementation would've been quite convenient
in my juliahub case.

Toggle quote (7 lines)
>
> Thanks,
> Ludo’.
>
>
>

--
Best regards,
Nicolas Graves
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 62202@patchwise.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 62202
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch