[RFC PATCH 0/3] Decentralized substitute distribution with ERIS

  • Open
  • quality assurance status badge
Details
3 participants
  • Ludovic Courtès
  • Maxime Devos
  • pukkamustard
Owner
unassigned
Submitted by
pukkamustard
Severity
important

Debbugs page

P
P
pukkamustard wrote on 16 Dec 2021 08:17
(address . guix-patches@gnu.org)
20211216161724.547-1-pukkamustard@posteo.net
Hello Guix,

This is an initial patch and proposal towards decentralizing substitute
distribution with ERIS.

ERIS (Encoding for Robust Immutable Storage) [1] is an encoding of content into
uniformly sized, encryped and content-addressed blocks. The original content
can be reconstructed only with access to a read capability, which can be
encoded as an URN.

One key advantage of ERIS is that the encoding is protocol agnostic. Any
protocol that can transfer small (32KiB) sized blocks referenced by the hash of
their content will do. This can be done with things such as GNUNet, IPFS,
OpenDHT, HTTP or a USB stick on a bicycle.

The following patch allows substitutes to be published over IPFS using ERIS.
This is inspired and very similar to previous work on distributing substitutes
over IPFS [2].

The narinfos served by `guix publish` look like this:

Toggle snippet (16 lines)
StorePath: /gnu/store/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
URL: nar/gzip/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
Compression: gzip
FileSize: 67363
ERIS: urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDVDAGJUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE
URL: nar/zstd/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
Compression: zstd
FileSize: 64917
ERIS: urn:erisx2:BIBO7KS7SAWHDNC43DVILOSQ3F3SRRHEV6YPLDCSZ7MMD6LZVCHQMEQ6FUBTJAPSNFF7XR5XPTP4OQ72OPABNEO7UYBUN42O46ARKHBTGM
NarHash: sha256:1sagsz1mnlqkr8r8s6gwkzvvhq619rlzhpbxl3h0b111n5hn2w9w
NarSize: 220704
References: 2fk1gz2s7ppdicynscra9b19byrrr866-glibc-2.33 81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10 90lbavffg0csrf208nw0ayj1bz5knl47-gcc-10.3.0-lib
Deriver: 260bk0ch4np4h2yz5yqhf8hjbsyhwpmr-hello-2.10.drv
Signature: 1;strawberry;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjNDk4ODkwODZDNTY4MzQyRENFQzk3QzA3NDE4NEQ1RkRCOTNCNDA2MUNCRDM4MUExRjVBQzVDODI0MTgwMTU3OSMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMEU2NDlFODE4QzRFNjNGNEY2OUQ5QTAwRjUwNjRDMzQ3QjY3RDM0RTM0NTg2MkI4NTc3RTg5MUY5Q0Q3NDhBQiMpCiAgIChzICMwMTZGRjA1MDdCQjZGMzA2NUEzMjYzRDA2MTAyRDc5MTBEOEZGODc5RTdENjREODRFODBENDBGMTJFMTBBOTQ1IykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjMDRDMkY4ODk1QTU0NDNGNTlCODk2NDEwMEI1MDY0NzU4RjQ1N0YzMENEREE1MTQyQzE0MDc0NjExNTA1NTc5MCMpCiAgICkKICApCiApCg==

For every compressed nar the ERIS URN is computed and added.

If the `--ipfs` is used for `guix publish` then the encoded blocks are also
uploaded to the IPFS daemon. The nar could then be retrieved from anywhere like
this:

Toggle snippet (8 lines)
(use-modules (eris)
(eris blocks ipfs))

(eris-decode->bytevector
"urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDVDAGJUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE"
eris-blocks-ipfs-ref)

These patches do not yet retrieve content from IPFS (TODO). But in principle,
anybody connected to IPFS can get the nar with the ERIS URN. This could be used
to reduce load on substitute server as they would only need to publish the ERIS
URN directly - substitutes could be delivered much more peer-to-peer.

Other transports that I have been looking in to and am pretty sure will work
include: HTTP (with RFC 2169 [3]), GNUNet, OpenDHT. This is, imho, the
advantage of ERIS over IPFS directly or GNUNet directly. The encoding and
identifiers (URN) are abstracted away from specific transports (and also
applications). ERIS is almost exactly the same encoding as used in GNUNet
(ECRS).

Blocks can be stored in any kind of databases (see for example the GDBM
bindings [4]).

A tricky things is figuring out how to multiplex all these different
transports and storages...

The ERIS specification is still considered "experimental". However we feel
confident to stabilize it and intend to do so around February/March 2022 with a
release 1.0.0 of the specification. This will ensure that the identifiers
remain stable for the forseeable future (until the crypto breaks). Before that
there is also a small external security audit of the specification planned
(thanks to NGI0/NLnet!).

This is just a little demo of the idea and some food for thought and
discussion. Give it a try and let me know what you think!

I've also pushed the patches to my personal Guix mirror if you want to check it
out from there:


Also CCing ~pukkamustard/eris@lists.sr.ht where there is some general ERIS
related discussion.

Thanks,
-pukkamustard



pukkamustard (3):
publish: Add ERIS URN to narinfo
WIP: gnu: guile-eris: Update to unreleased git version.
publish: Add IPFS support.

configure.ac | 5 ++
gnu/packages/guile-xyz.scm | 10 ++--
gnu/packages/package-management.scm | 1 +
guix/narinfo.scm | 10 ++--
guix/scripts/publish.scm | 79 ++++++++++++++++++++++-------
5 files changed, 79 insertions(+), 26 deletions(-)

--
2.34.0
P
P
pukkamustard wrote on 16 Dec 2021 08:20
[RFC PATCH 1/3] publish: Add ERIS URN to narinfo
(address . 52555@debbugs.gnu.org)
20211216162036.694-1-pukkamustard@posteo.net
* guix/scripts/publish.scm: (bake-narinfo+nar): Compute ERIS URN of compressed nars.
(narinfo-string): Add #:eris-urns parameter and honor it.
(store-item->recutils): Add #:eris-urn parameter and honor it.
* guix/scripts/narinfo.scm: (<narinfo>)[eris-urns]: New field.
(narinfo-maker): Handle ERIS URN.
* configure.ac: (HAVE_GUILE_ERIS): New conditional.
* gnu/packages/package-management.scm: (guix)[native-inputs]: Add guile-eris.
---
configure.ac | 5 +++++
gnu/packages/package-management.scm | 1 +
guix/narinfo.scm | 10 ++++++----
guix/scripts/publish.scm | 31 +++++++++++++++++++++--------
4 files changed, 35 insertions(+), 12 deletions(-)

Toggle diff (168 lines)
diff --git a/configure.ac b/configure.ac
index 341cff8fbd..72396be8aa 100644
--- a/configure.ac
+++ b/configure.ac
@@ -170,6 +170,11 @@ GUILE_MODULE_AVAILABLE([have_guile_avahi], [(avahi)])
AM_CONDITIONAL([HAVE_GUILE_AVAHI],
[test "x$have_guile_avahi" = "xyes"])
+dnl Check for Guile-eris.
+GUILE_MODULE_AVAILABLE([have_guile_eris], [(eris)])
+AM_CONDITIONAL([HAVE_GUILE_ERIS],
+ [test "x$have_guile_eris" = "xyes"])
+
dnl Guile-newt is used by the graphical installer.
GUILE_MODULE_AVAILABLE([have_guile_newt], [(newt)])
diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm
index 9496499850..5c49167782 100644
--- a/gnu/packages/package-management.scm
+++ b/gnu/packages/package-management.scm
@@ -394,6 +394,7 @@ (define code
("guile-zstd" ,guile-zstd)
("guile-ssh" ,guile-ssh)
("guile-git" ,guile-git)
+ ("guile-eris" ,guile-eris)
;; XXX: Keep the development inputs here even though
;; they're unnecessary, just so that 'guix environment
diff --git a/guix/narinfo.scm b/guix/narinfo.scm
index 4fc550aa6c..0972ede3c1 100644
--- a/guix/narinfo.scm
+++ b/guix/narinfo.scm
@@ -67,13 +67,14 @@ (define-module (guix narinfo)
equivalent-narinfo?))
(define-record-type <narinfo>
- (%make-narinfo path uri-base uris compressions file-sizes file-hashes
- nar-hash nar-size references deriver system
+ (%make-narinfo path uri-base uris eris-urns compressions file-sizes
+ file-hashes nar-hash nar-size references deriver system
signature contents)
narinfo?
(path narinfo-path)
(uri-base narinfo-uri-base) ;URI of the cache it originates from
(uris narinfo-uris) ;list of strings
+ (eris-urns narinfo-eris-urns) ;list of (strings | #f)
(compressions narinfo-compressions) ;list of strings
(file-sizes narinfo-file-sizes) ;list of (integers | #f)
(file-hashes narinfo-file-hashes)
@@ -134,7 +135,7 @@ (define (narinfo-signature->canonical-sexp str)
(define (narinfo-maker str cache-url)
"Return a narinfo constructor for narinfos originating from CACHE-URL. STR
must contain the original contents of a narinfo file."
- (lambda (path urls compressions file-hashes file-sizes
+ (lambda (path urls eris-urns compressions file-hashes file-sizes
nar-hash nar-size references deriver system
signature)
"Return a new <narinfo> object."
@@ -148,6 +149,7 @@ (define len (length urls))
(string-append cache-url url)
(string-append cache-url "/" url)))))
urls)
+ eris-urns
compressions
(match file-sizes
(() (make-list len #f))
@@ -186,7 +188,7 @@ (define* (read-narinfo port #:optional url
"FileHash" "FileSize" "NarHash" "NarSize"
"References" "Deriver" "System"
"Signature")
- '("URL" "Compression" "FileSize" "FileHash"))))
+ '("URL" "ERIS" "Compression" "FileSize" "FileHash"))))
(define (narinfo-sha256 narinfo)
"Return the sha256 hash of NARINFO as a bytevector, or #f if NARINFO lacks a
diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm
index 6e2b4368da..8e4b90789b 100644
--- a/guix/scripts/publish.scm
+++ b/guix/scripts/publish.scm
@@ -58,6 +58,7 @@ (define-module (guix scripts publish)
#:use-module (guix workers)
#:use-module (guix store)
#:use-module ((guix serialization) #:select (write-file))
+ #:use-module (eris)
#:use-module (zlib)
#:autoload (lzlib) (call-with-lzip-output-port
make-lzip-output-port)
@@ -308,7 +309,7 @@ (define* (store-item->recutils store-item
#:key
(nar-path "nar")
(compression %no-compression)
- file-size)
+ file-size eris-urn)
"Return the 'Compression' and 'URL' fields of the narinfo for STORE-ITEM,
with COMPRESSION, starting at NAR-PATH."
(let ((url (encode-and-join-uri-path
@@ -319,19 +320,22 @@ (define* (store-item->recutils store-item
(($ <compression> type)
(list (symbol->string type))))
,(basename store-item)))))
- (format #f "URL: ~a~%Compression: ~a~%~@[FileSize: ~a~%~]"
- url (compression-type compression) file-size)))
+ (format #f "URL: ~a~%Compression: ~a~%~@[FileSize: ~a~%~]~@[ERIS: ~a~%~]"
+ url (compression-type compression) file-size eris-urn)))
(define* (narinfo-string store store-path
#:key (compressions (list %no-compression))
- (nar-path "nar") (file-sizes '()))
+ (nar-path "nar") (file-sizes '()) (eris-urns '()))
"Generate a narinfo key/value string for STORE-PATH; an exception is raised
if STORE-PATH is invalid. Produce a URL that corresponds to COMPRESSION. The
narinfo is signed with KEY. NAR-PATH specifies the prefix for nar URLs.
Optionally, FILE-SIZES is a list of compression/integer pairs, where the
integer is size in bytes of the compressed NAR; it informs the client of how
-much needs to be downloaded."
+much needs to be downloaded.
+
+Optionally, ERIS-URNS is a list of compression/string pairs, where the
+string is the ERIS URN of the compressed NAR."
(let* ((path-info (query-path-info store store-path))
(compressions (actual-compressions store-path compressions))
(hash (bytevector->nix-base32-string
@@ -352,9 +356,12 @@ (define* (narinfo-string store store-path
store-path
(map (lambda (compression)
(let ((size (assoc-ref file-sizes
- compression)))
+ compression))
+ (eris-urn (assoc-ref eris-urns
+ compression)))
(store-item->recutils store-path
#:file-size size
+ #:eris-urn eris-urn
#:nar-path nar-path
#:compression
compression)))
@@ -632,6 +639,12 @@ (define (compressed-nar-size compression)
(and stat
(cons compression (stat:size stat)))))
+ (define (compressed-eris-urn compression)
+ (let* ((nar (nar-cache-file cache item #:compression compression))
+ (stat (stat nar #f)))
+ (and stat
+ (cons compression (call-with-input-file nar eris-encode->urn)))))
+
(let ((compression (actual-compressions item compressions)))
(for-each (cut compress-nar cache item <>) compressions)
@@ -646,11 +659,13 @@ (define (compressed-nar-size compression)
;; thread's connection to the store since we would end up sending
;; stuff concurrently on the same channel.
(with-store store
- (let ((sizes (filter-map compressed-nar-size compression)))
+ (let ((sizes (filter-map compressed-nar-size compression))
+ (eris-urns (filter-map compressed-eris-urn compression)))
(display (narinfo-string store item
#:nar-path nar-path
#:compressions compressions
- #:file-sizes sizes)
+ #:file-sizes sizes
+ #:eris-urns eris-urns)
port)))
;; Make the cached narinfo world-readable, contrary to what
--
2.34.0
P
P
pukkamustard wrote on 16 Dec 2021 08:20
[RFC PATCH 2/3] WIP: gnu: guile-eris: Update to unreleased git version.
(address . 52555@debbugs.gnu.org)
20211216162036.694-2-pukkamustard@posteo.net
* gnu/packages/guile-xyz.schm (guile-eris): Update to unreleased git version.
[source]: Update source URI.
[propagated-inputs]: Add guile-json-4 and guile-gdbm-ffi.
---
gnu/packages/guile-xyz.scm | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)

Toggle diff (35 lines)
diff --git a/gnu/packages/guile-xyz.scm b/gnu/packages/guile-xyz.scm
index 8346d99996..5e6f31a6e2 100644
--- a/gnu/packages/guile-xyz.scm
+++ b/gnu/packages/guile-xyz.scm
@@ -4325,15 +4325,15 @@ (define-public guile-sodium
(define-public guile-eris
(package
(name "guile-eris")
- (version "0.2.0")
+ (version "f1e4dd87988f9a80b05a8051d7f5ba3daf79dcc1")
(source
(origin
(method git-fetch)
(uri (git-reference
- (url "https://inqlab.net/git/eris.git")
- (commit (string-append "v" version))))
+ (url "https://inqlab.net/git/guile-eris.git")
+ (commit version)))
(file-name (git-file-name name version))
- (sha256 (base32 "1ijglmwkdy1l87gj429qfjis0v8b1zlxhbyfhx5za8664h68nqka"))))
+ (sha256 (base32 "0kgm4b4qn2s74wjvxy273gdi1l1m81i2k4kkk1zc6vlcg3np7p06"))))
(build-system gnu-build-system)
(arguments '())
(native-inputs
@@ -4345,7 +4345,7 @@ (define-public guile-eris
guile-srfi-180))
(inputs (list guile-3.0))
(propagated-inputs
- (list guile-sodium))
+ (list guile-sodium guile-json-4 guile-gdbm-ffi))
(synopsis "Guile implementation of the Encoding for Robust Immutable Storage (ERIS)")
(description
"Guile-ERIS is the reference implementation of the Encoding for Robust
--
2.34.0
P
P
pukkamustard wrote on 16 Dec 2021 08:20
[RFC PATCH 3/3] publish: Add IPFS support.
(address . 52555@debbugs.gnu.org)
20211216162036.694-3-pukkamustard@posteo.net
* guix/scripts/publish.scm: (show-help, %options): Add '--ipfs'.
(render-narinfo/cached, bake-narinfo+nar, make-request-handler, run-publish-server): Add #:ipfs? and honor it.
(guix-publish): Honor '--ipfs' and parameterize %ipfs-base-url.
---
guix/scripts/publish.scm | 52 +++++++++++++++++++++++++++++++---------
1 file changed, 41 insertions(+), 11 deletions(-)

Toggle diff (178 lines)
diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm
index 8e4b90789b..8e7fb47b9e 100644
--- a/guix/scripts/publish.scm
+++ b/guix/scripts/publish.scm
@@ -41,6 +41,8 @@ (define-module (guix scripts publish)
#:use-module (srfi srfi-26)
#:use-module (srfi srfi-34)
#:use-module (srfi srfi-37)
+ #:use-module (srfi srfi-71)
+ #:use-module (srfi srfi-171)
#:use-module (web http)
#:use-module (web request)
#:use-module (web response)
@@ -52,6 +54,7 @@ (define-module (guix scripts publish)
#:use-module (guix base64)
#:use-module (guix config)
#:use-module (guix derivations)
+ #:use-module ((guix ipfs) #:prefix ipfs:)
#:use-module (gcrypt hash)
#:use-module (guix pki)
#:use-module (gcrypt pk-crypto)
@@ -59,6 +62,8 @@ (define-module (guix scripts publish)
#:use-module (guix store)
#:use-module ((guix serialization) #:select (write-file))
#:use-module (eris)
+ #:use-module (eris read-capability)
+ #:use-module (eris blocks ipfs)
#:use-module (zlib)
#:autoload (lzlib) (call-with-lzip-output-port
make-lzip-output-port)
@@ -83,6 +88,7 @@ (define-module (guix scripts publish)
run-publish-server
guix-publish))
+
(define (show-help)
(format #t (G_ "Usage: guix publish [OPTION]...
Publish ~a over HTTP.\n") %store-directory)
@@ -102,6 +108,8 @@ (define (show-help)
(display (G_ "
--cache-bypass-threshold=SIZE
serve store items below SIZE even when not cached"))
+ (display (G_ "
+ --ipfs[=GATEWAY] publish items over IPFS via GATEWAY"))
(display (G_ "
--workers=N use N workers to bake items"))
(display (G_ "
@@ -220,6 +228,10 @@ (define %options
(lambda (opt name arg result)
(alist-cons 'cache-bypass-threshold (size->number arg)
result)))
+ (option '("ipfs") #f #t
+ (lambda (opt name arg result)
+ (alist-cons 'ipfs (or arg (ipfs:%ipfs-base-url))
+ result)))
(option '("workers") #t #f
(lambda (opt name arg result)
(alist-cons 'workers (string->number* arg)
@@ -526,7 +538,7 @@ (define (bypass-cache? store item)
(define* (render-narinfo/cached store request hash
#:key ttl (compressions (list %no-compression))
(nar-path "nar") negative-ttl
- cache pool)
+ cache pool ipfs?)
"Respond to the narinfo request for REQUEST. If the narinfo is available in
CACHE, then send it; otherwise, return 404 and \"bake\" that nar and narinfo
requested using POOL."
@@ -571,7 +583,8 @@ (define (delete-entry narinfo)
(bake-narinfo+nar cache item
#:ttl ttl
#:compressions compressions
- #:nar-path nar-path)))
+ #:nar-path nar-path
+ #:ipfs? ipfs?)))
(when ttl
(single-baker 'cache-cleanup
@@ -631,7 +644,7 @@ (define (write-compressed-file call-with-compressed-output-port)
(define* (bake-narinfo+nar cache item
#:key ttl (compressions (list %no-compression))
- (nar-path "/nar"))
+ (nar-path "/nar") ipfs?)
"Write the narinfo and nar for ITEM to CACHE."
(define (compressed-nar-size compression)
(let* ((nar (nar-cache-file cache item #:compression compression))
@@ -641,9 +654,19 @@ (define (compressed-nar-size compression)
(define (compressed-eris-urn compression)
(let* ((nar (nar-cache-file cache item #:compression compression))
- (stat (stat nar #f)))
+ (stat (stat nar #f))
+ (block-reducer (if ipfs?
+ (eris-blocks-ipfs-reducer
+ #:ipfs-base-url (ipfs:%ipfs-base-url))
+ rcount)))
(and stat
- (cons compression (call-with-input-file nar eris-encode->urn)))))
+ (cons compression
+ (call-with-input-file nar
+ (lambda (port)
+ (let ((read-cap _
+ (eris-encode port #:block-reducer
+ block-reducer)))
+ (read-capability->string read-cap))))))))
(let ((compression (actual-compressions item compressions)))
@@ -1115,7 +1138,8 @@ (define* (make-request-handler store
cache pool
narinfo-ttl narinfo-negative-ttl
(nar-path "nar")
- (compressions (list %no-compression)))
+ (compressions (list %no-compression))
+ ipfs?)
(define compression-type?
string->compression-type)
@@ -1147,7 +1171,8 @@ (define (handle request body)
#:ttl narinfo-ttl
#:negative-ttl narinfo-negative-ttl
#:nar-path nar-path
- #:compressions compressions)
+ #:compressions compressions
+ #:ipfs? ipfs?)
(render-narinfo store request hash
#:ttl narinfo-ttl
#:negative-ttl narinfo-negative-ttl
@@ -1218,7 +1243,7 @@ (define* (run-publish-server socket store
advertise? port
(compressions (list %no-compression))
(nar-path "nar") narinfo-ttl narinfo-negative-ttl
- cache pool)
+ cache pool ipfs?)
(when advertise?
(let ((name (service-name)))
;; XXX: Use a callback from Guile-Avahi here, as Avahi can pick a
@@ -1234,7 +1259,8 @@ (define* (run-publish-server socket store
#:nar-path nar-path
#:narinfo-ttl narinfo-ttl
#:narinfo-negative-ttl narinfo-negative-ttl
- #:compressions compressions)
+ #:compressions compressions
+ #:ipfs? ipfs?)
concurrent-http-server
`(#:socket ,socket)))
@@ -1296,6 +1322,8 @@ (define-command (guix-publish . args)
(repl-port (assoc-ref opts 'repl))
(cache (assoc-ref opts 'cache))
(workers (assoc-ref opts 'workers))
+ (ipfs (assoc-ref opts 'ipfs))
+ (ipfs? (if ipfs #t #f))
;; Read the key right away so that (1) we fail early on if we can't
;; access them, and (2) we can then drop privileges.
@@ -1315,7 +1343,8 @@ (define-command (guix-publish . args)
(%private-key private-key)
(cache-bypass-threshold
(or (assoc-ref opts 'cache-bypass-threshold)
- (cache-bypass-threshold))))
+ (cache-bypass-threshold)))
+ (ipfs:%ipfs-base-url ipfs))
(info (G_ "publishing ~a on ~a, port ~d~%")
%store-directory
(inet-ntop (sockaddr:fam address) (sockaddr:addr address))
@@ -1344,7 +1373,8 @@ (define-command (guix-publish . args)
#:nar-path nar-path
#:compressions compressions
#:narinfo-negative-ttl negative-ttl
- #:narinfo-ttl ttl))))))
+ #:narinfo-ttl ttl
+ #:ipfs? ipfs?))))))
;;; Local Variables:
;;; eval: (put 'single-baker 'scheme-indent-function 1)
--
2.34.0
L
L
Ludovic Courtès wrote on 18 Dec 2021 12:01
control message for bug #52555
(address . control@debbugs.gnu.org)
87czltof7d.fsf@gnu.org
severity 52555 important
quit
L
L
Ludovic Courtès wrote on 20 Dec 2021 08:25
Re: bug#52555: [RFC PATCH 0/3] Decentralized substitute distribution with ERIS
(name . pukkamustard)(address . pukkamustard@posteo.net)
87h7b3gs64.fsf@gnu.org
Hi pukkamustard,

pukkamustard <pukkamustard@posteo.net> skribis:

Toggle quote (3 lines)
> This is an initial patch and proposal towards decentralizing substitute
> distribution with ERIS.

Woohoo, sounds exciting!

Toggle quote (10 lines)
> ERIS (Encoding for Robust Immutable Storage) [1] is an encoding of content into
> uniformly sized, encryped and content-addressed blocks. The original content
> can be reconstructed only with access to a read capability, which can be
> encoded as an URN.
>
> One key advantage of ERIS is that the encoding is protocol agnostic. Any
> protocol that can transfer small (32KiB) sized blocks referenced by the hash of
> their content will do. This can be done with things such as GNUNet, IPFS,
> OpenDHT, HTTP or a USB stick on a bicycle.

Yes, that’s nice.

Toggle quote (16 lines)
> The following patch allows substitutes to be published over IPFS using ERIS.
> This is inspired and very similar to previous work on distributing substitutes
> over IPFS [2].
>
> The narinfos served by `guix publish` look like this:
>
> StorePath: /gnu/store/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
> URL: nar/gzip/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
> Compression: gzip
> FileSize: 67363
> ERIS: urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDVDAGJUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE
> URL: nar/zstd/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
> Compression: zstd
> FileSize: 64917
> ERIS: urn:erisx2:BIBO7KS7SAWHDNC43DVILOSQ3F3SRRHEV6YPLDCSZ7MMD6LZVCHQMEQ6FUBTJAPSNFF7XR5XPTP4OQ72OPABNEO7UYBUN42O46ARKHBTGM

Do we really need one URN per compression method? Couldn’t we leave
compression (of individual chunks, possibly) as a “detail” handled by
the encoding or the transport layer?

Toggle quote (16 lines)
> If the `--ipfs` is used for `guix publish` then the encoded blocks are also
> uploaded to the IPFS daemon. The nar could then be retrieved from anywhere like
> this:
>
> (use-modules (eris)
> (eris blocks ipfs))
>
> (eris-decode->bytevector
> "urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDVDAGJUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE"
> eris-blocks-ipfs-ref)
>
> These patches do not yet retrieve content from IPFS (TODO). But in principle,
> anybody connected to IPFS can get the nar with the ERIS URN. This could be used
> to reduce load on substitute server as they would only need to publish the ERIS
> URN directly - substitutes could be delivered much more peer-to-peer.

Nice. So adjusting ‘guix substitute’ should be relatively easy?

Toggle quote (7 lines)
> Other transports that I have been looking in to and am pretty sure will work
> include: HTTP (with RFC 2169 [3]), GNUNet, OpenDHT. This is, imho, the
> advantage of ERIS over IPFS directly or GNUNet directly. The encoding and
> identifiers (URN) are abstracted away from specific transports (and also
> applications). ERIS is almost exactly the same encoding as used in GNUNet
> (ECRS).

As a first step, ‘guix publish’ could implement RFC 2169, too.

I gather implementing the HTTP and IPFS backends in ‘guix substitute’
should be relatively easy, right?

Toggle quote (6 lines)
> Blocks can be stored in any kind of databases (see for example the GDBM
> bindings [4]).
>
> A tricky things is figuring out how to multiplex all these different
> transports and storages...

Yes. We don’t know yet what performance and data availability will be
like on IPFS, for instance, so it’s important for users to be able to
set priorities. It’s also important to gracefully fall back to direct
HTTP downloads when fancier p2p methods fail, regardless of how they
fail.

Toggle quote (7 lines)
> The ERIS specification is still considered "experimental". However we feel
> confident to stabilize it and intend to do so around February/March 2022 with a
> release 1.0.0 of the specification. This will ensure that the identifiers
> remain stable for the forseeable future (until the crypto breaks). Before that
> there is also a small external security audit of the specification planned
> (thanks to NGI0/NLnet!).

Neat.

This is all very exciting. I look forward to playing around with it!

Ludo’.
P
P
pukkamustard wrote on 23 Dec 2021 03:42
(name . Ludovic Courtès)(address . ludo@gnu.org)
86bl17ms56.fsf@posteo.net
Hi Ludo,

Thanks for your comments!

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (15 lines)
>> StorePath: /gnu/store/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
>> URL: nar/gzip/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
>> Compression: gzip
>> FileSize: 67363
>> ERIS: urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDVDAGJUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE
>> URL: nar/zstd/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
>> Compression: zstd
>> FileSize: 64917
>> ERIS: urn:erisx2:BIBO7KS7SAWHDNC43DVILOSQ3F3SRRHEV6YPLDCSZ7MMD6LZVCHQMEQ6FUBTJAPSNFF7XR5XPTP4OQ72OPABNEO7UYBUN42O46ARKHBTGM
>
> Do we really need one URN per compression method? Couldn’t we leave
> compression (of individual chunks, possibly) as a “detail” handled by
> the encoding or the transport layer?
>

I agree that it would be nice to leave this to the encoding layer as
that would allow certain optimizations (e.g. de-duplication).

Unfortunately, we haven't figured out yet what the most suitable
compression/format would be. Something like EROSFS seems good (as it
aligns data to fixed block sizes) [1]. But this seems a bit "clunky" for
just an archive format and there do not seem to be any libraries that we
could use to neatly integrate. It seems possible to block-align a Tar
archive, but that seems a bit hackey [2]. Other things to look into
might be Tarlz [3] and ZPAQ [4].

To get started I suggest just using one of the compressions/formats
already in Guix. zstd seems to be a reasonable choice (for the same
reasons why it makes sense to use zstd with `--discover` [5]).

Does that sound like a plan?


Toggle quote (18 lines)
>> If the `--ipfs` is used for `guix publish` then the encoded blocks are also
>> uploaded to the IPFS daemon. The nar could then be retrieved from anywhere like
>> this:
>>
>> (use-modules (eris)
>> (eris blocks ipfs))
>>
>> (eris-decode->bytevector
>> "urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDVDAGJUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE"
>> eris-blocks-ipfs-ref)
>>
>> These patches do not yet retrieve content from IPFS (TODO). But in principle,
>> anybody connected to IPFS can get the nar with the ERIS URN. This could be used
>> to reduce load on substitute server as they would only need to publish the ERIS
>> URN directly - substitutes could be delivered much more peer-to-peer.
>
> Nice. So adjusting ‘guix substitute’ should be relatively easy?

Yes, relatively! :)

I meant to send in a V2 that does this before going on holidays, but I'm
afraid I won't make it. V2 will come in early January!

Toggle quote (12 lines)
>> Other transports that I have been looking in to and am pretty sure will work
>> include: HTTP (with RFC 2169 [3]), GNUNet, OpenDHT. This is, imho, the
>> advantage of ERIS over IPFS directly or GNUNet directly. The encoding and
>> identifiers (URN) are abstracted away from specific transports (and also
>> applications). ERIS is almost exactly the same encoding as used in GNUNet
>> (ECRS).
>
> As a first step, ‘guix publish’ could implement RFC 2169, too.
>
> I gather implementing the HTTP and IPFS backends in ‘guix substitute’
> should be relatively easy, right?

Yes, those seem to be the two easiest backends to implement.

Toggle quote (9 lines)
>> A tricky things is figuring out how to multiplex all these different
>> transports and storages...
>
> Yes. We don’t know yet what performance and data availability will be
> like on IPFS, for instance, so it’s important for users to be able to
> set priorities. It’s also important to gracefully fall back to direct
> HTTP downloads when fancier p2p methods fail, regardless of how they
> fail.

Agree.

Thanks,
-pukkamustard
L
L
Ludovic Courtès wrote on 24 Dec 2021 06:48
(name . pukkamustard)(address . pukkamustard@posteo.net)
87ilveukhr.fsf@gnu.org
Hi!

pukkamustard <pukkamustard@posteo.net> skribis:

Toggle quote (28 lines)
> Ludovic Courtès <ludo@gnu.org> writes:
>
>>> StorePath: /gnu/store/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
>>> URL: nar/gzip/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
>>> Compression: gzip
>>> FileSize: 67363
>>> ERIS: urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDVDAGJUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE
>>> URL: nar/zstd/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
>>> Compression: zstd
>>> FileSize: 64917
>>> ERIS: urn:erisx2:BIBO7KS7SAWHDNC43DVILOSQ3F3SRRHEV6YPLDCSZ7MMD6LZVCHQMEQ6FUBTJAPSNFF7XR5XPTP4OQ72OPABNEO7UYBUN42O46ARKHBTGM
>>
>> Do we really need one URN per compression method? Couldn’t we leave
>> compression (of individual chunks, possibly) as a “detail” handled by
>> the encoding or the transport layer?
>>
>
> I agree that it would be nice to leave this to the encoding layer as
> that would allow certain optimizations (e.g. de-duplication).
>
> Unfortunately, we haven't figured out yet what the most suitable
> compression/format would be. Something like EROSFS seems good (as it
> aligns data to fixed block sizes) [1]. But this seems a bit "clunky" for
> just an archive format and there do not seem to be any libraries that we
> could use to neatly integrate. It seems possible to block-align a Tar
> archive, but that seems a bit hackey [2]. Other things to look into
> might be Tarlz [3] and ZPAQ [4].

Yeah. Though it may be that deduplication at the block level doesn’t
buy us much. That was the conclusion I reached a long time ago[a], and
also seems to be supported by the recent guix-daemon deduplication
improvements[b].


Toggle quote (6 lines)
> To get started I suggest just using one of the compressions/formats
> already in Guix. zstd seems to be a reasonable choice (for the same
> reasons why it makes sense to use zstd with `--discover` [5]).
>
> Does that sound like a plan?

Sure!

Toggle quote (3 lines)
> I meant to send in a V2 that does this before going on holidays, but I'm
> afraid I won't make it. V2 will come in early January!

Alright, we’ll see! :-)

Until then, enjoy your holidays!

Ludo’.
P
P
pukkamustard wrote on 25 Jan 2022 11:21
[RFC PATCH v2 0/5] Decentralized substitute distribution with ERIS
(address . 52555@debbugs.gnu.org)
20220125192201.7582-1-pukkamustard@posteo.net
Hello Guix,

Here comes the V2 of a proposal towards decentralizing substitute distribution
with ERIS.

A quick summary (as this has become quite long):

- This adds support for publishing and getting substitutes over IPFS.
- By using the ERIS encoding we are not limited to using IPFS as transport. We
can also use GNUNet, Named Data Networking (possibly) or just plain old HTTP. Support
for these can be added in (guix eris).
- These patches are still very rough and we need better logic for when to use
IPFS et. al. and when to fallback to HTTP.
- There might be performance issues when using IPFS via the IPFS daemon HTTP
API.

I found the setup for testing this a bit tricky. I will try and describe how I
have been testing it. Please let me know how this can be improved!

** Authorize local substitutes

We will be running a local substitute server so we need to add the local
signing key to the list of authorized keys. In the system configurations:

#+BEGIN_SRC scheme
(modify-services %base-services
(guix-service-type
config =>
(guix-configuration
(inherit config)
(authorized-keys
(cons*
;; allow substitutes from ourselves for testing purposes
(local-file "/etc/signing-key.pub")
%default-authorized-guix-keys)))))
#+END_SRC

** Configure the local Guix checkout

#+BEGIN_SRC shell
./bootstrap && ./configure --localstatedir=/var --sysconfdir=/etc
#+END_SRC

The ~--sysconfdir~ is required so that guix will use the ACL in ~/etc/guix/acl~.

** Start the IPFS daemon

#+BEGIN_SRC shell
guix shell go-ipfs -- ipfs daemon
#+END_SRC

Start a local substitute server:

#+BEGIN_SRC shell
sudo -E ./pre-inst-env guix publish --public-key=/etc/guix/signing-key.pub --private-key=/etc/guix/signing-key.sec --cache=/tmp/guix-publish-cache/ --port=8081 --compression=zstd:19
#+END_SRC

We use port 8081 as IPFS is running on 8080.

We use the temporary cache directory ~/tmp/guix-publish-cache~.

** Build some package locally

First we build some package:

#+BEGIN_SRC shell
./pre-inst-env guix build hello --no-substitutes --no-offload
#+END_SRC

#+RESULTS:
: /gnu/store/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11

** Trigger the substitute server to "bake" a susbtitute

#+BEGIN_SRC shell
#+END_SRC
Toggle snippet (10 lines)
StorePath: /gnu/store/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11
URL: nar/zstd/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11
Compression: zstd
NarHash: sha256:11pk3jsh4zk0gigyjk881ay1nnvjfgpd3xpb4rmbaljhbiis4jbm
NarSize: 190480
References: 094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib 5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.33 khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11
Deriver: mc7i1cdi42gy89mxl48nhdhgrfa9lpq6-hello-2.11.drv
Signature: 1;strawberry;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjOTE0QTVGNTE4NUZGRUIzMzc4QTEwMzgzQzdFMEU1NDI1MEUyREZDRjk1RDUwOTNCMzU4QTFBNDE4OUFBRDVGNCMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMDkxMDA2NDlCMkMyMzhEQzE2ODhFQTgyQTdCOEJFMTc5MTVBMjVDQjc1NzcwQjlGRkNGOTFDRTg2MDgyNzAwQiMpCiAgIChzICMwMUFBQ0VERjY0N0VENTQyRTIwNENDMEM1M0VDMEY0QjQ4QzdEOTAyRkFEQTkxREI4NzRGQjE2MTQ4QTIzNUI2IykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjMDRDMkY4ODk1QTU0NDNGNTlCODk2NDEwMEI1MDY0NzU4RjQ1N0YzMENEREE1MTQyQzE0MDc0NjExNTA1NTc5MCMpCiAgICkKICApCiApCg==

If you do this again after a few seconds you will get a different response that
has the ERIS URN and the FileSizes. The reason for this is that Guix publish
bakes the nars asyncrhonisly in the background:

#+BEGIN_SRC shell
#+END_SRC

#+RESULTS:
Toggle snippet (13 lines)
StorePath: /gnu/store/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11
URL: nar/zstd/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11
Compression: zstd
FileSize: 57691
NarHash: sha256:11pk3jsh4zk0gigyjk881ay1nnvjfgpd3xpb4rmbaljhbiis4jbm
NarSize: 190480
ERISFormat: application/x-nix-archive+zstd-19
ERIS: urn:erisx2:B4AYPTXLTACB6WJYJ74RKBCVU3RBLHA4PY6HATUWRZNJ6THVSDUFM34K2ASUF3B6EOYEEBRZ5XEUR4PAAAIED7G7YSEZVZ5V7WWZ2PSC7Q
References: 094bbaq6glba86h1d4cj16xhdi6fk2jl-gcc-10.3.0-lib 5h2w4qi9hk1qzzgi1w83220ydslinr4s-glibc-2.33 khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11
Deriver: mc7i1cdi42gy89mxl48nhdhgrfa9lpq6-hello-2.11.drv
Signature: 1;strawberry;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjOTE0QTVGNTE4NUZGRUIzMzc4QTEwMzgzQzdFMEU1NDI1MEUyREZDRjk1RDUwOTNCMzU4QTFBNDE4OUFBRDVGNCMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMDkxMDA2NDlCMkMyMzhEQzE2ODhFQTgyQTdCOEJFMTc5MTVBMjVDQjc1NzcwQjlGRkNGOTFDRTg2MDgyNzAwQiMpCiAgIChzICMwMUFBQ0VERjY0N0VENTQyRTIwNENDMEM1M0VDMEY0QjQ4QzdEOTAyRkFEQTkxREI4NzRGQjE2MTQ4QTIzNUI2IykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjMDRDMkY4ODk1QTU0NDNGNTlCODk2NDEwMEI1MDY0NzU4RjQ1N0YzMENEREE1MTQyQzE0MDc0NjExNTA1NTc5MCMpCiAgICkKICApCiApCg==

These patches have added the ERIS and ERISFormat fields. Eventually we would
have figured out what the best format for use over ERIS is, for now we encode
it in the ERISFormat field.

** Removing a package from the store

This is necessary in order to make guix look for a substitute.

#+BEGIN_SRC shell
./pre-inst-env guix gc -D /gnu/store/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11
#+END_SRC

** Start the Guix daemon from the repository

#+BEGIN_SRC shell
sudo -E ./pre-inst-env guix-daemon --build-users-group=guixbuild --debug --substitute-urls=http://localhost:8081/
#+END_SRC

Note this will probably stop your system Guix daemon. Run ~sudo herd restart
guix-daemon~ to restart it.

#+BEGIN_SRC shell
./pre-inst-env guix build hello
#+END_SRC
Toggle snippet (7 lines)
substituting /gnu/store/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11...
downloading from urn:erisx2:B4AYPTXLTACB6WJYJ74RKBCVU3RBLHA4PY6HATUWRZNJ6THVSDUFM34K2ASUF3B6EOYEEBRZ5XEUR4PAAAIED7G7YSEZVZ5V7WWZ2PSC7Q ...
urn:erisx2:B4AYPTXLTACB6WJYJ74RKBCVU3RBLHA4PY6HATUWRZNJ6THVSDUFM34K2ASUF3B6EOYEEBRZ5XEUR4PAAAIED7G7YSEZVZ5V7WWZ2PSC7Q 502KiB/s 00:00 | 56KiB transferred

/gnu/store/khaaib6s836bk5kbik239hlk6n6ianc4-hello-2.11

We have just retreived the substitute for the hello package from IPFS. Hello
decentralized substitutes!

I have only tested this for fairly small packages (up to a few MB).

One issue with IPFS might be that we have to create a new HTTP connection to
the IPFS daemon for every single block (32KiB). The IPFS daemon does not seem
to support HTTP connection re-use and neither does the Guile (web client). I
fear this might become a performance issue. It seems possible to use IPFS more
directly by exposing the Go code as a C library and then using that with the
Guile FFI [1]. This is however a bit complicated and adds a lot of
dependencies. In particular, this should not become a dependency of Guix
itself. The performance of IPFS itself also needs to be evaluated, maybe the
IPFS HTTP API will not be the bottle-neck.

As mentioned in previous mail a simple HTTP transport for blocks would be a
good fallback. This would allow users to get missing blocks (things that
somehow got dropped from IPFS) directly from a substitute server. This is
different then getting the entire NAR from a substitute server. A user might be
missing a single 32KiB block and should be able to get only that. However, such
a HTTP fallback would also suffer from the one-connection-per-block issue. As
part of general ERIS research we are investigating CoAP as a better fallback
transport.

In any case, it would be necessary for the substitute server to store encoded
blocks of the NAR. For this I think it makes sense to use a small database. We
have bindings to use ERIS with GDBM [2]. It might also make sense to use
SQLite, especially if there are other use-cases for such a database.

I will be looking into the HTTP fallback and also using BitTorrent and GNUNet
as transports.

Thanks for making it so far and happy hacking!
-pukkamustard



pukkamustard (5):
WIP: gnu: guile-eris: Update to unreleased git version.
publish: Add ERIS URN to narinfo
Add (guix eris).
publish: Add support for storing ERIS encoded blocks to IPFS.
substitute: Fetch substitutes using ERIS.

Makefile.am | 1 +
configure.ac | 5 +++
gnu/packages/guile-xyz.scm | 10 ++---
gnu/packages/package-management.scm | 1 +
guix/eris.scm | 60 +++++++++++++++++++++++++++++
guix/narinfo.scm | 14 +++++--
guix/scripts/publish.scm | 32 +++++++++++++--
guix/scripts/substitute.scm | 21 +++++++---
8 files changed, 126 insertions(+), 18 deletions(-)
create mode 100644 guix/eris.scm

--
2.34.0
P
P
pukkamustard wrote on 25 Jan 2022 11:21
[RFC PATCH v2 2/5] publish: Add ERIS URN to narinfo
(address . 52555@debbugs.gnu.org)
20220125192201.7582-3-pukkamustard@posteo.net
* guix/scripts/publish.scm: (bake-narinfo+nar): Compute ERIS URN of compressed nars.
(narinfo-string): Add #:eris-urn parameter and honor it.
* guix/scripts/narinfo.scm: (<narinfo>)[eris-format,eris-urn]: New fields.
(narinfo-maker): Handle ERIS URN and ERIS format.
* configure.ac: (HAVE_GUILE_ERIS): New conditional.
* gnu/packages/package-management.scm: (guix)[native-inputs]: Add guile-eris.
---
configure.ac | 5 +++++
gnu/packages/package-management.scm | 1 +
guix/narinfo.scm | 14 ++++++++++----
guix/scripts/publish.scm | 24 ++++++++++++++++++++----
4 files changed, 36 insertions(+), 8 deletions(-)

Toggle diff (167 lines)
diff --git a/configure.ac b/configure.ac
index 341cff8fbd..72396be8aa 100644
--- a/configure.ac
+++ b/configure.ac
@@ -170,6 +170,11 @@ GUILE_MODULE_AVAILABLE([have_guile_avahi], [(avahi)])
AM_CONDITIONAL([HAVE_GUILE_AVAHI],
[test "x$have_guile_avahi" = "xyes"])
+dnl Check for Guile-eris.
+GUILE_MODULE_AVAILABLE([have_guile_eris], [(eris)])
+AM_CONDITIONAL([HAVE_GUILE_ERIS],
+ [test "x$have_guile_eris" = "xyes"])
+
dnl Guile-newt is used by the graphical installer.
GUILE_MODULE_AVAILABLE([have_guile_newt], [(newt)])
diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm
index 05795824b5..a9094b8b7f 100644
--- a/gnu/packages/package-management.scm
+++ b/gnu/packages/package-management.scm
@@ -404,6 +404,7 @@ (define code
("guile-zstd" ,guile-zstd)
("guile-ssh" ,guile-ssh)
("guile-git" ,guile-git)
+ ("guile-eris" ,guile-eris)
;; XXX: Keep the development inputs here even though
;; they're unnecessary, just so that 'guix environment
diff --git a/guix/narinfo.scm b/guix/narinfo.scm
index 4fc550aa6c..a6a5d3b84b 100644
--- a/guix/narinfo.scm
+++ b/guix/narinfo.scm
@@ -45,6 +45,8 @@ (define-module (guix narinfo)
narinfo-file-sizes
narinfo-hash
narinfo-size
+ narinfo-eris-format
+ narinfo-eris-urn
narinfo-references
narinfo-deriver
narinfo-system
@@ -68,8 +70,8 @@ (define-module (guix narinfo)
(define-record-type <narinfo>
(%make-narinfo path uri-base uris compressions file-sizes file-hashes
- nar-hash nar-size references deriver system
- signature contents)
+ nar-hash nar-size eris-format eris-urn references deriver
+ system signature contents)
narinfo?
(path narinfo-path)
(uri-base narinfo-uri-base) ;URI of the cache it originates from
@@ -79,6 +81,8 @@ (define-record-type <narinfo>
(file-hashes narinfo-file-hashes)
(nar-hash narinfo-hash)
(nar-size narinfo-size)
+ (eris-format narinfo-eris-format)
+ (eris-urn narinfo-eris-urn)
(references narinfo-references)
(deriver narinfo-deriver)
(system narinfo-system)
@@ -135,7 +139,7 @@ (define (narinfo-maker str cache-url)
"Return a narinfo constructor for narinfos originating from CACHE-URL. STR
must contain the original contents of a narinfo file."
(lambda (path urls compressions file-hashes file-sizes
- nar-hash nar-size references deriver system
+ nar-hash nar-size eris-format eris-urn references deriver system
signature)
"Return a new <narinfo> object."
(define len (length urls))
@@ -157,6 +161,8 @@ (define len (length urls))
((lst ...) (map string->number lst)))
nar-hash
(and=> nar-size string->number)
+ eris-format
+ (if eris-urn (string->uri eris-urn) #f)
(string-tokenize references)
(match deriver
((or #f "") #f)
@@ -184,7 +190,7 @@ (define* (read-narinfo port #:optional url
(narinfo-maker str url)
'("StorePath" "URL" "Compression"
"FileHash" "FileSize" "NarHash" "NarSize"
- "References" "Deriver" "System"
+ "ERISFormat" "ERIS" "References" "Deriver" "System"
"Signature")
'("URL" "Compression" "FileSize" "FileHash"))))
diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm
index 6e2b4368da..9c83f5183d 100644
--- a/guix/scripts/publish.scm
+++ b/guix/scripts/publish.scm
@@ -58,6 +58,7 @@ (define-module (guix scripts publish)
#:use-module (guix workers)
#:use-module (guix store)
#:use-module ((guix serialization) #:select (write-file))
+ #:use-module (eris)
#:use-module (zlib)
#:autoload (lzlib) (call-with-lzip-output-port
make-lzip-output-port)
@@ -146,6 +147,9 @@ (define %default-gzip-compression
;; Since we compress on the fly, default to fast compression.
(compression 'gzip 3))
+(define %eris-zstd-compression
+ (compression 'zstd 19))
+
(define (default-compression type)
(compression type 3))
@@ -324,7 +328,8 @@ (define* (store-item->recutils store-item
(define* (narinfo-string store store-path
#:key (compressions (list %no-compression))
- (nar-path "nar") (file-sizes '()))
+ (nar-path "nar") (file-sizes '())
+ eris-urn)
"Generate a narinfo key/value string for STORE-PATH; an exception is raised
if STORE-PATH is invalid. Produce a URL that corresponds to COMPRESSION. The
narinfo is signed with KEY. NAR-PATH specifies the prefix for nar URLs.
@@ -347,7 +352,7 @@ (define* (narinfo-string store store-path
StorePath: ~a
~{~a~}\
NarHash: sha256:~a
-NarSize: ~d
+NarSize: ~d~@[~%ERISFormat: application/x-nix-archive+zstd-19~%ERIS: ~a~]
References: ~a~%"
store-path
(map (lambda (compression)
@@ -359,7 +364,7 @@ (define* (narinfo-string store store-path
#:compression
compression)))
compressions)
- hash size references))
+ hash size eris-urn references))
;; Do not render a "Deriver" line if we are rendering info for a
;; derivation. Also do not render a "System" line that would be
;; expensive to compute and is currently unused.
@@ -632,6 +637,16 @@ (define (compressed-nar-size compression)
(and stat
(cons compression (stat:size stat)))))
+ (define (eris-encode-nar compressions)
+ (and (member %eris-zstd-compression compressions)
+ (let* ((nar (nar-cache-file cache item
+ #:compression %eris-zstd-compression))
+ (stat (stat nar #f)))
+ (and stat
+ (call-with-input-file nar
+ (cut eris-encode->string <>
+ #:block-size %eris-block-size-large))))))
+
(let ((compression (actual-compressions item compressions)))
(for-each (cut compress-nar cache item <>) compressions)
@@ -650,7 +665,8 @@ (define (compressed-nar-size compression)
(display (narinfo-string store item
#:nar-path nar-path
#:compressions compressions
- #:file-sizes sizes)
+ #:file-sizes sizes
+ #:eris-urn (eris-encode-nar compression))
port)))
;; Make the cached narinfo world-readable, contrary to what
--
2.34.0
P
P
pukkamustard wrote on 25 Jan 2022 11:21
[RFC PATCH v2 3/5] Add (guix eris).
(address . 52555@debbugs.gnu.org)
20220125192201.7582-4-pukkamustard@posteo.net
* guix/ipfs.scm: New file.
* Makefile.am (MODULES): Add it.
---
Makefile.am | 1 +
guix/eris.scm | 55 +++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 56 insertions(+)
create mode 100644 guix/eris.scm

Toggle diff (75 lines)
diff --git a/Makefile.am b/Makefile.am
index a10aeb817b..7219386361 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -131,6 +131,7 @@ MODULES = \
guix/cve.scm \
guix/workers.scm \
guix/ipfs.scm \
+ guix/eris.scm \
guix/build-system.scm \
guix/build-system/android-ndk.scm \
guix/build-system/ant.scm \
diff --git a/guix/eris.scm b/guix/eris.scm
new file mode 100644
index 0000000000..163bbe05ac
--- /dev/null
+++ b/guix/eris.scm
@@ -0,0 +1,55 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 pukkamustard <pukkamustard@posteo.net>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix eris)
+ #:use-module (eris)
+ #:use-module (eris blocks ipfs)
+ #:use-module (web client)
+ #:use-module (web response)
+ #:use-module (srfi srfi-71)
+
+ #:export (guix-eris-block-reducer))
+
+(define (ipfs-daemon-alive?)
+ "Attempt to connect to the IPFS daemon. Returns #t if the daemon is alive
+and #f else."
+ (with-exception-handler
+ (const #f)
+ (lambda _
+ (let ((response _
+ (http-post (string-append (%ipfs-base-url)
+ "/api/v0/version"))))
+ (equal? 200 (response-code response))))
+ #:unwind? #t))
+
+(define guix-eris-block-reducer
+ (case-lambda
+
+ ;; Check if IPFS Daemon is running.
+ (() (if (ipfs-daemon-alive?)
+ (eris-blocks-ipfs-reducer)
+ #f))
+
+ ;; Completion. Nothing to do.
+ ((_) #t)
+
+ ((ipfs ref-block)
+ ;; If IPFS has been initialized store block there
+ (if ipfs
+ (eris-blocks-ipfs-reducer ipfs ref-block)
+ ipfs))))
--
2.34.0
P
P
pukkamustard wrote on 25 Jan 2022 11:22
[RFC PATCH v2 4/5] publish: Add support for storing ERIS encoded blocks to IPFS.
(address . 52555@debbugs.gnu.org)
20220125192201.7582-5-pukkamustard@posteo.net
* guix/scripts/publish.scm (bake-narinfo+nar): Use guix-eris-block-reducer
from (guix eris).
---
guix/scripts/publish.scm | 12 ++++++++++--
1 file changed, 10 insertions(+), 2 deletions(-)

Toggle diff (39 lines)
diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm
index 9c83f5183d..556107ab7d 100644
--- a/guix/scripts/publish.scm
+++ b/guix/scripts/publish.scm
@@ -41,6 +41,7 @@ (define-module (guix scripts publish)
#:use-module (srfi srfi-26)
#:use-module (srfi srfi-34)
#:use-module (srfi srfi-37)
+ #:use-module (srfi srfi-71)
#:use-module (web http)
#:use-module (web request)
#:use-module (web response)
@@ -58,6 +59,7 @@ (define-module (guix scripts publish)
#:use-module (guix workers)
#:use-module (guix store)
#:use-module ((guix serialization) #:select (write-file))
+ #:use-module (guix eris)
#:use-module (eris)
#:use-module (zlib)
#:autoload (lzlib) (call-with-lzip-output-port
@@ -644,8 +646,14 @@ (define (eris-encode-nar compressions)
(stat (stat nar #f)))
(and stat
(call-with-input-file nar
- (cut eris-encode->string <>
- #:block-size %eris-block-size-large))))))
+ (lambda (port)
+ (let ((eris-urn _
+ (eris-encode port
+ #:block-size
+ %eris-block-size-large
+ #:block-reducer
+ guix-eris-block-reducer)))
+ (uri->string eris-urn))))))))
(let ((compression (actual-compressions item compressions)))
--
2.34.0
P
P
pukkamustard wrote on 25 Jan 2022 11:21
[RFC PATCH v2 1/5] WIP: gnu: guile-eris: Update to unreleased git version.
(address . 52555@debbugs.gnu.org)
20220125192201.7582-2-pukkamustard@posteo.net
* gnu/packages/guile-xyz.schm (guile-eris): Update to unreleased git version.
[source]: Update source URI.
[propagated-inputs]: Add guile-json-4 and guile-gdbm-ffi.
---
gnu/packages/guile-xyz.scm | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)

Toggle diff (35 lines)
diff --git a/gnu/packages/guile-xyz.scm b/gnu/packages/guile-xyz.scm
index e2cf793acc..66ac486a74 100644
--- a/gnu/packages/guile-xyz.scm
+++ b/gnu/packages/guile-xyz.scm
@@ -4374,15 +4374,15 @@ (define-public guile-sodium
(define-public guile-eris
(package
(name "guile-eris")
- (version "0.2.0")
+ (version "bcbbcbc88f3ec1f2fafcd034ce5b620516bff105")
(source
(origin
(method git-fetch)
(uri (git-reference
- (url "https://inqlab.net/git/eris.git")
- (commit (string-append "v" version))))
+ (url "https://codeberg.org/eris/guile-eris")
+ (commit version)))
(file-name (git-file-name name version))
- (sha256 (base32 "1ijglmwkdy1l87gj429qfjis0v8b1zlxhbyfhx5za8664h68nqka"))))
+ (sha256 (base32 "17v3h2hqx080739rl57nfradp5vlmy24fgqdxry1zal5z9d3i8sr"))))
(build-system gnu-build-system)
(arguments '())
(native-inputs
@@ -4394,7 +4394,7 @@ (define-public guile-eris
guile-srfi-180))
(inputs (list guile-3.0))
(propagated-inputs
- (list guile-sodium))
+ (list guile-sodium guile-json-4 guile-gdbm-ffi))
(synopsis "Guile implementation of the Encoding for Robust Immutable Storage (ERIS)")
(description
"Guile-ERIS is the reference implementation of the Encoding for Robust
--
2.34.0
P
P
pukkamustard wrote on 25 Jan 2022 11:22
[RFC PATCH v2 5/5] substitute: Fetch substitutes using ERIS.
(address . 52555@debbugs.gnu.org)
20220125192201.7582-6-pukkamustard@posteo.net
* guix/scripts/substitute.scm (process-substitution): Fetch substitutes using ERIS.
* guix/eris.scm (guix-eris-block-ref): New procedure.
---
guix/eris.scm | 7 ++++++-
guix/scripts/substitute.scm | 21 ++++++++++++++++-----
2 files changed, 22 insertions(+), 6 deletions(-)

Toggle diff (70 lines)
diff --git a/guix/eris.scm b/guix/eris.scm
index 163bbe05ac..0999564c1f 100644
--- a/guix/eris.scm
+++ b/guix/eris.scm
@@ -23,7 +23,8 @@ (define-module (guix eris)
#:use-module (web response)
#:use-module (srfi srfi-71)
- #:export (guix-eris-block-reducer))
+ #:export (guix-eris-block-reducer
+ guix-eris-block-ref))
(define (ipfs-daemon-alive?)
"Attempt to connect to the IPFS daemon. Returns #t if the daemon is alive
@@ -53,3 +54,7 @@ (define guix-eris-block-reducer
(if ipfs
(eris-blocks-ipfs-reducer ipfs ref-block)
ipfs))))
+
+(define (guix-eris-block-ref ref)
+ "Dereference a block for decoding ERIS content"
+ (eris-blocks-ipfs-ref ref))
diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
index 908a8334a8..852264976e 100755
--- a/guix/scripts/substitute.scm
+++ b/guix/scripts/substitute.scm
@@ -62,6 +62,8 @@ (define-module (guix scripts substitute)
#:use-module (srfi srfi-35)
#:use-module (web uri)
#:use-module (guix http-client)
+ #:use-module (guix eris)
+ #:use-module (eris)
#:export (%allow-unauthenticated-substitutes?
%reply-file-descriptor
@@ -486,18 +488,27 @@ (define (fetch uri)
#:port port
#:keep-alive? #t
#:buffered? #f)))))
+
(else
- (leave (G_ "unsupported substitute URI scheme: ~a~%")
- (uri->string uri)))))
+ (if (and (eris-read-capability? uri))
+ (values (eris-decode->port uri
+ #:block-ref
+ guix-eris-block-ref) #f)
+ (leave (G_ "unsupported substitute URI scheme: ~a~%")
+ (uri->string uri))))))
+
+ (define* (best-uri narinfo #:key (eris? #f))
+ (if (and eris? (narinfo-eris-urn narinfo))
+ (values (narinfo-eris-urn narinfo) "zstd" #f)
+ (narinfo-best-uri narinfo #:fast-decompression?
+ %prefer-fast-decompression?)))
(unless narinfo
(leave (G_ "no valid substitute for '~a'~%")
store-item))
(let-values (((uri compression file-size)
- (narinfo-best-uri narinfo
- #:fast-decompression?
- %prefer-fast-decompression?)))
+ (best-uri narinfo #:eris? #t)))
(unless print-build-trace?
(format (current-error-port)
(G_ "Downloading ~a...~%") (uri->string uri)))
--
2.34.0
M
M
Maxime Devos wrote on 29 Jan 2022 13:00
Re: [bug#52555] [RFC PATCH v2 0/5] Decentralized substitute distribution with ERIS
(address . ~pukkamustard/eris@lists.sr.ht)
cc7ec17cded7fda6689d23a163df2ba72058a8aa.camel@telenet.be
pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]:
Toggle quote (18 lines)
> ** Authorize local substitutes
>
> We will be running a local substitute server so we need to add the
> local
> signing key to the list of authorized keys. In the system
> configurations:
>
> #+BEGIN_SRC scheme
> (modify-services %base-services
> (guix-service-type config => [...]))
> #+END_SRC
> [...]
> ** Start the IPFS daemon
>
> #+BEGIN_SRC shell
> guix shell go-ipfs -- ipfs daemon
> #+END_SRC

There's an ipfs-service-type nowadays, so starting the daemon manually
isn't required (if using Guix System).

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfWq7xccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7kWHAQD8B6ufMY7MntcR+Q44j+roW/MG
/b3y8wn1dPnDxb27cQEA+fgcZyuRGc458oFJKqyMP2xsoTGUYQmMC1Vc2aiYRwA=
=/y6d
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 29 Jan 2022 13:08
(address . ~pukkamustard/eris@lists.sr.ht)
37651a21b0699b78b0282284b7d7322cb1a8c320.camel@telenet.be
pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]:
Toggle quote (3 lines)
> I will be looking into the HTTP fallback and also using BitTorrent and GNUNet
> as transports.

I have been writing a (Guile) Scheme port of GNUnet's client libraries
supported, but I'm working on DHT. DHT search/put already works to a
degree (see examples/web.scm), but there are plenty of sharp edges
(see TODOs about disconnecting, reconnecting and stopping fibers,
and see guix.scm for Guile bugs that are patched out and extra guile-
fibers features).

Tests are being written a edge cases will be addressed.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfWs1BccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7iUHAP9GaXu5iS0pWHmlWrCS/xPjChSf
5aDwgq8ex1CbWgGslwEA2yz95+6uNajWeX9xbu6TV6b+AcAPwVUC4p2HgSvxBwA=
=tbfK
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 29 Jan 2022 13:09
Re: [bug#52555] [RFC PATCH v2 2/5] publish: Add ERIS URN to narinfo
(address . ~pukkamustard/eris@lists.sr.ht)
d0882404dc5153c7cf75846bb5cc70cb90138b78.camel@telenet.be
pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]:
Toggle quote (5 lines)
> +dnl Check for Guile-eris.
> +GUILE_MODULE_AVAILABLE([have_guile_eris], [(eris)])
> +AM_CONDITIONAL([HAVE_GUILE_ERIS],
> +  [test "x$have_guile_eris" = "xyes"])

This could to be documented in (guix)Requirements.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfWtJxccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7qqUAQCEB1li7ZBR4r0xjFIQwDM+1Z2d
DmdnE6ogz16oO4WWjAD9HT5dzf21xMKPiGXXFNsOLwJKtc+2i4ZLOTTOvlCnTgk=
=ZFnB
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 29 Jan 2022 13:15
(address . ~pukkamustard/eris@lists.sr.ht)
46558e7148b91f12cdaa92f32adb91bbcb3080bd.camel@telenet.be
pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]:
Toggle quote (10 lines)
> +  (define (eris-encode-nar compressions)
> +    (and (member %eris-zstd-compression compressions)
> +         (let* ((nar (nar-cache-file cache item
> +                                     #:compression %eris-zstd-compression))
> +                (stat (stat nar #f)))
> +           (and stat
> +                (call-with-input-file nar
> +                  (cut eris-encode->string <>
> +                       #:block-size %eris-block-size-large))))))

Why are exceptions turned into #f (in (stat nar #f))?
Should this be done for all I/O errors, including, say, EOVERFLOW,
ENOMEM or ENAMETOOLONG, or only for ENOENT?

Is a race condition possible here? If so, maybe consider doing
something like

(catch 'system-error
(lambda () (call-with-input-file ...))
(lambda exception
(and it-is-a-ENOENT
(apply throw exception))))

to avoid it?

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfWudxccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7ka/AP0QTYuG8PJyBF3QVw+5ZnP0029l
Ax9ryDs/xHGvJ9rfgwEA40JgFHgUhz0M23qld+IUbNztFjshJjFr+Frzf69aow8=
=Imy3
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 29 Jan 2022 13:23
Re: [bug#52555] [RFC PATCH v2 3/5] Add (guix eris).
(address . ~pukkamustard/eris@lists.sr.ht)
0b977ae15a49b051c04922d234149a8e4762404f.camel@telenet.be
pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]:
Toggle quote (12 lines)
> +(define (ipfs-daemon-alive?)
> +  "Attempt to connect to the IPFS daemon. Returns #t if the daemon is alive
> +and #f else."
> +  (with-exception-handler
> +      (const #f)
> +    (lambda _
> +      (let ((response _
> +                      (http-post (string-append (%ipfs-base-url)
> +                                                "/api/v0/version"))))
> +        (equal? 200 (response-code response))))
> +    #:unwind? #t))

This should preferably only be catching exceptions indicating that
the daemon is down (exceptions indicating 404s, or system-errors
indicating network errors, ...).

Toggle quote (18 lines)
> +
> +(define guix-eris-block-reducer
> +  (case-lambda
> +
> +    ;; Check if IPFS Daemon is running.
> +    (() (if (ipfs-daemon-alive?)
> +            (eris-blocks-ipfs-reducer)
> +            #f))
> +
> +    ;; Completion. Nothing to do.
> +    ((_) #t)
> +
> +    ((ipfs ref-block)
> +     ;; If IPFS has been initialized store block there
> +     (if ipfs
> +         (eris-blocks-ipfs-reducer ipfs ref-block)
> +         ipfs))))

This (ipfs-daemon-alive?) seems racy, although it's probably not.
Can we do

(define guix-eris-block-reducer
(case-lambda
(() (guard (c (oops-it-fails-because-the-daemon-cannot-be-
contacted? c)
#false)
(eris-block-ipfs-reducer))
[...]))

instead? (I don't think this will work as-is, because from the name and
thunkiness, it would appear that eris-block-ipfs-reducer returns a
procedure ...

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfWwRBccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7nxXAP9CAcbVz3Isk6TX52V5SGhjdExt
PLcXaJq7dAgpB44VfwEAqjgtVq1V7j2KDmmrZ9Om8e3E6+ys8okKmYBIMJia/QM=
=8qD5
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 29 Jan 2022 13:24
(address . ~pukkamustard/eris@lists.sr.ht)
96a3137b8b746fcad9ef1eb75e1abfda5d01bfdb.camel@telenet.be
pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]:
Toggle quote (3 lines)
> +  #:use-module (eris)
> +  #:use-module (eris blocks ipfs)

guile-eris is an optional dependency, so this needs to be autoloaded.
Or guix/eris.scm must only be compiled when guile-eris is available.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfWwhRccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7tniAP9kASEY2aOmsyLv2tvwFXDDOimF
ogDnNWhDkbmvosQthAD/TR5q03yjYDu3Nv/gitR+b5d0ZeZS9eGYT+tQ4PI66A8=
=RBt/
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 29 Jan 2022 13:28
Re: [bug#52555] [RFC PATCH v2 4/5] publish: Add support for storing ERIS encoded blocks to IPFS.
(address . ~pukkamustard/eris@lists.sr.ht)
c6841903c177f934c7fc9d42a270907dbba59554.camel@telenet.be
pukkamustard schreef op di 25-01-2022 om 19:22 [+0000]:
Toggle quote (8 lines)
> +                    (let ((eris-urn _
> +                                    (eris-encode port
> +                                                 #:block-size
> +                                                 %eris-block-size-large
> +                                                 #:block-reducer
> +                                                 guix-eris-block-reducer)))
> + (uri->string eris-urn))))))))

IIUC (and quite probably I don't, because I've only being reading
things cursorly), eris-encode returns #false when the IPFS daemon is
down (because then guix-eris-block-reducer returns #false).

In that case, (uri->string eris-urn) = (uri->string #false) would
throw an exception:

scheme@(guile-user)> ((@ (web uri) uri->string) #false)
ice-9/boot-9.scm:1669:16: In procedure raise-exception:
In procedure struct-vtable: Wrong type argument in position 1
(expecting struct): #f

Entering a new prompt. Type `,bt' for a backtrace or `,q' to continue.
scheme@(guile-user) [1]> ,bt
In web/uri.scm:
336:17 1 (uri->string #f #:include-fragment? _)
In ice-9/boot-9.scm:
1669:16 0 (raise-exception _ #:continuable? _)

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfWxaxccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7hHyAP9u5jfIW9GOQmUDoxjLJrBrB3cA
/zhz/MksltEGl16dtwD+M9uQeCUfT4Wqig+4eAxGYD952sqhmdmp8/9iI8W8ngM=
=b/X2
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 29 Jan 2022 13:29
Re: [bug#52555] [RFC PATCH v2 5/5] substitute: Fetch substitutes using ERIS.
(address . ~pukkamustard/eris@lists.sr.ht)
00bb85fa4fb62498b354ecb78d248df5209b6f29.camel@telenet.be
pukkamustard schreef op di 25-01-2022 om 19:22 [+0000]:
Toggle quote (4 lines)
> +(define (guix-eris-block-ref ref)
> +  "Dereference a block for decoding ERIS content"
> +  (eris-blocks-ipfs-ref ref))

'guix-eris-block-ref' just calls 'eris-blocks-ipfs-ref',
so I'm not seeing the point of this procedure.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfWxzhccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7uMzAP4u7P60iQCm0ohzXL8c0ZG6Zl0a
66kylFJ4J3x2BE+6UgEAgsWqyNKl0p3k/gg68VcHfsVnJMd0a+zy7V4houhF7As=
=iiau
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 29 Jan 2022 13:33
(address . ~pukkamustard/eris@lists.sr.ht)
fbe6d8d556a796b580d0263cac87369bdc62b0d7.camel@telenet.be
pukkamustard schreef op di 25-01-2022 om 19:22 [+0000]:
Toggle quote (1 lines)
> +
Superfluous new empty line?

Toggle quote (5 lines)
>        (else
> -       (leave (G_ "unsupported substitute URI scheme: ~a~%")
> -              (uri->string uri)))))
> +       (if (and (eris-read-capability? uri))

guile-eris (which has the eris-read-capability? procedure) is an
optional dependency, so you have to check here is Guix was compiled
with guile-eris.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfWyxxccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7v7+AQCV4ylW2AvzZfYFWUufqDp6Fm2+
neoJm3Ip51Rj9gX/EgD9GWPVNtgpSMGfs3nbd9Vg2785Z3K0SsseF3FGI8o/sgA=
=1GlX
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 29 Jan 2022 13:38
(address . ~pukkamustard/eris@lists.sr.ht)
0b0401a711d2196400b05a7f98ef5b757c99bf4b.camel@telenet.be
pukkamustard schreef op di 25-01-2022 om 19:22 [+0000]:
Toggle quote (6 lines)
> +  (define* (best-uri narinfo #:key (eris? #f))
> +    (if (and eris? (narinfo-eris-urn narinfo))
> +        (values (narinfo-eris-urn narinfo) "zstd" #f)
> +        (narinfo-best-uri narinfo #:fast-decompression?
> +                          %prefer-fast-decompression?)))

When Guix is compiled without guile-eris support,
'(and eris? (narinfo-eris-urn narinfo))' is the worst, not the best.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfWz0xccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7hfIAP4gQtcdPHk1ATJQ6Y2i1FdWxKRD
Hzwh+VxRPOOqAoleBQD+N+YpX65UT4K6zjdj7rmVzn/44mwqm+RRlGa2Kb0NeQg=
=9kGm
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 29 Jan 2022 13:40
(address . ~pukkamustard/eris@lists.sr.ht)
d6bb147c902c0067d923321daf6d8db4a06fa537.camel@telenet.be
pukkamustard schreef op di 25-01-2022 om 19:22 [+0000]:
Toggle quote (6 lines)
> +  (define* (best-uri narinfo #:key (eris? #f))
> +    (if (and eris? (narinfo-eris-urn narinfo))
> +        (values (narinfo-eris-urn narinfo) "zstd" #f)
> +        (narinfo-best-uri narinfo #:fast-decompression?
> +                          %prefer-fast-decompression?)))

Why is ERIS the best here? Fast download speeds, decentralisation,
less network I/O, less heat production, more pronouncable than HTTPS?
I would add a comment here.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfW0ThccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7rl8AP9JdJvZVUngfsqUuONq1psXpouT
FB5+4GT1KNZ8p15qwgEA0BH/56Hi+ezFF4RuMfpqYZkY/yF7KtM6j0a0va/qNgo=
=48HT
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 29 Jan 2022 13:40
(address . ~pukkamustard/eris@lists.sr.ht)
e0156ce82b5e9a990d05c0b7bfb6626af5ce171a.camel@telenet.be
Maxime Devos schreef op za 29-01-2022 om 22:38 [+0100]:
Toggle quote (10 lines)
> pukkamustard schreef op di 25-01-2022 om 19:22 [+0000]:
> > +  (define* (best-uri narinfo #:key (eris? #f))
> > +    (if (and eris? (narinfo-eris-urn narinfo))
> > +        (values (narinfo-eris-urn narinfo) "zstd" #f)
> > +        (narinfo-best-uri narinfo #:fast-decompression?
> > +                          %prefer-fast-decompression?)))
>
> When Guix is compiled without guile-eris support,
> '(and eris? (narinfo-eris-urn narinfo))' is the worst, not the best.

Nevermind, that's what (eris? #f) is for, I presume?

Greetings,
Maxime.
M
M
Maxime Devos wrote on 29 Jan 2022 13:52
Re: [bug#52555] [RFC PATCH v2 0/5] Decentralized substitute distribution with ERIS
(address . ~pukkamustard/eris@lists.sr.ht)
73b50ffdca94407ef9fd7ef4875985a3b1c3c568.camel@telenet.be
Hi,

Is it possible for the following situation to happen?
If so, why not?

1. server A is authentic
2. server M is malicious, it tries to trick the client into
installing an incorrect substitute
3. (key of) server A is authorised
4. (key of) server M is _not_ authorised
5. server A and M are both in substitute-urls
6. server A only serves ‘classical’ substitutes, server B also serves
via ERIS+ipfs
7. Both A and M set the same FileHash, References, etc. in the
narinfo
8. However, M set an ERIS URN pointing to a backdoored substitute.
9. The client trusts A, and A and B have the same FileHash etc.,
so the client considers the narinfo of B to be authentic
because it has the same FileHash.
10. The client prefers ERIS above HTTP(S), so it downloads via M.
11. The client now installed a backdoored substitute!

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfW3KhccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7hJxAQDlxH9zz6F8LpkiMv07Hcyt35CG
YFb2CeCw2dbFO7qXNwD9HeDCIHVHzYQD/EvcGcYhQIocR7cUvrSnvPyFVHFErQ8=
=4RmQ
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 30 Jan 2022 03:46
(address . ~pukkamustard/eris@lists.sr.ht)
ba490f009be9908063609f71800a2f00b4b1c1ae.camel@telenet.be
pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]:
Toggle quote (6 lines)
> I have only tested this for fairly small packages (up to a few MB).
>
> One issue with IPFS might be that we have to create a new HTTP connection to
> the IPFS daemon for every single block (32KiB). The IPFS daemon does not seem
> to support HTTP connection re-use

daemon supports connection reuse according to some people and doesn't
according to other people.

Toggle quote (2 lines)
> and neither does the Guile (web client).

Guix supports connection reuse, see 'call-with-cached-connection'
in (guix scripts substitute).

Toggle quote (2 lines)
> I fear this might become a performance issue.

IIUC, the performance problem primarily lies in the round-tripping
between the client and the server. If the client and the server are on
the same machine, then this round trip time is presumably small
compared to, say, localhost contacting ci.guix.gnu.org.

Still, connection reuse would be nice.

Toggle quote (6 lines)
> It seems possible to use IPFS more directly by exposing the Go code as a
> C library and then using that with the Guile FFI [1]. This is however a bit
> complicated and adds a lot of dependencies. In particular, this should not become
> a dependency of Guix itself. The performance of IPFS itself also needs to be
> evaluated, maybe the IPFS HTTP API will not be the bottle-neck.

Security-wise, libipfs doesn't seem great: libipfs starts the IPFS
daemon inside the process and guix/scripts/substitute.scm is run
as root.

Toggle quote (4 lines)
> As mentioned in previous mail a simple HTTP transport for blocks would be a
> good fallback. This would allow users to get missing blocks (things that
> somehow got dropped from IPFS) directly from a substitute server. [...]

Seems a good idea to me -- DHTs can be unreliable. I presume this will
be implemented with some kind of timeout: if no block is received
within N seconds, fallback to HTTP?

Also, don't forget to insert this missing block back into
IPFS/GNUnet/BitTorrent/..., otherwise less and less blocks will be
available until nothing is available anymore.

Toggle quote (5 lines)
> In any case, it would be necessary for the substitute server to store encoded
> blocks of the NAR. For this I think it makes sense to use a small database. We
> have bindings to use ERIS with GDBM [2]. It might also make sense to use
> SQLite, especially if there are other use-cases for such a database.

Wouldn't this be a huge database? IIRC, according to logs.guix.gnu.org
the size of the nars of the substitute servers are somewhere in the
200G-2T range or something like that.

To reduce the size of the database, perhaps you could let the database
be a mapping from block ids to the name of the nar + the position in
the nar, and encode the block on-demand?

The database doesn't seem necessary, the substitute server could have
some end-point

/publish-this-nar-again-into-IPFS/name-of-the-nar

which, when contacted, inserts the nar again into IPFS. Then when a
block was unavailable, the client contacts this end-point and retries.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfZ6jxccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7rANAP9328I61zxzX3OGMvtg/yulC2fN
hA2FYGhhmzeZUgJfWAD+Nz3A9TmanG1T0/QvW6bgstsy7kILYqCvXo7xF8onkAM=
=psyK
-----END PGP SIGNATURE-----


P
P
pukkamustard wrote on 2 Feb 2022 01:50
(name . Maxime Devos)(address . maximedevos@telenet.be)
86a6f9iofg.fsf@posteo.net
Hi Maxime,

Maxime Devos <maximedevos@telenet.be> writes:

Toggle quote (3 lines)
> There's an ipfs-service-type nowadays, so starting the daemon manually
> isn't required (if using Guix System).

Good point. Starting the daemon manually is only necessary if you don't
use the service. I don't use the IPFS service.

-pukkamustard
P
P
pukkamustard wrote on 2 Feb 2022 01:56
(name . Maxime Devos)(address . maximedevos@telenet.be)
865ypxinr8.fsf@posteo.net
Maxime Devos <maximedevos@telenet.be> writes:

Toggle quote (13 lines)
> [[PGP Signed Part:Undecided]]
> pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]:
>> I will be looking into the HTTP fallback and also using BitTorrent and GNUNet
>> as transports.
>
> I have been writing a (Guile) Scheme port of GNUnet's client libraries
> (https://git.gnunet.org/gnunet-scheme.git/). Currently only NSE is
> supported, but I'm working on DHT. DHT search/put already works to a
> degree (see examples/web.scm), but there are plenty of sharp edges
> (see TODOs about disconnecting, reconnecting and stopping fibers,
> and see guix.scm for Guile bugs that are patched out and extra guile-
> fibers features).

Very interesting! I have been following your work on that a bit.

From what I understand gnunet-scheme interacts with the GNUNet services
and sends messages to the various GNUNet services. Is that correct?

Have you considered implementing the GNUNet protocols themeselves in
Guile? I.e. instead of connecting with the GNUNet services and sending
messages, implement R5N completely in Guile. IMHO this would be very
nice as one could use GNUNet protocols completely in Guile and not rely
on the GNUNet C code.

I believe this is somewhat the direction being taken with the GNUNet Go
implementation (https://github.com/bfix/gnunet-go)and also in line with
recent efforts to specify the individual GNUNet components and protocols
more independantly of one another (e.g. R5N is specified to work over IP

-pukkamustard
P
P
pukkamustard wrote on 2 Feb 2022 02:11
Re: [bug#52555] [RFC PATCH v2 5/5] substitute: Fetch substitutes using ERIS.
(name . Maxime Devos)(address . maximedevos@telenet.be)
861r0linh1.fsf@posteo.net
Maxime Devos <maximedevos@telenet.be> writes:

Toggle quote (9 lines)
> [[PGP Signed Part:Undecided]]
> pukkamustard schreef op di 25-01-2022 om 19:22 [+0000]:
>> +(define (guix-eris-block-ref ref)
>> +  "Dereference a block for decoding ERIS content"
>> +  (eris-blocks-ipfs-ref ref))
>
> 'guix-eris-block-ref' just calls 'eris-blocks-ipfs-ref',
> so I'm not seeing the point of this procedure.

Yes, currently it is an unnecessary level of abstraction.

The idea is that when there are multiple backends/transports they are
multiplexed here. E.g. guix-eris-block-ref would attempt to use
HTTP/IPFS or whatever to get the block. Whatever calls
guix-eris-block-ref does not need to know from where the blocks come.

I hope to make this more clear in a V3 that will add HTTP transport.

-pukkamustard
P
P
pukkamustard wrote on 2 Feb 2022 02:16
Re: [bug#52555] [RFC PATCH v2 2/5] publish: Add ERIS URN to narinfo
(name . Maxime Devos)(address . maximedevos@telenet.be)
86wnidh8ht.fsf@posteo.net
Maxime Devos <maximedevos@telenet.be> writes:

Toggle quote (26 lines)
> pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]:
>> +  (define (eris-encode-nar compressions)
>> +    (and (member %eris-zstd-compression compressions)
>> +         (let* ((nar (nar-cache-file cache item
>> +                                     #:compression %eris-zstd-compression))
>> +                (stat (stat nar #f)))
>> +           (and stat
>> +                (call-with-input-file nar
>> +                  (cut eris-encode->string <>
>> +                       #:block-size %eris-block-size-large))))))
>
> Why are exceptions turned into #f (in (stat nar #f))?
> Should this be done for all I/O errors, including, say, EOVERFLOW,
> ENOMEM or ENAMETOOLONG, or only for ENOENT?
>
> Is a race condition possible here? If so, maybe consider doing
> something like
>
> (catch 'system-error
> (lambda () (call-with-input-file ...))
> (lambda exception
> (and it-is-a-ENOENT
> (apply throw exception))))
>
> to avoid it?

A valid question. But (stat nar #f) is not something I introduced. It is
already in guix/scripts/publish.scm like that.

To me turning all exceptions to #f makes sense. Here we only want to
know if the file is readable. As the NAR is baked in the background by
another thread the case where the compressed NAR does exist yet will
happen. In that case we don't worry, we just don't publish the
`FileSize` and `ERIS` fields in the narinfo.

-pukkamustard
P
P
pukkamustard wrote on 2 Feb 2022 02:28
Re: [bug#52555] [RFC PATCH v2 3/5] Add (guix eris).
(name . Maxime Devos)(address . maximedevos@telenet.be)
86sft1h7wh.fsf@posteo.net
Maxime Devos <maximedevos@telenet.be> writes:

Toggle quote (17 lines)
> pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]:
>> +(define (ipfs-daemon-alive?)
>> +  "Attempt to connect to the IPFS daemon. Returns #t if the daemon is alive
>> +and #f else."
>> +  (with-exception-handler
>> +      (const #f)
>> +    (lambda _
>> +      (let ((response _
>> +                      (http-post (string-append (%ipfs-base-url)
>> +                                                "/api/v0/version"))))
>> +        (equal? 200 (response-code response))))
>> +    #:unwind? #t))
>
> This should preferably only be catching exceptions indicating that
> the daemon is down (exceptions indicating 404s, or system-errors
> indicating network errors, ...).

Yes, I guess it could be checked a bit finer. But at the end if an
exception happens then the IPFS daemon is probably not reachable, right?
If we don't care about the reason why it is not reachable then why
bother with catching finer grained exceptions?

Toggle quote (33 lines)
>> +
>> +(define guix-eris-block-reducer
>> +  (case-lambda
>> +
>> +    ;; Check if IPFS Daemon is running.
>> +    (() (if (ipfs-daemon-alive?)
>> +            (eris-blocks-ipfs-reducer)
>> +            #f))
>> +
>> +    ;; Completion. Nothing to do.
>> +    ((_) #t)
>> +
>> +    ((ipfs ref-block)
>> +     ;; If IPFS has been initialized store block there
>> +     (if ipfs
>> +         (eris-blocks-ipfs-reducer ipfs ref-block)
>> +         ipfs))))
>
> This (ipfs-daemon-alive?) seems racy, although it's probably not.
> Can we do
>
> (define guix-eris-block-reducer
> (case-lambda
> (() (guard (c (oops-it-fails-because-the-daemon-cannot-be-
> contacted? c)
> #false)
> (eris-block-ipfs-reducer))
> [...]))
>
> instead? (I don't think this will work as-is, because from the name and
> thunkiness, it would appear that eris-block-ipfs-reducer returns a
> procedure ...

Yes, eris-block-ipfs-reducer returns and SRFI-171 reducer. This is a
3-arity procedure that is either initialized, called with a block to
reduce and finalized.

The #f that the initialization case returns (0-arity call) is the state
of the reducer. In the block reducing case (2-ary call) the state (the
ipfs variable) is checked if ipfs is alive. If not the blocks are just
forgotten.

So guix-eris-block-reducer always returns a SRF-171 reducer, regardless
of if IPFS is alive or not. This is important as the ERIS URN can still
be computed without the IPFS daemon running.

-pukkamustard
P
P
pukkamustard wrote on 2 Feb 2022 02:24
Re: [bug#52555] [RFC PATCH v2 4/5] publish: Add support for storing ERIS encoded blocks to IPFS.
(name . Maxime Devos)(address . maximedevos@telenet.be)
86o83ph7uz.fsf@posteo.net
Maxime Devos <maximedevos@telenet.be> writes:
Toggle quote (13 lines)
> [[PGP Signed Part:Undecided]]
> pukkamustard schreef op di 25-01-2022 om 19:22 [+0000]:
>> +                    (let ((eris-urn _
>> +                                    (eris-encode port
>> +                                                 #:block-size
>> +                                                 %eris-block-size-large
>> +                                                 #:block-reducer
>> +                                                 guix-eris-block-reducer)))
>> + (uri->string eris-urn))))))))
>
> IIUC (and quite probably I don't, because I've only being reading
> things cursorly), eris-encode returns #false when the IPFS daemon is
> down (because then guix-eris-block-reducer returns #false).
No, eris-encode will still return the ERIS URN. The blocks will just not
be stored in IPFS if the IPFS daemon is not running. See also my
response to the questions on guix-eris-block-reducer in (guix eris).
-pukkamustard
P
P
pukkamustard wrote on 2 Feb 2022 02:38
Re: [bug#52555] [RFC PATCH v2 5/5] substitute: Fetch substitutes using ERIS.
(name . Maxime Devos)(address . maximedevos@telenet.be)
86k0edh79j.fsf@posteo.net
Maxime Devos <maximedevos@telenet.be> writes:

Toggle quote (12 lines)
> [[PGP Signed Part:Undecided]]
> pukkamustard schreef op di 25-01-2022 om 19:22 [+0000]:
>> +  (define* (best-uri narinfo #:key (eris? #f))
>> +    (if (and eris? (narinfo-eris-urn narinfo))
>> +        (values (narinfo-eris-urn narinfo) "zstd" #f)
>> +        (narinfo-best-uri narinfo #:fast-decompression?
>> +                          %prefer-fast-decompression?)))
>
> Why is ERIS the best here? Fast download speeds, decentralisation,
> less network I/O, less heat production, more pronouncable than HTTPS?
> I would add a comment here.

Those are all possible reasons. I think we first need to do some
experiments to see if any of those claims can be justified.

In general, the logic for when to use ERIS transports needs more
thought.

For one, I think it should be user configurable. I can imagine that
certain users do not want to use decentralized substitutes at all. Users
should be able to deactivate the entire ERIS thing.

For the default, I personally think it would be ok to try and use
ERIS. The only thing we absolutely need is a clean fallback logic that
transparently falls back to getting the entire NAR by HTTP. Currently
such a fallback is not yet implemented.

-pukkamustard
M
M
Maxime Devos wrote on 2 Feb 2022 03:09
Re: [bug#52555] [RFC PATCH v2 0/5] Decentralized substitute distribution with ERIS
(name . pukkamustard)(address . pukkamustard@posteo.net)
3d9821d7eac6c81a5b1c8a5304a78ec733b2dba4.camel@telenet.be
pukkamustard schreef op wo 02-02-2022 om 09:56 [+0000]:
Toggle quote (20 lines)
> Maxime Devos <maximedevos@telenet.be> writes:
>
> > [[PGP Signed Part:Undecided]]
> > pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]:
> > > I will be looking into the HTTP fallback and also using BitTorrent and GNUNet
> > > as transports.
> >
> > I have been writing a (Guile) Scheme port of GNUnet's client libraries
> > (https://git.gnunet.org/gnunet-scheme.git/). Currently only NSE is
> > supported, but I'm working on DHT. DHT search/put already works to a
> > degree (see examples/web.scm), but there are plenty of sharp edges
> > (see TODOs about disconnecting, reconnecting and stopping fibers,
> > and see guix.scm for Guile bugs that are patched out and extra guile-
> > fibers features).
>
> Very interesting! I have been following your work on that a bit.
>
> From what I understand gnunet-scheme interacts with the GNUNet services
> and sends messages to the various GNUNet services. Is that correct?

Yes, it works like the C GNUnet client libraries, except it's in Guile
Scheme and a few different design decisions were made, e.g. w.r.t.
concurrency.

Toggle quote (4 lines)
> Have you considered implementing the GNUNet protocols themeselves in
> Guile? I.e. instead of connecting with the GNUNet services and sending
> messages, implement R5N completely in Guile.

I didn't, at least not _yet_. As-is, things are already complicated
enough and the client code seems a lot simpler than the service code.
Though perhaps in the future ...

E.g., for testing the DHT service, the test code effectively creates a
tiny, limited, in-memory DHT service (not communicating to any peers)
that's buggy in some respects (not yet committed, but will be in
tests/distributed-hash-table.scm).

Toggle quote (3 lines)
> IMHO this would be very nice as one could use GNUNet protocols completely
> in Guile and not rely on the GNUNet C code.

While it's not a priority, I'm not opposed to someday implementing the
services in Guile and testing whether they can communicate with C
peers.

However, keep in mind that GNUnet is supposed to be able to eventually
replace the TCP/IP stack, and running the DHT, NSE, NAT, FS, CADET,
TRANSPORT ... services in every web browser, in every mail client, in
all "guix substitute" and "guix perform-download" processes, etc. is
rather wasteful (memory-wise and CPU-wise), so I'd prefer this not to
be the _default_ option.

(I'm not sure if you were referring to that.)

Greetings,
Maxme.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfpmXRccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7mEeAPwOAn/O9DonokYiuo2Yq5+oLnHK
fIr8apEPKdTzy/O5sAD/TeShsCEUX0+DdtJF1xDXsYY4uRDd1ubLC/dowbvehA8=
=I6pm
-----END PGP SIGNATURE-----


P
P
pukkamustard wrote on 2 Feb 2022 02:51
(name . Maxime Devos)(address . maximedevos@telenet.be)
86fsp1h6ce.fsf@posteo.net
Maxime Devos <maximedevos@telenet.be> writes:

Toggle quote (12 lines)
> [[PGP Signed Part:Undecided]]
> pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]:
>> I have only tested this for fairly small packages (up to a few MB).
>>
>> One issue with IPFS might be that we have to create a new HTTP connection to
>> the IPFS daemon for every single block (32KiB). The IPFS daemon does not seem
>> to support HTTP connection re-use
>
> According to <https://github.com/ipfs/go-ipfs/issues/3767>, the IPFS
> daemon supports connection reuse according to some people and doesn't
> according to other people.

Hm, from what I understand connection re-use is something introduced in
HTTP/2 and go-ipfs does not do HTTP/2

Toggle quote (5 lines)
>> and neither does the Guile (web client).
>
> Guix supports connection reuse, see 'call-with-cached-connection'
> in (guix scripts substitute).

Ah ok. Cool!

Toggle quote (9 lines)
>> I fear this might become a performance issue.
>
> IIUC, the performance problem primarily lies in the round-tripping
> between the client and the server. If the client and the server are on
> the same machine, then this round trip time is presumably small
> compared to, say, localhost contacting ci.guix.gnu.org.
>
> Still, connection reuse would be nice.

Remains to be seen if this is a problem.

It is considerably more pronounced than with regular usage of IPFS as we
make a HTTP request to IPFS for every 32KiB block instead of for an
entire file (what most people do when using the IPFS daemon).

Toggle quote (10 lines)
>> It seems possible to use IPFS more directly by exposing the Go code as a
>> C library and then using that with the Guile FFI [1]. This is however a bit
>> complicated and adds a lot of dependencies. In particular, this should not become
>> a dependency of Guix itself. The performance of IPFS itself also needs to be
>> evaluated, maybe the IPFS HTTP API will not be the bottle-neck.
>
> Security-wise, libipfs doesn't seem great: libipfs starts the IPFS
> daemon inside the process and guix/scripts/substitute.scm is run
> as root.

I agree.

Toggle quote (8 lines)
>> As mentioned in previous mail a simple HTTP transport for blocks would be a
>> good fallback. This would allow users to get missing blocks (things that
>> somehow got dropped from IPFS) directly from a substitute server. [...]
>
> Seems a good idea to me -- DHTs can be unreliable. I presume this will
> be implemented with some kind of timeout: if no block is received
> within N seconds, fallback to HTTP?

Yes, exactly.

Toggle quote (4 lines)
> Also, don't forget to insert this missing block back into
> IPFS/GNUnet/BitTorrent/..., otherwise less and less blocks will be
> available until nothing is available anymore.

This might be a bit of a burden for users. As you mention the size of
such a database might become considerable.

Toggle quote (13 lines)
>> In any case, it would be necessary for the substitute server to store encoded
>> blocks of the NAR. For this I think it makes sense to use a small database. We
>> have bindings to use ERIS with GDBM [2]. It might also make sense to use
>> SQLite, especially if there are other use-cases for such a database.
>
> Wouldn't this be a huge database? IIRC, according to logs.guix.gnu.org
> the size of the nars of the substitute servers are somewhere in the
> 200G-2T range or something like that.
>
> To reduce the size of the database, perhaps you could let the database
> be a mapping from block ids to the name of the nar + the position in
> the nar, and encode the block on-demand?

Yes! I've also been thinking of this - a "in-file" block store. I think
this makes a lot of sense for Guix but also other things (e.g. sharing
your music collection).

Another problem with IPFS/GNUNet is that they have their own storage. So
even if are clever about storing blocks in Guix, IPFS and GNUNet will
have their own copy of the blocks on disk. I think it would be much
nicer if DHTs/transport layers don't do block storage but are provided
with a callback from where they can get stored blocks. I believe this is
what OpenDHT does

I think we should propose such a change to the GNUNet R5N specification

Toggle quote (8 lines)
> The database doesn't seem necessary, the substitute server could have
> some end-point
>
> /publish-this-nar-again-into-IPFS/name-of-the-nar
>
> which, when contacted, inserts the nar again into IPFS. Then when a
> block was unavailable, the client contacts this end-point and retries.

But for a HTTP block endpoint we would still need such a database/block
storage.

I think it is important that we do not rely on IPFS for block
storage. The decentralized block distribution should work even if the
IPFS daemon is not available.

-pukkamustard
P
P
pukkamustard wrote on 2 Feb 2022 03:10
(name . Maxime Devos)(address . maximedevos@telenet.be)
86bkzph5ux.fsf@posteo.net
Maxime Devos <maximedevos@telenet.be> writes:

Toggle quote (26 lines)
> [[PGP Signed Part:Undecided]]
> Hi,
>
> Is it possible for the following situation to happen?
> If so, why not?
>
> 1. server A is authentic
> 2. server M is malicious, it tries to trick the client into
> installing an incorrect substitute
> 3. (key of) server A is authorised
> 4. (key of) server M is _not_ authorised
> 5. server A and M are both in substitute-urls
> 6. server A only serves ‘classical’ substitutes, server B also serves
> via ERIS+ipfs
> 7. Both A and M set the same FileHash, References, etc. in the
> narinfo
> 8. However, M set an ERIS URN pointing to a backdoored substitute.
> 9. The client trusts A, and A and B have the same FileHash etc.,
> so the client considers the narinfo of B to be authentic
> because it has the same FileHash.
> 10. The client prefers ERIS above HTTP(S), so it downloads via M.
> 11. The client now installed a backdoored substitute!
>
> Greetings,
> Maxime.

No this should not work.

The ERIS URN is only used if the entire narinfo is signed with a
authorized signature. The FileHash is not used when getting substitutes
via ERIS (being able to decode ERIS content implies integrity).

The interesting case that would be allowed with ERIS is following:

1. Server A is authentic and its key is authorized.
2. Servers M1 to MN are potentially malicious and their keys are not
authorized.
3. Server A and servers M1 to MN are in the substitute-urls.
4. Client gets Narinfo from server A and uses the ERIS URN from there.
5. Client can get blocks simultaneously from Server A and servers M1 to
MN.
6. Client decodes content with the ERIS URN and can be sure that they
have the valid substitute.

So client only needs to trust A but can use M1-MN (simultaneously) for
fetching the content.

-pukkamustard
M
M
Maxime Devos wrote on 2 Feb 2022 03:27
(name . pukkamustard)(address . pukkamustard@posteo.net)
846716544b4424f02e383114ebcb52957b43dd4d.camel@telenet.be
pukkamustard schreef op wo 02-02-2022 om 10:51 [+0000]:
Toggle quote (19 lines)
> > The database doesn't seem necessary, the substitute server could
> > have
> > some end-point
> >
> >    /publish-this-nar-again-into-IPFS/name-of-the-nar
> >
> > which, when contacted, inserts the nar again into IPFS.  Then when
> > a
> > block was unavailable, the client contacts this end-point and
> > retries.
>
> But for a HTTP block endpoint we would still need such a
> database/block
> storage.
>
> I think it is important that we do not rely on IPFS for block
> storage. The decentralized block distribution should work even if the
> IPFS daemon is not available.

Do we need a database at all?

E.g., if the client cannot download the data in the range [start, end]
because the corresponding block has disappeared, can it not simply
download that range from https://ci.guix.gnu.org/nar/[...]
(not sure about the URI) using a HTTP range request?

(Afterwards, the client should insert the block(s) back into
IPFS/GNUnet/whatever, maybe using this proposed ‘in-file block store’
such that other clients (using the same DHT mechanism) can benefit.)

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfpqjxccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7q/tAQCx5U1HlZF9H7rfUztb00awEgdY
KRhb7frTXDXhQfzGwwD/Ro+r9CPRGyK2EQ1AqDDFgZUohQQrKHbAzL8yY+DG/Qk=
=93De
-----END PGP SIGNATURE-----


P
P
pukkamustard wrote on 2 Feb 2022 04:42
(name . Maxime Devos)(address . maximedevos@telenet.be)
8635l1h1mx.fsf@posteo.net
Maxime Devos <maximedevos@telenet.be> writes:

Toggle quote (11 lines)
>> I think it is important that we do not rely on IPFS for block
>> storage. The decentralized block distribution should work even if the
>> IPFS daemon is not available.
>
> Do we need a database at all?
>
> E.g., if the client cannot download the data in the range [start, end]
> because the corresponding block has disappeared, can it not simply
> download that range from https://ci.guix.gnu.org/nar/[...]
> (not sure about the URI) using a HTTP range request?

This does not work as the mapping from block reference to location in
NAR can not be known by the client who only holds the ERIS
URN. Furthermore, some blocks will be intermediary nodes - they hold
references to content blocks (or other intermediary nodes) but not
content itself.

Toggle quote (4 lines)
> (Afterwards, the client should insert the block(s) back into
> IPFS/GNUnet/whatever, maybe using this proposed ‘in-file block store’
> such that other clients (using the same DHT mechanism) can benefit.)

It might make sense for some clients to make content available to other
clients and to go trough the extra effort of putting blocks back into
IPFS/GNUNet/whatever. But this should be optional. Maybe we can call
such clients "caching peers"?

IMO A client should by default only deal with things that are strictly
necessary for getting substitutes. The substistute servers (and caching
peers) should make sure substitutes are available to clients, whether
over IPFS/GNUNet/whatever or plain old HTTP.

-pukkamustard
M
M
Maxime Devos wrote on 2 Feb 2022 07:07
(name . pukkamustard)(address . pukkamustard@posteo.net)
45e1dd2f05c6733eb74792af07194347fdd55ebd.camel@telenet.be
pukkamustard schreef op wo 02-02-2022 om 12:42 [+0000]:
Toggle quote (19 lines)
> > (Afterwards, the client should insert the block(s) back into
> > IPFS/GNUnet/whatever, maybe using this proposed ‘in-file block
> > store’
> > such that other clients (using the same DHT mechanism) can
> > benefit.)
>
> It might make sense for some clients to make content available to
> other
> clients and to go trough the extra effort of putting blocks back into
> IPFS/GNUNet/whatever. But this should be optional. Maybe we can call
> such clients "caching peers"?
>
> IMO A client should by default only deal with things that are
> strictly
> necessary for getting substitutes. The substistute servers (and
> caching
> peers) should make sure substitutes are available to clients, whether
> over IPFS/GNUNet/whatever or plain old HTTP.

If re-inserting missing blocks back into the IPFS/GNUnet/whatever is
made optional and is off by default, then almost nobody will enable the
‘caching peer’ option and we will have freeloaders, somewhat defeating
the point of GNUnet/whatever.

In a classic setting (‘plain old HTTP’), serving and downloading is a
separate thing. But in a P2P setting, downloading cannot be separated
from uploading -- strictly speaking, a peer might be able to download
without uploading (depending on the P2P system), but that's anti-
social, not something that should be done by default.

However, if re-inserting missing blocks is _on_ by default, then there
doesn't seem to be any trouble.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfqeNBccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7hI+AQCyLDdDhhLnScMBo6MNl55F+DKo
Yaje3fToELjEOBmOUAD/YOvJCAsYd7BILqDAImAt4T8s+UnANcKzVtbWKXbklwc=
=ttyT
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 2 Feb 2022 07:27
(name . pukkamustard)(address . pukkamustard@posteo.net)
84d930adac8c54120b176362e66a63ebf49179c6.camel@telenet.be
pukkamustard schreef op wo 02-02-2022 om 12:42 [+0000]:
Toggle quote (10 lines)
> > E.g., if the client cannot download the data in the range [start,
> > end]
> > because the corresponding block has disappeared, can it not simply
> > download that range from https://ci.guix.gnu.org/nar/[...]
> > (not sure about the URI) using a HTTP range request?
>
> This does not work as the mapping from block reference to location in
> NAR can not be known by the client who only holds the ERIS
> URN.

The client not only knows the ERIS URN, it also knows the location of
the nar (over classical HTTP) because it's in the narinfo.

Toggle quote (4 lines)
> Furthermore, some blocks will be intermediary nodes - they hold
> references to content blocks (or other intermediary nodes) but not
> content itself.

If an intermediary node (responsible for, say, bytes 900--10000)
is missing, then the bytes 900--10000 could be downloaded via HTTP.
Whether the node is close to the top, or close to the bottom, in ERIS'
variant of Merkle trees, doesn't matter much.

Granted, if the nar is, say, 1 GiB, and the top-level block is missing,
then we'll have to download 1 GiB over HTTP, even if most lower blocks
exist on IPFS/GNUnet/whatever, which isn't really great.

We could also do some combination of the GDBM database and HTTP
Content-Range requests: most nodes are leaf nodes (*). Instead of
representing all nodes in the database, we could include only
(intermediate) nodes responsible for data of size, say, 4MiB.

(*) At least, that's the case for binary trees, presumably something
similar holds for ERIS.

I don't know the specifics for ERIS, but for (balanced) binary trees,
not storing the leaf nodes would save about 50% (**), which is a rather
nice space saving.

(**) This assumes the ‘block size’ is the size for storing two pointers
to the children, but in practice the block size would be quite a bit
larger, so there would be more space savings?

Perhaps we are overthinking things and the GDBM (***) database isn't
overly large, or perhaps missing blocks are sufficiently rare such that
we could simply download the _entire_ nar from classical HTTP in case
of missing blocks ...

(***) Guix uses SQlite databases, so I would use SQLite instead of GDBM
unless there's a compelling reason to use GDBM instead.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfqi+BccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7luwAP4x9bFYhc+pM+oN9Jed5nNRDUSl
xTW+PIkxKhsPSSkOtAD8DtBwnt0Ju57kJ/OTzouZ0ZwGJRzBUCZ1bkRntYZZPg4=
=r/kQ
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 2 Feb 2022 07:36
Re: [bug#52555] [RFC PATCH v2 3/5] Add (guix eris).
(name . pukkamustard)(address . pukkamustard@posteo.net)
26bc58741a60c972226cd47832b27de955e4a57a.camel@telenet.be
pukkamustard schreef op wo 02-02-2022 om 10:28 [+0000]:
Toggle quote (25 lines)
> > pukkamustard schreef op di 25-01-2022 om 19:21 [+0000]:
> > > +(define (ipfs-daemon-alive?)
> > > +  "Attempt to connect to the IPFS daemon. Returns #t if the
> > > daemon is alive
> > > +and #f else."
> > > +  (with-exception-handler
> > > +      (const #f)
> > > +    (lambda _
> > > +      (let ((response _
> > > +                      (http-post (string-append (%ipfs-base-url)
> > > +                                               
> > > "/api/v0/version"))))
> > > +        (equal? 200 (response-code response))))
> > > +    #:unwind? #t))
> >
> > This should preferably only be catching exceptions indicating that
> > the daemon is down (exceptions indicating 404s, or system-errors
> > indicating network errors, ...).
>
> Yes, I guess it could be checked a bit finer. But at the end if an
> exception happens then the IPFS daemon is probably not reachable,
> right?
> If we don't care about the reason why it is not reachable then why
> bother with catching finer grained exceptions?

The exception could be caused by, say:

* an unbound variable
* wrong arity
* type error
* stack overflow
* prompt tag does not exist in current environment
* out of memory

Except for the last one, these causes are all bugs and hence shouldn't
be surpressed. Granted, this is a bit unlikely since this use of
'http-post' is very simple, but it's far from impossible for
(web client) to have a bug.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfqlGBccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7lMuAQCTlCl2Q5LRrRQiIKgiwuimw1Sm
NXVCP9Pf/eOoQZMWAAD+NpY1trIo+TZLI+tmUcKBARROcYliOgUMS0RgLj1l3Qk=
=sgcN
-----END PGP SIGNATURE-----


M
M
Maxime Devos wrote on 3 Feb 2022 12:36
Re: [bug#52555] [RFC PATCH v2 0/5] Decentralized substitute distribution with ERIS
(name . pukkamustard)(address . pukkamustard@posteo.net)
52ee517f75c66a8fd9e9823da016b5720b4d5d34.camel@telenet.be
pukkamustard schreef op wo 02-02-2022 om 11:10 [+0000]:
Toggle quote (3 lines)
> The ERIS URN is only used if the entire narinfo is signed with a
> authorized signature.

Perhaps I'm missing something here, but in that case, shouldn't "ERIS"
be added to %mandatory-fields in (guix narinfo)?

Anyway, I don't see what prevents an unauthorised narinfo with a ERIS
URN to be used: the narinfo is chosen with

(define narinfo
(lookup-narinfo cache-urls store-item
(if (%allow-unauthenticated-substitutes?)
(const #t)
(cut valid-narinfo? <> acl))))

where lookup-narinfo is a tiny wrapper around lookup-narinfos/diverse.
lookup-narinfos/diverse considers both unauthorised and authorised
narinfos, and can choose an unauthorised narinfo if it's ‘equivalent’
to an authorised narinfo (using equivalent-narinfo?)

equivalent-narinfo? only looks at the hash, path, references and size,
and ignores the ERIS. As such, an unauthorised narinfo with a
malicious ERIS URN could be selected.

However, it turns out that all this doesn't really matter: whether the
port returned by 'fetch' in (guix scripts substitute) came from
file://, http://, https://or ERIS, the file hash is verified later
anyway:

;; Compute the actual nar hash as we read it.
((algorithm expected)
(narinfo-hash-algorithm+value narinfo))
((hashed get-hash)
(open-hash-input-port algorithm input)))

[...]

;; Check whether we got the data announced in NARINFO.
(let ((actual (get-hash)))
(if (bytevector=? actual expected)
[...]

False alarm I guess!

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYfw80hccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7s/dAQDJWPpPuJCxdqcJv0tvWer5fKY/
Hr1YGeNT61UHexvazwD/X4/mwomnod+8Urvl9nhTeLUOa8Hs4iZ2tRYZR9nJ7wA=
=w/KP
-----END PGP SIGNATURE-----


P
P
pukkamustard wrote on 4 Feb 2022 02:20
(name . Maxime Devos)(address . maximedevos@telenet.be)
86tudeyknu.fsf@posteo.net
Maxime Devos <maximedevos@telenet.be> writes:

Toggle quote (25 lines)
> pukkamustard schreef op wo 02-02-2022 om 11:10 [+0000]:
>> The ERIS URN is only used if the entire narinfo is signed with a
>> authorized signature.
>
> Perhaps I'm missing something here, but in that case, shouldn't "ERIS"
> be added to %mandatory-fields in (guix narinfo)?
>
> Anyway, I don't see what prevents an unauthorised narinfo with a ERIS
> URN to be used: the narinfo is chosen with
>
> (define narinfo
> (lookup-narinfo cache-urls store-item
> (if (%allow-unauthenticated-substitutes?)
> (const #t)
> (cut valid-narinfo? <> acl))))
>
> where lookup-narinfo is a tiny wrapper around lookup-narinfos/diverse.
> lookup-narinfos/diverse considers both unauthorised and authorised
> narinfos, and can choose an unauthorised narinfo if it's ‘equivalent’
> to an authorised narinfo (using equivalent-narinfo?)
>
> equivalent-narinfo? only looks at the hash, path, references and size,
> and ignores the ERIS. As such, an unauthorised narinfo with a
> malicious ERIS URN could be selected.

You're right. I was not aware that parts of unauthorized narinfos are
used when they are deemed equavelent to authorized narinfos with
equivalent-narinfo?.

Toggle quote (21 lines)
>
> However, it turns out that all this doesn't really matter: whether the
> port returned by 'fetch' in (guix scripts substitute) came from
> file://, http://, https:// or ERIS, the file hash is verified later
> anyway:
>
> ;; Compute the actual nar hash as we read it.
> ((algorithm expected)
> (narinfo-hash-algorithm+value narinfo))
> ((hashed get-hash)
> (open-hash-input-port algorithm input)))
>
> [...]
>
> ;; Check whether we got the data announced in NARINFO.
> (let ((actual (get-hash)))
> (if (bytevector=? actual expected)
> [...]
>
> False alarm I guess!

Yeah, good that the hash is checked. Still, I think we should not even
try downloading a ERIS URN that is not authorized.

I think adding a check to equivalent-narinfo? that makes sure that the
ERIS URNs are equivalent if present would fix this. wdyt?

-pukkamustard
M
M
Maxime Devos wrote on 4 Feb 2022 08:16
(name . pukkamustard)(address . pukkamustard@posteo.net)
f05a19ad22a063662852f3799e465a6bb4d28ee8.camel@telenet.be
pukkamustard schreef op wo 02-02-2022 om 10:51 [+0000]:
Toggle quote (7 lines)
> > Also, don't forget to insert this missing block back into
> > IPFS/GNUnet/BitTorrent/..., otherwise less and less blocks will be
> > available until nothing is available anymore.
>
> This might be a bit of a burden for users. As you mention the size of
> such a database might become considerable.

At least in GNUnet, there are quota on the size of the datastore
(and presumably, whatever the DHT service uses as database). When it's
exceeded, old blocks are removed. So I don't see a burden here,
assuming that the quota aren't overly large by default.

Greetings,
Maxime.
-----BEGIN PGP SIGNATURE-----

iI0EABYKADUWIQTB8z7iDFKP233XAR9J4+4iGRcl7gUCYf1RXxccbWF4aW1lZGV2
b3NAdGVsZW5ldC5iZQAKCRBJ4+4iGRcl7tbgAQCIiwpFI5Vkp58Yf2ShVd29ShAW
qWSqyii0NGzBDXhwvQD+LNNDNEyqN59Ln2/gcNgL0dQw+u01A9WtwZ/IhWXJhAw=
=ky4X
-----END PGP SIGNATURE-----


P
P
pukkamustard wrote on 29 Dec 2022 10:13
[PATCH v3 0/8] Decentralized substitute distribution with ERIS
(address . 52555@debbugs.gnu.org)(name . pukkamustard)(address . pukkamustard@posteo.net)
20221229181327.758-1-pukkamustard@posteo.net
Hello Guix,

I'm very happy to present a V3 of a proposal towards decentralizing substitute
distribution with ERIS. An initial version [1] and a V2 [2] are now almost a
years old!

The idea is to use ERIS (Encoding for Robust Immutable Storage) [3] to allow
more decentralized substitute distribution. ERIS defines an encoding of
content into uniformly sized, encrypted and content-addressed blocks
(32KiB). The content can be decoded from the blocks given a short identifier
called the read capability. It allows a network-optimized form of
content-addressing.

Blocks can be transported over many different transport protocols, such as
HTTP, CoAP, GNUnet, IPFS or a SD card sent via pigeons. Only the read
capability must be transmited securely. Given an authentic read capability,
the content can be decoded correctly from blocks that might have been
transported over unreliable (and untrusted) protocols.

For Guix, substitutes (Nar files) are encoded using ERIS and the read
capability is added to the signed Narinfos. The read capability published in
the Narinfo can then be used to decode the Nar from blocks that are fetched
from many peers over many different protocols.

This version of the patches allows blocks to be fetched over HTTP and IPFS.

A summary of this patch series:

- Use the stable version of the ERIS encoding (version 1.0.0)
- Add two fields to Narinfos (ERIS and ERISFormat)
- Store blocks of published substitutes in a local block store (in
`/var/guix/eris`)
- Add an endpoint for resolving ERIS blocks over HTTP (a la RFC 2169)
- Use ERIS when fetching substitutes
- Use IPFS for de-referencing ERIS blocks

Testing procedure is a bit tedious, but described in the V2:

Please give it a try or a read.

Two commits are marked as WIP and have some todos regarding
configuration. Suggestions in how to go about Guix wide configuration are very
welcome.

Some further ideas include:

- We've been researching CoAP (RFC 7252) as a transport protocol for
blocks. It is lightweight, performant, well-specified and optimized for
small implementations, meaning it can be reasonably implemented in pure
Guile. This would allow more light-weight peer connections (over UDP) and
also discovery of local peers via CoAP service discovery [4] - very similar
to how Avahi is currently used.
- Publish substitutes to IPFS. Currently substitutes are not published to
IPFS. It might make sense to add a CLI command that does this. In a previous
verion of the series this was done when a substitute is requested over
HTTP. But it seems to make sense to publish on IPFS (or other protocols)
independantly of the HTTP endpoints.
- Use an optimized format for encoding Nars. Based on some investigation into
SquashFS, EROFS and de-duplication of files in substitutes [5], we have an
initial proposal of how to encode Nars (and in general file system trees) in
ERIS so that common files are de-duplicated:
- More transport protocols.

Thanks for making it so far! :) I'd be very happy for your thoughs and
ideas. There might also be opportunity to discuss these ideas at Guix Days in
February.

Best regards and a happy new year,
pukkamustard



pukkamustard (8):
publish: Add ERIS URN to narinfo.
publish: Store ERIS encoded blocks to a local block store.
publish: Add HTTP endpoint for resolving ERIS blocks.
WIP: substitute: Fetch substitutes using ERIS.
eris/http: Add HTTP block de-referencer.
WIP: eris: Use HTTP to get ERIS blocks.
eris: Use parameterized %eris-peers when getting blocks.
eris: Use IPFS to get ERIS blocks.

Makefile.am | 4 +
configure.ac | 5 +
gnu/packages/package-management.scm | 1 +
guix/eris.scm | 95 ++++++++++++
guix/eris/fs-store.scm | 66 +++++++++
guix/eris/http.scm | 82 +++++++++++
guix/eris/ipfs.scm | 214 ++++++++++++++++++++++++++++
guix/narinfo.scm | 16 ++-
guix/scripts/publish.scm | 55 ++++++-
guix/scripts/substitute.scm | 44 ++++--
10 files changed, 563 insertions(+), 19 deletions(-)
create mode 100644 guix/eris.scm
create mode 100644 guix/eris/fs-store.scm
create mode 100644 guix/eris/http.scm
create mode 100644 guix/eris/ipfs.scm


base-commit: c37b6fba1af0d823bd18c753484459b40fc8c041
prerequisite-patch-id: 38bf058b6e298a205d7c9bc19f6358ee3cd5257b
prerequisite-patch-id: b2bb0971706132f1f02d99a9a7504fb195d6a74a
prerequisite-patch-id: 60537c1eb34acc412769470c7ee90043bde600f9
prerequisite-patch-id: 71cb6edb430a0b8371b4c06d171b51e781bfe348
prerequisite-patch-id: 38a019a8380ac8dfb77bfb6304159ce8ca6d32dd
prerequisite-patch-id: a53a66ddac48c405f30debc74b267de5895394c9
prerequisite-patch-id: 165aa2669327788a9e51436b8000643da742b16e
prerequisite-patch-id: 2812cebf583538f6bdd2acccd6e0825ceca7ba25
prerequisite-patch-id: 0d04d66369e86604a7bbd59c28cafe3ee3bd7730
prerequisite-patch-id: ccb6e14312ab11227f5547d17cd7c5df2341238a
prerequisite-patch-id: 0b7662f5b8b11edd52fd902d8e70827a6661323c
prerequisite-patch-id: 26ee9dc07a521052655c3bdaf849febaf0464e50
prerequisite-patch-id: 8b2b3209e7f0053d75b52c2475a76feed2114de2
prerequisite-patch-id: 2fdb157131a90a1542357acaf17cb34db4ca7f9b
prerequisite-patch-id: 1ddca41aca5544ab9fa3fa868271976293db1518
prerequisite-patch-id: 04eb88d682cb51708eba1dcd5e20e7b468daf1ce
prerequisite-patch-id: 531d5e5039f76948cdf3fc21ccd9dc61fd9682f0
prerequisite-patch-id: 17a795a4022a0c80b2df3f362fa965ad65f42191
prerequisite-patch-id: 4216fc57a30c8c1d3db94a48aae78b73d9a2c77e
prerequisite-patch-id: 305b8cf69729cb3d6167bb60189e6a9e1d228fd8
prerequisite-patch-id: 60de21549abd45931b5ecad335acb88a5369696c
prerequisite-patch-id: 27c2475ac075bd74804106bbfab927cc168333a4
prerequisite-patch-id: 5f8bd6b339c6fc14deba1929a5cd934e3a8bbb68
prerequisite-patch-id: 93ad6954d493f65bf032525c09996de41c51de1d
prerequisite-patch-id: 6195fafe0e46bf1b6ae15f08a1d1fa3c98b89cab
prerequisite-patch-id: ee11d213a4ee466dff6379eeaeea292b6ba93fb0
prerequisite-patch-id: 857e9a0f3d1bc9ad5584c44bc767683a53e44972
prerequisite-patch-id: b87a0615ac48fe8950281f797cd2d096a98f4a2c
prerequisite-patch-id: c554e46d5d315cd349df21ba153ef6ed580f0b3c
prerequisite-patch-id: 232075806f66963ae7aa260cb6fd3e537f30f1e4
prerequisite-patch-id: 4c5479903595202ac84dcd5978c7c58f542110a4
prerequisite-patch-id: 9e68c0bacd9228f62607d52285f587988a49a6c6
prerequisite-patch-id: b6bcee4d778d2cbf9bcd3d55bb39a65fa5659a61
prerequisite-patch-id: d6702e7370f001971201f5f751719f888b1b9354
prerequisite-patch-id: ead326db75ec4a111dd64f56c985853319a3d55c
prerequisite-patch-id: 2633f1445ab40e07536b7c0d0c7df2ca8699bc57
prerequisite-patch-id: 05955431fa785aabe4c11545fdbce46aff5c3deb
prerequisite-patch-id: 4b212323a81b84bebacdeab2bda9f3efff768542
prerequisite-patch-id: 38106d6e8e5c6bcabed45f2a469ba2609ef8a904
prerequisite-patch-id: 482714ab5f307400a641012270afe5e6a392245c
prerequisite-patch-id: 70739c3de04e8d66181af4850973a778de614385
prerequisite-patch-id: 464301961e7060a9d80879881114b76df3314d5c
prerequisite-patch-id: 9096c39f5006697843fc26c1e32735f5bbc65da8
prerequisite-patch-id: 240bac4d775f14296009b6e8f41dc7fee540724a
prerequisite-patch-id: 797bca2dd99a05e6886436a3eadf09037c19f174
prerequisite-patch-id: 0f37a06624e40cc1ea41c9f9dcc4097e98f0bc38
prerequisite-patch-id: e238c0b3d0889467b98693cd01db27331d1dad1b
prerequisite-patch-id: 8fdcafd9def3b02af3a8df411ea6b2372b8ac0fb
prerequisite-patch-id: 470061e10a39b26c52d53cce5c65a2295feefbd2
prerequisite-patch-id: 6aa5474a3378080a6536d46a3616833df711e65c
prerequisite-patch-id: c8247c4fb50de660e145ff4b1bdb586d23cc9c59
prerequisite-patch-id: 7fdbe491552bcc0b718f3d82418a49bc3b225f79
prerequisite-patch-id: 1f26481841f7d528c236a07c988c6410f4b1ec1e
prerequisite-patch-id: 9a4c598e9efeedce6f577cba794a5f3f6762c0c0
prerequisite-patch-id: cc12fbcd08457f67b9f0a947088fe69d4b593e5c
prerequisite-patch-id: 736edff89d49035031cbd647fa99ed57e6e4f71e
prerequisite-patch-id: d6e0c13f594235607d36a8d7cdf7390e4530dfdf
prerequisite-patch-id: edbcbb20b76e5c64eb0d9c3971101c7e67aed63f
prerequisite-patch-id: 6381b67d4b369d5f6ab111446e5082f179c906c8
prerequisite-patch-id: 7744b2dd228f3963c73acbf4dad19884ca4ab819
prerequisite-patch-id: 77fd89d1fcc9cb2d08c72f13c76c1c1b5bf09f0d
prerequisite-patch-id: 5721541afe5a74e0cab98717a0b684f24dfb0bd9
prerequisite-patch-id: 16c506e2b7929ed0f63484be8af3e46d83b7e9c1
prerequisite-patch-id: 8a2c2e203e13fdb52cd2249e9e966ad89834f468
prerequisite-patch-id: 134efcd91e8eddb574dd7ca84fbf97c7dc55dab3
prerequisite-patch-id: 0b80209b680bc0da99f29a088b63ad529390475f
prerequisite-patch-id: c206070aeaea4969fb718a1eab2a672eb6887be3
prerequisite-patch-id: 06e9164d837f5fe0fed7f53a6520eb119ad9c07d
prerequisite-patch-id: c5cd707027c858c8c1360fca569036fd0e98d7c8
prerequisite-patch-id: 561ffda841ebf652dd4f0d6e59dffcc5b26aed3f
prerequisite-patch-id: f96dc76e70647bd2e4097b36fae74ecd2b4ef706
prerequisite-patch-id: a31940d97c85bb1126e4d5e613f73a0d4e6e3ce3
prerequisite-patch-id: 204940d7d89e985068e5ab6cbd0c9753d5e6cbb6
prerequisite-patch-id: b9a0811fa8aa30c6bb67b3e6d76e2e0d73e5e300
prerequisite-patch-id: 3b683f608d01c23bcde0c179cfdc8f83ba430f9d
prerequisite-patch-id: 9c7ad92ccae90d137e22c6f5dd2df6ba5006b10a
prerequisite-patch-id: 07256b1e50c1a9ffcad34724bd2a0e9c8fc2dec1
prerequisite-patch-id: e48b0932283f2e81e5226b1e9eaeff4d25200aa9
prerequisite-patch-id: bf53daf969eddd302c084b2f77c707005fd6226a
prerequisite-patch-id: 3aaeaf0c560dd750c0000f9871b461c348060fb8
prerequisite-patch-id: fb09c518a1e1eab86bd0e3dc9337d01abfb3aede
prerequisite-patch-id: a6d38e15437ed0e19ca8de6d7ea8bec9f403a347
prerequisite-patch-id: 0582bad0ad1f76d7f3bb32ff9e1b986708fe92ab
prerequisite-patch-id: 8c93bd940ae761f1ee27b3514c751a6b67fe648c
prerequisite-patch-id: 547a83c461648121d964e0d677271ffae513ded0
prerequisite-patch-id: 93f020e5ace6a0034683807ab1acbaacd14c6c49
prerequisite-patch-id: 8d871c380f84927be637e51dbd79298030a26211
prerequisite-patch-id: bf4084d999bb48633f9b313fd5c1611a7f6880e0
prerequisite-patch-id: 5e105f911cecdb68d91b3b0e698351366af8c0ea
prerequisite-patch-id: 428685a07eca306ff9b7804835d92a414651182c
prerequisite-patch-id: 0bc436f66c4256680592ec5e11c3bca9e6735430
prerequisite-patch-id: 129c52d7ee4eba0666bcd088c85292a2230e714f
prerequisite-patch-id: f42112e78fb5483035525aff61cc06a7dfd16e92
prerequisite-patch-id: 5dee96fef77d3ad04e7b8fb96f7f75d6d02cd287
prerequisite-patch-id: d36b9670d2734ece43529aa7b502eb5d142cd6b9
prerequisite-patch-id: e6cc80f307552a886e39113007dc62b326f32adf
prerequisite-patch-id: ef3fa2f68f986c7ec54e9eca0446e30ee72c847e
prerequisite-patch-id: 4e9255e48ae5452b35e058f149e77745c04fc436
prerequisite-patch-id: 1054691ce867cd6085bde390306e8df1cdc61e48
prerequisite-patch-id: f544cb093aaa2aa36783b05ea1c09fae389bf6bf
prerequisite-patch-id: a0002a2545cbc48285dbea6f636cca51c7886560
prerequisite-patch-id: dfe2ed731d951b076a5896df61e08c604d1d9efb
prerequisite-patch-id: 177b76db09d5e96ff0191e54d62f969c62239d43
prerequisite-patch-id: 392ee4da803f87deb0175842281dc404d65a2ee4
prerequisite-patch-id: a1a7038d4d35ed6b6352d2300c9c4fe8cd7d4c3e
prerequisite-patch-id: e63d8db9db9124fb8ad50c384f7468980ba25b47
prerequisite-patch-id: 878f0ef3ffaabbc15d2ee4ee0564347f82926a8d
prerequisite-patch-id: e8f134c6370d435f1716cfe9647d965ae7a4b1a9
prerequisite-patch-id: 7470f65f04ba4451f50e6c5f18ec14b51a858d01
prerequisite-patch-id: 20a28e6b1d370ba73029f263f6fa9e9a8d1160c1
prerequisite-patch-id: 16ccc436c7c92fb9e41dd9634d211576e4190337
prerequisite-patch-id: be1b5a39649b32ab1361c95d1b4736cf0074179d
prerequisite-patch-id: 6ebca863c5453ecd1b9023646a229215ebd13452
prerequisite-patch-id: c6a0cd51aa6890d7ba6abbc6f08519a946ecbb0e
prerequisite-patch-id: 1cbcafd9adf73b1664bf416a983b64781ccb3de1
prerequisite-patch-id: 5beac63517a1c0b8361e48cdb76031353571f899
prerequisite-patch-id: a28e0e677e8e70bde296b9eaad861da05f3a7572
prerequisite-patch-id: dbb7feae5390a6aec98ab0618a477983d217882f
prerequisite-patch-id: faba2d31c61942f6f3ded0d4f6ed20ff52ebe865
prerequisite-patch-id: e9c4b4b08a2f4e9bfa08fba2c96f5acb2ff1361c
prerequisite-patch-id: f95e278262ce40c3c304c3b0b97cdac02ab6cdc1
prerequisite-patch-id: 015cece7899c9b8af24dd542ceca65dafa2e6d4b
prerequisite-patch-id: d57c54b60fd3fd151e7c3318df13cf2901c042a8
prerequisite-patch-id: 9a48f08769b01f60fa369881c0e1c0f4f5a62601
prerequisite-patch-id: 45d3527827fb17fb8946943ebe29fe61285ff8fc
prerequisite-patch-id: a78104964358350ebb7ba4071d51b857a563b389
prerequisite-patch-id: 4c1247c66fa368185d6c5f1b91136cace31d2996
prerequisite-patch-id: ee23d06d8d676221dedefd0beeace4b2249f6203
prerequisite-patch-id: 76f871743dd063789d1ba543131205072da2e514
prerequisite-patch-id: 9f83e1a4b61a58e28055e02a3ea5f0d936c4a272
prerequisite-patch-id: 26a603cc8e14ff53e47cd23b88f380129b84a687
prerequisite-patch-id: 6f2ad843478baf980135f9f0996d0dbdfb2fe260
prerequisite-patch-id: db2c8aae4743505ad5a36fd6586e651206406a58
prerequisite-patch-id: 217b648e20131c63182a317e8aa963c3440e9956
prerequisite-patch-id: 02a38f9e6b58d115e7b1310933e72a6a718c8002
prerequisite-patch-id: 8bbedde3db4944f76e546a085dfc53937e733b0a
prerequisite-patch-id: 27cd4a8b3b7daa02fa2aa549ed2c584ac58b118d
prerequisite-patch-id: 6796a96b4cc332e170c9893515351a133bf3cbd9
prerequisite-patch-id: 028b42964bbcdb390f77c3fec97db30f42b29478
prerequisite-patch-id: c5d78227865719237dcb0356811008a5fa8c8a04
prerequisite-patch-id: 12e290b368ffbf85afcdbc66b5eeba744d015814
prerequisite-patch-id: d2a58d75b6c38a03da5a74c67195f6ed17840d29
prerequisite-patch-id: 4814320bf2499370e3ea6fb3bb6899239c3ae6ee
prerequisite-patch-id: 956a1c74c9772838f4b1a46baa99375bc594e51b
prerequisite-patch-id: b667030ab4aef0d15c68aac6ca1de8b9fef4eb19
prerequisite-patch-id: 9f9f39201b649d16cbe75da5e073fd5a9d12f97e
prerequisite-patch-id: f669c71c4852e5ee6d46a20c3f3386f19df5d07a
prerequisite-patch-id: 8122fd0b90e20029a339833b1a055da7dedfa21a
prerequisite-patch-id: 94584e23cc9f1e37bbc3c54ac950d1ea76f5e900
prerequisite-patch-id: 184fcb20369b70452f1b9dbb9973f8870b2595ce
prerequisite-patch-id: 6499fe37c121b9843b6d1988cfb1f11ef1193939
prerequisite-patch-id: 83407b3f6fb4d24f9947cdbcef079744bab36da8
prerequisite-patch-id: ba131ace8fbc158e1ab7b5c1854eb80fafb8cefe
prerequisite-patch-id: 9706095b8330b04e91cbc43064cc3e769807840e
prerequisite-patch-id: 2e8d8005aca3e5d66768cf667d1261762609c61f
prerequisite-patch-id: 95747be36e38ae854fd4fd65f23194c2ec60eb87
prerequisite-patch-id: d389d08e983165dfb1fef9e7cbe38481efd29ccf
prerequisite-patch-id: 66a0804e05983681af77fb26a0ce409ce050b733
prerequisite-patch-id: 82caea0441d0954e3e159de8708e24ca83514c1e
prerequisite-patch-id: a966347d564354f512f71a875be7bd9d610fb4e6
prerequisite-patch-id: bd318b9b11b1e4568986bba81063269db42fcd4e
prerequisite-patch-id: ec228f3dfab23d4d166023d327f0efe271fa20f1
prerequisite-patch-id: 33d563137eaea059421d7c800a92296a8b670ee6
prerequisite-patch-id: 597038c4efb635d3afa6882f7b380816639926dc
prerequisite-patch-id: e50ae62c7af2f3c014bbd424f9dc6a916abfbb8e
prerequisite-patch-id: c229274d1cd5f8ed8b7429560ce4cd7ad788af65
prerequisite-patch-id: 263f50ff048ac6964c4af48104123f0a6f55cb95
prerequisite-patch-id: a514d6fa23dd6a2e4001d89715342155849e0dfc
prerequisite-patch-id: 325a0dfe5940ee861ad1729a305ba3d13d61bb85
prerequisite-patch-id: 2b0f90f5f5ad85b7fab1ee2340b6fa88255800b3
prerequisite-patch-id: 23d1e2410c7f933f6b2f467378c11e7d66c7a653
prerequisite-patch-id: de4ea7504ff38ab882b4e5e3087116246d91465a
prerequisite-patch-id: 42d09d8fca07831b7eb57909e11a5f47c1dcad9d
prerequisite-patch-id: 839d672c34031518dc716a7d5b2f36c174c1a19d
prerequisite-patch-id: ac6cb5491164b907cd0b4802a06568d2a8671017
prerequisite-patch-id: 9da135c032972d11f7682f3c5e3f695741ea8fbd
prerequisite-patch-id: ef7989acb42b3da2c340634ee07895770d26b88e
prerequisite-patch-id: 221b01cd4f38453080f721f4f07e0fb5e1108ab9
prerequisite-patch-id: 93c27c637a45114ee87fbe02beec8475da9e3592
prerequisite-patch-id: 16a07eb39ffcb7fecc9de389fbce66455bd8815e
prerequisite-patch-id: b05d09d133e8131d600fe9c087703327633372aa
prerequisite-patch-id: 649e822307b5adb031dad10dc38739155bf5ae27
prerequisite-patch-id: 0c87eb233edb1362d249f4828486d1526e33afa3
prerequisite-patch-id: 2b62e27c777741fcc8c28f61ea9cd48fb99e894f
prerequisite-patch-id: 063f7ffd5ff936f8493205d5aefea1421d52a8ca
prerequisite-patch-id: 622cbebc7a82fb9789506c31329d9c9f4fbabe95
prerequisite-patch-id: 7bef9850ce8d1cd904a32f18c2202a8c8319091b
prerequisite-patch-id: 3b454a5ddafb74a01e56515d3c22da496d2a1626
prerequisite-patch-id: 6269a9d9e5e76fbc034aa68dafd8e159912bbb2b
prerequisite-patch-id: 77e4d852c6ff934e64cfc67e9ae350e26a83008a
prerequisite-patch-id: 0d682ba3d1fd449326b1fb78b8a2f26f4cc695e1
prerequisite-patch-id: 4a02b8ff27880d5e182b5b0a62949f1861f7b396
prerequisite-patch-id: 2a0cc98bcf98efae96bc66f6d008ffdb85242952
prerequisite-patch-id: 5a62132a7f9889af3c45030163cf0ca5e96e24e9
prerequisite-patch-id: f63e64247a3106ce799c326608dc92e7a625bc84
prerequisite-patch-id: 320967154285a6b2bee8fe8a5687a84712f2cff9
prerequisite-patch-id: dee3e1bb39bb08e09bfa27f86cbe042e23f10d04
prerequisite-patch-id: 340e81de5468a8af063674aeae405b184315e886
prerequisite-patch-id: 8cf7228cca56823149287eb677441f2b831d7127
prerequisite-patch-id: ff8fa83584b7025c9d55880e81be8e612e5190ed
prerequisite-patch-id: 3338fcba95ed51c00ffbf601093428915f89ecb4
prerequisite-patch-id: b735a0ba361b3dc3dbe366649535887901518f85
prerequisite-patch-id: 46d3971d3bfbc67438ffce11be263981ae9b4621
prerequisite-patch-id: cba3717d29ab9acfef78f7e70269d8deffc2b91b
prerequisite-patch-id: 238d14ef175c0368e0c16f1d2ef9a5ec2d24efca
prerequisite-patch-id: 1d51aa1c1e3a82251d203de985be62a0b9402752
prerequisite-patch-id: d9dc5987e2ce1d8d3d45227ba89cc1bb688fd312
prerequisite-patch-id: be97255da5e7c9f3bc824785fcd7d58863ed821b
prerequisite-patch-id: 9d47fce0cc920022068265c72b9d72f02c8f75c3
prerequisite-patch-id: 0159d32f88bfd90eb4ff1eec43640c2c5482a4f9
prerequisite-patch-id: 2500f00e09cde491099d34567ca7e356d96a0b64
prerequisite-patch-id: 167fba505b6a0791ba4eea4de37ab8509c5f53b3
prerequisite-patch-id: 3e9769087da4ee6b1557eb5e1cbc3e8527520701
prerequisite-patch-id: aedc29d90774552741df4ff3c77a234397246f2c
prerequisite-patch-id: 40f53217c453e002092469a32b43d317d77848d6
prerequisite-patch-id: a7c13980dbc2933e77706296a18a607e02aaf39b
prerequisite-patch-id: 652ffa323a151f3d116b02b5b342cba239cd88ac
prerequisite-patch-id: db6a4954d8ca87b0f7437e4c35c20be142f49bd7
prerequisite-patch-id: 5e4d4bf7ffcbc391bd8e473e347d2ad4ce70dec9
prerequisite-patch-id: be894f7c7db836724d92ff46d1cf7772899f99a0
prerequisite-patch-id: 7f8f6c49ef0e526c4920744a61132f798dcabc41
prerequisite-patch-id: db4b76df1c799482628afca46efef0fa687c546a
prerequisite-patch-id: 9a2d794ab564ad0cd7aad5c8551fc2ffae9e5db4
prerequisite-patch-id: 017c8c510a7e683119f8f720c9c13a111b9de9a5
prerequisite-patch-id: b5dd7d30ca7d567e6877e0f6ae6b942abcafbf4b
prerequisite-patch-id: a9a37c8beaebab4207cab2447b30e2b559d76e49
prerequisite-patch-id: 7a11e631dd100776319d05bc31d5dca4c90b686d
prerequisite-patch-id: 4c84745aee8ba2835a8b425ca090e3ec1ad7205c
prerequisite-patch-id: eace61f02ecbb0b5802b342f34ca5a331421edfe
prerequisite-patch-id: 955131799e4b4b8fc5e06de951875c61c8e06883
prerequisite-patch-id: c0255702fd789d06f0f0096c949dca551956df16
prerequisite-patch-id: 03d030fdb65c756f6870a639a2f19d9b2bd7884b
prerequisite-patch-id: 44686e39403fd7624d1b023f80170123c6cc6aa5
prerequisite-patch-id: b6c23623df08ac834bc1b5f172eede03543822a2
prerequisite-patch-id: c4a56f193a7df3b4da4057fc3376411363403f24
prerequisite-patch-id: 26a2355cd6f9d9fcb120c97b07fce4c7b41f4658
prerequisite-patch-id: f2c32e18e50c3dc1c41cb17173b8633d049f66da
prerequisite-patch-id: 7c0686a24cd8dcc8431dd5922d76dad9bcb2788a
prerequisite-patch-id: d134f3f1cf36457db602408790e279b102362dc4
prerequisite-patch-id:
This message was truncated. Download the full message here.
P
P
pukkamustard wrote on 29 Dec 2022 10:13
[PATCH v3 1/8] publish: Add ERIS URN to narinfo.
(address . 52555@debbugs.gnu.org)(name . pukkamustard)(address . pukkamustard@posteo.net)
20221229181327.758-2-pukkamustard@posteo.net
* guix/scripts/publish.scm: (bake-narinfo+nar): Compute ERIS URN of compressed nars.
(narinfo-string): Add #:eris-urn parameter and honor it.
* guix/scripts/narinfo.scm: (<narinfo>)[eris-format,eris-urn]: New fields.
(narinfo-maker): Handle ERIS URN and ERIS format.
* configure.ac: (HAVE_GUILE_ERIS): New conditional.
* gnu/packages/package-management.scm: (guix)[native-inputs]: Add guile-eris.
---
configure.ac | 5 +++++
gnu/packages/package-management.scm | 1 +
guix/narinfo.scm | 16 +++++++++-----
guix/scripts/publish.scm | 34 ++++++++++++++++++++++++-----
4 files changed, 46 insertions(+), 10 deletions(-)

Toggle diff (188 lines)
diff --git a/configure.ac b/configure.ac
index 92dede8014..52a37a6113 100644
--- a/configure.ac
+++ b/configure.ac
@@ -173,6 +173,11 @@ GUILE_MODULE_AVAILABLE([have_guile_avahi], [(avahi)])
AM_CONDITIONAL([HAVE_GUILE_AVAHI],
[test "x$have_guile_avahi" = "xyes"])
+dnl Check for Guile-eris.
+GUILE_MODULE_AVAILABLE([have_guile_eris], [(eris)])
+AM_CONDITIONAL([HAVE_GUILE_ERIS],
+ [test "x$have_guile_eris" = "xyes"])
+
dnl Guile-newt is used by the graphical installer.
GUILE_MODULE_AVAILABLE([have_guile_newt], [(newt)])
diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm
index 2ffaa12247..3585f6c073 100644
--- a/gnu/packages/package-management.scm
+++ b/gnu/packages/package-management.scm
@@ -419,6 +419,7 @@ (define code
("guile-zstd" ,guile-zstd)
("guile-ssh" ,guile-ssh)
("guile-git" ,guile-git)
+ ("guile-eris" ,guile-eris)
;; XXX: Keep the development inputs here even though
;; they're unnecessary, just so that 'guix environment
diff --git a/guix/narinfo.scm b/guix/narinfo.scm
index 741c7ad406..d7f8c88f8f 100644
--- a/guix/narinfo.scm
+++ b/guix/narinfo.scm
@@ -45,6 +45,8 @@ (define-module (guix narinfo)
narinfo-file-sizes
narinfo-hash
narinfo-size
+ narinfo-eris-format
+ narinfo-eris-urn
narinfo-references
narinfo-deriver
narinfo-system
@@ -68,8 +70,8 @@ (define-module (guix narinfo)
(define-record-type <narinfo>
(%make-narinfo path uri-base uris compressions file-sizes file-hashes
- nar-hash nar-size references deriver system
- signature contents)
+ nar-hash nar-size eris-format eris-urn references deriver
+ system signature contents)
narinfo?
(path narinfo-path)
(uri-base narinfo-uri-base) ;URI of the cache it originates from
@@ -79,6 +81,8 @@ (define-record-type <narinfo>
(file-hashes narinfo-file-hashes)
(nar-hash narinfo-hash)
(nar-size narinfo-size)
+ (eris-format narinfo-eris-format)
+ (eris-urn narinfo-eris-urn)
(references narinfo-references)
(deriver narinfo-deriver)
(system narinfo-system)
@@ -135,8 +139,8 @@ (define (narinfo-maker str cache-url)
"Return a narinfo constructor for narinfos originating from CACHE-URL. STR
must contain the original contents of a narinfo file."
(lambda (path urls compressions file-hashes file-sizes
- nar-hash nar-size references deriver system
- signature)
+ nar-hash nar-size eris-format eris-urn references deriver
+ system signature)
"Return a new <narinfo> object."
(define len (length urls))
(%make-narinfo path cache-url
@@ -157,6 +161,8 @@ (define len (length urls))
((lst ...) (map string->number lst)))
nar-hash
(and=> nar-size string->number)
+ eris-format
+ (if eris-urn (string->uri eris-urn) #f)
(string-tokenize references)
(match deriver
((or #f "") #f)
@@ -184,7 +190,7 @@ (define* (read-narinfo port #:optional url
(narinfo-maker str url)
'("StorePath" "URL" "Compression"
"FileHash" "FileSize" "NarHash" "NarSize"
- "References" "Deriver" "System"
+ "ERISFormat" "ERIS" "References" "Deriver" "System"
"Signature")
'("URL" "Compression" "FileSize" "FileHash"))))
diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm
index 3bf3bd9c7c..76b5a429f4 100644
--- a/guix/scripts/publish.scm
+++ b/guix/scripts/publish.scm
@@ -40,6 +40,7 @@ (define-module (guix scripts publish)
#:use-module (srfi srfi-34)
#:use-module (srfi srfi-37)
#:use-module (srfi srfi-71)
+ #:use-module (srfi srfi-171)
#:use-module (web http)
#:use-module (web request)
#:use-module (web response)
@@ -57,6 +58,7 @@ (define-module (guix scripts publish)
#:use-module (guix workers)
#:use-module (guix store)
#:use-module ((guix serialization) #:select (write-file))
+ #:use-module (eris)
#:use-module (zlib)
#:autoload (lzlib) (call-with-lzip-output-port
make-lzip-output-port)
@@ -145,6 +147,9 @@ (define %default-gzip-compression
;; Since we compress on the fly, default to fast compression.
(compression 'gzip 3))
+(define %eris-zstd-compression
+ (compression 'zstd 19))
+
(define (default-compression type)
(compression type 3))
@@ -323,7 +328,8 @@ (define* (store-item->recutils store-item
(define* (narinfo-string store store-path
#:key (compressions (list %no-compression))
- (nar-path "nar") (file-sizes '()))
+ (nar-path "nar") (file-sizes '())
+ eris-urn)
"Generate a narinfo key/value string for STORE-PATH; an exception is raised
if STORE-PATH is invalid. Produce a URL that corresponds to COMPRESSION. The
narinfo is signed with KEY. NAR-PATH specifies the prefix for nar URLs.
@@ -345,10 +351,10 @@ (define* (narinfo-string store store-path
"\
StorePath: ~a
NarHash: sha256:~a
-NarSize: ~d
+NarSize: ~d~@[~%ERISFormat: application/x-nix-archive+zstd-19~%ERIS: ~a~]
References: ~a~%"
store-path
- hash size references))
+ hash size eris-urn references))
;; Do not render a "Deriver" line if we are rendering info for a
;; derivation. Also do not render a "System" line that would be
;; expensive to compute and is currently unused.
@@ -633,6 +639,22 @@ (define (compressed-nar-size compression)
(and stat
(cons compression (stat:size stat)))))
+ (define (eris-encode-nar compressions)
+ (and (member %eris-zstd-compression compressions)
+ (let* ((nar (nar-cache-file cache item
+ #:compression %eris-zstd-compression))
+ (stat (stat nar #f)))
+ (and stat
+ (call-with-input-file nar
+ (lambda (port)
+ (let ((eris-urn _
+ (eris-encode port
+ #:block-size 'large
+ #:block-reducer rcount
+ #:convergence-secret
+ %null-convergence-secret)))
+ eris-urn)))))))
+
(let ((compression (actual-compressions item compressions)))
(for-each (cut compress-nar cache item <>) compressions)
@@ -640,7 +662,8 @@ (define (compressed-nar-size compression)
(match compressions
((main others ...)
(let ((narinfo (narinfo-cache-file cache item
- #:compression main)))
+ #:compression main))
+ (eris-urn (eris-encode-nar compressions)))
(with-atomic-file-output narinfo
(lambda (port)
;; Open a new connection to the store. We cannot reuse the main
@@ -651,7 +674,8 @@ (define (compressed-nar-size compression)
(display (narinfo-string store item
#:nar-path nar-path
#:compressions compressions
- #:file-sizes sizes)
+ #:file-sizes sizes
+ #:eris-urn eris-urn)
port)))
;; Make the cached narinfo world-readable, contrary to what
--
2.38.1
P
P
pukkamustard wrote on 29 Dec 2022 10:13
[PATCH v3 2/8] publish: Store ERIS encoded blocks to a local block store.
(address . 52555@debbugs.gnu.org)(name . pukkamustard)(address . pukkamustard@posteo.net)
20221229181327.758-3-pukkamustard@posteo.net
* guix/eris.scm: New file.
* guix/eris/fs-store.scm: New file.
* Makefile.am (MODULES): Add new files.
* guix/scripts/publish.scm (bake-narinfo+nar): Use guix-eris-block-reducer.
---
Makefile.am | 2 ++
guix/eris.scm | 36 +++++++++++++++++++++
guix/eris/fs-store.scm | 67 ++++++++++++++++++++++++++++++++++++++++
guix/scripts/publish.scm | 14 +++++----
4 files changed, 113 insertions(+), 6 deletions(-)
create mode 100644 guix/eris.scm
create mode 100644 guix/eris/fs-store.scm

Toggle diff (162 lines)
diff --git a/Makefile.am b/Makefile.am
index b54288c0fc..c549fc8580 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -132,6 +132,8 @@ MODULES = \
guix/least-authority.scm \
guix/read-print.scm \
guix/ipfs.scm \
+ guix/eris.scm \
+ guix/eris/fs-store.scm \
guix/platform.scm \
guix/platforms/arm.scm \
guix/platforms/mips.scm \
diff --git a/guix/eris.scm b/guix/eris.scm
new file mode 100644
index 0000000000..29d5e7b1db
--- /dev/null
+++ b/guix/eris.scm
@@ -0,0 +1,36 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 pukkamustard <pukkamustard@posteo.net>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix eris)
+ #:use-module (eris)
+
+ #:use-module (guix config)
+ #:use-module (guix eris fs-store)
+
+ #:export (guix-eris-block-reducer
+
+ %eris-block-store-directory))
+
+(define %eris-block-store-directory
+ (make-parameter
+ (or (getenv "GUIX_ERIS_BLOCK_STORE_DIRECTORY")
+ (string-append %state-directory "/eris"))))
+
+(define (guix-eris-block-reducer)
+ "Returns a block reducer that stores blocks of ERIS encoded content."
+ (eris-fs-store-reducer (%eris-block-store-directory)))
diff --git a/guix/eris/fs-store.scm b/guix/eris/fs-store.scm
new file mode 100644
index 0000000000..2ef7607988
--- /dev/null
+++ b/guix/eris/fs-store.scm
@@ -0,0 +1,67 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 pukkamustard <pukkamustard@posteo.net>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix eris fs-store)
+ #:use-module (rnrs io ports)
+ #:use-module (guix build utils) ; for mkdir-p
+ #:use-module (eris utils base32)
+
+ #:export (eris-fs-store-reducer
+ eris-fs-store-ref))
+
+;;; Commentary:
+;;;
+;;; This module provides a file-system based store of ERIS encoded blocks.
+;;;
+;;; Code:
+
+(define (eris-fs-store-reducer store-directory)
+ (case-lambda
+ (() (mkdir-p store-directory))
+
+ ((result) result)
+
+ ((_ ref-block)
+ (let* ((ref (car ref-block))
+ (b32 (base32-encode ref))
+ (pre (substring b32 0 2))
+ (suf (substring b32 2))
+ (pre-dir (string-append store-directory "/" pre))
+ (path (string-append pre-dir "/" suf))
+ (block (cdr ref-block)))
+
+ (mkdir-p pre-dir)
+
+ (unless (file-exists? path)
+ (call-with-output-file path
+ (lambda (port) (put-bytevector port block))
+ #:binary #t))
+
+ #t))))
+
+(define (eris-fs-store-ref store-directory)
+ (lambda (ref)
+ (let* ((b32 (base32-encode ref))
+ (pre (substring b32 0 2))
+ (suf (substring b32 2))
+ (path (string-append store-directory "/" pre "/" suf)))
+ (if (file-exists? path)
+ (call-with-input-file path
+ (lambda (port) (get-bytevector-all port))
+ #:binary #t)
+ #f))))
diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm
index 76b5a429f4..7f14e4d4d4 100644
--- a/guix/scripts/publish.scm
+++ b/guix/scripts/publish.scm
@@ -52,6 +52,7 @@ (define-module (guix scripts publish)
#:use-module (guix base64)
#:use-module (guix config)
#:use-module (guix derivations)
+ #:use-module (guix eris)
#:use-module (gcrypt hash)
#:use-module (guix pki)
#:use-module (gcrypt pk-crypto)
@@ -647,12 +648,13 @@ (define (eris-encode-nar compressions)
(and stat
(call-with-input-file nar
(lambda (port)
- (let ((eris-urn _
- (eris-encode port
- #:block-size 'large
- #:block-reducer rcount
- #:convergence-secret
- %null-convergence-secret)))
+ (let ((eris-urn
+ _ (eris-encode port
+ #:block-size 'large
+ #:block-reducer
+ (guix-eris-block-reducer)
+ #:convergence-secret
+ %null-convergence-secret)))
eris-urn)))))))
(let ((compression (actual-compressions item compressions)))
--
2.38.1
P
P
pukkamustard wrote on 29 Dec 2022 10:13
[PATCH v3 3/8] publish: Add HTTP endpoint for resolving ERIS blocks.
(address . 52555@debbugs.gnu.org)(name . pukkamustard)(address . pukkamustard@posteo.net)
20221229181327.758-4-pukkamustard@posteo.net
* guix/eris/http.scm: New file.
* Makefile.am (MODULES): Add it.
* guix/scripts/publish.scm (make-request-handler): Add handler for RFC 2169
URN resolution endpoint.
---
Makefile.am | 1 +
guix/eris/http.scm | 40 ++++++++++++++++++++++++++++++++++++++++
guix/scripts/publish.scm | 20 ++++++++++++++++++++
3 files changed, 61 insertions(+)
create mode 100644 guix/eris/http.scm

Toggle diff (105 lines)
diff --git a/Makefile.am b/Makefile.am
index c549fc8580..373f6b7c27 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -134,6 +134,7 @@ MODULES = \
guix/ipfs.scm \
guix/eris.scm \
guix/eris/fs-store.scm \
+ guix/eris/http.scm \
guix/platform.scm \
guix/platforms/arm.scm \
guix/platforms/mips.scm \
diff --git a/guix/eris/http.scm b/guix/eris/http.scm
new file mode 100644
index 0000000000..a8a9520197
--- /dev/null
+++ b/guix/eris/http.scm
@@ -0,0 +1,40 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 pukkamustard <pukkamustard@posteo.net>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix eris http)
+ #:use-module (eris utils base32)
+ #:use-module (web uri)
+
+ #:use-module (srfi srfi-2)
+
+ #:export (make-blake2b-urn-query-ref))
+
+(define blake2b-uri-path-regexp
+ (make-regexp "^blake2b:" regexp/icase))
+
+(define (make-blake2b-urn-query-ref block-ref)
+ (lambda (query)
+ (and-let* ((_ query) ; ensure query is not false
+ (urn (string->uri query))
+ (_ (uri? urn))
+ (_ (eqv? (uri-scheme urn) 'urn))
+ (_ (regexp-exec blake2b-uri-path-regexp
+ (uri-path urn)))
+ (blake2b-ref (base32-decode
+ (string-drop (uri-path urn) 8))))
+ (block-ref blake2b-ref))))
diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm
index 7f14e4d4d4..15bdf02670 100644
--- a/guix/scripts/publish.scm
+++ b/guix/scripts/publish.scm
@@ -53,6 +53,8 @@ (define-module (guix scripts publish)
#:use-module (guix config)
#:use-module (guix derivations)
#:use-module (guix eris)
+ #:use-module (guix eris fs-store)
+ #:use-module (guix eris http)
#:use-module (gcrypt hash)
#:use-module (guix pki)
#:use-module (gcrypt pk-crypto)
@@ -1083,6 +1085,14 @@ (define nar-path?
(let ((expected (split-and-decode-uri-path nar-path)))
(cut equal? expected <>)))
+ ;; Get ERIS blocks directly from the filesystem store.
+ (define eris-block-ref
+ (eris-fs-store-ref (%eris-block-store-directory)))
+
+ ;; Create a handler for resolving blake2b URN queries.
+ (define blake2b-urn-query-ref
+ (make-blake2b-urn-query-ref eris-block-ref))
+
(define (handle request body)
(format #t "~a ~a~%"
(request-method request)
@@ -1125,6 +1135,16 @@ (define (handle request body)
(("log" name)
(render-log-file store request name))
+ ;; /uri-res/N2R - RFC2169 URN resolution
+ (("uri-res" "N2R")
+ (let ((block (blake2b-urn-query-ref
+ (uri-query (request-uri request)))))
+ (if block
+ (values `((content-type . (application/octet-stream
+ (charset . "ISO-8859-1"))))
+ block)
+ (not-found request))))
+
;; Use different URLs depending on the compression type. This
;; guarantees that /nar URLs remain valid even when 'guix publish'
;; is restarted with different compression parameters.
--
2.38.1
P
P
pukkamustard wrote on 29 Dec 2022 10:13
[PATCH v3 5/8] eris/http: Add HTTP block de-referencer.
(address . 52555@debbugs.gnu.org)(name . pukkamustard)(address . pukkamustard@posteo.net)
20221229181327.758-6-pukkamustard@posteo.net
* guix/eris/http.scm (eris-http-block-ref): New procedure.
---
guix/eris/http.scm | 46 ++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 44 insertions(+), 2 deletions(-)

Toggle diff (66 lines)
diff --git a/guix/eris/http.scm b/guix/eris/http.scm
index a8a9520197..354d3be3be 100644
--- a/guix/eris/http.scm
+++ b/guix/eris/http.scm
@@ -18,11 +18,17 @@
(define-module (guix eris http)
#:use-module (eris utils base32)
+
#:use-module (web uri)
+ #:use-module (web client)
+ #:use-module (web response)
- #:use-module (srfi srfi-2)
+ #:use-module (rnrs base)
+ #:use-module (srfi srfi-2) ; and-let*
+ #:use-module (srfi srfi-71) ; extended let-syntax for multiple values
- #:export (make-blake2b-urn-query-ref))
+ #:export (make-blake2b-urn-query-ref
+ eris-http-block-ref))
(define blake2b-uri-path-regexp
(make-regexp "^blake2b:" regexp/icase))
@@ -38,3 +44,39 @@ (define (make-blake2b-urn-query-ref block-ref)
(blake2b-ref (base32-decode
(string-drop (uri-path urn) 8))))
(block-ref blake2b-ref))))
+
+(define (make-blake2b-urn ref)
+ (string-append "urn:blake2b:" (base32-encode ref)))
+
+(define (make-request-uri host ref)
+ (build-uri (uri-scheme host)
+ #:userinfo (uri-userinfo host)
+ #:host (uri-host host)
+ #:port (uri-port host)
+ #:path "/uri-res/N2R"
+ #:query (make-blake2b-urn ref)))
+
+(define* (eris-http-block-ref
+ ref
+ #:key host
+ (open-connection (lambda (host) (open-socket-for-uri host))))
+
+ (let* ((uri (make-request-uri host ref))
+ (_ (format #t "URI: ~a\n" uri))
+ (response body
+ (http-get uri
+ #:decode-body? #f
+ #:port (open-connection host)
+ #:keep-alive? #t)))
+ (if (eqv? (response-code response) 200)
+ body
+ #f)))
+
+;; (use-modules (eris)
+;; (rnrs bytevectors))
+
+;; (bytevector?
+;; (eris-http-block-ref
+;; (base32-decode
+;; "4ACQM2Q5IB3DBHGKHM2WRTAXDBPMVZ7F6MG2TUXZ5QOJHW7P4N7Q")
+;; #:host (string->uri "http://localhost:8081")))
--
2.38.1
P
P
pukkamustard wrote on 29 Dec 2022 10:13
[PATCH v3 4/8] WIP: substitute: Fetch substitutes using ERIS.
(address . 52555@debbugs.gnu.org)(name . pukkamustard)(address . pukkamustard@posteo.net)
20221229181327.758-5-pukkamustard@posteo.net
* guix/scripts/substitute.scm (process-substitution): Fetch substitutes using ERIS.
* guix/eris.scm (call-with-eris-block-ref): New procedure.

TODO:

- When to set prefer-eris? in scripts/substitute.scm?
---
guix/eris.scm | 7 +++++++
guix/scripts/substitute.scm | 42 +++++++++++++++++++++++++++++--------
2 files changed, 40 insertions(+), 9 deletions(-)

Toggle diff (125 lines)
diff --git a/guix/eris.scm b/guix/eris.scm
index 29d5e7b1db..d9a0914b67 100644
--- a/guix/eris.scm
+++ b/guix/eris.scm
@@ -23,6 +23,7 @@ (define-module (guix eris)
#:use-module (guix eris fs-store)
#:export (guix-eris-block-reducer
+ call-with-eris-block-ref
%eris-block-store-directory))
@@ -34,3 +35,9 @@ (define %eris-block-store-directory
(define (guix-eris-block-reducer)
"Returns a block reducer that stores blocks of ERIS encoded content."
(eris-fs-store-reducer (%eris-block-store-directory)))
+
+(define (call-with-eris-block-ref f)
+ (let ((fs-store-block-ref
+ (eris-fs-store-ref
+ (%eris-block-store-directory))))
+ (f fs-store-block-ref)))
diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
index 0efa61b0d7..8cf011d7e6 100755
--- a/guix/scripts/substitute.scm
+++ b/guix/scripts/substitute.scm
@@ -55,12 +55,16 @@ (define-module (guix scripts substitute)
#:use-module (ice-9 ftw)
#:use-module (rnrs bytevectors)
#:use-module (srfi srfi-1)
+ #:use-module (srfi srfi-2)
#:use-module (srfi srfi-19)
#:use-module (srfi srfi-26)
#:use-module (srfi srfi-34)
#:use-module (srfi srfi-35)
#:use-module (srfi srfi-71)
#:use-module (web uri)
+ #:use-module (eris)
+ #:use-module (eris read-capability)
+ #:use-module (guix eris)
#:use-module (guix http-client)
#:export (%allow-unauthenticated-substitutes?
%reply-file-descriptor
@@ -439,11 +443,12 @@ (define-syntax-rule (with-cached-connection uri port exp ...)
(define* (download-nar narinfo destination
#:key status-port
- deduplicate? print-build-trace?)
+ deduplicate? print-build-trace?
+ prefer-eris?)
"Download the nar prescribed in NARINFO, which is assumed to be authentic
and authorized, and write it to DESTINATION. When DEDUPLICATE? is true, and
-if DESTINATION is in the store, deduplicate its files. Print a status line to
-STATUS-PORT."
+if DESTINATION is in the store, deduplicate its files. When PREFER-ERIS? is
+true, attempt to ERIS to get the nar. Print a status line to STATUS-PORT."
(define destination-in-store?
(string-prefix? (string-append (%store-prefix) "/")
destination))
@@ -474,14 +479,29 @@ (define (fetch uri)
#:port port
#:keep-alive? #t
#:buffered? #f))))
+ ((urn)
+ (let ((read-capability (->eris-read-capability uri)))
+ (if (eris-read-capability? read-capability)
+ (call-with-eris-block-ref
+ (lambda (block-ref)
+ (values (open-eris-input-port
+ read-capability
+ #:block-ref block-ref)
+ #f)))
+ (leave (G_ "unsupported substitute URI scheme: ~a~%")
+ (uri->string uri)))))
(else
(leave (G_ "unsupported substitute URI scheme: ~a~%")
(uri->string uri)))))
(let ((uri compression file-size
- (narinfo-best-uri narinfo
- #:fast-decompression?
- %prefer-fast-decompression?)))
+ (if (and prefer-eris? (narinfo-eris-urn narinfo))
+ (values (narinfo-eris-urn narinfo) "zstd" #f)
+ (narinfo-best-uri narinfo
+ #:fast-decompression?
+ %prefer-fast-decompression?))))
+
+
(unless print-build-trace?
(format (current-error-port)
(G_ "Downloading ~a...~%") (uri->string uri)))
@@ -631,7 +651,8 @@ (define* (process-substitution/fallback port narinfo destination
(define* (process-substitution port store-item destination
#:key cache-urls acl
- deduplicate? print-build-trace?)
+ deduplicate? print-build-trace?
+ prefer-eris?)
"Substitute STORE-ITEM (a store file name) from CACHE-URLS, and write it to
DESTINATION as a nar file. Verify the substitute against ACL, and verify its
hash against what appears in the narinfo. When DEDUPLICATE? is true, and if
@@ -660,7 +681,8 @@ (define narinfo
(download-nar narinfo destination
#:status-port port
#:deduplicate? deduplicate?
- #:print-build-trace? print-build-trace?)))
+ #:print-build-trace? print-build-trace?
+ #:prefer-eris? prefer-eris?)))
;;;
@@ -858,7 +880,9 @@ (define reply-port
#:acl (current-acl)
#:deduplicate? deduplicate?
#:print-build-trace?
- print-build-trace?)
+ print-build-trace?
+ ;; TODO when to prefer ERIS?
+ #:prefer-eris? #t)
(loop))))))
(opts
(leave (G_ "~a: unrecognized options~%") opts))))))
--
2.38.1
P
P
pukkamustard wrote on 29 Dec 2022 10:13
[PATCH v3 7/8] eris: Use parameterized %eris-peers when getting blocks.
(address . 52555@debbugs.gnu.org)(name . pukkamustard)(address . pukkamustard@posteo.net)
20221229181327.758-8-pukkamustard@posteo.net
---
guix/eris.scm | 25 ++++++++++++++++++++-----
1 file changed, 20 insertions(+), 5 deletions(-)

Toggle diff (42 lines)
diff --git a/guix/eris.scm b/guix/eris.scm
index 4af17c2807..d56643bec4 100644
--- a/guix/eris.scm
+++ b/guix/eris.scm
@@ -54,15 +54,30 @@ (define* (try-in-order ref #:key block-refs)
(try-in-order ref #:block-refs rest))))
(() #f)))
+(define* (peer->block-ref peer #:key open-connection)
+ (case (uri-scheme peer)
+
+ ((http https)
+ (lambda (ref)
+ (eris-http-block-ref ref
+ #:host peer
+ #:open-connection open-connection)))
+
+ ;; unsupported ERIS peer URL
+ (else (lambda (_) #f))))
+
(define* (guix-eris-block-ref ref #:key open-connection)
(try-in-order
ref
#:block-refs
- (list
+ (cons
+
+ ;; first try and get block from local block store
(lambda (ref)
(eris-fs-store-ref ref
#:store-directory (%eris-block-store-directory)))
- (lambda (ref)
- (eris-http-block-ref ref
- #:host (string->uri "http://localhost:8081")
- #:open-connection open-connection)))))
+
+ ;; then try peers
+ (map (lambda (peer)
+ (peer->block-ref peer #:open-connection open-connection))
+ (%eris-peers)))))
--
2.38.1
P
P
pukkamustard wrote on 29 Dec 2022 10:13
[PATCH v3 6/8] WIP: eris: Use HTTP to get ERIS blocks.
(address . 52555@debbugs.gnu.org)(name . pukkamustard)(address . pukkamustard@posteo.net)
20221229181327.758-7-pukkamustard@posteo.net
- guix/eris.scm (guix-eris-block-ref): Refactor from call-with-eris-block-ref
and use eris-http-block-ref.
- guix/eris/fs-store.scm (eris-fs-store-ref): Refactor to take keyword
arguments.
- guix/scripts/substitute.scm (download-nar): Use guix-eris-block-ref.
- guix/scripts/publish.scm (make-request-handler): Use refactored
eris-fs-store-ref.

TODO:

- default value for %eris-peers parameter in (eris)
---
guix/eris.scm | 37 +++++++++++++++++++++++++++++++------
guix/eris/fs-store.scm | 21 ++++++++++-----------
guix/scripts/publish.scm | 9 ++++-----
guix/scripts/substitute.scm | 14 ++++++++------
4 files changed, 53 insertions(+), 28 deletions(-)

Toggle diff (132 lines)
diff --git a/guix/eris.scm b/guix/eris.scm
index d9a0914b67..4af17c2807 100644
--- a/guix/eris.scm
+++ b/guix/eris.scm
@@ -21,9 +21,13 @@ (define-module (guix eris)
#:use-module (guix config)
#:use-module (guix eris fs-store)
+ #:use-module (guix eris http)
+
+ #:use-module (web uri)
+ #:use-module (ice-9 match)
#:export (guix-eris-block-reducer
- call-with-eris-block-ref
+ guix-eris-block-ref
%eris-block-store-directory))
@@ -36,8 +40,29 @@ (define (guix-eris-block-reducer)
"Returns a block reducer that stores blocks of ERIS encoded content."
(eris-fs-store-reducer (%eris-block-store-directory)))
-(define (call-with-eris-block-ref f)
- (let ((fs-store-block-ref
- (eris-fs-store-ref
- (%eris-block-store-directory))))
- (f fs-store-block-ref)))
+(define %eris-peers
+ (make-parameter
+ ;; TODO
+ (list (string->uri "http://localhost:8081"))))
+
+(define* (try-in-order ref #:key block-refs)
+ (match block-refs
+ ((block-ref . rest)
+ (let ((block (block-ref ref)))
+ (if block
+ block
+ (try-in-order ref #:block-refs rest))))
+ (() #f)))
+
+(define* (guix-eris-block-ref ref #:key open-connection)
+ (try-in-order
+ ref
+ #:block-refs
+ (list
+ (lambda (ref)
+ (eris-fs-store-ref ref
+ #:store-directory (%eris-block-store-directory)))
+ (lambda (ref)
+ (eris-http-block-ref ref
+ #:host (string->uri "http://localhost:8081")
+ #:open-connection open-connection)))))
diff --git a/guix/eris/fs-store.scm b/guix/eris/fs-store.scm
index 2ef7607988..38f5926280 100644
--- a/guix/eris/fs-store.scm
+++ b/guix/eris/fs-store.scm
@@ -54,14 +54,13 @@ (define (eris-fs-store-reducer store-directory)
#t))))
-(define (eris-fs-store-ref store-directory)
- (lambda (ref)
- (let* ((b32 (base32-encode ref))
- (pre (substring b32 0 2))
- (suf (substring b32 2))
- (path (string-append store-directory "/" pre "/" suf)))
- (if (file-exists? path)
- (call-with-input-file path
- (lambda (port) (get-bytevector-all port))
- #:binary #t)
- #f))))
+(define* (eris-fs-store-ref ref #:key store-directory)
+ (let* ((b32 (base32-encode ref))
+ (pre (substring b32 0 2))
+ (suf (substring b32 2))
+ (path (string-append store-directory "/" pre "/" suf)))
+ (if (file-exists? path)
+ (call-with-input-file path
+ (lambda (port) (get-bytevector-all port))
+ #:binary #t)
+ #f)))
diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm
index 15bdf02670..0ce50b2942 100644
--- a/guix/scripts/publish.scm
+++ b/guix/scripts/publish.scm
@@ -1085,13 +1085,12 @@ (define nar-path?
(let ((expected (split-and-decode-uri-path nar-path)))
(cut equal? expected <>)))
- ;; Get ERIS blocks directly from the filesystem store.
- (define eris-block-ref
- (eris-fs-store-ref (%eris-block-store-directory)))
-
;; Create a handler for resolving blake2b URN queries.
(define blake2b-urn-query-ref
- (make-blake2b-urn-query-ref eris-block-ref))
+ (make-blake2b-urn-query-ref
+ (lambda (ref) (eris-fs-store-ref
+ ref
+ #:store-directory (%eris-block-store-directory)))))
(define (handle request body)
(format #t "~a ~a~%"
diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
index 8cf011d7e6..14a21f6c37 100755
--- a/guix/scripts/substitute.scm
+++ b/guix/scripts/substitute.scm
@@ -482,12 +482,14 @@ (define (fetch uri)
((urn)
(let ((read-capability (->eris-read-capability uri)))
(if (eris-read-capability? read-capability)
- (call-with-eris-block-ref
- (lambda (block-ref)
- (values (open-eris-input-port
- read-capability
- #:block-ref block-ref)
- #f)))
+ (values
+ (open-eris-input-port
+ read-capability
+ #:block-ref (lambda (ref)
+ (guix-eris-block-ref
+ ref
+ #:open-connection open-connection-for-uri/cached)))
+ #f)
(leave (G_ "unsupported substitute URI scheme: ~a~%")
(uri->string uri)))))
(else
--
2.38.1
P
P
pukkamustard wrote on 29 Dec 2022 10:13
[PATCH v3 8/8] eris: Use IPFS to get ERIS blocks.
(address . 52555@debbugs.gnu.org)(name . pukkamustard)(address . pukkamustard@posteo.net)
20221229181327.758-9-pukkamustard@posteo.net
* guix/eris/ipfs.scm: New files.
* Makefile.am (MODULES): Add it.
* guix/eris.scm (%eris-peers): Add IPFS.
(peer->block-ref): Handle IPFS peer.
---
Makefile.am | 1 +
guix/eris.scm | 32 ++++---
guix/eris/ipfs.scm | 214 +++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 237 insertions(+), 10 deletions(-)
create mode 100644 guix/eris/ipfs.scm

Toggle diff (295 lines)
diff --git a/Makefile.am b/Makefile.am
index 373f6b7c27..6f648a40a3 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -135,6 +135,7 @@ MODULES = \
guix/eris.scm \
guix/eris/fs-store.scm \
guix/eris/http.scm \
+ guix/eris/ipfs.scm \
guix/platform.scm \
guix/platforms/arm.scm \
guix/platforms/mips.scm \
diff --git a/guix/eris.scm b/guix/eris.scm
index d56643bec4..5b0c1ee36b 100644
--- a/guix/eris.scm
+++ b/guix/eris.scm
@@ -22,6 +22,7 @@ (define-module (guix eris)
#:use-module (guix config)
#:use-module (guix eris fs-store)
#:use-module (guix eris http)
+ #:use-module (guix eris ipfs)
#:use-module (web uri)
#:use-module (ice-9 match)
@@ -42,8 +43,10 @@ (define (guix-eris-block-reducer)
(define %eris-peers
(make-parameter
- ;; TODO
- (list (string->uri "http://localhost:8081"))))
+ ;; TODO: make ERIS peers configurable somewhere
+ (list
+ (string->uri "http://localhost:8081")
+ 'ipfs)))
(define* (try-in-order ref #:key block-refs)
(match block-refs
@@ -55,18 +58,27 @@ (define* (try-in-order ref #:key block-refs)
(() #f)))
(define* (peer->block-ref peer #:key open-connection)
- (case (uri-scheme peer)
+ (cond
+ ((uri? peer) (case (uri-scheme peer)
- ((http https)
- (lambda (ref)
- (eris-http-block-ref ref
- #:host peer
- #:open-connection open-connection)))
+ ((http https)
+ (lambda (ref)
+ (eris-http-block-ref ref
+ #:host peer
+ #:open-connection open-connection)))
- ;; unsupported ERIS peer URL
- (else (lambda (_) #f))))
+ ;; unsupported ERIS peer URL
+ (else (lambda (_) #f))))
+
+ ((eqv? 'ipfs peer)
+ (lambda (ref)
+ (eris-ipfs-ref ref #:open-connection open-connection)))))
(define* (guix-eris-block-ref ref #:key open-connection)
+ "Attempts to dereference a block of some ERIS encoded content with reference
+REF. First the local block store is checked, followed by remote peers as
+configured in the parameter %eris-peers (in order). Returns #f if the block
+could not be de-referenced."
(try-in-order
ref
#:block-refs
diff --git a/guix/eris/ipfs.scm b/guix/eris/ipfs.scm
new file mode 100644
index 0000000000..9771414e7b
--- /dev/null
+++ b/guix/eris/ipfs.scm
@@ -0,0 +1,214 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2018 Ludovic Courtès <ludo@gnu.org>
+;;; Copyright © 2022 pukkamustard <pukkamustard@posteo.net>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+
+;;; Commentary:
+;;;
+;;; This module provides an interface to the IPFS daemons HTTP API for storing
+;;; and retrieving blocks. This can be used to store blocks of ERIS encoded
+;;; content.
+;;;
+;;; See also the IPFS API documentation:
+;;; https://docs.ipfs.io/reference/http/api/#api-v0-block-put
+
+(define-module (guix eris ipfs)
+ #:use-module (eris utils base32)
+ #:use-module (sodium generichash)
+ #:use-module (json)
+ #:use-module (web uri)
+ #:use-module (web client)
+ #:use-module (web response)
+ #:use-module (srfi srfi-71)
+ #:use-module (rnrs io ports)
+ #:use-module (rnrs bytevectors)
+
+ #:use-module ((guix build download)
+ #:select ((open-connection-for-uri
+ . guix:open-connection-for-uri)))
+ #:export (%ipfs-base-url
+
+ eris-ipfs-reducer
+ eris-ipfs-ref))
+
+
+;; CID encoding
+
+;; Multicodec codes
+;; (https://github.com/multiformats/multicodec/blob/master/table.csv)
+(define multicodec-raw-code #x55)
+(define multicodec-blake2b-256-code #xb220)
+
+(define (blake2b-256->binary-cid hash)
+ "Encode a Blake2b-256 hash as binary CID"
+ (call-with-values
+ (lambda () (open-bytevector-output-port))
+ (lambda (port get-bytevector)
+ ;; CID version
+ (put-u8 port 1)
+ ;; multicoded content-type
+ (put-u8 port multicodec-raw-code)
+ ;; set multihash to blake2b-256. This is the manually encoded varint of
+ ;; 0xb220
+ (put-u8 port 160) (put-u8 port 228) (put-u8 port 2)
+ ;; set hash lenght
+ (put-u8 port 32)
+ ;; and finally the hash itself
+ (put-bytevector port hash)
+
+ ;; finalize and get the bytevector
+ (get-bytevector))))
+
+(define (binary-cid->cid bcid)
+ "Encode a binary CID as Base32 encoded CID"
+ ;; 'b' is the multibsae code for base32
+ (string-append "b"
+ ;; the IPFS daemon uses lower-case, so to be consistent we
+ ;; also.
+ (string-downcase
+ ;; base32 encode the binary cid
+ (base32-encode bcid))))
+
+(define blake2b-256->cid
+ (compose binary-cid->cid blake2b-256->binary-cid))
+
+
+;; IPFS API
+
+(define %ipfs-base-url
+ ;; URL of the IPFS gateway.
+ (make-parameter "http://localhost:5001"))
+
+(define* (call url decode
+ #:optional
+ (method http-post)
+ #:key port body (false-if-404? #t) (headers '())
+ (keep-alive #t)
+ (open-connection guix:open-connection-for-uri)
+ (timeout 10))
+ "Invoke the endpoint at URL using METHOD. Decode the resulting JSON body
+using DECODE, a one-argument procedure that takes an input port; when DECODE
+is false, return the input port. When FALSE-IF-404? is true, return #f upon
+404 responses."
+ (let* ((url (if (string? url) (string->uri url) url))
+ (port (or port (open-connection url #:timeout timeout)))
+ (response response-port
+ (if keep-alive
+ (method url #:streaming? #t
+ #:body body
+ #:port port
+ #:keep-alive? #t)
+ (method url #:streaming? #t
+ #:body body
+ #:port port
+ ;; IPFS daemon seems to responds with bad
+ ;; request if PUT requests are kept alive and
+ ;; do not have "Connection: close" header.
+ #:keep-alive? #f
+ #:headers `((connection close)
+ ,@headers)))))
+ (cond ((= 200 (response-code response))
+ (if decode
+ (let ((result (decode response-port)))
+ (close-port response-port)
+ result)
+ response-port))
+ ((and false-if-404?
+ (= 404 (response-code response)))
+ (close-port response-port)
+ #f)
+ (else
+ (close-port response-port)
+ (format #t "~a\n" response)
+ (throw 'ipfs-error url response)))))
+
+(define-syntax-rule (false-if-ipfs-error exp)
+ "Return $f if EXP triggers a network related or IPFS related exception."
+ (with-exception-handler
+ (lambda (exn)
+ (let ((kind (exception-kind exn))
+ (errno (system-error-errno
+ (cons 'system-error (exception-args exn)))))
+ (cond
+ ((= errno ECONNREFUSED) #f)
+ (else (raise-exception exp)))))
+ (lambda () exp)
+ #:unwind? #t))
+
+(define %multipart-boundary
+ ;; XXX: We might want to find a more reliable boundary.
+ (string-append (make-string 24 #\-) "2698127afd7425a6"))
+
+(define (bytevector->form-data bv port)
+ "Write to PORT a 'multipart/form-data' representation of BV."
+ (display (string-append "--" %multipart-boundary "\r\n"
+ "Content-Disposition: form-data\r\n"
+ "Content-Type: application/octet-stream\r\n\r\n")
+ port)
+ (put-bytevector port bv)
+ (display (string-append "\r\n--" %multipart-boundary "--\r\n")
+ port))
+
+(define (ipfs-block-put bv)
+ "Store a block on IPFS and return the CID of the block"
+ (call (string-append (%ipfs-base-url)
+ "/api/v0/block/put"
+ "?format=raw&mhtype=blake2b-256")
+ (lambda (port) (assoc-ref (json->scm port) "Key"))
+ #:headers `((content-type
+ . (multipart/form-data
+ (boundary . ,%multipart-boundary))))
+ #:body (call-with-bytevector-output-port
+ (lambda (port) (bytevector->form-data bv port)))
+ ;; IPFS daemon does not seem to accept connection re-use when putting
+ ;; blocks.
+ #:keep-alive #f))
+
+(define* (ipfs-block-get cid #:key
+ (open-connection guix:open-connection-for-uri))
+ "Get a block from IPFS via the HTTP API"
+ (false-if-ipfs-error
+ (call (string-append (%ipfs-base-url)
+ "/api/v0/block/get"
+ "?arg=" cid)
+ get-bytevector-all
+ #:timeout 5
+ #:open-connection open-connection)))
+
+;; ERIS block reducer
+
+(define eris-ipfs-reducer
+ (case-lambda
+ ;; initialization. Nothing to do here. In an improved implementation we
+ ;; might create a single HTTP connection and reuse it for all blocks.
+ (() '())
+
+ ;; Completion. Again, nothing to do.
+ ((_) 'done)
+
+ ;; store a block
+ ((_ ref-block)
+ ;; ref-block is a pair consisting of the reference to the block and the
+ ;; block itself.
+ (ipfs-block-put (cdr ref-block)))))
+
+(define* (eris-ipfs-ref ref #:key
+ (open-connection guix:open-connection-for-uri))
+ "Dereference a block from IPFS"
+ (ipfs-block-get (blake2b-256->cid ref)
+ #:open-connection open-connection))
--
2.38.1
L
L
Ludovic Courtès wrote on 14 Jan 2023 10:25
Re: bug#52555: [RFC PATCH 0/3] Decentralized substitute distribution with ERIS
(name . pukkamustard)(address . pukkamustard@posteo.net)(address . 52555@debbugs.gnu.org)
87bkn17z2i.fsf_-_@gnu.org
Hello!

pukkamustard <pukkamustard@posteo.net> skribis:

Toggle quote (4 lines)
> I'm very happy to present a V3 of a proposal towards decentralizing substitute
> distribution with ERIS. An initial version [1] and a V2 [2] are now almost a
> years old!

Woohoo! 👍

Toggle quote (13 lines)
> The idea is to use ERIS (Encoding for Robust Immutable Storage) [3] to allow
> more decentralized substitute distribution. ERIS defines an encoding of
> content into uniformly sized, encrypted and content-addressed blocks
> (32KiB). The content can be decoded from the blocks given a short identifier
> called the read capability. It allows a network-optimized form of
> content-addressing.
>
> Blocks can be transported over many different transport protocols, such as
> HTTP, CoAP, GNUnet, IPFS or a SD card sent via pigeons. Only the read
> capability must be transmited securely. Given an authentic read capability,
> the content can be decoded correctly from blocks that might have been
> transported over unreliable (and untrusted) protocols.

Neat! (Do we have an implementation yet for SD-cards-over-pigeons?)

Toggle quote (7 lines)
> For Guix, substitutes (Nar files) are encoded using ERIS and the read
> capability is added to the signed Narinfos. The read capability published in
> the Narinfo can then be used to decode the Nar from blocks that are fetched
> from many peers over many different protocols.
>
> This version of the patches allows blocks to be fetched over HTTP and IPFS.

Nice.

Toggle quote (13 lines)
> A summary of this patch series:
>
> - Use the stable version of the ERIS encoding (version 1.0.0)
> - Add two fields to Narinfos (ERIS and ERISFormat)
> - Store blocks of published substitutes in a local block store (in
> `/var/guix/eris`)
> - Add an endpoint for resolving ERIS blocks over HTTP (a la RFC 2169)
> - Use ERIS when fetching substitutes
> - Use IPFS for de-referencing ERIS blocks
>
> Testing procedure is a bit tedious, but described in the V2:
> https://issues.guix.gnu.org/52555#8-lineno16

OK. There are still I guess a number of unknowns, including the cost of
ERIS support for publishers (how much disk space ‘guix publish’ will use
for blocks, how much CPU is needed to compute those blocks, bandwidth
usage over IFPS/HTTP) and for consumers (performance of substitution
over HTTP+ERIS or IPFS+ERIS CPU-wise and bandwidth-wise).

To address that, we’ll need to make it easy to test, and to make it easy
to disable it if things don’t work as expected.

(Perhaps this is already the case, I’m thinking out loud.)

Toggle quote (4 lines)
> Thanks for making it so far! :) I'd be very happy for your thoughs and
> ideas. There might also be opportunity to discuss these ideas at Guix Days in
> February.

Definitely! I think it would be great if you could present what you’re
up to and maybe make a quick demo.

Thanks for the great news to begin the year!

Ludo’.
L
L
Ludovic Courtès wrote on 14 Jan 2023 10:34
(name . pukkamustard)(address . pukkamustard@posteo.net)(address . 52555@debbugs.gnu.org)
877cxp7yox.fsf_-_@gnu.org
pukkamustard <pukkamustard@posteo.net> skribis:

Toggle quote (7 lines)
> * guix/scripts/publish.scm: (bake-narinfo+nar): Compute ERIS URN of compressed nars.
> (narinfo-string): Add #:eris-urn parameter and honor it.
> * guix/scripts/narinfo.scm: (<narinfo>)[eris-format,eris-urn]: New fields.
> (narinfo-maker): Handle ERIS URN and ERIS format.
> * configure.ac: (HAVE_GUILE_ERIS): New conditional.
> * gnu/packages/package-management.scm: (guix)[native-inputs]: Add guile-eris.

I’d suggest separating the (guix narinfo) part from the rest.

Toggle quote (9 lines)
> @@ -135,8 +139,8 @@ (define (narinfo-maker str cache-url)
> "Return a narinfo constructor for narinfos originating from CACHE-URL. STR
> must contain the original contents of a narinfo file."
> (lambda (path urls compressions file-hashes file-sizes
> - nar-hash nar-size references deriver system
> - signature)
> + nar-hash nar-size eris-format eris-urn references deriver
> + system signature)

Maybe make ‘eris-format’ and ‘eris-urn’ named parameters.

Toggle quote (16 lines)
> + (define (eris-encode-nar compressions)
> + (and (member %eris-zstd-compression compressions)
> + (let* ((nar (nar-cache-file cache item
> + #:compression %eris-zstd-compression))
> + (stat (stat nar #f)))
> + (and stat
> + (call-with-input-file nar
> + (lambda (port)
> + (let ((eris-urn _
> + (eris-encode port
> + #:block-size 'large
> + #:block-reducer rcount
> + #:convergence-secret
> + %null-convergence-secret)))
> + eris-urn)))))))

I think you remove the ‘let’ and just call ‘eris-encode’ in tail
position (Guile truncates multiple-value returns.)

Where is ‘eris-encode’ storing the blocks? I don’t see any parameter
here.

I guess it should go to /var/cache/guix/publish/eris or similar.

Toggle quote (21 lines)
> (let ((compression (actual-compressions item compressions)))
>
> (for-each (cut compress-nar cache item <>) compressions)
> @@ -640,7 +662,8 @@ (define (compressed-nar-size compression)
> (match compressions
> ((main others ...)
> (let ((narinfo (narinfo-cache-file cache item
> - #:compression main)))
> + #:compression main))
> + (eris-urn (eris-encode-nar compressions)))
> (with-atomic-file-output narinfo
> (lambda (port)
> ;; Open a new connection to the store. We cannot reuse the main
> @@ -651,7 +674,8 @@ (define (compressed-nar-size compression)
> (display (narinfo-string store item
> #:nar-path nar-path
> #:compressions compressions
> - #:file-sizes sizes)
> + #:file-sizes sizes
> + #:eris-urn eris-urn)

Would be nice to make ERIS encoding optional, with a command-line
switch to turn it on.

Ludo’.
L
L
Ludovic Courtès wrote on 14 Jan 2023 10:42
(name . pukkamustard)(address . pukkamustard@posteo.net)(address . 52555@debbugs.gnu.org)
87358d7yak.fsf_-_@gnu.org
pukkamustard <pukkamustard@posteo.net> skribis:

Toggle quote (5 lines)
> * guix/eris.scm: New file.
> * guix/eris/fs-store.scm: New file.
> * Makefile.am (MODULES): Add new files.
> * guix/scripts/publish.scm (bake-narinfo+nar): Use guix-eris-block-reducer.

[...]

Toggle quote (9 lines)
> +(define %eris-block-store-directory
> + (make-parameter
> + (or (getenv "GUIX_ERIS_BLOCK_STORE_DIRECTORY")
> + (string-append %state-directory "/eris"))))
> +
> +(define (guix-eris-block-reducer)
> + "Returns a block reducer that stores blocks of ERIS encoded content."
> + (eris-fs-store-reducer (%eris-block-store-directory)))

Maybe this should be private to (guix scripts publish)?

Also, the store directory should be /var/cache/guix/publish/eris by
default IMO.

Toggle quote (36 lines)
> +(define (eris-fs-store-reducer store-directory)
> + (case-lambda
> + (() (mkdir-p store-directory))
> +
> + ((result) result)
> +
> + ((_ ref-block)
> + (let* ((ref (car ref-block))
> + (b32 (base32-encode ref))
> + (pre (substring b32 0 2))
> + (suf (substring b32 2))
> + (pre-dir (string-append store-directory "/" pre))
> + (path (string-append pre-dir "/" suf))
> + (block (cdr ref-block)))
> +
> + (mkdir-p pre-dir)
> +
> + (unless (file-exists? path)
> + (call-with-output-file path
> + (lambda (port) (put-bytevector port block))
> + #:binary #t))
> +
> + #t))))
> +
> +(define (eris-fs-store-ref store-directory)
> + (lambda (ref)
> + (let* ((b32 (base32-encode ref))
> + (pre (substring b32 0 2))
> + (suf (substring b32 2))
> + (path (string-append store-directory "/" pre "/" suf)))
> + (if (file-exists? path)
> + (call-with-input-file path
> + (lambda (port) (get-bytevector-all port))
> + #:binary #t)
> + #f))))

Could you add docstrings, remove tabs, and use (ice-9 match) instead of
car/cdr? :-)

Whole files (blocks, right?) get loaded in memory. Is that OK or should
it be avoided?

There are time-of-check-to-time-of-use race conditions with those
‘file-exists?’ calls. In the case of ‘ref’, you can instead write:

(catch 'system-error
(lambda ()
(call-with-input-file …))
(lambda args
(if (= ENOENT (system-error-errno args))
#f
(apply throw args))))

In the case of ‘reducer’, perhaps it’d be safer to write the block
atomically with ‘with-atomic-file-output’? (And remove the
‘file-exists?’ check too?)

Write “file” rather than “path” (the latter is used to refer to “search
paths”). :-)

Ludo’.
P
P
pukkamustard wrote on 28 Dec 2023 01:40
[PATCH v4 1/7] narinfo: Add ERIS field.
(address . 52555@debbugs.gnu.org)
7c1c7108a369b608e8cebbe09c4cbfe24d032067.1703316055.git.pukkamustard@posteo.net
* guix/narinfo.scm (<narinfo>)[eris-urn]: New field.
(narinfo-maker): Handle new field.
(read-narifno): Handle new field.
(equivalent-narinfo?): Require ERIS field to be equal.
---
guix/narinfo.scm | 17 +++++++++++++----
1 file changed, 13 insertions(+), 4 deletions(-)

Toggle diff (72 lines)
diff --git a/guix/narinfo.scm b/guix/narinfo.scm
index a149d9a901..5f5ed8be6f 100644
--- a/guix/narinfo.scm
+++ b/guix/narinfo.scm
@@ -45,6 +45,7 @@ (define-module (guix narinfo)
narinfo-file-sizes
narinfo-hash
narinfo-size
+ narinfo-eris-urn
narinfo-references
narinfo-deriver
narinfo-system
@@ -69,7 +70,7 @@ (define-module (guix narinfo)
(define-record-type <narinfo>
(%make-narinfo path uri-base uris compressions file-sizes file-hashes
- nar-hash nar-size references deriver system
+ nar-hash nar-size eris-urn references deriver system
signature contents)
narinfo?
(path narinfo-path)
@@ -80,6 +81,7 @@ (define-record-type <narinfo>
(file-hashes narinfo-file-hashes)
(nar-hash narinfo-hash)
(nar-size narinfo-size)
+ (eris-urn narinfo-eris-urn)
(references narinfo-references)
(deriver narinfo-deriver)
(system narinfo-system)
@@ -136,7 +138,7 @@ (define (narinfo-maker str cache-url)
"Return a narinfo constructor for narinfos originating from CACHE-URL. STR
must contain the original contents of a narinfo file."
(lambda (path urls compressions file-hashes file-sizes
- nar-hash nar-size references deriver system
+ nar-hash nar-size eris-urn references deriver system
signature)
"Return a new <narinfo> object."
(define len (length urls))
@@ -158,6 +160,7 @@ (define (narinfo-maker str cache-url)
((lst ...) (map string->number lst)))
nar-hash
(and=> nar-size string->number)
+ (if eris-urn (string->uri eris-urn) #f)
(string-tokenize references)
(match deriver
((or #f "") #f)
@@ -185,7 +188,7 @@ (define* (read-narinfo port #:optional url
(narinfo-maker str url)
'("StorePath" "URL" "Compression"
"FileHash" "FileSize" "NarHash" "NarSize"
- "References" "Deriver" "System"
+ "ERIS" "References" "Deriver" "System"
"Signature")
'("URL" "Compression" "FileSize" "FileHash"))))
@@ -271,7 +274,13 @@ (define (equivalent-narinfo? narinfo1 narinfo2)
(narinfo-references narinfo2))
(= (narinfo-size narinfo1)
- (narinfo-size narinfo2))))
+ (narinfo-size narinfo2))
+
+ ;; When downloading substitutes with ERIS the hash is not checked. To
+ ;; avoid malicious substitutes we must ensure that the ERIS URN is equal.
+ ;; See also <https://issues.guix.gnu.org/52555#43>.
+ (equal? (narinfo-eris-urn narinfo1)
+ (narinfo-eris-urn narinfo2))))
(define %compression-methods
;; Known compression methods and a thunk to determine whether they're
--
2.41.0
P
P
pukkamustard wrote on 28 Dec 2023 01:40
[PATCH v4 2/7] gnu: Add guile-coap.
(address . 52555@debbugs.gnu.org)
d8b139db4128d495f50a70ef8df009762928409f.1703316055.git.pukkamustard@posteo.net
* gnu/packages/guile-xyz.scm (guile-coap): New variable.
---
gnu/packages/guile-xyz.scm | 32 ++++++++++++++++++++++++++++++++
1 file changed, 32 insertions(+)

Toggle diff (45 lines)
diff --git a/gnu/packages/guile-xyz.scm b/gnu/packages/guile-xyz.scm
index 7b44dc3167..2fc6d079a1 100644
--- a/gnu/packages/guile-xyz.scm
+++ b/gnu/packages/guile-xyz.scm
@@ -1457,6 +1457,38 @@ (define-public guile-aws
the Guile compiler tower to generate the DSL from AWS JSON specifications.")
(license license:gpl3+))))
+(define-public guile-coap
+ (let ((commit "1218d4f98210a14b52cf8185c9a7d39ed8b28643")
+ (revision "0"))
+ (package
+ (name "guile-coap")
+ (version (git-version "0.2.0-dev" revision commit))
+ (source
+ (origin
+ (method git-fetch)
+ (uri (git-reference
+ (url "https://codeberg.org/eris/guile-coap.git")
+ (commit commit)))
+ (file-name (git-file-name name version))
+ (sha256 (base32 "0mk9s4vzsi5y3sk8rs4a0jdcn6qj54nh7nwicdrsj77b9nghqwb3"))))
+ (build-system gnu-build-system)
+ (native-inputs
+ (list autoconf
+ automake
+ pkg-config
+ texinfo))
+ (inputs (list guile-3.0))
+ (propagated-inputs (list guile-fibers))
+ (synopsis "Guile implementation of the Constrained Application Protocol (CoAP)")
+ (description "Gulie-CoAP is a Guile implementation of the Constrained
+Application Protocol (CoAP). CoAP is a network transport protocol specialized
+for use with constrained nodes and constrained networks (e.g. low-power,
+lousy). This library implements basic serialization of CoAP messages over UDP
+(RFC 7252) and TCP (RFC 8323) as well as an asynchronous TCP client (using
+@code{guile-fibers})." )
+ (home-page "https://codeberg.org/eris/guile-coap")
+ (license license:gpl3+))))
+
(define-public guile-simple-zmq
(let ((commit "d25d1865e3378d93c44e2b4f5246a70b078a489d")
(revision "11"))
--
2.41.0
P
P
pukkamustard wrote on 28 Dec 2023 01:40
[PATCH v4 3/7] gnu: guile-eris: Update to 1.2.0-dev.
(address . 52555@debbugs.gnu.org)
a4344c72782e94f9ca228ff169827200dde06fae.1703316055.git.pukkamustard@posteo.net
* gnu/packages/guile-xyz.scm (guile-eris): Update to 1.2.0-dev.
[propagated-inputs]: Add guile-coap, guile-fibers, guile-sqlite3, guile-zstd
and guile-cbor.
---
gnu/packages/guile-xyz.scm | 66 +++++++++++++++++++++-----------------
1 file changed, 37 insertions(+), 29 deletions(-)

Toggle diff (82 lines)
diff --git a/gnu/packages/guile-xyz.scm b/gnu/packages/guile-xyz.scm
index 2fc6d079a1..a66a6d8ad5 100644
--- a/gnu/packages/guile-xyz.scm
+++ b/gnu/packages/guile-xyz.scm
@@ -5152,38 +5152,46 @@ (define-public guile-sodium
(license license:gpl3+)))
(define-public guile-eris
- (package
- (name "guile-eris")
- (version "1.0.0")
- (source
- (origin
- (method git-fetch)
- (uri (git-reference
- (url "https://codeberg.org/eris/guile-eris.git")
- (commit (string-append "v" version))))
- (file-name (git-file-name name version))
- (sha256 (base32 "0d4wbjwwaxk0zn5gjhl86qhvk1aisgzp1vnvy4xbvrv5ydqpgyqm"))))
- (build-system gnu-build-system)
- (arguments '())
- (native-inputs
- (list autoconf
- automake
- pkg-config
- texinfo
- ;; test dependency
- guile-srfi-180
- guile-quickcheck))
- (inputs (list guile-3.0))
- (propagated-inputs
- (list guile-sodium))
- (synopsis "Guile implementation of the Encoding for Robust Immutable Storage (ERIS)")
- (description
- "Guile-ERIS is a Guile implementation of the @url{http://purl.org/eris,
+ (let ((commit "98ec63cc04c7c59be8d09beed99a0ab1975829ce")
+ (revision "0"))
+ (package
+ (name "guile-eris")
+ (version (git-version "1.2.0-dev" revision commit))
+ (source
+ (origin
+ (method git-fetch)
+ (uri (git-reference
+ (url "https://codeberg.org/eris/guile-eris.git")
+ (commit commit)))
+ (file-name (git-file-name name version))
+ (sha256 (base32 "0n4crjqhj0ndni02xyrhwpzd325668vjbj3c1asy8wh2j50rvcs7"))))
+ (build-system gnu-build-system)
+ (arguments '())
+ (native-inputs
+ (list autoconf
+ automake
+ pkg-config
+ texinfo
+ ;; test dependency
+ guile-srfi-180
+ guile-quickcheck))
+ (inputs (list guile-3.0))
+ (propagated-inputs
+ (list guile-sodium
+ guile-coap
+ guile-fibers
+ guile-sqlite3
+ guile-zstd
+ guile-cbor))
+ (synopsis "Guile implementation of the Encoding for Robust Immutable
+Storage (ERIS)")
+ (description
+ "Guile-ERIS is a Guile implementation of the @url{http://purl.org/eris,
Encoding for Robust Immutable Storage (ERIS)}. ERIS allows arbitrary content
to be encoded into uniformly sized, encrypted blocks that can be reassembled
using a short read-capability.")
- (home-page "https://codeberg.org/eris/guile-eris")
- (license license:gpl3+)))
+ (home-page "https://codeberg.org/eris/guile-eris")
+ (license license:gpl3+))))
(define-public guile-r6rs-protobuf
(package
--
2.41.0
P
P
pukkamustard wrote on 28 Dec 2023 01:40
[PATCH v4 4/7] publish: Add ERIS URN to narinfo.
(address . 52555@debbugs.gnu.org)
000b47eb98b4f22a24f246cc12bb405c65efdf57.1703316055.git.pukkamustard@posteo.net
* guix/scripts/publish.scm (bake-narinfo+nar): Encode store item using ERIS.
(show-help, %options): Add '--eris'.
(guix-publish): Honor '--eris'.
* gnu/packages/package-management.scm (guix): Add guile-eris to native-inputs.
* guix/eris.scm: New file.
* Makefile.am (MODULES): Add new file.
---
Makefile.am | 1 +
configure.ac | 5 ++
gnu/packages/package-management.scm | 1 +
guix/eris.scm | 73 +++++++++++++++++++++++++++++
guix/scripts/publish.scm | 51 ++++++++++++++------
5 files changed, 117 insertions(+), 14 deletions(-)
create mode 100644 guix/eris.scm

Toggle diff (304 lines)
diff --git a/Makefile.am b/Makefile.am
index b64dcaa77c..86da4560e4 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -136,6 +136,7 @@ MODULES = \
guix/least-authority.scm \
guix/read-print.scm \
guix/ipfs.scm \
+ guix/eris.scm \
guix/platform.scm \
guix/platforms/arm.scm \
guix/platforms/avr.scm \
diff --git a/configure.ac b/configure.ac
index ecbd596a34..7e25099c4c 100644
--- a/configure.ac
+++ b/configure.ac
@@ -173,6 +173,11 @@ GUILE_MODULE_AVAILABLE([have_guile_avahi], [(avahi)])
AM_CONDITIONAL([HAVE_GUILE_AVAHI],
[test "x$have_guile_avahi" = "xyes"])
+dnl Check for Guile-ERIS.
+GUILE_MODULE_AVAILABLE([have_guile_eris], [(eris)])
+AM_CONDITIONAL([HAVE_GUILE_ERIS],
+ [test "x$have_guile_eris" = "xyes"])
+
dnl Guile-newt is used by the graphical installer.
GUILE_MODULE_AVAILABLE([have_guile_newt], [(newt)])
diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm
index 97ea41df66..9c0afb70dd 100644
--- a/gnu/packages/package-management.scm
+++ b/gnu/packages/package-management.scm
@@ -478,6 +478,7 @@ (define-public guix
("guile-zstd" ,guile-zstd)
("guile-ssh" ,guile-ssh)
("guile-git" ,guile-git)
+ ("guile-eris" ,guile-eris)
;; XXX: Keep the development inputs here even though
;; they're unnecessary, just so that 'guix environment
diff --git a/guix/eris.scm b/guix/eris.scm
new file mode 100644
index 0000000000..d98a9a62bd
--- /dev/null
+++ b/guix/eris.scm
@@ -0,0 +1,73 @@
+;;; GNU Guix --- Functional package management for GNU
+;;; Copyright © 2022 pukkamustard <pukkamustard@posteo.net>
+;;;
+;;; This file is part of GNU Guix.
+;;;
+;;; GNU Guix is free software; you can redistribute it and/or modify it
+;;; under the terms of the GNU General Public License as published by
+;;; the Free Software Foundation; either version 3 of the License, or (at
+;;; your option) any later version.
+;;;
+;;; GNU Guix is distributed in the hope that it will be useful, but
+;;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;;; GNU General Public License for more details.
+;;;
+;;; You should have received a copy of the GNU General Public License
+;;; along with GNU Guix. If not, see <http://www.gnu.org/licenses/>.
+
+(define-module (guix eris)
+
+ #:use-module (eris)
+ #:use-module (eris fs)
+ #:use-module (eris sqlite)
+ #:use-module (eris read-capability)
+
+ #:use-module (web uri)
+ #:use-module (ice-9 match)
+ #:use-module (srfi srfi-171)
+
+ #:export (%eris-store-url
+ eris-encode-store-item))
+
+(define %eris-store-url
+ (make-parameter
+ (getenv "ERIS_STORE_URL")
+ (lambda (val)
+ (cond
+ ((uri? val) val)
+ ((string? val) (string->uri val))
+ (else #f)))))
+
+(define %guix-eris-convergence-secret
+ (make-parameter %null-convergence-secret))
+
+(define (guix-eris-block-reducer)
+ "Returns an ERIS block reducer."
+ (if (uri? (%eris-store-url))
+ (match (uri-scheme (%eris-store-url))
+
+ ;; Store blocks in an SQLite database (see
+ ;; https://eris.codeberg.page/eer/sqlite.xml)
+ ('sqlite
+ (eris-sqlite-block-reducer (uri-path (%eris-store-url))))
+
+ ;; TODO
+ ;; ('coap+unix #f)
+ ;; ('coap+tcp #f)
+
+ (_ (error "Don't know how to handle ERIS store URL "
+ (uri->string (%eris-store-url)))))
+
+ ;; If no ERIS store URL is provided we just compute the ERIS URN without
+ ;; storing the blocks anywhere. As dummy block-reducer we use `rcount` from
+ ;; SRFI-171 that counts the number of blocks.
+ rcount))
+
+(define* (eris-encode-store-item item)
+ "Encodes the store item ITEM using ERIS and returns the read capability as
+string."
+ (eris-read-capability->string
+ (eris-fs-encode item
+ #:convergence-secret (%guix-eris-convergence-secret)
+ #:block-reducer (guix-eris-block-reducer))))
diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm
index 4457be1fce..2e7138f3c7 100644
--- a/guix/scripts/publish.scm
+++ b/guix/scripts/publish.scm
@@ -53,6 +53,7 @@ (define-module (guix scripts publish)
#:use-module (guix workers)
#:use-module (guix store)
#:use-module ((guix serialization) #:select (write-file))
+ #:use-module (guix eris)
#:use-module (zlib)
#:autoload (lzlib) (call-with-lzip-output-port
make-lzip-output-port)
@@ -96,6 +97,8 @@ (define (show-help)
(display (G_ "
--cache-bypass-threshold=SIZE
serve store items below SIZE even when not cached"))
+ (display (G_ "
+ --eris[=STORE] encode items using ERIS and publish blocks to STORE"))
(display (G_ "
--workers=N use N workers to bake items"))
(display (G_ "
@@ -218,6 +221,9 @@ (define %options
(lambda (opt name arg result)
(alist-cons 'workers (string->number* arg)
result)))
+ (option '("eris") #f #t
+ (lambda (opt name arg result)
+ (alist-cons 'eris (or arg #t) result)))
(option '("ttl") #t #f
(lambda (opt name arg result)
(let ((duration (string->duration arg)))
@@ -319,7 +325,8 @@ (define* (store-item->recutils store-item
(define* (narinfo-string store store-path
#:key (compressions (list %no-compression))
- (nar-path "nar") (file-sizes '()))
+ (nar-path "nar") (file-sizes '())
+ eris-urn)
"Generate a narinfo key/value string for STORE-PATH; an exception is raised
if STORE-PATH is invalid. Produce a URL that corresponds to COMPRESSION. The
narinfo is signed with KEY. NAR-PATH specifies the prefix for nar URLs.
@@ -341,10 +348,10 @@ (define* (narinfo-string store store-path
"\
StorePath: ~a
NarHash: sha256:~a
-NarSize: ~d
+NarSize: ~d~@[~%ERIS: ~a~]
References: ~a~%"
store-path
- hash size references))
+ hash size eris-urn references))
;; Do not render a "Deriver" line if we are rendering info for a
;; derivation. Also do not render a "System" line that would be
;; expensive to compute and is currently unused.
@@ -530,7 +537,7 @@ (define (bypass-cache? store item)
(define* (render-narinfo/cached store request hash
#:key ttl (compressions (list %no-compression))
(nar-path "nar") negative-ttl
- cache pool)
+ cache pool eris?)
"Respond to the narinfo request for REQUEST. If the narinfo is available in
CACHE, then send it; otherwise, return 404 and \"bake\" that nar and narinfo
requested using POOL."
@@ -575,7 +582,8 @@ (define* (render-narinfo/cached store request hash
(bake-narinfo+nar cache item
#:ttl ttl
#:compressions compressions
- #:nar-path nar-path)))
+ #:nar-path nar-path
+ #:eris? eris?)))
(when ttl
(single-baker 'cache-cleanup
@@ -636,7 +644,8 @@ (define (compress-nar cache item compression)
(define* (bake-narinfo+nar cache item
#:key ttl (compressions (list %no-compression))
- (nar-path "/nar"))
+ (nar-path "/nar")
+ (eris? #f))
"Write the narinfo and nar for ITEM to CACHE."
(define (compressed-nar-size compression)
(let* ((nar (nar-cache-file cache item #:compression compression))
@@ -644,7 +653,10 @@ (define* (bake-narinfo+nar cache item
(and stat
(cons compression (stat:size stat)))))
- (let ((compression (actual-compressions item compressions)))
+ (let ((compression (actual-compressions item compressions))
+ (eris-urn (if eris?
+ (eris-encode-store-item item)
+ #f)))
(for-each (cut compress-nar cache item <>) compressions)
@@ -662,7 +674,8 @@ (define* (bake-narinfo+nar cache item
(display (narinfo-string store item
#:nar-path nar-path
#:compressions compressions
- #:file-sizes sizes)
+ #:file-sizes sizes
+ #:eris-urn eris-urn)
port)))
;; Make the cached narinfo world-readable, contrary to what
@@ -1060,7 +1073,8 @@ (define* (make-request-handler store
cache pool
narinfo-ttl narinfo-negative-ttl
(nar-path "nar")
- (compressions (list %no-compression)))
+ (compressions (list %no-compression))
+ (eris? #f))
(define compression-type?
string->compression-type)
@@ -1092,7 +1106,8 @@ (define* (make-request-handler store
#:ttl narinfo-ttl
#:negative-ttl narinfo-negative-ttl
#:nar-path nar-path
- #:compressions compressions)
+ #:compressions compressions
+ #:eris? eris?)
(render-narinfo store request hash
#:ttl narinfo-ttl
#:negative-ttl narinfo-negative-ttl
@@ -1162,7 +1177,7 @@ (define* (run-publish-server socket store
advertise? port
(compressions (list %no-compression))
(nar-path "nar") narinfo-ttl narinfo-negative-ttl
- cache pool)
+ cache pool eris?)
(when advertise?
(let ((name (service-name)))
;; XXX: Use a callback from Guile-Avahi here, as Avahi can pick a
@@ -1178,7 +1193,8 @@ (define* (run-publish-server socket store
#:nar-path nar-path
#:narinfo-ttl narinfo-ttl
#:narinfo-negative-ttl narinfo-negative-ttl
- #:compressions compressions)
+ #:compressions compressions
+ #:eris? eris?)
concurrent-http-server
`(#:socket ,socket)))
@@ -1262,6 +1278,7 @@ (define-command (guix-publish . args)
(repl-port (assoc-ref opts 'repl))
(cache (assoc-ref opts 'cache))
(workers (assoc-ref opts 'workers))
+ (eris? (assoc-ref opts 'eris))
;; Read the key right away so that (1) we fail early on if we can't
;; access them, and (2) we can then drop privileges.
@@ -1281,7 +1298,8 @@ (define-command (guix-publish . args)
(%private-key private-key)
(cache-bypass-threshold
(or (assoc-ref opts 'cache-bypass-threshold)
- (cache-bypass-threshold))))
+ (cache-bypass-threshold)))
+ (%eris-store-url (assoc-ref opts 'eris)))
(if (eq? style 'systemd)
(info (G_ "publishing (started via socket activation)~%"))
(info (G_ "publishing ~a on ~a, port ~d~%")
@@ -1289,6 +1307,10 @@ (define-command (guix-publish . args)
(inet-ntop (sockaddr:fam address) (sockaddr:addr address))
(sockaddr:port address)))
+ (when (string? (assoc-ref opts 'eris))
+ (info (G_ "publishing ERIS blocks to ~a~%")
+ (assoc-ref opts 'eris)))
+
(for-each (lambda (compression)
(info (G_ "using '~a' compression method, level ~a~%")
(compression-type compression)
@@ -1312,7 +1334,8 @@ (define-command (guix-publish . args)
#:nar-path nar-path
#:compressions compressions
#:narinfo-negative-ttl negative-ttl
- #:narinfo-ttl ttl))))))
+ #:narinfo-ttl ttl
+ #:eris? eris?))))))
;;; Local Variables:
;;; eval: (put 'single-baker 'scheme-indent-function 1)
--
2.41.0
P
P
pukkamustard wrote on 28 Dec 2023 01:40
[PATCH v4 5/7] eris: Connect with an ERIS Store over CoAP+Unix.
(address . 52555@debbugs.gnu.org)
cd7432fcc871296d8a14b902e65190669ef89d62.1703316055.git.pukkamustard@posteo.net
* guix/eris.scm (open-coap-unix-socket): New procedure.
(guix-eris-block-reducer): Handle coap+unix URIs.
* guix/publish.scm (bake-narinfo+nar): Start a fibers scheduler when encoding
item with ERIS.
---
guix/eris.scm | 63 ++++++++++++++++++++++++++++++----------
guix/scripts/publish.scm | 9 +++++-
2 files changed, 56 insertions(+), 16 deletions(-)

Toggle diff (118 lines)
diff --git a/guix/eris.scm b/guix/eris.scm
index d98a9a62bd..3fbedd0cb7 100644
--- a/guix/eris.scm
+++ b/guix/eris.scm
@@ -21,8 +21,11 @@ (define-module (guix eris)
#:use-module (eris)
#:use-module (eris fs)
#:use-module (eris sqlite)
+ #:use-module (eris coap)
#:use-module (eris read-capability)
+ #:use-module (coap tcp)
+
#:use-module (web uri)
#:use-module (ice-9 match)
#:use-module (srfi srfi-171)
@@ -42,27 +45,57 @@ (define %eris-store-url
(define %guix-eris-convergence-secret
(make-parameter %null-convergence-secret))
+(define (open-coap-unix-socket path)
+ (let ((sock (socket PF_UNIX SOCK_STREAM 0)))
+ ;; Release FD on exec
+ (fcntl sock F_SETFD FD_CLOEXEC)
+ ;; Set to non-blocking
+ (fcntl sock F_SETFL (logior O_NONBLOCK (fcntl sock F_GETFL)))
+ ;; Connect
+ (connect sock AF_UNIX path)
+
+ ;; Initialize the CoAP connection
+ (open-socket-for-uri #f
+ #:socket sock
+ ;; Allow up to 64 in-flight requests
+ #:nstart 64)))
+
(define (guix-eris-block-reducer)
"Returns an ERIS block reducer."
- (if (uri? (%eris-store-url))
- (match (uri-scheme (%eris-store-url))
+ (let ((store-url (%eris-store-url)))
+ (if (uri? store-url)
+ (match (uri-scheme store-url)
+
+ ;; Store blocks in an SQLite database (see
+ ;; https://eris.codeberg.page/eer/sqlite.xml)
+ ('sqlite
+ (eris-sqlite-block-reducer (uri-path store-url)))
- ;; Store blocks in an SQLite database (see
- ;; https://eris.codeberg.page/eer/sqlite.xml)
- ('sqlite
- (eris-sqlite-block-reducer (uri-path (%eris-store-url))))
+ ;; Connect to a CoAP ERIS store over a Unix socket
+ ('coap+unix
+ ;; Wrap the eris-coap-block-reducer to close the provided connection.
+ (let ((ecbr (eris-coap-block-reducer
+ (build-uri 'coap #:path ".well-known/eris")
+ #:nstart 64
+ #:connection (open-coap-unix-socket
+ (uri-path store-url)))))
+ (case-lambda
+ (() (ecbr))
+ ((conn ref-block) (ecbr conn ref-block))
+ ((conn)
+ (ecbr conn)
+ (close-port conn)))))
- ;; TODO
- ;; ('coap+unix #f)
- ;; ('coap+tcp #f)
+ ;; TODO
+ ;; ('coap+tcp #f)
- (_ (error "Don't know how to handle ERIS store URL "
- (uri->string (%eris-store-url)))))
+ (_ (error "Don't know how to handle ERIS store URL "
+ (uri->string (%eris-store-url)))))
- ;; If no ERIS store URL is provided we just compute the ERIS URN without
- ;; storing the blocks anywhere. As dummy block-reducer we use `rcount` from
- ;; SRFI-171 that counts the number of blocks.
- rcount))
+ ;; If no ERIS store URL is provided we just compute the ERIS URN without
+ ;; storing the blocks anywhere. As dummy block-reducer we use `rcount` from
+ ;; SRFI-171 that counts the number of blocks.
+ rcount)))
(define* (eris-encode-store-item item)
"Encodes the store item ITEM using ERIS and returns the read capability as
diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm
index 2e7138f3c7..0b61354327 100644
--- a/guix/scripts/publish.scm
+++ b/guix/scripts/publish.scm
@@ -54,6 +54,7 @@ (define-module (guix scripts publish)
#:use-module (guix store)
#:use-module ((guix serialization) #:select (write-file))
#:use-module (guix eris)
+ #:use-module (fibers)
#:use-module (zlib)
#:autoload (lzlib) (call-with-lzip-output-port
make-lzip-output-port)
@@ -655,7 +656,13 @@ (define* (bake-narinfo+nar cache item
(let ((compression (actual-compressions item compressions))
(eris-urn (if eris?
- (eris-encode-store-item item)
+ ;; Encode with fibers.
+ ;; XXX: There seems to be some buggy interactions when
+ ;; running a fibers scheduler and connecting to the
+ ;; store.
+ (run-fibers (lambda ()
+ (eris-encode-store-item item))
+ #:drain? #t)
#f)))
(for-each (cut compress-nar cache item <>) compressions)
--
2.41.0
P
P
pukkamustard wrote on 28 Dec 2023 01:40
[PATCH v4 6/7] substitute: Decode substitutes using ERIS.
(address . 52555@debbugs.gnu.org)
deebb9389ad0f32f9bcfb0d7ca6ed9cc04a980d2.1703316055.git.pukkamustard@posteo.net
* guix/scripts/substitute.scm: Decode substitutes using ERIS.
* guix/eris.scm (eris-decode-store-item): New function.
* nix/nix-daemon/guix-daemon.cc (options): Add eris-store-url option.
---
guix/eris.scm | 45 +++++++++++++++++----
guix/scripts/substitute.scm | 76 +++++++++++++++++++++++++++--------
nix/nix-daemon/guix-daemon.cc | 5 +++
3 files changed, 102 insertions(+), 24 deletions(-)

Toggle diff (226 lines)
diff --git a/guix/eris.scm b/guix/eris.scm
index 3fbedd0cb7..1aa52e69dd 100644
--- a/guix/eris.scm
+++ b/guix/eris.scm
@@ -21,6 +21,7 @@ (define-module (guix eris)
#:use-module (eris)
#:use-module (eris fs)
#:use-module (eris sqlite)
+ #:use-module (sqlite3)
#:use-module (eris coap)
#:use-module (eris read-capability)
@@ -31,7 +32,8 @@ (define-module (guix eris)
#:use-module (srfi srfi-171)
#:export (%eris-store-url
- eris-encode-store-item))
+ eris-encode-store-item
+ eris-decode-store-item))
(define %eris-store-url
(make-parameter
@@ -83,8 +85,7 @@ (define (guix-eris-block-reducer)
(() (ecbr))
((conn ref-block) (ecbr conn ref-block))
((conn)
- (ecbr conn)
- (close-port conn)))))
+ (ecbr conn)))))
;; TODO
;; ('coap+tcp #f)
@@ -97,10 +98,40 @@ (define (guix-eris-block-reducer)
;; SRFI-171 that counts the number of blocks.
rcount)))
+(define (call-with-guix-eris-block-ref proc)
+ (let ((store-url (%eris-store-url)))
+ (if (uri? store-url)
+ (match (uri-scheme store-url)
+
+ ('sqlite
+ (let ((db (eris-sqlite-open (uri-path store-url))))
+ (proc (lambda (ref) (eris-sqlite-ref db ref)))
+ (sqlite-close db)))
+
+ ('coap+unix
+ (let ((conn (open-coap-unix-socket (uri-path store-url)))
+ (req-uri (build-uri 'coap #:path ".well-known/eris")))
+ (proc
+ (lambda (ref)
+ (eris-coap-block-ref req-uri ref #:connection conn)))
+ (close-port conn)))
+
+ (_ (error "Don't know how to handle ERIS store URL "
+ (uri->string (%eris-store-url)))))
+
+ (error "No ERIS store to get blocks."))))
+
(define* (eris-encode-store-item item)
"Encodes the store item ITEM using ERIS and returns the read capability as
string."
- (eris-read-capability->string
- (eris-fs-encode item
- #:convergence-secret (%guix-eris-convergence-secret)
- #:block-reducer (guix-eris-block-reducer))))
+ (eris-fs-encode item
+ #:convergence-secret (%guix-eris-convergence-secret)
+ #:block-reducer (guix-eris-block-reducer)))
+
+(define* (eris-decode-store-item eris-urn destination)
+ "Decode a store item with read-capability ERIS-URN to DESTINATION."
+ (call-with-guix-eris-block-ref
+ (lambda (block-ref)
+ (eris-fs-decode eris-urn destination
+ #:block-ref block-ref)
+ #t)))
diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
index 37cd08e289..3c060f1c89 100755
--- a/guix/scripts/substitute.scm
+++ b/guix/scripts/substitute.scm
@@ -45,6 +45,8 @@ (define-module (guix scripts substitute)
. guix:open-connection-for-uri)))
#:autoload (gnutls) (error/invalid-session error/again error/interrupted)
#:use-module (guix progress)
+ #:use-module (guix eris)
+ #:use-module (fibers)
#:use-module ((guix build syscalls)
#:select (set-thread-name))
#:use-module (ice-9 rdelim)
@@ -656,9 +658,11 @@ (define* (process-substitution/fallback port narinfo destination
(()
(loop rest)))))))
+
(define* (process-substitution port store-item destination
#:key cache-urls acl
- deduplicate? print-build-trace?)
+ deduplicate? print-build-trace?
+ (eris? #f))
"Substitute STORE-ITEM (a store file name) from CACHE-URLS, and write it to
DESTINATION as a nar file. Verify the substitute against ACL, and verify its
hash against what appears in the narinfo. When DEDUPLICATE? is true, and if
@@ -674,20 +678,56 @@ (define* (process-substitution port store-item destination
(leave (G_ "no valid substitute for '~a'~%")
store-item))
- (guard (c ((network-error? c)
- (format (current-error-port)
- (G_ "retrying download of '~a' with other substitute URLs...~%")
- store-item)
- (process-substitution/fallback port narinfo destination
- #:cache-urls cache-urls
- #:acl acl
- #:deduplicate? deduplicate?
- #:print-build-trace?
- print-build-trace?)))
- (download-nar narinfo destination
- #:status-port port
- #:deduplicate? deduplicate?
- #:print-build-trace? print-build-trace?)))
+ (if (and eris?
+ (%eris-store-url)
+ (narinfo-eris-urn narinfo))
+
+ (unless
+
+ ;; Attempt to fetch substitute via ERIS
+ (let ((eris-urn (narinfo-eris-urn narinfo)))
+ (format (current-error-port)
+ (G_ "Downloading ~a...~%") (uri->string eris-urn))
+ (run-fibers
+ (lambda ()
+ (guard
+ (c (else
+ (format (current-error-port)
+ (G_ "failed to decode substitute from ERIS URN ~a: ~a"
+ (uri->string eris-urn)
+ c))
+ #f))
+ (eris-decode-store-item eris-urn destination)
+ ;; Tell the daemon that we're done.
+ (format port "success ~a ~a~%"
+ (narinfo-hash narinfo) (narinfo-size narinfo))))))
+
+ ;; Retry without ERIS on failure.
+ (process-substitution port store-item destination
+ #:cache-urls cache-urls
+ #:acl acl
+ #:deduplicate? deduplicate?
+ #:print-build-trace? print-build-trace?
+ #:eris? #f))
+
+ (guard (c ((network-error? c)
+ (format (current-error-port)
+ (G_ "retrying download of '~a' with other substitute URLs...~%")
+ store-item)
+ (process-substitution/fallback port narinfo destination
+ #:cache-urls cache-urls
+ #:acl acl
+ #:deduplicate? deduplicate?
+ #:print-build-trace?
+ print-build-trace?)))
+ (download-nar narinfo destination
+ #:status-port port
+ #:deduplicate? deduplicate?
+ #:print-build-trace? print-build-trace?))))
+
+
+
+
;;;
@@ -876,7 +916,8 @@ (define-command (guix-substitute . args)
;; Download STORE-PATH and store it as a Nar in file DESTINATION.
;; Specify the number of columns of the terminal so the progress
;; report displays nicely.
- (parameterize ((current-terminal-columns (client-terminal-columns)))
+ (parameterize ((current-terminal-columns (client-terminal-columns))
+ (%eris-store-url (find-daemon-option "eris-store-url")))
(let loop ()
(match (read-line)
((? eof-object?)
@@ -887,7 +928,8 @@ (define-command (guix-substitute . args)
#:acl (current-acl)
#:deduplicate? deduplicate?
#:print-build-trace?
- print-build-trace?)
+ print-build-trace?
+ #:eris? #t)
(loop))))))
(opts
(leave (G_ "~a: unrecognized options~%") opts))))))
diff --git a/nix/nix-daemon/guix-daemon.cc b/nix/nix-daemon/guix-daemon.cc
index d7ab9c5e64..d6b054bc6c 100644
--- a/nix/nix-daemon/guix-daemon.cc
+++ b/nix/nix-daemon/guix-daemon.cc
@@ -90,6 +90,7 @@ builds derivations on behalf of its clients.");
#define GUIX_OPT_MAX_SILENT_TIME 19
#define GUIX_OPT_LOG_COMPRESSION 20
#define GUIX_OPT_DISCOVER 21
+#define GUIX_OPT_ERIS_STORE_URL 22
static const struct argp_option options[] =
{
@@ -132,6 +133,8 @@ static const struct argp_option options[] =
n_("use the specified compression type for build logs") },
{ "discover", GUIX_OPT_DISCOVER, "yes/no", OPTION_ARG_OPTIONAL,
n_("use substitute servers discovered on the local network") },
+ { "eris-store-url", GUIX_OPT_ERIS_STORE_URL, n_("URL"), 0,
+ n_("use URL to retrieve blocks of ERIS encoded substitutes") },
/* '--disable-deduplication' was known as '--disable-store-optimization'
up to Guix 0.7 included, so keep the alias around. */
@@ -270,6 +273,8 @@ parse_opt (int key, char *arg, struct argp_state *state)
useDiscover = string_to_bool (arg);
settings.set ("discover", useDiscover ? "true" : "false");
break;
+ case GUIX_OPT_ERIS_STORE_URL:
+ settings.set ("eris-store-url", arg);
case GUIX_OPT_DEBUG:
verbosity = lvlDebug;
break;
--
2.41.0
P
P
pukkamustard wrote on 28 Dec 2023 01:40
[PATCH v4 0/7] Decentralized substitute distribution with ERIS
(address . 52555@debbugs.gnu.org)
cover.1703316055.git.pukkamustard@posteo.net
Dear Guix,

This is the V4 of a proposal towards decentralized subsitute distirbution
using the ERIS encoding. The initial proposal was submitted in December 2021,
V2 and V3 a year later in 2022. All is still very much work-in-progress, but
I'm happy to submit it as is to keep up with good old traditions.

Thank you also for all the valuable comments and feedback via mail and live at
Guix Days last year. Sorry for being so slow to react.

The general idea of the proposal is to use ERIS (http://purl.org/eris)for
substitute distribution. This allows substitutes to be shared over various
protocols such as IPFS, GNUnet, NNCP, HTTP, CoAP, a USB stick or some
Spritely-esque fun.

ERIS itself defines an encoding of content into uniformly sized, encrypted and
content-addressed blocks. The original content can be decoded with a short
read capability that can be encoded as an URN and access to the blocks that
make up the content. Blocks can be shared over different protocols with low
security requirements on the transport layer itself. Read capabilities need to
be shared in a secure manner.

This series does following:

- Adds an `ERIS` field holding the ERIS read capability of a substitute to the
Narinfo as published by `guix publish`
- Allows the daemon/substitute script to decode content using the ERIS URN
instead of fetching a substitute via HTTP.
## Changes to previous version

### Fibers

When encoding and decoding it is important for performance to be able to fetch
multiple blocks concurrently (from potentially multiple peers). For this we
use Guile fibers.

Fiberization is currently very crude as there seem to be weird interactions
with the parallization strategies currently used by `guix publish` and `guix
substitute`. Help here is extremely welcome.

### ERIS-FS

In previous versions we encoded the Nar-file of the susbstitute. In this
version we use a specialized encoding of file-system trees for ERIS: ERIS-FS

ERIS-FS allows de-duplication of files and much easier parlallized decoding of
substitutes. The ideas are similar to the custom encoding defined for

One major consequence is that the SHA256 sum of the Nar file as published in
the Narinfo is no longer used to verify integrity of downloaded
susbtitute. Instead we must ensure that the `ERIS` field in the Narinfo is
trusted, `equivalent-narinfo?` and such have been changed towards this. Thanks
to Maxime for pointing out this subtlety!

### External Block Store Daemon

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (3 lines)
> Also, the store directory should be /var/cache/guix/publish/eris by
> default IMO.

This comment caused a lot of pondering and hacking! :)

The problem seems to be that two processes need to be able to access the block
store securely: `guix publish` when publishing substitutes and the Guix daemon
when fetching substitutes (blocks need to be stored when fetching for
de-duplication). `guix publish` is usually run as a non-privileged user,
whereas the Guix daemon runs with much higher privileges. The block store
needs to be secure in the sense that manipulating blocks should be prevented
by less privileged processes.

I'm afraid the only solution I came up with is that the block store is not a
Unix directory or database, but a daemon that listens on a Unix Socket.

Enter Kapla (https://codeberg.org/eris/kapla).Kapla is a Guile program that
stores and transports blocks. The Guix daemon (via substitute scripts) and the
`guix publish` server talk to Kapla over a Unix socket. Kapla stores blocks in
a database (SQlite) and connects with multiple peers and/or services such as
IPFS/GNUnet to get blocks.

I was fighting such an architecture for a long time. I would have preferred if
the Guix daemon and `guix publish` simply use a library for everything instead
of relying on another external service. However, there seem to be some advantages:

- Managing peers seem to be quite complex and stateful. Maybe better if this
is externalized from Guix proper.
- The block storage and transport mechanisms can be used to share any files
(try `kapla encode my-file` and `kapla decode ...`).
- We can experiment with more transports quicker in a service external to Guix
and include dependencies that might not make sense to have in Guix proper.
The protocol that is used to talk to the external block store is CoAP over
Unix Sockets or TCP. Kapla is not the only software that speaks this
protocol. There is also a Golang block store that can be used

### CoAP for block transport

Currently the protocol over which blocks are transported over networks is the
Constrained Application Protocol (CoAP)
(https://eris.codeberg.page/eer/coap.xml).CoAP is well suited as it allows
multiple asynchronous in-flight requests (unlike HTTP 1.1). This is important
for parallizing block retrieval. It also allows bi-directional requests,
making it attractive for more peer-to-peer systems where some peers might not
have a public IP.

As it is designed for constrained environment it is quite simple and the
implementation we use is in pure Guile.

More block transports can be added by implementing them in Kapla (e.g. IPFS or
GNUnet) or using something completely different than Kapla that speaks CoAP
over Unix Sockets (such as eris-go that also can use NNCP for block
transport).

### Comments to V3

I hope to have addressed all comments to the V3 (Thanks Ludo!). This included
things like organizing commits differently, using (ice-9 match) and how to
handle multiple return values.

## Testing

Testing is a bit complicated. We need to find a better and more systematic way
of doing so. Suggestions are very welcome!

Manually this is how to do it:

1. Authorize local substitutes

We will be running a local substitute server so we need to add the local
signing key to the list of authorized keys. In the system configurations:

```
(modify-services %base-services
(guix-service-type
config =>
(guix-configuration
(inherit config)
(authorized-keys
(cons*
;; allow substitutes from ourselves for testing purposes
(local-file "/etc/signing-key.pub")
%default-authorized-guix-keys)))))
```

2. Configure the local Guix checkout

#+BEGIN_SRC shell
./bootstrap && ./configure --localstatedir=/var --sysconfdir=/etc && make
#+END_SRC

The ~--sysconfdir~ is required so that guix will use the ACL in ~/etc/guix/acl~.

3. Start a Kapla daemon:

```
./pre-inst-env guix build kapla -- kaplad -d
```

The `-d` option enables debug output. Kapla will listen for connection on a
Unix socket. The path will be output, e.g.:

```
2023-12-23 15:10:23 (INFO): Listening on Unix socket /run/user/1000/eris.sock
```

3. Build some package that we will

```
$ ./pre-inst-env guix build libchop --no-grafts
/gnu/store/60m6qih391rq95ck64am8ir64z0sv0zr-libchop-0.5.2
```

4. Start a local publish server

```
sudo -E ./pre-inst-env guix publish --public-key=/etc/guix/signing-key.pub --private-key=/etc/guix/signing-key.sec --cache=/tmp/guix-publish-cache/ --eris=coap+unix:///run/user/1000/eris.sock
```

I need to run it with sudo in order to be able to use the proper signing keys.

The `--eris=STORE_URL` option defines where to store blocks. `--eris` can also
be provided without a URL which will cause the ERIS read capability to be
computed but blocks won't be stored anywhere. See "Proposed Deployment" on
when this might make sense.

5. Get the narinfo from the publish server

```
StorePath: /gnu/store/60m6qih391rq95ck64am8ir64z0sv0zr-libchop-0.5.2
NarHash: sha256:1i2hhzw81qfsba0d1b09ax13694imgjrpay0122gqll85dx7k7ml
NarSize: 1021984
ERIS: urn:eris:BIARSURIJ3WYLEINE6W5ZF7LKTIHL42AE367TQ355ORW5UZVSTQGT5H5T2OLKF7XICML3VHLTLMDWXLUCQVKHRKNVREV3GMVX3J5RMT4GU
References: 1i0iz5rgixyva0zy4bmaasjil2683xrn-mit-krb5-1.20 2w976k6g70gkfih9wwhalqsni209vcqz-gdbm-1.23 4p1l5bdxxbyyqc3wh0d07jv9rp1pdcy7-guile-2.0.14 60m6qih391rq95ck64am8ir64z0sv0zr-libchop-0.5.2 620h3panf2lss42ns625rlay522g2hza-tdb-1.4.7 8y0pwifz8a3d7zbdfzsawa1amf4afx1s-libgcrypt-1.10.1 930nwsiysdvy2x5zv1sf6v7ym75z8ayk-gcc-11.3.0-lib gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35 m9wi9hcrf7f9dm4ri32vw1jrbh1csywi-libgpg-error-1.45 pl09vk5g3cl8fxfln2hjk996pyahqk8m-bzip2-1.0.8 r5saysi65chivbv3y65nzsisx8rypp76-libtirpc-1.3.1 rib9g2ig1xf3kclyl076w28parmncg4k-bash-minimal-5.1.16 slzq3zqwj75lbrg4ly51hfhbv2vhryv5-zlib-1.2.13 z4likj1rnshd81hr0vwvwhdxpfwa1rz7-lzo-2.10
Deriver: zcly5fm6rsqis9csk33i7zfqjidjzy1a-libchop-0.5.2.drv
Signature: 1;strawberry;KHNpZ25hdHVyZSAKIChkYXRhIAogIChmbGFncyByZmM2OTc5KQogIChoYXNoIHNoYTI1NiAjRjFEQTI3RkJDQzA4RTI2MzZDQkY3NEE5NkMwNjQ5QzZBQ0U3QjEzNDRGNzJFNEM1OUMwMzIyRUIzMjFFMUY4RCMpCiAgKQogKHNpZy12YWwgCiAgKGVjZHNhIAogICAociAjMDQwQzYwNzBBOTlBQjI1NEFDNkMwOTdGREIxMUREMkZBNEExNTFGNzYyOTBBRUNENzdCQUMzQ0M0REVERkI0RCMpCiAgIChzICMwQzZGMEFGOTgzODg1NDE5NzAwNzI0NEE5NDU2RTYyMDAxNEE2NUI5NDgxODc4QTlFNjJDOTA2RDQ3NTVGM0YyIykKICAgKQogICkKIChwdWJsaWMta2V5IAogIChlY2MgCiAgIChjdXJ2ZSBFZDI1NTE5KQogICAocSAjMDRDMkY4ODk1QTU0NDNGNTlCODk2NDEwMEI1MDY0NzU4RjQ1N0YzMENEREE1MTQyQzE0MDc0NjExNTA1NTc5MCMpCiAgICkKICApCiApCg==
URL: nar/gzip/60m6qih391rq95ck64am8ir64z0sv0zr-libchop-0.5.2
Compression: gzip
FileSize: 345576
```

Note the new `ERIS` field. You might have to get the narinfo twice for all the
fields to appear.

The ERIS read capability URN (urn:eris:BIARSURIJ...) contains enough
information to decode the substitute. If you look at the kaplad log output you
will see that some blocks were stored. You could now run:

```
kapla decode urn:eris:BIARSURIJ3WYLEINE6W5ZF7LKTIHL42AE367TQ355ORW5UZVSTQGT5H5T2OLKF7XICML3VHLTLMDWXLUCQVKHRKNVREV3GMVX3J5RMT4GU out
```

to decode the substitute to the folder out. The Guix daemon can do the same
when getting substitutes.

6. Remove the libchop store item:

```
guix gc -D /gnu/store/60m6qih391rq95ck64am8ir64z0sv0zr-libchop-0.5.2
```

7. Start the guix daemon from the checkout:


```
sudo -E ./pre-inst-env guix-daemon --build-users-group=guixbuild --eris-store-url=coap+unix:///run/user/1001/eris.sock --substitute-urls=http://localhost:8080/
```

Note that the `--eris-store-url` argument points to the same store as what we
used for the publish server - the blocks come go to and come from the same
place.

We need to use the local publish server for getting the narinfo with the
`ERIS` field.

8. Get the substitute:

```
./pre-inst-env guix build libchop --no-grafts
substitute: updating substitutes from 'http://localhost:8080/'.substitute:updating substitutes from 'http://localhost:8080/'.substitute: updating substitutes from 'http://localhost:8080/'.substitute: updating substitutes from 'http://localhost:8080/'.substitute: updating substitutes from 'http://localhost:8080/'.substitute: updating substitutes from 'http://localhost:8080/'.substitute: updating substitutes from 'http://localhost:8080/'.substitute: updating substitutes from 'http://localhost:8080/'.substitute: updating substitutes from 'http://localhost:8080/'... 100.0%
0.3 MB will be downloaded:
/gnu/store/60m6qih391rq95ck64am8ir64z0sv0zr-libchop-0.5.2
substituting /gnu/store/60m6qih391rq95ck64am8ir64z0sv0zr-libchop-0.5.2...
Downloading urn:eris:BIARSURIJ3WYLEINE6W5ZF7LKTIHL42AE367TQ355ORW5UZVSTQGT5H5T2OLKF7XICML3VHLTLMDWXLUCQVKHRKNVREV3GMVX3J5RMT4GU...

/gnu/store/60m6qih391rq95ck64am8ir64z0sv0zr-libchop-0.5.2
```

Substitute was fetched using ERIS!

The daemon automatically tries to use ERIS when the `--eris-store-url` is
passed and the narinfo has a signed ERIS field. If not it will fetch
substitutes using HTTP. On failures while using ERIS it will also revert back
to getting substitutes via HTTP.

9. Restart your system guix daemon

```
sudo herd restart guix-daemon
```

10. Figure out a better way to test this.

## Proposed Deployment

As an initial deployment it would be nice if the official Guix substitute
servers include the ERIS read capability in the Narinfo withouth making blocks
available. This only requires a bit of CPU for computing the ERIS read
capability. No additional disk space or bandwidth is required by the official
substitute servers.

Users can get the signed ERIS read capability from the official Guix
susbtitute servers but can fetch blocks from anywhere while still being sure
to get the right substitutes.

This would allow community members to make blocks available independently and
experiment with various transports while not changing the trust-model when
fetching substitutes.

## TODOs

- [ ] Better fiberization of `guix publish` and `guix/scripts/substitute.scm`
- [ ] Debug issue where blocks of large substitutes are not properly stored
(possibly related to hackey fiberization).
- [ ] Add Guix tests
- [ ] Add documentation.
- [ ] Write a RFC along the proposed Request-For-Comment process
- [ ] Allow encoding of blocks without running `guix publish` (maybe an `eris`
format for `guix pack`).
- [ ] Dependencies (guile-eris and guile-coap) contain some fixes that need to
be properly released.
- [ ] Implement CoAP peer connections and maybe IPFS transport in Kapla.
- [ ] Release initial version of Kapla.
- [ ] Update service definitions for `guix-publish`, `guix-daemon` with an
option to enable decentralized substitute stuff and add service
definition for `kapla`.

## Hic Sunt Dracones

Everything is still quite fragile. The fiberization of `guix substitute` and
`guix publish` is hackey. Kapla is still very limited and unreleased. But I
hope to have been able to capture the current state of things and paint a
picture of the plan.

Thanks for making it so far and for your comments, questions and hackings.

Greetings,
pukkamustard


pukkamustard (7):
narinfo: Add ERIS field.
gnu: Add guile-coap.
gnu: guile-eris: Update to 1.2.0-dev.
publish: Add ERIS URN to narinfo.
eris: Connect with an ERIS Store over CoAP+Unix.
substitute: Decode substitutes using ERIS.
gnu: Add kapla.

Makefile.am | 1 +
configure.ac | 5 +
gnu/packages/guile-xyz.scm | 144 ++++++++++++++++++++++------
gnu/packages/package-management.scm | 1 +
guix/eris.scm | 137 ++++++++++++++++++++++++++
guix/narinfo.scm | 17 +++-
guix/scripts/publish.scm | 58 ++++++++---
guix/scripts/substitute.scm | 76 +++++++++++----
nix/nix-daemon/guix-daemon.cc | 5 +
9 files changed, 380 insertions(+), 64 deletions(-)
create mode 100644 guix/eris.scm


base-commit: 4a1b3830a8ff6b05ad9a2b27c8a2cdbd00a45787
--
2.41.0
P
P
pukkamustard wrote on 28 Dec 2023 01:40
[PATCH v4 7/7] gnu: Add kapla.
(address . 52555@debbugs.gnu.org)
80c5a32fbbe24b50cb5c6e2eedd29f8d0bbac44a.1703316055.git.pukkamustard@posteo.net
* gnu/packages/guile-xyz.scm (kapla): New variable.
---
gnu/packages/guile-xyz.scm | 46 ++++++++++++++++++++++++++++++++++++++
1 file changed, 46 insertions(+)

Toggle diff (66 lines)
diff --git a/gnu/packages/guile-xyz.scm b/gnu/packages/guile-xyz.scm
index a66a6d8ad5..f3d97cca37 100644
--- a/gnu/packages/guile-xyz.scm
+++ b/gnu/packages/guile-xyz.scm
@@ -102,6 +102,7 @@ (define-module (gnu packages guile-xyz)
#:use-module (gnu packages multiprecision)
#:use-module (gnu packages ncurses)
#:use-module (gnu packages networking)
+ #:use-module (gnu packages ninja)
#:use-module (gnu packages noweb)
#:use-module (gnu packages nss)
#:use-module (gnu packages package-management)
@@ -5193,6 +5194,51 @@ (define-public guile-eris
(home-page "https://codeberg.org/eris/guile-eris")
(license license:gpl3+))))
+(define-public kapla
+ (let ((commit "fa72bcb116ed6ea20f76206145766cb46fa4f30d")
+ (revision "0"))
+ (package
+ (name "kapla")
+ (version (git-version "0.1.0-dev" revision commit))
+ (source
+ (origin
+ (method git-fetch)
+ (uri (git-reference
+ (url "https://codeberg.org/eris/kapla.git")
+ (commit commit)))
+ (file-name (git-file-name name version))
+ (sha256 (base32 "15g9dg6i93n5wkkma983hvcl5mgvw13yxxrv8ka5n5mvslk9f0wa"))))
+ (build-system gnu-build-system)
+ (arguments
+ (list
+ #:phases
+ #~(modify-phases
+ %standard-phases
+ (delete 'patch-source-shebangs)
+ (replace 'configure
+ (lambda* (#:key outputs #:allow-other-keys)
+ (invoke "guile"
+ "--no-auto-compile"
+ "configure"
+ (string-append "--prefix=" #$output))))
+ (replace 'build (lambda _ (invoke "ninja")))
+ (delete 'check)
+ (delete 'install))))
+ (native-inputs (list ninja))
+ (inputs (list guile-3.0))
+ (propagated-inputs
+ (list guile-coap
+ guile-sqlite3
+ guile-eris
+ guile-lib
+ guile-fibers))
+ (synopsis "Block storage and transport for ERIS encoded content")
+ (description "Kapla is a tool for encoding content with ERIS and
+managing blocks of ERIS encoded content. It allows blocks to be stored in a
+local database and be shared over the network.")
+ (home-page "https://codeberg.org/eris/kapla")
+ (license license:agpl3+))))
+
(define-public guile-r6rs-protobuf
(package
(name "guile-r6rs-protobuf")
--
2.41.0
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 52555@patchwise.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 52555
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch