GNU bug report logs

#52555 [RFC PATCH 0/3] Decentralized substitute distribution with ERIS

PackageSource(s)Maintainer(s)
guix-patches PTS Buildd Popcon
Full log

Message #145 received at 52555@debbugs.gnu.org (full text, mbox, reply):

Received: (at 52555) by debbugs.gnu.org; 29 Dec 2022 18:14:30 +0000
From debbugs-submit-bounces@debbugs.gnu.org Thu Dec 29 13:14:30 2022
Received: from localhost ([127.0.0.1]:32798 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces@debbugs.gnu.org>)
	id 1pAxQD-0007Og-BY
	for submit@debbugs.gnu.org; Thu, 29 Dec 2022 13:14:30 -0500
Received: from mout02.posteo.de ([185.67.36.66]:34393)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <pukkamustard@posteo.net>) id 1pAxQB-0007OG-Mo
 for 52555@debbugs.gnu.org; Thu, 29 Dec 2022 13:14:28 -0500
Received: from submission (posteo.de [185.67.36.169]) 
 by mout02.posteo.de (Postfix) with ESMTPS id 1F77C2401A9
 for <52555@debbugs.gnu.org>; Thu, 29 Dec 2022 19:14:22 +0100 (CET)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017;
 t=1672337662; bh=lrwWE98dyL4vG9CQTOKjY0FM2je3h7b72QM1GOagd7I=;
 h=From:To:Cc:Subject:Date:From;
 b=rY/0OHumeRqmQHm9ycrIyREYwhfVM4WFZHzDCG5fb7wY7E7FLI9NjGqFeCIRkxTFP
 7opk0CVeyfn9FQc3ITWWnwdXqAe3yfNkL6CQTpxqzckPElDbPWni1mkx+ffjW7pWXo
 qL7tCFc7e7rjpyxh7ZVNn3iiUyMyhLJ+BgyhQecUCsVbSKWsvLLvecmC48xZjMa+Ma
 BboQRiziZ1Pqt/Kebv5WIKp8m+Yb0LmQ3Xet8DLxQx+p9/yL8j3Dj5wn8DkCc9zenx
 E9UAbBOqX1FQxCKW/5nkBesBl2uQ9t/kZxBIN7ZsHTa9+qQh1U+pfTI6PkYEG5doSJ
 OFPXpbZR0hu4A==
Received: from customer (localhost [127.0.0.1])
 by submission (posteo.de) with ESMTPSA id 4Njc3d3ZBBz6tmY;
 Thu, 29 Dec 2022 19:14:21 +0100 (CET)
From: pukkamustard <pukkamustard@posteo.net>
To: 52555@debbugs.gnu.org
Subject: [PATCH v3 1/8] publish: Add ERIS URN to narinfo.
Date: Thu, 29 Dec 2022 18:13:20 +0000
Message-Id: <20221229181327.758-2-pukkamustard@posteo.net>
In-Reply-To: <20221229181327.758-1-pukkamustard@posteo.net>
References: <20221229181327.758-1-pukkamustard@posteo.net>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 52555
Cc: pukkamustard <pukkamustard@posteo.net>
X-BeenThere: debbugs-submit@debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request@debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit@debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request@debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request@debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces@debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces@debbugs.gnu.org>
X-Spam-Score: -3.3 (---)
* guix/scripts/publish.scm: (bake-narinfo+nar): Compute ERIS URN of compressed nars.
(narinfo-string): Add #:eris-urn parameter and honor it.
* guix/scripts/narinfo.scm: (<narinfo>)[eris-format,eris-urn]: New fields.
(narinfo-maker): Handle ERIS URN and ERIS format.
* configure.ac: (HAVE_GUILE_ERIS): New conditional.
* gnu/packages/package-management.scm: (guix)[native-inputs]: Add guile-eris.
---
 configure.ac                        |  5 +++++
 gnu/packages/package-management.scm |  1 +
 guix/narinfo.scm                    | 16 +++++++++-----
 guix/scripts/publish.scm            | 34 ++++++++++++++++++++++++-----
 4 files changed, 46 insertions(+), 10 deletions(-)

diff --git a/configure.ac b/configure.ac
index 92dede8014..52a37a6113 100644
--- a/configure.ac
+++ b/configure.ac
@@ -173,6 +173,11 @@ GUILE_MODULE_AVAILABLE([have_guile_avahi], [(avahi)])
 AM_CONDITIONAL([HAVE_GUILE_AVAHI],
   [test "x$have_guile_avahi" = "xyes"])
 
+dnl Check for Guile-eris.
+GUILE_MODULE_AVAILABLE([have_guile_eris], [(eris)])
+AM_CONDITIONAL([HAVE_GUILE_ERIS],
+  [test "x$have_guile_eris" = "xyes"])
+
 dnl Guile-newt is used by the graphical installer.
 GUILE_MODULE_AVAILABLE([have_guile_newt], [(newt)])
 
diff --git a/gnu/packages/package-management.scm b/gnu/packages/package-management.scm
index 2ffaa12247..3585f6c073 100644
--- a/gnu/packages/package-management.scm
+++ b/gnu/packages/package-management.scm
@@ -419,6 +419,7 @@ (define code
                        ("guile-zstd" ,guile-zstd)
                        ("guile-ssh" ,guile-ssh)
                        ("guile-git" ,guile-git)
+                       ("guile-eris" ,guile-eris)
 
                        ;; XXX: Keep the development inputs here even though
                        ;; they're unnecessary, just so that 'guix environment
diff --git a/guix/narinfo.scm b/guix/narinfo.scm
index 741c7ad406..d7f8c88f8f 100644
--- a/guix/narinfo.scm
+++ b/guix/narinfo.scm
@@ -45,6 +45,8 @@ (define-module (guix narinfo)
             narinfo-file-sizes
             narinfo-hash
             narinfo-size
+            narinfo-eris-format
+            narinfo-eris-urn
             narinfo-references
             narinfo-deriver
             narinfo-system
@@ -68,8 +70,8 @@ (define-module (guix narinfo)
 
 (define-record-type <narinfo>
   (%make-narinfo path uri-base uris compressions file-sizes file-hashes
-                 nar-hash nar-size references deriver system
-                 signature contents)
+                 nar-hash nar-size eris-format eris-urn references deriver
+                 system signature contents)
   narinfo?
   (path         narinfo-path)
   (uri-base     narinfo-uri-base)        ;URI of the cache it originates from
@@ -79,6 +81,8 @@ (define-record-type <narinfo>
   (file-hashes  narinfo-file-hashes)
   (nar-hash     narinfo-hash)
   (nar-size     narinfo-size)
+  (eris-format  narinfo-eris-format)
+  (eris-urn     narinfo-eris-urn)
   (references   narinfo-references)
   (deriver      narinfo-deriver)
   (system       narinfo-system)
@@ -135,8 +139,8 @@ (define (narinfo-maker str cache-url)
   "Return a narinfo constructor for narinfos originating from CACHE-URL.  STR
 must contain the original contents of a narinfo file."
   (lambda (path urls compressions file-hashes file-sizes
-                nar-hash nar-size references deriver system
-                signature)
+                nar-hash nar-size eris-format eris-urn references deriver
+                system signature)
     "Return a new <narinfo> object."
     (define len (length urls))
     (%make-narinfo path cache-url
@@ -157,6 +161,8 @@ (define len (length urls))
                      ((lst ...) (map string->number lst)))
                    nar-hash
                    (and=> nar-size string->number)
+                   eris-format
+                   (if eris-urn (string->uri eris-urn) #f)
                    (string-tokenize references)
                    (match deriver
                      ((or #f "") #f)
@@ -184,7 +190,7 @@ (define* (read-narinfo port #:optional url
                    (narinfo-maker str url)
                    '("StorePath" "URL" "Compression"
                      "FileHash" "FileSize" "NarHash" "NarSize"
-                     "References" "Deriver" "System"
+                     "ERISFormat" "ERIS" "References" "Deriver" "System"
                      "Signature")
                    '("URL" "Compression" "FileSize" "FileHash"))))
 
diff --git a/guix/scripts/publish.scm b/guix/scripts/publish.scm
index 3bf3bd9c7c..76b5a429f4 100644
--- a/guix/scripts/publish.scm
+++ b/guix/scripts/publish.scm
@@ -40,6 +40,7 @@ (define-module (guix scripts publish)
   #:use-module (srfi srfi-34)
   #:use-module (srfi srfi-37)
   #:use-module (srfi srfi-71)
+  #:use-module (srfi srfi-171)
   #:use-module (web http)
   #:use-module (web request)
   #:use-module (web response)
@@ -57,6 +58,7 @@ (define-module (guix scripts publish)
   #:use-module (guix workers)
   #:use-module (guix store)
   #:use-module ((guix serialization) #:select (write-file))
+  #:use-module (eris)
   #:use-module (zlib)
   #:autoload   (lzlib) (call-with-lzip-output-port
                         make-lzip-output-port)
@@ -145,6 +147,9 @@ (define %default-gzip-compression
   ;; Since we compress on the fly, default to fast compression.
   (compression 'gzip 3))
 
+(define %eris-zstd-compression
+  (compression 'zstd 19))
+
 (define (default-compression type)
   (compression type 3))
 
@@ -323,7 +328,8 @@ (define* (store-item->recutils store-item
 
 (define* (narinfo-string store store-path
                          #:key (compressions (list %no-compression))
-                         (nar-path "nar") (file-sizes '()))
+                         (nar-path "nar") (file-sizes '())
+                         eris-urn)
   "Generate a narinfo key/value string for STORE-PATH; an exception is raised
 if STORE-PATH is invalid.  Produce a URL that corresponds to COMPRESSION.  The
 narinfo is signed with KEY.  NAR-PATH specifies the prefix for nar URLs.
@@ -345,10 +351,10 @@ (define* (narinfo-string store store-path
                              "\
 StorePath: ~a
 NarHash: sha256:~a
-NarSize: ~d
+NarSize: ~d~@[~%ERISFormat: application/x-nix-archive+zstd-19~%ERIS: ~a~]
 References: ~a~%"
                              store-path
-                             hash size references))
+                             hash size eris-urn references))
          ;; Do not render a "Deriver" line if we are rendering info for a
          ;; derivation.  Also do not render a "System" line that would be
          ;; expensive to compute and is currently unused.
@@ -633,6 +639,22 @@ (define (compressed-nar-size compression)
       (and stat
            (cons compression (stat:size stat)))))
 
+  (define (eris-encode-nar compressions)
+    (and (member %eris-zstd-compression compressions)
+         (let* ((nar (nar-cache-file cache item
+                                     #:compression %eris-zstd-compression))
+                (stat (stat nar #f)))
+           (and stat
+                (call-with-input-file nar
+                  (lambda (port)
+                    (let ((eris-urn _
+                                    (eris-encode port
+                                                 #:block-size 'large
+                                                 #:block-reducer rcount
+                                                 #:convergence-secret
+                                                 %null-convergence-secret)))
+                      eris-urn)))))))
+
   (let ((compression (actual-compressions item compressions)))
 
     (for-each (cut compress-nar cache item <>) compressions)
@@ -640,7 +662,8 @@ (define (compressed-nar-size compression)
     (match compressions
       ((main others ...)
        (let ((narinfo (narinfo-cache-file cache item
-                                          #:compression main)))
+                                          #:compression main))
+             (eris-urn (eris-encode-nar compressions)))
          (with-atomic-file-output narinfo
            (lambda (port)
              ;; Open a new connection to the store.  We cannot reuse the main
@@ -651,7 +674,8 @@ (define (compressed-nar-size compression)
                  (display (narinfo-string store item
                                           #:nar-path nar-path
                                           #:compressions compressions
-                                          #:file-sizes sizes)
+                                          #:file-sizes sizes
+                                          #:eris-urn eris-urn)
                           port)))
 
              ;; Make the cached narinfo world-readable, contrary to what
-- 
2.38.1





Send a report that this bug log contains spam.


debbugs.gnu.org maintainers <help-debbugs@gnu.org>. Last modified: Mon Sep 8 11:24:09 2025; Machine Name: wallace-server

GNU bug tracking system

Debbugs is free software and licensed under the terms of the GNU Public License version 2. The current version can be obtained from https://bugs.debian.org/debbugs-source/.

Copyright © 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson, 2005-2017 Don Armstrong, and many other contributors.