#52555 [RFC PATCH 0/3] Decentralized substitute distribution with ERIS

Message #172 received at 52555@debbugs.gnu.org (full text, mbox, reply):

Received: (at 52555) by debbugs.gnu.org; 14 Jan 2023 18:34:18 +0000
From debbugs-submit-bounces@debbugs.gnu.org Sat Jan 14 13:34:18 2023
Received: from localhost ([127.0.0.1]:55540 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces@debbugs.gnu.org>)
	id 1pGlMA-0005Zg-8h
	for submit@debbugs.gnu.org; Sat, 14 Jan 2023 13:34:18 -0500
Received: from eggs.gnu.org ([209.51.188.92]:54678)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@gnu.org>) id 1pGlM8-0005ZU-BV
 for 52555@debbugs.gnu.org; Sat, 14 Jan 2023 13:34:16 -0500
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@gnu.org>)
 id 1pGlM2-0002vj-MD; Sat, 14 Jan 2023 13:34:10 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To:
 From; bh=SFTfqQoee9qpEyWGtHXAD/XLzDkFAkcQbvVEndvv5xw=; b=BsjzBRezuNKoAEzw+n9/
 te4Wq3z7KExrDw98ADv1iIJlwTi2AwcyFApC6lWY5mu+w7Z11EJepRIbPm0HITtRE6YjudYF9J8RO
 nfxauRqctxBRHsNNCeGi/7ITbkvno0UP2pfhGqb39LCfFgBdq1Ws7Z4OMil3kvMdzkZbDS4f1gDy6
 QNzmXtzso4GFLVaxN/vWfyIjsty+jfn8A57oK6Mey89xwgYw+q6b3EkSUXPjTzvcVnL271YUlaHaM
 asp51l0x88aNVdoi8bIKyJM2bqYfzpNleHv82n3NKioaloeHx0JngKHzuf2buVzJj9VPvoVnslr0H
 V5l0BzT6sZjIfg==;
Received: from 91-160-117-201.subs.proxad.net ([91.160.117.201] helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@gnu.org>)
 id 1pGlM0-00036V-3D; Sat, 14 Jan 2023 13:34:10 -0500
From: Ludovic Courtès <ludo@gnu.org>
To: pukkamustard <pukkamustard@posteo.net>
Subject: Re: bug#52555: [RFC PATCH 0/3] Decentralized substitute
 distribution with ERIS
References: <20221229181327.758-1-pukkamustard@posteo.net>
 <20221229181327.758-2-pukkamustard@posteo.net>
Date: Sat, 14 Jan 2023 19:34:06 +0100
In-Reply-To: <20221229181327.758-2-pukkamustard@posteo.net>
 (pukkamustard@posteo.net's message of "Thu, 29 Dec 2022 18:13:20
 +0000")
Message-ID: <877cxp7yox.fsf_-_@gnu.org>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 52555
Cc: 52555@debbugs.gnu.org
X-BeenThere: debbugs-submit@debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request@debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit@debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request@debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request@debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces@debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces@debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

pukkamustard <pukkamustard@posteo.net> skribis:

> * guix/scripts/publish.scm: (bake-narinfo+nar): Compute ERIS URN of compressed nars.
> (narinfo-string): Add #:eris-urn parameter and honor it.
> * guix/scripts/narinfo.scm: (<narinfo>)[eris-format,eris-urn]: New fields.
> (narinfo-maker): Handle ERIS URN and ERIS format.
> * configure.ac: (HAVE_GUILE_ERIS): New conditional.
> * gnu/packages/package-management.scm: (guix)[native-inputs]: Add guile-eris.

I’d suggest separating the (guix narinfo) part from the rest.

> @@ -135,8 +139,8 @@ (define (narinfo-maker str cache-url)
>    "Return a narinfo constructor for narinfos originating from CACHE-URL.  STR
>  must contain the original contents of a narinfo file."
>    (lambda (path urls compressions file-hashes file-sizes
> -                nar-hash nar-size references deriver system
> -                signature)
> +                nar-hash nar-size eris-format eris-urn references deriver
> +                system signature)

Maybe make ‘eris-format’ and ‘eris-urn’ named parameters.

> +  (define (eris-encode-nar compressions)
> +    (and (member %eris-zstd-compression compressions)
> +         (let* ((nar (nar-cache-file cache item
> +                                     #:compression %eris-zstd-compression))
> +                (stat (stat nar #f)))
> +           (and stat
> +                (call-with-input-file nar
> +                  (lambda (port)
> +                    (let ((eris-urn _
> +                                    (eris-encode port
> +                                                 #:block-size 'large
> +                                                 #:block-reducer rcount
> +                                                 #:convergence-secret
> +                                                 %null-convergence-secret)))
> +                      eris-urn)))))))

I think you remove the ‘let’ and just call ‘eris-encode’ in tail
position (Guile truncates multiple-value returns.)

Where is ‘eris-encode’ storing the blocks?  I don’t see any parameter
here.

I guess it should go to /var/cache/guix/publish/eris or similar.

>    (let ((compression (actual-compressions item compressions)))
>  
>      (for-each (cut compress-nar cache item <>) compressions)
> @@ -640,7 +662,8 @@ (define (compressed-nar-size compression)
>      (match compressions
>        ((main others ...)
>         (let ((narinfo (narinfo-cache-file cache item
> -                                          #:compression main)))
> +                                          #:compression main))
> +             (eris-urn (eris-encode-nar compressions)))
>           (with-atomic-file-output narinfo
>             (lambda (port)
>               ;; Open a new connection to the store.  We cannot reuse the main
> @@ -651,7 +674,8 @@ (define (compressed-nar-size compression)
>                   (display (narinfo-string store item
>                                            #:nar-path nar-path
>                                            #:compressions compressions
> -                                          #:file-sizes sizes)
> +                                          #:file-sizes sizes
> +                                          #:eris-urn eris-urn)

Would be nice to make ERIS encoding optional, with a command-line
switch to turn it on.

Ludo’.

Send a report that this bug log contains spam.