GNU bug report logs

#52555 [RFC PATCH 0/3] Decentralized substitute distribution with ERIS

PackageSource(s)Maintainer(s)
guix-patches PTS Buildd Popcon
Full log

Message #25 received at 52555@debbugs.gnu.org (full text, mbox, reply):

Received: (at 52555) by debbugs.gnu.org; 24 Dec 2021 14:48:45 +0000
From debbugs-submit-bounces@debbugs.gnu.org Fri Dec 24 09:48:45 2021
Received: from localhost ([127.0.0.1]:35945 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces@debbugs.gnu.org>)
	id 1n0lsD-0005En-H1
	for submit@debbugs.gnu.org; Fri, 24 Dec 2021 09:48:45 -0500
Received: from hera.aquilenet.fr ([185.233.100.1]:57460)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@gnu.org>) id 1n0lsA-0005EQ-Mr
 for 52555@debbugs.gnu.org; Fri, 24 Dec 2021 09:48:43 -0500
Received: from localhost (localhost [127.0.0.1])
 by hera.aquilenet.fr (Postfix) with ESMTP id 1392B1B2;
 Fri, 24 Dec 2021 15:48:36 +0100 (CET)
X-Virus-Scanned: Debian amavisd-new at aquilenet.fr
Received: from hera.aquilenet.fr ([127.0.0.1])
 by localhost (hera.aquilenet.fr [127.0.0.1]) (amavisd-new, port 10024)
 with ESMTP id qtDbPZFk31Mm; Fri, 24 Dec 2021 15:48:35 +0100 (CET)
Received: from ribbon (unknown [78.199.208.147])
 by hera.aquilenet.fr (Postfix) with ESMTPSA id 424BABE;
 Fri, 24 Dec 2021 15:48:34 +0100 (CET)
From: Ludovic Courtès <ludo@gnu.org>
To: pukkamustard <pukkamustard@posteo.net>
Subject: Re: bug#52555: [RFC PATCH 0/3] Decentralized substitute
 distribution with ERIS
References: <20211216161724.547-1-pukkamustard@posteo.net>
 <87h7b3gs64.fsf@gnu.org> <86bl17ms56.fsf@posteo.net>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: 4 Nivôse an 230 de la Révolution
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Fri, 24 Dec 2021 15:48:32 +0100
In-Reply-To: <86bl17ms56.fsf@posteo.net> (pukkamustard@posteo.net's message of
 "Thu, 23 Dec 2021 11:42:46 +0000")
Message-ID: <87ilveukhr.fsf@gnu.org>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spamd-Bar: /
Authentication-Results: hera.aquilenet.fr;
	none
X-Rspamd-Server: hera
X-Rspamd-Queue-Id: 1392B1B2
X-Spamd-Result: default: False [-0.10 / 15.00]; ARC_NA(0.00)[];
 RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_HAS_DN(0.00)[];
 RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[];
 TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain];
 FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+];
 RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[];
 MID_RHS_MATCH_FROM(0.00)[]
X-Spam-Score: 1.0 (+)
X-Debbugs-Envelope-To: 52555
Cc: ~pukkamustard/eris@lists.sr.ht, 52555@debbugs.gnu.org
X-BeenThere: debbugs-submit@debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request@debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit@debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request@debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request@debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces@debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces@debbugs.gnu.org>
X-Spam-Score: -0.0 (/)
Hi!

pukkamustard <pukkamustard@posteo.net> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>>> StorePath: /gnu/store/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
>>> URL: nar/gzip/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
>>> Compression: gzip
>>> FileSize: 67363
>>> ERIS: urn:erisx2:BIBC2LUTIQH43S2KRIAV7TBXNUUVPZTMV6KFA2M7AL5V6FNE77VNUDDVDAGJUEEAFATVO2QQT67SMOPTO3LGWCJFU7BZVCF5VXEQQW25BE
>>> URL: nar/zstd/81bdcd5x4v50i28h98bfkvvkx9cky63w-hello-2.10
>>> Compression: zstd
>>> FileSize: 64917
>>> ERIS: urn:erisx2:BIBO7KS7SAWHDNC43DVILOSQ3F3SRRHEV6YPLDCSZ7MMD6LZVCHQMEQ6FUBTJAPSNFF7XR5XPTP4OQ72OPABNEO7UYBUN42O46ARKHBTGM
>>
>> Do we really need one URN per compression method?  Couldn’t we leave
>> compression (of individual chunks, possibly) as a “detail” handled by
>> the encoding or the transport layer?
>>
>
> I agree that it would be nice to leave this to the encoding layer as
> that would allow certain optimizations (e.g. de-duplication).
>
> Unfortunately, we haven't figured out yet what the most suitable
> compression/format would be. Something like EROSFS seems good (as it
> aligns data to fixed block sizes) [1]. But this seems a bit "clunky" for
> just an archive format and there do not seem to be any libraries that we
> could use to neatly integrate. It seems possible to block-align a Tar
> archive, but that seems a bit hackey [2]. Other things to look into
> might be Tarlz [3] and ZPAQ [4].

Yeah.  Though it may be that deduplication at the block level doesn’t
buy us much.  That was the conclusion I reached a long time ago[a], and
also seems to be supported by the recent guix-daemon deduplication
improvements[b].

[a] https://hal.inria.fr/hal-00187069/en
[b] https://issues.guix.gnu.org/24937#20-lineno0

> To get started I suggest just using one of the compressions/formats
> already in Guix. zstd seems to be a reasonable choice (for the same
> reasons why it makes sense to use zstd with `--discover` [5]).
>
> Does that sound like a plan?

Sure!

> I meant to send in a V2 that does this before going on holidays, but I'm
> afraid I won't make it. V2 will come in early January!

Alright, we’ll see!  :-)

Until then, enjoy your holidays!

Ludo’.




Send a report that this bug log contains spam.


debbugs.gnu.org maintainers <help-debbugs@gnu.org>. Last modified: Mon Sep 8 11:22:38 2025; Machine Name: wallace-server

GNU bug tracking system

Debbugs is free software and licensed under the terms of the GNU Public License version 2. The current version can be obtained from https://bugs.debian.org/debbugs-source/.

Copyright © 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson, 2005-2017 Don Armstrong, and many other contributors.