#52555 - [RFC PATCH 0/3] Decentralized substitute distribution with ERIS

Package	Source(s)		Maintainer(s)
guix-patches		PTS Buildd Popcon

Message #127 received at 52555@debbugs.gnu.org (full text, mbox, reply):

Received: (at 52555) by debbugs.gnu.org; 2 Feb 2022 15:28:04 +0000
From debbugs-submit-bounces@debbugs.gnu.org Wed Feb 02 10:28:04 2022
Received: from localhost ([127.0.0.1]:53095 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces@debbugs.gnu.org>)
	id 1nFHYC-0005V3-5e
	for submit@debbugs.gnu.org; Wed, 02 Feb 2022 10:28:04 -0500
Received: from andre.telenet-ops.be ([195.130.132.53]:48458)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <maximedevos@telenet.be>) id 1nFHY7-0005Ub-3o
 for 52555@debbugs.gnu.org; Wed, 02 Feb 2022 10:28:02 -0500
Received: from ptr-bvsjgyhxw7psv60dyze.18120a2.ip6.access.telenet.be
 ([IPv6:2a02:1811:8c09:9d00:3c5f:2eff:feb0:ba5a])
 by andre.telenet-ops.be with bizsmtp
 id qFTw2600T4UW6Th01FTwYL; Wed, 02 Feb 2022 16:27:57 +0100
Message-ID: <84d930adac8c54120b176362e66a63ebf49179c6.camel@telenet.be>
Subject: Re: [bug#52555] [RFC PATCH v2 0/5] Decentralized substitute
 distribution with ERIS
From: Maxime Devos <maximedevos@telenet.be>
To: pukkamustard <pukkamustard@posteo.net>
Date: Wed, 02 Feb 2022 16:27:52 +0100
In-Reply-To: <8635l1h1mx.fsf@posteo.net>
References: <20211216161724.547-1-pukkamustard@posteo.net>
 <20220125192201.7582-1-pukkamustard@posteo.net>
 <ba490f009be9908063609f71800a2f00b4b1c1ae.camel@telenet.be>
 <86fsp1h6ce.fsf@posteo.net>
 <846716544b4424f02e383114ebcb52957b43dd4d.camel@telenet.be>
 <8635l1h1mx.fsf@posteo.net>
Content-Type: multipart/signed; micalg="pgp-sha512";
 protocol="application/pgp-signature"; boundary="=-MLPmcf3VaezyhnWN/EcJ"
User-Agent: Evolution 3.38.3-1 
MIME-Version: 1.0
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telenet.be; s=r22;
 t=1643815677; bh=L1cE7iw9wpDs9RBnZWsEw4oIK8YvYcba5uknLmImRGU=;
 h=Subject:From:To:Cc:Date:In-Reply-To:References;
 b=ElS/TxmLltpabQrQQ28GDGY/JGBbtspCujTfS0pWMjJ2JWZi80ZcumfDMT/28eNVa
 KhuLco/tCtEg77KqCfTtW5zTD6B39RO4RZLJEIMDO5zY9n03knG8nKXe+ihMZxJnnc
 IiiKrQbe3uh4Tf2DLipmBYsHNRxtJQWPnZLRp1iZoPFgof0/IV/vLa6X+bmSvJackl
 Xb7XpmF+bKN1wM+BAoUrpXKRuntXchFhhi1qN4XZ7XhFeNWWU8no+F+G3KgG6QMy6x
 G/0OTRGeZZcs7jo18t7Zkr2rExGt1LxbmcLtDNoKwhltjCHxeZy+FLEDl5PUHZMEyl
 sKz98EGtJqKqQ==
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 52555
Cc: ~pukkamustard/eris@lists.sr.ht, 52555@debbugs.gnu.org
X-BeenThere: debbugs-submit@debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request@debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit@debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request@debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request@debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces@debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces@debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

[Message part 1 (text/plain, inline)]

pukkamustard schreef op wo 02-02-2022 om 12:42 [+0000]:
> > E.g., if the client cannot download the data in the range [start,
> > end]
> > because the corresponding block has disappeared, can it not simply
> > download that range from https://ci.guix.gnu.org/nar/[...]
> > (not sure about the URI) using a HTTP range request?
> 
> This does not work as the mapping from block reference to location in
> NAR can not be known by the client who only holds the ERIS
> URN.

The client not only knows the ERIS URN, it also knows the location of
the nar (over classical HTTP) because it's in the narinfo.

> Furthermore, some blocks will be intermediary nodes - they hold
> references to content blocks (or other intermediary nodes) but not
> content itself.

If an intermediary node (responsible for, say, bytes 900--10000)
is missing, then the bytes 900--10000 could be downloaded via HTTP.
Whether the node is close to the top, or close to the bottom, in ERIS'
variant of Merkle trees, doesn't matter much.

Granted, if the nar is, say, 1 GiB, and the top-level block is missing,
then we'll have to download 1 GiB over HTTP, even if most lower blocks
exist on IPFS/GNUnet/whatever, which isn't really great.

We could also do some combination of the GDBM database and HTTP
Content-Range requests: most nodes are leaf nodes (*).  Instead of
representing all nodes in the database, we could include only
(intermediate) nodes responsible for data of size, say, 4MiB.

(*) At least, that's the case for binary trees, presumably something
similar holds for ERIS.

I don't know the specifics for ERIS, but for (balanced) binary trees,
not storing the leaf nodes would save about 50% (**), which is a rather
nice space saving.

(**) This assumes the ‘block size’ is the size for storing two pointers
to the children, but in practice the block size would be quite a bit
larger, so there would be more space savings?

Perhaps we are overthinking things and the GDBM (***) database isn't
overly large, or perhaps missing blocks are sufficiently rare such that
we could simply download the _entire_ nar from classical HTTP in case
of missing blocks ...

(***) Guix uses SQlite databases, so I would use SQLite instead of GDBM
unless there's a compelling reason to use GDBM instead.

Greetings,
Maxime.

[signature.asc (application/pgp-signature, inline)]

Display info messages

Send a report that this bug log contains spam.

debbugs.gnu.org maintainers <help-debbugs@gnu.org>. Last modified: Tue Sep 9 02:38:19 2025; Machine Name: wallace-server

GNU bug tracking system

Debbugs is free software and licensed under the terms of the GNU Public License version 2. The current version can be obtained from https://bugs.debian.org/debbugs-source/.

#52555 [RFC PATCH 0/3] Decentralized substitute distribution with ERIS