GNU bug report logs

#24937 "deleting unused links" GC phase is too slow

PackageSource(s)Maintainer(s)
guix PTS Buildd Popcon
Full log

Message #76 received at 24937@debbugs.gnu.org (full text, mbox, reply):

Received: (at 24937) by debbugs.gnu.org; 16 Nov 2021 13:54:24 +0000
From debbugs-submit-bounces@debbugs.gnu.org Tue Nov 16 08:54:24 2021
Received: from localhost ([127.0.0.1]:57556 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces@debbugs.gnu.org>)
	id 1mmyul-0007Dh-M1
	for submit@debbugs.gnu.org; Tue, 16 Nov 2021 08:54:24 -0500
Received: from eggs.gnu.org ([209.51.188.92]:41576)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@gnu.org>) id 1mmyuj-0007DG-Ot
 for 24937@debbugs.gnu.org; Tue, 16 Nov 2021 08:54:22 -0500
Received: from [2001:470:142:3::e] (port=47430 helo=fencepost.gnu.org)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@gnu.org>) id 1mmyue-0006yg-Fx
 for 24937@debbugs.gnu.org; Tue, 16 Nov 2021 08:54:16 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:In-Reply-To:Date:References:Subject:To:
 From; bh=rVquBRw2cCTTy+pnADwfs0DRheds1f144iMWY+3qSSU=; b=LLcTqmL+L4y5O1C5SxJA
 u5Z99NCo4i1Ux3jdCUOIpsXDSqyOsBQfX2s5qORk8R+K+XH+bgV9k+Ob7NXxQiy0I8a99YTAbFLdC
 vrhv12iTbH2fS9Xvs8n6sqTBr/LYZTYFbFi4Zxw2Pb63Mq61uGxFwRRdyBKD/7N6tZ2TSLteBcRRW
 +4ETgFtKmwgK65g3NnnrFmeG/8zitZAyrkNYhBrA8/x0gLqSWJoj5R0fXVLDVEU53eBKu7daMtW1d
 5iwS4LsYE7y4W8wzxJrWgl5U7N7SpylwrcA7tqOn4IWhTmHGEgvMpTxFbGLu0AD3F9wIgsf8pXwKJ
 6WspnciYHBnXwg==;
Received: from [2001:660:6102:320:e120:2c8f:8909:cdfe] (port=46194 helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@gnu.org>) id 1mmyue-0002fd-8K
 for 24937@debbugs.gnu.org; Tue, 16 Nov 2021 08:54:16 -0500
From: Ludovic Courtès <ludo@gnu.org>
To: 24937@debbugs.gnu.org
Subject: Re: bug#24937: "deleting unused links" GC phase is too slow
References: <87v90wat9n.fsf@gnu.org> <20211113213745.2601-1-ludo@gnu.org>
 <20211113213745.2601-2-ludo@gnu.org>
Date: Tue, 16 Nov 2021 14:54:13 +0100
In-Reply-To: <20211113213745.2601-2-ludo@gnu.org> ("Ludovic Courtès"'s message
 of "Sat, 13 Nov 2021 22:37:45 +0100")
Message-ID: <87h7cc2ol6.fsf_-_@gnu.org>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-Debbugs-Envelope-To: 24937
X-BeenThere: debbugs-submit@debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request@debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit@debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request@debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request@debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces@debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces@debbugs.gnu.org>
X-Spam-Score: -3.3 (---)
Hi,

Ludovic Courtès <ludo@gnu.org> skribis:

> Files smaller than 4 KiB typically represent ~60% of the entries in
> /gnu/store/.links but only contribute to ~2.5% of the space savings
> afforded by deduplication.
>
> Not considering these files for deduplication speeds up file insertion
> in the store and, more importantly, leaves 'removeUnusedLinks' with
> fewer entries to traverse, thereby speeding it up proportionally.
>
> Partly fixes <https://issues.guix.gnu.org/24937>.

Pushed a variant of this as commit
472a0e82a52a3d5d841e1dfad6b13e26082a5750, with a threshold of 8 KiB.

Concretely, the number of .links entries shrinks by ~70%, from
2M to 700K on my laptop, and (presumably) from 64M to 19M on berlin.

I’ll deploy it within a few days on berlin.  I hope the speedup will
reduce pressure there, though obviously it’ll still be an expensive
operation (but fundamentally I think it’ll always be linear in the size
of the store.)

I’m preparing an update of the ‘guix’ package to make this readily
available.  When you deploy the new daemon, .links will be trimmed of
entries for files smaller than 8 KiB the first time you run ‘guix gc’.

Ludo’.




Send a report that this bug log contains spam.


debbugs.gnu.org maintainers <help-debbugs@gnu.org>. Last modified: Sun Sep 7 11:41:33 2025; Machine Name: wallace-server

GNU bug tracking system

Debbugs is free software and licensed under the terms of the GNU Public License version 2. The current version can be obtained from https://bugs.debian.org/debbugs-source/.

Copyright © 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson, 2005-2017 Don Armstrong, and many other contributors.