#24937 - "deleting unused links" GC phase is too slow

Package	Source(s)		Maintainer(s)
guix		PTS Buildd Popcon

Message #43 received at 24937@debbugs.gnu.org (full text, mbox, reply):

Received: (at 24937) by debbugs.gnu.org; 16 Apr 2020 13:27:43 +0000
From debbugs-submit-bounces@debbugs.gnu.org Thu Apr 16 09:27:43 2020
Received: from localhost ([127.0.0.1]:37743 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces@debbugs.gnu.org>)
	id 1jP4YQ-0008AH-Qq
	for submit@debbugs.gnu.org; Thu, 16 Apr 2020 09:27:43 -0400
Received: from sender4-of-o53.zoho.com ([136.143.188.53]:21335)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <rekado@elephly.net>) id 1jP4YO-0008A8-3V
 for 24937@debbugs.gnu.org; Thu, 16 Apr 2020 09:27:41 -0400
ARC-Seal: i=1; a=rsa-sha256; t=1587043654; cv=none; 
 d=zohomail.com; s=zohoarc; 
 b=gjN/fdRn5klYUMhJRP97MIgV4+WAtR/CaZQ2/2yhOAa6P2RPZ+0wZ6k8Q2GZTg7763GtZBmeoDbBa4W54uvRVxbbGk3/KdnCr3ppVU8WU/0xzkNA7AdHPta4Ylhy3VpddwZV7O42g2U0sRZikl6Q9g3PN+GmGXDfJOzW7FDgOyY=
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com;
 s=zohoarc; t=1587043654;
 h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To;
 bh=CetexPByQizFw7YwqtgKpHwuPuDD5g2j6jbh87dcg8M=; 
 b=AvmFJPJOMF63AWdEtAVLdjkUJEoIXsyrXTO//Zm3lxMWYynInn0jDuoBoleGDdZfrQH7NSGmuXbdI7Lk5kY3p5daPdwfDLt1S5i1RZR2aBQRtGVsRiqDK5y4dBCU3y4pmyE/iuZK6t5dapzmC/g4WJ/GcKIGwEiL6X8qdtJ59qg=
ARC-Authentication-Results: i=1; mx.zohomail.com;
 dkim=pass  header.i=elephly.net;
 spf=pass  smtp.mailfrom=rekado@elephly.net;
 dmarc=pass header.from=<rekado@elephly.net> header.from=<rekado@elephly.net>
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1587043654; 
 s=zoho; d=elephly.net; i=rekado@elephly.net;
 h=References:From:To:Cc:Subject:In-reply-to:Date:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding;
 bh=CetexPByQizFw7YwqtgKpHwuPuDD5g2j6jbh87dcg8M=;
 b=MMQAYyVGh1bN/4EQ5+y0e1EPX/H4ZpAm0ssLY+9J0Bi2PM3smeXfLnPT2Zt7A1uD
 I0UKJIB0UIcYEXaSWXu/XUNBxLPyTVIi+dZh+6bgaKsDe/oiu7yEs34qrBiS5i3Gurx
 +aVXq6Sj8Q+2zSzcELfCKZRV2Szfl4xClBnMmOyk=
Received: from localhost (p54AD4D27.dip0.t-ipconnect.de [84.173.77.39]) by
 mx.zohomail.com with SMTPS id 1587043621480388.8881527687968;
 Thu, 16 Apr 2020 06:27:01 -0700 (PDT)
References: <87wpg7ffbm.fsf@gnu.org> <87lgvm4lzu.fsf@gnu.org>
 <87twaaa6j9.fsf@netris.org> <87twaa2vjx.fsf@gnu.org>
 <87lgvm9sgq.fsf@netris.org> <87d1gwvgu0.fsf@gnu.org>
 <87wpf4yoz0.fsf@netris.org> <87fulrsqxx.fsf@gnu.org>
 <87vaunbvcu.fsf@mdc-berlin.de>
User-agent: mu4e 1.2.0; emacs 26.3
From: Ricardo Wurmus <rekado@elephly.net>
To: Ludovic Courtès <ludo@gnu.org>
Subject: Re: bug#24937: "deleting unused links" GC phase is too slow
In-reply-to: <87vaunbvcu.fsf@mdc-berlin.de>
X-URL: https://elephly.net
X-PGP-Key: https://elephly.net/rekado.pubkey
X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
Date: Thu, 16 Apr 2020 15:26:58 +0200
Message-ID: <87ftd3muhp.fsf@elephly.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-ZohoMailClient: External
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 24937
Cc: 24937@debbugs.gnu.org
X-BeenThere: debbugs-submit@debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request@debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit@debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request@debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request@debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces@debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces@debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Ricardo Wurmus <ricardo.wurmus@mdc-berlin.de> writes:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> Ricardo, Roel: would you be able to run that links-traversal.c from
>> <https://debbugs.gnu.org/cgi/bugreport.cgi?filename=links-traversal.c;bug=24937;msg=25;att=1>
>> on a machine with a big store, as described at
>> <https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24937#25>?
>
> I just ran this on my workstation in the office where I regularly build
> packages.  Here’s the output of “df -i /gnu”
>
>   Filesystem               Inodes   IUsed   IFree IUse% Mounted on
>   /dev/mapper/fedora-root 3301376 1098852 2202524   34% /
>
> Probably not large enough to derive conclusions about hydra’s behaviour.
>
> [I can’t run it on the shared store at the MDC because NFS performance is
> too poor.  I recently ran “guix gc --optimize” to dedupe the shared
> store (post-build deduplication is disabled since a few weeks) and it’s
> at 3,197,489 used inodes.]
>
> Here are the results of running the link-traversal code on my
> workstation:
>
> --8<---------------cut here---------------start------------->8---
> rwurmus in ~: gcc -std=gnu99 -Wall links-traversal.c  -DMODE=3
> rwurmus in ~: sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
> rwurmus in ~: time ./a.out
> 412825 dir_entries, 107 seconds
> stat took 0 seconds
>
> real	1m47.264s
> user	0m0.214s
> sys	0m1.314s
>
> rwurmus in ~: gcc -std=gnu99 -Wall links-traversal.c  -DMODE=2
> rwurmus in ~: sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'
> rwurmus in ~: time ./a.out
> 12821 dir_entries, 107 seconds (including stat)
>
> real	1m46.475s
> user	0m0.201s
> sys	0m1.309s
> --8<---------------cut here---------------end--------------->8---

I ran this for the first time on ci.guix.gnu.org, which has a very big
store (currently at around 29TB).

df -i /gnu:

  Filesystem        Inodes     IUsed     IFree IUse% Mounted on
  /dev/sdb1      610021376 132350406 477670970   22% /gnu

I had to increase the number of MAX_ENTRIES to 135000000.

I forgot to drop caches initially.  This is the first run:

--8<---------------cut here---------------start------------->8---
root@berlin ~ [env]# gcc links-traversal.c -DMODE=3 -o links-traversal
root@berlin ~ [env]# time ./links-traversal
57079502 dir_entries, 3906 seconds
stat took 136 seconds

real    67m48.145s
user    0m59.575s
sys     2m30.065s
--8<---------------cut here---------------end--------------->8---

I aborted the run after I dropped caches after 67 minutes.

I’m going to continue testing on one of the build nodes, and I’ll try
using statx.

--
Ricardo

Display info messages

Send a report that this bug log contains spam.

debbugs.gnu.org maintainers <help-debbugs@gnu.org>. Last modified: Sun Sep 7 08:38:06 2025; Machine Name: wallace-server

GNU bug tracking system

Debbugs is free software and licensed under the terms of the GNU Public License version 2. The current version can be obtained from https://bugs.debian.org/debbugs-source/.

#24937 "deleting unused links" GC phase is too slow