#48005 - low performance when building profile of already built packages

Package	Source(s)		Maintainer(s)
guix		PTS Buildd Popcon

Message #5 received at submit@debbugs.gnu.org (full text, mbox, reply):

Received: (at submit) by debbugs.gnu.org; 24 Apr 2021 20:39:16 +0000
From debbugs-submit-bounces@debbugs.gnu.org Sat Apr 24 16:39:16 2021
Received: from localhost ([127.0.0.1]:41580 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces@debbugs.gnu.org>)
	id 1laP3b-0007JZ-Qb
	for submit@debbugs.gnu.org; Sat, 24 Apr 2021 16:39:16 -0400
Received: from lists.gnu.org ([209.51.188.17]:41314)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <rekado@elephly.net>) id 1laP3Y-0007JQ-8W
 for submit@debbugs.gnu.org; Sat, 24 Apr 2021 16:39:14 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10]:39280)
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <rekado@elephly.net>)
 id 1laP3X-0008DT-SG
 for bug-guix@gnu.org; Sat, 24 Apr 2021 16:39:12 -0400
Received: from sender4-of-o51.zoho.com ([136.143.188.51]:21154)
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <rekado@elephly.net>)
 id 1laP3V-0004Q0-Pq
 for bug-guix@gnu.org; Sat, 24 Apr 2021 16:39:11 -0400
ARC-Seal: i=1; a=rsa-sha256; t=1619296743; cv=none; 
 d=zohomail.com; s=zohoarc; 
 b=R3t2kuMFLHPUvYA7S8aodivRH/PrhHdXntQhQEBmUC/hBcwujNUFb1/9zpuPk4BMVu8cHS/N51uo46tHS9Ta7VjICb8UltAOZYaWd9/gt7TWfXiqRjW91o/OEij1e4gMeyLpN7lqw81r5eFmVqaz1tdJai7IeOHcYqybJJO9Pi0=
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com;
 s=zohoarc; t=1619296743;
 h=Content-Type:Content-Transfer-Encoding:Date:From:MIME-Version:Message-ID:Subject:To;
 bh=sI0vQYrIZpHYJmB4h9CVDSgtiWmeKqAl4eoi3lHCYXo=; 
 b=Y8mJ5OiZl+bm5sEOjIU7M+NvFnQeO3nUeGX8QDKAIH+/YFyuMG6NRmGB3Z+HreyT12dsWwI9i3hSFNtj+NqJJMttFNbUPrMTX+S9FQemB+/a+NMwbUUHZUgAoWTnJZ5EnGU++xmdv5HY1RgK7BimOOGnRWc0dh5nwePgQUDY90Q=
ARC-Authentication-Results: i=1; mx.zohomail.com;
 dkim=pass  header.i=elephly.net;
 spf=pass  smtp.mailfrom=rekado@elephly.net;
 dmarc=pass header.from=<rekado@elephly.net> header.from=<rekado@elephly.net>
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1619296743; 
 s=zoho; d=elephly.net; i=rekado@elephly.net;
 h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding;
 bh=sI0vQYrIZpHYJmB4h9CVDSgtiWmeKqAl4eoi3lHCYXo=;
 b=aMWInhCrxsRskEgeLrVKlPQ8WWgip4GFSurjzjh/6IV9oqYMIrau5vus240dumOe
 errVOq1f3YHoNh7z0ulbPpB7gHoxACV9SolEvNQAdTSnA12EmJNC0OVpr0iPbIU1mlc
 6sV5GXei1PUk9/B6X5gsd2Z4RcEyICpZDjIFNRNw=
Received: from localhost (p4fd5ab0c.dip0.t-ipconnect.de [79.213.171.12]) by
 mx.zohomail.com with SMTPS id 1619296741170139.57686629621355;
 Sat, 24 Apr 2021 13:39:01 -0700 (PDT)
User-agent: mu4e 1.4.15; emacs 27.2
From: Ricardo Wurmus <rekado@elephly.net>
To: bug-guix@gnu.org
Subject: low performance when building profile of already built packages
X-URL: https://elephly.net
X-PGP-Key: https://elephly.net/rekado.pubkey
X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
Date: Sat, 24 Apr 2021 22:38:58 +0200
Message-ID: <87zgxnpgd9.fsf@elephly.net>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: quoted-printable
X-ZohoMailClient: External
Received-SPF: pass client-ip=136.143.188.51; envelope-from=rekado@elephly.net;
 helo=sender4-of-o51.zoho.com
X-Spam_score_int: -20
X-Spam_score: -2.1
X-Spam_bar: --
X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,
 RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001,
 SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -1.3 (-)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit@debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request@debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit@debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request@debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request@debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces@debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces@debbugs.gnu.org>
X-Spam-Score: -2.3 (--)

Here is a manifest that isn’t unusual for a bioinformatics 
workflow:

--8<---------------cut here---------------start------------->8---
(define packages
 (list "bash-minimal"
       "fastqc"
       "gzip"
       "htseq"
       "multiqc"
       "r-corrplot"
       "r-crosstalk"
       "r-data-table"
       "r-deseq2"
       "r-dt"
       "r-genomicalignments"
       "r-genomicranges"
       "r-ggplot2"
       "r-ggrepel"
       "r-gprofiler"
       "r-knitr"
       "r-pheatmap"
       "r-plotly"
       "r-reshape2"
       "r-rmarkdown"
       "r-rsamtools"
       "r-rtracklayer"
       "r-s4vectors"
       "r-scales"
       "r-summarizedexperiment"
       "r-tximport"
       "samtools"
       "sed"
       "trim-galore"))

(specifications->manifest packages)
--8<---------------cut here---------------end--------------->8---

When all these packages already exist and nothing is to be done 
when Guix is asked to instantiate this manifest, more than 13 
seconds pass.

--8<---------------cut here---------------start------------->8---
$ [env] export GUIX_PROFILING="object-cache 
add-data-to-store-cache rpc"
$ [env] time guix package -m manifest-test.scm -p /tmp/foo
The following packages will be installed:
  bash-minimal           5.0.16
  fastqc                 0.11.5
  gzip                   1.10
  htseq                  0.9.1
  multiqc                1.5
  r-corrplot             0.84
  r-crosstalk            1.1.1
  r-data-table           1.14.0
  r-deseq2               1.30.1
  r-dt                   0.17
  r-genomicalignments    1.26.0
  r-genomicranges        1.42.0
  r-ggplot2              3.3.3
  r-ggrepel              0.9.1
  r-gprofiler            0.7.0
  r-knitr                1.31
  r-pheatmap             1.0.12
  r-plotly               4.9.3
  r-reshape2             1.4.4
  r-rmarkdown            2.7
  r-rsamtools            2.6.0
  r-rtracklayer          1.50.0
  r-s4vectors            0.28.1
  r-scales               1.1.1
  r-summarizedexperiment 1.20.0
  r-tximport             1.18.0
  samtools               1.11
  sed                    4.8
  trim-galore            0.6.6
[[most of the time passes here]]

nothing to be done
Store object cache:
 fresh caches:     2
 lookups:      45493
 hits:         43943 (96.6%)
 cache size:    1549 entries

'add-data-to-store' cache:
 lookups:       7755
 hits:          3411 (44.0%)
 .drv files:    4357 (56.2%)
 Scheme files:  3343 (43.1%)
Remote procedure call summary: 5283 RPCs
 built-in-builders              ...     1
 add-to-store/tree              ...    27
 add-to-store                   ...   213
 query-references               ...   698
 add-text-to-store              ...  4344

real	0m13.379s
user	0m16.414s
sys	0m0.253s
--8<---------------cut here---------------end--------------->8---

Some of these R packages have a lot of transitive inputs, such as 
r-deseq2 with 81 transitive propagated inputs.  So there’s a lot 
of propagation here, which I suspect contributes to things being 
so unexpectedly slow.

This report may be related to performance issues reported for 
“guix environment” in https://issues.guix.gnu.org/41702, which 
also involves R packages (such as r-learnr with 46 transitive 
propagated inputs).

-- 
Ricardo

Display info messages

Send a report that this bug log contains spam.

debbugs.gnu.org maintainers <help-debbugs@gnu.org>. Last modified: Sun Dec 22 11:56:46 2024; Machine Name: wallace-server

GNU bug tracking system

Debbugs is free software and licensed under the terms of the GNU Public License version 2. The current version can be obtained from https://bugs.debian.org/debbugs-source/.

#48005 low performance when building profile of already built packages