Poor 'guix substitute' performance when receiving Zstd-compressed substitutes

  • Done
  • quality assurance status badge
Details
2 participants
  • Ludovic Courtès
  • Maxim Cournoyer
Owner
unassigned
Submitted by
Maxim Cournoyer
Severity
normal

Debbugs page

M
M
Maxim Cournoyer wrote on 22 Jun 2021 11:50
(name . bug-guix)(address . bug-guix@gnu.org)
87fsx93f99.fsf@gmail.com
Hello,

It's something I've been observing for a while, but substitutes are very
IO intensive (as can be seen in iotop, the substitute process is waiting
on IO > 99% of the time) and is much slower than expected (3 minutes to
transfer 100 MiB uncompressed over a 50 mbps downstream link):

Toggle snippet (4 lines)
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
13934 be/4 root 1033.09 K/s 1485.06 K/s 0.00 % 93.36 % guile \ /gnu/store/vphx2839xv0qj9xwcwrb95592lzrrnx7-guix-1.3.0-3.50dfbbf/bin/guix substitute --substitute

The publisher (remote machine) is has its guix-daemon configured via:

Toggle snippet (7 lines)
(service guix-publish-service-type
(guix-publish-configuration
(advertise? #t)
(compression '(("zstd" 3)))
(host "0.0.0.0"))) ;listen on all interfaces

CPU is idling during on both sides during the transfer (all the work
appears to be in the IO on the receiving end).

The above example was observed downloading
/gnu/store/4fcwwlv9bzfrraxiz41b4vcv131930fx-libstdc++-doc-9.4.0, but I
do not think it is substitute-specific.

Maxim
L
L
Ludovic Courtès wrote on 23 Jun 2021 06:26
(name . Maxim Cournoyer)(address . maxim.cournoyer@gmail.com)(address . 49174@debbugs.gnu.org)
87eecsyang.fsf@gnu.org
Hi,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

Toggle quote (17 lines)
> It's something I've been observing for a while, but substitutes are very
> IO intensive (as can be seen in iotop, the substitute process is waiting
> on IO > 99% of the time) and is much slower than expected (3 minutes to
> transfer 100 MiB uncompressed over a 50 mbps downstream link):
>
> TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
> 13934 be/4 root 1033.09 K/s 1485.06 K/s 0.00 % 93.36 % guile \ /gnu/store/vphx2839xv0qj9xwcwrb95592lzrrnx7-guix-1.3.0-3.50dfbbf/bin/guix substitute --substitute
>
>
> The publisher (remote machine) is has its guix-daemon configured via:
>
> (service guix-publish-service-type
> (guix-publish-configuration
> (advertise? #t)
> (compression '(("zstd" 3)))
> (host "0.0.0.0"))) ;listen on all interfaces

Note that in this case nars are built and compressed on the fly on the
server side, which puts an upper bound on the bandwidth you can achieve.

I showed earlier how I profiled these things:


If the client is I/O-bound, that’s good: it means we can’t do any better
(unless we skip unpacking as demonstrated by distri).

If you can provide detailed profiles of either the server side or the
client side (but in that case, make sure the server is caching things),
that’d be great!

Otherwise I’m afraid this is not actionable. :-)

Thanks,
Ludo’.
L
L
Ludovic Courtès wrote on 3 Jul 2021 09:10
control message for bug #49174
(address . control@debbugs.gnu.org)
87czrzz8d6.fsf@gnu.org
tags 49174 + moreinfo
quit
M
M
Maxim Cournoyer wrote on 6 Oct 2022 18:11
Re: bug#49174: Poor 'guix substitute' performance when receiving Zstd-compressed substitutes
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 49174-done@debbugs.gnu.org)
87ilkw7803.fsf@gmail.com
Hi,

Ludovic Courtès <ludo@gnu.org> writes:

Toggle quote (37 lines)
> Hi,
>
> Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:
>
>> It's something I've been observing for a while, but substitutes are very
>> IO intensive (as can be seen in iotop, the substitute process is waiting
>> on IO > 99% of the time) and is much slower than expected (3 minutes to
>> transfer 100 MiB uncompressed over a 50 mbps downstream link):
>>
>> TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
>> 13934 be/4 root 1033.09 K/s 1485.06 K/s 0.00 % 93.36 % guile \ /gnu/store/vphx2839xv0qj9xwcwrb95592lzrrnx7-guix-1.3.0-3.50dfbbf/bin/guix substitute --substitute
>>
>>
>> The publisher (remote machine) is has its guix-daemon configured via:
>>
>> (service guix-publish-service-type
>> (guix-publish-configuration
>> (advertise? #t)
>> (compression '(("zstd" 3)))
>> (host "0.0.0.0"))) ;listen on all interfaces
>
> Note that in this case nars are built and compressed on the fly on the
> server side, which puts an upper bound on the bandwidth you can achieve.
>
> I showed earlier how I profiled these things:
>
> https://guix.gnu.org/en/blog/2021/getting-bytes-to-disk-more-quickly/
>
> If the client is I/O-bound, that’s good: it means we can’t do any better
> (unless we skip unpacking as demonstrated by distri).
>
> If you can provide detailed profiles of either the server side or the
> client side (but in that case, make sure the server is caching things),
> that’d be great!
>
> Otherwise I’m afraid this is not actionable. :-)

Since moving from HDDs to SSDs, I haven't seen this problem, so I
suspect the poor IO of the HDDs was really the culprit rather than
something to do with guile-zstd (and we had also benchmarked the late
some when I experimented with using zstd-compressed man pages).

Closing!

--
Thanks,
Maxim
Closed
?
Your comment

This issue is archived.

To comment on this conversation send an email to 49174@patchwise.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 49174
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch