'add-to-store' RPC loads files in memory (daemon-side)

  • Open
  • quality assurance status badge
Details
3 participants
  • Andreas Enge
  • Leo Famulari
  • Ludovic Courtès
Owner
unassigned
Submitted by
Andreas Enge
Severity
normal
Merged with

Debbugs page

A
A
Andreas Enge wrote on 31 May 2016 12:56
guix download fails for large files
(address . bug-guix@gnu.org)
20160531195631.GA22897@solar
Hello,

since I am regularly rebuilding texlive (who is not!), I would like to keep
the source tarball on my machine, and whenever I garbage collect it from the
store, add it back via
guix download file:texlive-20150523-texmf.tar.xz

On my Novena with a 32 bit processor and 4 GB of main memory, this fails with
guix download: error: sendfile: Broken pipe

The file itself has 1,8 GB.

Downloading the file via http as part of "guix build" works, and
"guix download file:" also works on my x86_64 machine with 8 GB of RAM.

Andreas
L
L
Leo Famulari wrote on 31 May 2016 13:26
(name . Andreas Enge)(address . andreas@enge.fr)(address . 23666@debbugs.gnu.org)
20160531202651.GA19070@jasmine
On Tue, May 31, 2016 at 09:56:31PM +0200, Andreas Enge wrote:
Toggle quote (13 lines)
> since I am regularly rebuilding texlive (who is not!), I would like to keep
> the source tarball on my machine, and whenever I garbage collect it from the
> store, add it back via
> guix download file:texlive-20150523-texmf.tar.xz
>
> On my Novena with a 32 bit processor and 4 GB of main memory, this fails with
> guix download: error: sendfile: Broken pipe
>
> The file itself has 1,8 GB.
>
> Downloading the file via http as part of "guix build" works, and
> "guix download file:" also works on my x86_64 machine with 8 GB of RAM.

As a temporary workaround, you could make the tarball a gcroot:

# ln -sf /gnu/store/...-texlive-20150523-texmf.tar.xz \
/var/guix/gcroots/texlive-texmf.tar.xz
L
L
Ludovic Courtès wrote on 1 Jun 2016 05:39
(name . Andreas Enge)(address . andreas@enge.fr)(address . 23666@debbugs.gnu.org)
878typkrqd.fsf@gnu.org
Andreas Enge <andreas@enge.fr> skribis:

Toggle quote (5 lines)
> On my Novena with a 32 bit processor and 4 GB of main memory, this fails with
> guix download: error: sendfile: Broken pipe
>
> The file itself has 1,8 GB.

This is implemented using the ‘add-to-store’ RPC, which, after all these
years, is still implemented like this (nix/libstore/local-store.cc):

Toggle snippet (11 lines)
Path LocalStore::addToStore(const string & name, const Path & _srcPath,
bool recursive, HashType hashAlgo, PathFilter & filter, bool repair)
{
Path srcPath(absPath(_srcPath));
debug(format("adding `%1%' to the store") % srcPath);

/* Read the whole path into memory. This is not a very scalable
method for very large paths, but `copyPath' is mainly used for
small files. */

This is undoubtedly what’s causing the failure.

Until this is fixed, you could wrap the file in a nar and import the nar
(nar import does not have this limitation, see ’LocalStore::importPath’
and co.).

To do that, you could do:

guix archive --export $(guix build -S texlive-texmf-whatever) > t.nar

and later:

guix archive --import < t.nar

HTH,
Ludo’.
L
L
Ludovic Courtès wrote on 1 Jun 2016 05:40
retitle
(address . request@debbugs.gnu.org)
874m9dkroo.fsf@gnu.org
retitle 23666 'add-to-store' RPC loads files in memory (daemon-side)
thanks
A
A
Andreas Enge wrote on 5 Jun 2016 13:58
Re: bug#23666: guix download fails for large files
(name . Ludovic Courtès)(address . ludo@gnu.org)(address . 23666@debbugs.gnu.org)
20160605205839.GB20831@solar
Thanks for your suggestions, Leo and Ludovic! I still see this as a bug;
should I report it upstream to Nix?

On Wed, Jun 01, 2016 at 02:39:54PM +0200, Ludovic Courtès wrote:
Toggle quote (15 lines)
> This is implemented using the ‘add-to-store’ RPC, which, after all these
> years, is still implemented like this (nix/libstore/local-store.cc):
>
> --8<---------------cut here---------------start------------->8---
> Path LocalStore::addToStore(const string & name, const Path & _srcPath,
> bool recursive, HashType hashAlgo, PathFilter & filter, bool repair)
> {
> Path srcPath(absPath(_srcPath));
> debug(format("adding `%1%' to the store") % srcPath);
>
> /* Read the whole path into memory. This is not a very scalable
> method for very large paths, but `copyPath' is mainly used for
> small files. */
> --8<---------------cut here---------------end--------------->8---

Something that mainly does not fail could indeed be seen as a bug...
But how come that "guix download http://" succeeds, where
"guix download file://" fails?

Andreas
L
L
Ludovic Courtès wrote on 6 Jun 2016 00:06
(name . Andreas Enge)(address . andreas@enge.fr)(address . 23666@debbugs.gnu.org)
87k2i27q5c.fsf@gnu.org
Andreas Enge <andreas@enge.fr> skribis:

Toggle quote (20 lines)
> On Wed, Jun 01, 2016 at 02:39:54PM +0200, Ludovic Courtès wrote:
>> This is implemented using the ‘add-to-store’ RPC, which, after all these
>> years, is still implemented like this (nix/libstore/local-store.cc):
>>
>> --8<---------------cut here---------------start------------->8---
>> Path LocalStore::addToStore(const string & name, const Path & _srcPath,
>> bool recursive, HashType hashAlgo, PathFilter & filter, bool repair)
>> {
>> Path srcPath(absPath(_srcPath));
>> debug(format("adding `%1%' to the store") % srcPath);
>>
>> /* Read the whole path into memory. This is not a very scalable
>> method for very large paths, but `copyPath' is mainly used for
>> small files. */
>> --8<---------------cut here---------------end--------------->8---
>
> Something that mainly does not fail could indeed be seen as a bug...
> But how come that "guix download http://" succeeds, where
> "guix download file://" fails?

Both use the ‘add-to-store’ RPC, so both have equal chances of failing.

Note that, fortunately, fixed-output derivations (‘guix build -S foo’)
do not use ‘add-to-store’. In general, except in the case of ‘guix
download’, ‘add-to-store’ is only used with small files like when adding
.scm files to the store, etc.

Ludo’.
L
L
Ludovic Courtès wrote on 24 Jul 2016 07:03
control message for bug #24053
(address . control@debbugs.gnu.org)
87oa5ncevv.fsf@gnu.org
merge 24053 23666
?
Your comment

Commenting via the web interface is currently disabled.

To comment on this conversation send an email to 23666@patchwise.org

To respond to this issue using the mumi CLI, first switch to it
mumi current 23666
Then, you may apply the latest patchset in this issue (with sign off)
mumi am -- -s
Or, compose a reply to this issue
mumi compose
Or, send patches to this issue
mumi send-email *.patch