#23666 'add-to-store' RPC loads files in memory (daemon-side)

Message #5 received at submit@debbugs.gnu.org (full text, mbox, reply):

From: Andreas Enge <andreas@enge.fr>

To: bug-guix@gnu.org

Subject: guix download fails for large files

Date: Tue, 31 May 2016 21:56:31 +0200

Hello,

since I am regularly rebuilding texlive (who is not!), I would like to keep
the source tarball on my machine, and whenever I garbage collect it from the
store, add it back via
   guix download file:texlive-20150523-texmf.tar.xz

On my Novena with a 32 bit processor and 4 GB of main memory, this fails with
   guix download: error: sendfile: Broken pipe

The file itself has 1,8 GB.

Downloading the file via http as part of "guix build" works, and
"guix download file:" also works on my x86_64 machine with 8 GB of RAM.

Andreas

Message #8 received at 23666@debbugs.gnu.org (full text, mbox, reply):

From: Leo Famulari <leo@famulari.name>

To: Andreas Enge <andreas@enge.fr>

Cc: 23666@debbugs.gnu.org

Subject: Re: bug#23666: guix download fails for large files

Date: Tue, 31 May 2016 16:26:51 -0400

On Tue, May 31, 2016 at 09:56:31PM +0200, Andreas Enge wrote:
> since I am regularly rebuilding texlive (who is not!), I would like to keep
> the source tarball on my machine, and whenever I garbage collect it from the
> store, add it back via
>    guix download file:texlive-20150523-texmf.tar.xz
>
> On my Novena with a 32 bit processor and 4 GB of main memory, this fails with
>    guix download: error: sendfile: Broken pipe
> 
> The file itself has 1,8 GB.
> 
> Downloading the file via http as part of "guix build" works, and
> "guix download file:" also works on my x86_64 machine with 8 GB of RAM.

As a temporary workaround, you could make the tarball a gcroot:

# ln -sf /gnu/store/...-texlive-20150523-texmf.tar.xz \
/var/guix/gcroots/texlive-texmf.tar.xz

Message #11 received at 23666@debbugs.gnu.org (full text, mbox, reply):

From: ludo@gnu.org (Ludovic Courtès)

To: Andreas Enge <andreas@enge.fr>

Cc: 23666@debbugs.gnu.org

Subject: Re: bug#23666: guix download fails for large files

Date: Wed, 01 Jun 2016 14:39:54 +0200

Andreas Enge <andreas@enge.fr> skribis:

> On my Novena with a 32 bit processor and 4 GB of main memory, this fails with
>    guix download: error: sendfile: Broken pipe
>
> The file itself has 1,8 GB.

This is implemented using the ‘add-to-store’ RPC, which, after all these
years, is still implemented like this (nix/libstore/local-store.cc):

--8<---------------cut here---------------start------------->8---
Path LocalStore::addToStore(const string & name, const Path & _srcPath,
    bool recursive, HashType hashAlgo, PathFilter & filter, bool repair)
{
    Path srcPath(absPath(_srcPath));
    debug(format("adding `%1%' to the store") % srcPath);

    /* Read the whole path into memory. This is not a very scalable
       method for very large paths, but `copyPath' is mainly used for
       small files. */
--8<---------------cut here---------------end--------------->8---

This is undoubtedly what’s causing the failure.

Until this is fixed, you could wrap the file in a nar and import the nar
(nar import does not have this limitation, see ’LocalStore::importPath’
and co.).

To do that, you could do:

  guix archive --export $(guix build -S texlive-texmf-whatever) > t.nar

and later:

  guix archive --import < t.nar

HTH,
Ludo’.

Message #16 received at 23666@debbugs.gnu.org (full text, mbox, reply):

From: Andreas Enge <andreas@enge.fr>

To: Ludovic Courtès <ludo@gnu.org>

Cc: 23666@debbugs.gnu.org

Subject: Re: bug#23666: guix download fails for large files

Date: Sun, 5 Jun 2016 22:58:39 +0200

Thanks for your suggestions, Leo and Ludovic! I still see this as a bug;
should I report it upstream to Nix?

On Wed, Jun 01, 2016 at 02:39:54PM +0200, Ludovic Courtès wrote:
> This is implemented using the ‘add-to-store’ RPC, which, after all these
> years, is still implemented like this (nix/libstore/local-store.cc):
> 
> --8<---------------cut here---------------start------------->8---
> Path LocalStore::addToStore(const string & name, const Path & _srcPath,
>     bool recursive, HashType hashAlgo, PathFilter & filter, bool repair)
> {
>     Path srcPath(absPath(_srcPath));
>     debug(format("adding `%1%' to the store") % srcPath);
> 
>     /* Read the whole path into memory. This is not a very scalable
>        method for very large paths, but `copyPath' is mainly used for
>        small files. */
> --8<---------------cut here---------------end--------------->8---

Something that mainly does not fail could indeed be seen as a bug...
But how come that "guix download http://" succeeds, where
"guix download file://" fails?

Andreas

Message #19 received at 23666@debbugs.gnu.org (full text, mbox, reply):

From: ludo@gnu.org (Ludovic Courtès)

To: Andreas Enge <andreas@enge.fr>

Cc: 23666@debbugs.gnu.org

Subject: Re: bug#23666: guix download fails for large files

Date: Mon, 06 Jun 2016 09:06:07 +0200

Andreas Enge <andreas@enge.fr> skribis:

> On Wed, Jun 01, 2016 at 02:39:54PM +0200, Ludovic Courtès wrote:
>> This is implemented using the ‘add-to-store’ RPC, which, after all these
>> years, is still implemented like this (nix/libstore/local-store.cc):
>> 
>> --8<---------------cut here---------------start------------->8---
>> Path LocalStore::addToStore(const string & name, const Path & _srcPath,
>>     bool recursive, HashType hashAlgo, PathFilter & filter, bool repair)
>> {
>>     Path srcPath(absPath(_srcPath));
>>     debug(format("adding `%1%' to the store") % srcPath);
>> 
>>     /* Read the whole path into memory. This is not a very scalable
>>        method for very large paths, but `copyPath' is mainly used for
>>        small files. */
>> --8<---------------cut here---------------end--------------->8---
>
> Something that mainly does not fail could indeed be seen as a bug...
> But how come that "guix download http://" succeeds, where
> "guix download file://" fails?

Both use the ‘add-to-store’ RPC, so both have equal chances of failing.

Note that, fortunately, fixed-output derivations (‘guix build -S foo’)
do not use ‘add-to-store’.  In general, except in the case of ‘guix
download’, ‘add-to-store’ is only used with small files like when adding
.scm files to the store, etc.

Ludo’.

Send a report that this bug log contains spam.