#47867 [1.2.1 pre-release testing] substitute downloading and TLS errors

Message #54 received at 47867@debbugs.gnu.org (full text, mbox, reply):

Received: (at 47867) by debbugs.gnu.org; 20 Apr 2021 22:39:09 +0000
From debbugs-submit-bounces@debbugs.gnu.org Tue Apr 20 18:39:09 2021
Received: from localhost ([127.0.0.1]:56330 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces@debbugs.gnu.org>)
	id 1lYz1R-00076j-1k
	for submit@debbugs.gnu.org; Tue, 20 Apr 2021 18:39:09 -0400
Received: from eggs.gnu.org ([209.51.188.92]:49944)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@gnu.org>) id 1lYz1O-00076M-Mm
 for 47867@debbugs.gnu.org; Tue, 20 Apr 2021 18:39:08 -0400
Received: from fencepost.gnu.org ([2001:470:142:3::e]:44086)
 by eggs.gnu.org with esmtp (Exim 4.90_1)
 (envelope-from <ludo@gnu.org>)
 id 1lYz1I-0004cP-UF; Tue, 20 Apr 2021 18:39:00 -0400
Received: from [2a01:e0a:1d:7270:af76:b9b:ca24:c465] (port=52838 helo=ribbon)
 by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256)
 (Exim 4.82) (envelope-from <ludo@gnu.org>)
 id 1lYz1I-0004ed-IG; Tue, 20 Apr 2021 18:39:00 -0400
From: Ludovic Courtès <ludo@gnu.org>
To: "pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de>
Subject: Re: bug#47867: [1.2.1 pre-release testing] substitute downloading
 and TLS errors
References: <87lf9hsh9t.fsf@gmail.com>
 <20210418093856.a6r2zqw2gks56dg5@pelzflorian.localdomain>
 <87a6pvnbjg.fsf@gnu.org>
 <20210418113337.z3afnp4lmt55cjvw@pelzflorian.localdomain>
 <20210418121006.4ihop3umb2oyofum@pelzflorian.localdomain>
 <20210419080516.piqb52s32h3fe5ag@pelzflorian.localdomain>
 <20210420012113.jcwuzfx6k3by5nvo@pelzflorian.localdomain>
 <20210420152754.a3ucylrdugrmxu57@pelzflorian.localdomain>
Date: Wed, 21 Apr 2021 00:38:58 +0200
In-Reply-To: <20210420152754.a3ucylrdugrmxu57@pelzflorian.localdomain>
 (pelzflorian@pelzflorian.de's message of "Tue, 20 Apr 2021 17:27:54
 +0200")
Message-ID: <8735vk8ttp.fsf@gnu.org>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux)
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="=-=-="
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 47867
Cc: guix-devel@gnu.org, jcguu95 <jcguu95@gmail.com>, 47867@debbugs.gnu.org
X-BeenThere: debbugs-submit@debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request@debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit@debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request@debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request@debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces@debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces@debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

[Message part 1 (text/plain, inline)]

Hi Florian,

"pelzflorian (Florian Pelz)" <pelzflorian@pelzflorian.de> skribis:

> On Tue, Apr 20, 2021 at 03:21:13AM +0200, pelzflorian (Florian Pelz) wrote:
>> > git revert be5a75ebb5988b87b2392e2113f6590f353dd6cd
>
> It seems this is the bad commit.  Downloading the enlightenment
> substitute got stuck and after a few minutes displayed the usual TLS
> error.

Note that on master there have been changes in this area since this
commit, in particular 20c08a8a45d0f137ead7c05e720456b2aea44402.

I assume the error we’re after is still this one:

>>    |    substitute: updating substitutes from 'https://ci.guix.gnu.org'... 0.0%guix substitute: error: TSL error in procedure 'write_to_session_port': Resource temporarily unavailable, try again.

I believe the attached patch “addresses” this problem.

Now, I’m a bit skeptical: the error above is GNUTLS_E_AGAIN, which
happens if sendmsg(2) returns EAGAIN, which supposedly only happens on
datagram (UDP) sockets or on non-blocking sockets, neither of which
applies here.

This change should be safe, though since I’m not sure why we’re getting
EAGAIN in the first place, I wonder what could happen (infinite loop?).

Could you give it a try?

Can you reproduce the substitute issue in a simpler environment?
For instance, by running:

  rm -rf ~/.cache/guix/substitute/
  ./pre-inst-env guix weather $(guix package -A |head -2000 |cut -f1) 

This makes 2000 pipelined GETs to https://ci.guix.gnu.org, which is
probably similar to what you observe during system installation.

Thanks for the thorough debugging and bisecting, as always!

Ludo’.

[Message part 2 (text/x-patch, inline)]

diff --git a/guix/http-client.scm b/guix/http-client.scm
index a2e11a1b73..51bba250d5 100644
--- a/guix/http-client.scm
+++ b/guix/http-client.scm
@@ -38,7 +38,7 @@
   #:use-module (guix utils)
   #:use-module (guix base64)
   #:autoload   (gcrypt hash) (sha256)
-  #:autoload   (gnutls) (error/invalid-session)
+  #:autoload   (gnutls) (error/invalid-session error/again)
   #:use-module ((guix build utils)
                 #:select (mkdir-p dump-port))
   #:use-module ((guix build download)
@@ -163,7 +163,8 @@ reusing stale cached connections."
       (if (or (and (eq? key 'system-error)
                    (= EPIPE (system-error-errno `(,key ,@args))))
               (and (eq? key 'gnutls-error)
-                   (eq? (first args) error/invalid-session))
+                   (memq (first args)
+                         (list error/again error/invalid-session)))
               (memq key
                     '(bad-response bad-header bad-header-component)))
           #f
diff --git a/guix/scripts/substitute.scm b/guix/scripts/substitute.scm
index 48309f9b3a..65940591a9 100755
--- a/guix/scripts/substitute.scm
+++ b/guix/scripts/substitute.scm
@@ -45,7 +45,7 @@
                 #:select (uri-abbreviation nar-uri-abbreviation
                           (open-connection-for-uri
                            . guix:open-connection-for-uri)))
-  #:autoload   (gnutls) (error/invalid-session)
+  #:autoload   (gnutls) (error/invalid-session error/again)
   #:use-module (guix progress)
   #:use-module ((guix build syscalls)
                 #:select (set-thread-name))
@@ -417,7 +417,8 @@ server certificates."
         (if (or (and (eq? key 'system-error)
                      (= EPIPE (system-error-errno `(,key ,@args))))
                 (and (eq? key 'gnutls-error)
-                     (eq? (first args) error/invalid-session))
+                     (memq (first args)
+                           (list error/again error/invalid-session)))
                 (memq key '(bad-response bad-header bad-header-component)))
             (proc (open-connection-for-uri/cached uri
                                                   #:verify-certificate? #f

Send a report that this bug log contains spam.