GNU bug report logs

#66684 [shepherd] Altering system time renders herd unresponsive

PackageSource(s)Maintainer(s)
guix PTS Buildd Popcon
Full log

Message #5 received at submit@debbugs.gnu.org (full text, mbox, reply):

Received: (at submit) by debbugs.gnu.org; 22 Oct 2023 16:41:14 +0000
From debbugs-submit-bounces@debbugs.gnu.org Sun Oct 22 12:41:14 2023
Received: from localhost ([127.0.0.1]:47963 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces@debbugs.gnu.org>)
	id 1qubVp-0001BU-E9
	for submit@debbugs.gnu.org; Sun, 22 Oct 2023 12:41:14 -0400
Received: from lists.gnu.org ([2001:470:142::17]:34830)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <vladilen.kozin@gmail.com>) id 1quYkc-0000WF-5M
 for submit@debbugs.gnu.org; Sun, 22 Oct 2023 09:44:18 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10])
 by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <vladilen.kozin@gmail.com>)
 id 1quYk3-00043B-SK
 for bug-guix@gnu.org; Sun, 22 Oct 2023 09:43:43 -0400
Received: from mail-ed1-x532.google.com ([2a00:1450:4864:20::532])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
 (Exim 4.90_1) (envelope-from <vladilen.kozin@gmail.com>)
 id 1quYk2-0003Gw-7d
 for bug-guix@gnu.org; Sun, 22 Oct 2023 09:43:43 -0400
Received: by mail-ed1-x532.google.com with SMTP id
 4fb4d7f45d1cf-53e2dc8fa02so3530897a12.2
 for <bug-guix@gnu.org>; Sun, 22 Oct 2023 06:43:40 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=gmail.com; s=20230601; t=1697982219; x=1698587019; darn=gnu.org;
 h=to:subject:message-id:date:from:mime-version:from:to:cc:subject
 :date:message-id:reply-to;
 bh=RYnLd9oKa9vC2fjy/Lmye3ugKSJY5zV2aDajbK4cTjY=;
 b=aT5Yyv6qrzIEPxSoSjAzW30dIW12Q2NOB3uVLgYdlfIBjNZW+X7/KrHSfXwvGgs3Qc
 iEcg4/XFmJLBMEhND8AQzQxrKIGJDwXoAA4rit3Mze0RqUMDfgfTNfZPyceb6Yy4XXzV
 Kmt3hjrX2LRQ9HjFcKgpBFeOauzlhs6uffX+TDkBMYIlS4kH+ezxj272m3RH5SO2lNC9
 cL5hjgu4Q79b4l+QG5fHUXX/8WcYPq4kwIq/eptWXBKMi6xDCXi50aOyUo/vjj2SuOTP
 diLCr/RNACpjzLaaToaoBoKP7NC/efjXc/u8Sa6TuxX/P4EwTQw35t7Ddzai1oM1Oxr9
 VQ5Q==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20230601; t=1697982219; x=1698587019;
 h=to:subject:message-id:date:from:mime-version:x-gm-message-state
 :from:to:cc:subject:date:message-id:reply-to;
 bh=RYnLd9oKa9vC2fjy/Lmye3ugKSJY5zV2aDajbK4cTjY=;
 b=kMnDfXERl3aMFvXijoNbRC38/dDidhxM+Oq3vNFq3pNJY9+pQhji51mY91f/ik3S2V
 5CYlsXK9DdfXaHdkNIOmll1B/NGDGVdkcPtrdaNNU3lhnx4SsatAjsQRruEiyo2RVmBU
 OwfWsnHP3nUSDwB2YENQEAXarr9yNfS1nw/KzgYso3zk4Ekxgt5UDWLdmzCCYf+BmPtd
 KH8N4ln1cHCVMWcKy5Ex2AQA1Rat8asVN4OhK3kijqlZVptPJGcbTW6MYddX74PXHOsj
 Qf/kVk/EvdVWFLtRhmHLuw37Prit5W7RfPkhoEDESqL4zj2AotZkVY2wsDl3BnnmSPpx
 mMnw==
X-Gm-Message-State: AOJu0YxjXUMpuVHPtNbmNYBEuj5Qo6H3kGWtd3dpRNmUemOHqWPDFoiw
 kUWo6tbvr4yyLR07zE08BRhbCRPhKvnNbOzsKhXyPoiwP2c=
X-Google-Smtp-Source: AGHT+IGhI5LaIU2Ibp1XjIJKTcv7sbC0xxKAF+ZnhYt+sTu27Iah/hU5nqWLqH48TQUVBUvjiXMZve53I5PQ4uuEQKM=
X-Received: by 2002:a50:aad8:0:b0:53f:9ced:e5c2 with SMTP id
 r24-20020a50aad8000000b0053f9cede5c2mr4097460edc.12.1697982218943; Sun, 22
 Oct 2023 06:43:38 -0700 (PDT)
MIME-Version: 1.0
From: Vladilen Kozin <vladilen.kozin@gmail.com>
Date: Sun, 22 Oct 2023 14:43:28 +0100
Message-ID: <CACw=CXN8dbRb8RmiHimqTs6J_QtSz5HuXaxf0mkRJeEEX1Wy7w@mail.gmail.com>
Subject: [shepherd] Altering system time renders herd unresponsive
To: bug-guix@gnu.org
Content-Type: text/plain; charset="UTF-8"
Received-SPF: pass client-ip=2a00:1450:4864:20::532;
 envelope-from=vladilen.kozin@gmail.com; helo=mail-ed1-x532.google.com
X-Spam_score_int: -20
X-Spam_score: -2.1
X-Spam_bar: --
X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001,
 RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001,
 SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: 1.0 (+)
X-Debbugs-Envelope-To: submit
X-Mailman-Approved-At: Sun, 22 Oct 2023 12:41:10 -0400
X-BeenThere: debbugs-submit@debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request@debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit@debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request@debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request@debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces@debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces@debbugs.gnu.org>
X-Spam-Score: -0.0 (/)
Hello guix.

My server would consistently run with system time 1h ahead of actual.
Both `date` and `hwclock` would show the same time off by 1hr, while
BIOS showed me correct time. I'm not sure why, but some services won't
run if time difference is e.g. over 15min or smth, so.

$ sudo date -s '-1 hour'

fixes time but causes `herd` to become unresponsive as in you type a
command, any command and stare at tty stuck. Also ssh'ing into the
system becomes impossible. Any attempt gets logged in
/var/log/messages - I can see that, but you again just stare at
unresponsive terminal. Initially I thought it fried shepherd
completely, so I powercycle the system to get it back. `sudo reboot`
being an alias to `herd` command will of course not work - so you have
to do it physically. Annoying but feasible on a desktop system -
complete nightmare on a physical server which may take up to 20min to
reboot due to inventory lifecycle and such.

By chance, I got distracted this time and just left it hanging. Lo and
behold it unfroze some 15-20min later. What gives I've no clue.

I hope I won't be seeing this particular issue again, cause I followed
system clock alteration with:
$ sudo hwclock -w
and reboot shows correct time.

In general my experience with shepherd has been less than stellar.
IMO, this just shouldn't happen with PID 1 ever - cause there isn't
anything you can do at this point. Not the first time it became
unresponsive. On occasion after pull that changes some user service
code, followed by system reconfigure those services would start
failing to find their binaries - best guess I have there is that those
specific services depend on user-home service or some such and
something happens that prevents discovery of said binaries in PATH -
binaries in those services aren't referenced by absolute path in GNU
store. Separate issue.

Generation 8 Oct 14 2023 00:22:53 (current)
  file name: /var/guix/profiles/system-8-link
  canonical file name: /gnu/store/j9i2w1zacw7sl8vlb7k1g7p0vnd58ns7-system
  label: GNU with Linux 6.4.16
  bootloader: grub
  root device: label: "r720-guix-0"
  kernel: /gnu/store/cbc7x9in2dnjrnh840c21ivgygnndp1c-linux-6.4.16/bzImage
  channels:
    guix:
      repository URL: https://git.savannah.gnu.org/git/guix.git
      branch: master
      commit: 3963fa1a465708690cd1554d911613f1c92f5eef

Thank you

-- 
Best regards
Vlad Kozin




Send a report that this bug log contains spam.


debbugs.gnu.org maintainers <help-debbugs@gnu.org>. Last modified: Sun Dec 22 16:16:32 2024; Machine Name: wallace-server

GNU bug tracking system

Debbugs is free software and licensed under the terms of the GNU Public License version 2. The current version can be obtained from https://bugs.debian.org/debbugs-source/.

Copyright © 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson, 2005-2017 Don Armstrong, and many other contributors.