#33848 Store references in SBCL-compiled code are "invisible"

Message #46 received at 33848@debbugs.gnu.org (full text, mbox, reply):

Received: (at 33848) by debbugs.gnu.org; 27 Dec 2018 13:53:13 +0000
From debbugs-submit-bounces@debbugs.gnu.org Thu Dec 27 08:53:13 2018
Received: from localhost ([127.0.0.1]:38272 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces@debbugs.gnu.org>)
	id 1gcW64-0000SO-JX
	for submit@debbugs.gnu.org; Thu, 27 Dec 2018 08:53:12 -0500
Received: from dd26836.kasserver.com ([85.13.145.193]:50892)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <dannym@scratchpost.org>) id 1gcW61-0000SE-8u
 for 33848@debbugs.gnu.org; Thu, 27 Dec 2018 08:53:11 -0500
Received: from localhost (77.116.200.150.wireless.dyn.drei.com
 [77.116.200.150])
 by dd26836.kasserver.com (Postfix) with ESMTPSA id E30FC3360147;
 Thu, 27 Dec 2018 14:53:06 +0100 (CET)
Date: Thu, 27 Dec 2018 14:52:58 +0100
From: Danny Milosavljevic <dannym@scratchpost.org>
To: Mark H Weaver <mhw@netris.org>
Subject: Re: bug#33848: Store references in SBCL-compiled code are "invisible"
Message-ID: <20181227145258.0c420eac@scratchpost.org>
In-Reply-To: <87tvj2yesd.fsf@netris.org>
References: <87r2e8jpfx.fsf@gnu.org> <877eg0i43j.fsf@netris.org>
 <87d0psi1xo.fsf@gnu.org> <874lb3kin6.fsf@ambrevar.xyz>
 <87sgynezha.fsf@gnu.org> <87tvj2yesd.fsf@netris.org>
X-Mailer: Claws Mail 3.17.1 (GTK+ 2.24.32; x86_64-unknown-linux-gnu)
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha256;
 boundary="Sig_/psKMAB3Tik_KCO1Se691Qt1"; protocol="application/pgp-signature"
X-Spam-Score: -0.7 (/)
X-Debbugs-Envelope-To: 33848
Cc: Ludovic Courtès <ludo@gnu.org>,
 Pierre Neidhardt <mail@ambrevar.xyz>, 33848@debbugs.gnu.org
X-BeenThere: debbugs-submit@debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request@debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit@debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request@debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request@debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces@debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces@debbugs.gnu.org>
X-Spam-Score: -1.7 (-)

[Message part 1 (text/plain, inline)]

Hi Mark,

On Mon, 24 Dec 2018 13:12:23 -0500
Mark H Weaver <mhw@netris.org> wrote:

> Of course, the usual reason to choose UTF-32 is to support non-ASCII
> characters while retaining fixed-width code points, so that string
> lookups are straightforward and efficient.

This kind of lookup is almost never what is necessary.  There are many
people who assume character is the same as codepoint and to those people
UTF-32 brings something to the table, but it's really not useful if people
do text processing correctly, see below.

(Of course whether packages actually do this remains to be seen)

>  Using UTF-8 improves space efficiency, but at the cost of extra code
>complexity.

I agree.

>  That extra
> complexity is what I guess we would need to add to each program that
> currently uses UTF-32.

Yes, but they usually have to do stream processing even with UTF-32 (because
a character can be composed of possibly infinite number of codepoints),
so the infrastructure should be already there and the effort should be
minimal.

>  Alternatively, we could extend the on-disk
> format to support UTF-8 and then add some kind of "load hook" that
> converts the string to UTF-32 at load time.  Either way, it's likely to
> be a can of worms.

If it ever came to that, a pluggable reference scanner would be 
preferrable.  But really, it would irk me to have so much complexity
in something so basic (the reference scanner) for no end-user gain
(as a distribution we could just mandate UTF-8 for references and the
problem would be gone for the user with no loss of functionality).

It's always easy to add special cases - but more code means more bugs
and I think if possible it's best to have only the simple case implemented
in the core - because it's less complicated which means more likely
to be correct (for the case it does handle).  In the end it depends on
what would be more code, and more widely used.

Also, if we wanted to debug reference errors, we couldn't use grep anymore
because it can't handle utf-32 either (neither can any of the other UNIX tools).

Also, I really don't want to return to the time where I had to call iconv
once every three commands to be able to do anything useful on UNIX.

Also, the build daemon is written in C++ and C++ strings are widely
known to have very very bad codepoint awareness (to say nothing about
the horrible conversion facilities).

Also, if both UTF-32 and UTF-8 are used on disk, care needs to not misdetect
an UTF-8 sequence as an UTF-32 sequence of different text - or the other way
around -, but that's unlikely for ASCII strings.

> I really think it would be a mistake to try to force every program and
> language implementation to use our preferred string representation.  I
> suspect it would be vastly easier to compromise and support a few other
> popular string representations in Guix, namely UTF-16 and UTF-32.

In 1992, UTF-8 was invented.  Subsequently, most of the Internet,
all new GNU Linux distributions etc, all UNIX GUI frameworks, Subversion
etc standardized on UTF-8, with the eventual goal of standardizing all
network transfer and storage to UTF-8.  I think that by now the outliers
are the ones who need to change, otherwise these senseless encoding
conversions will never cease.  It's not like different encodings allow for
better expression of writings or anything useful to the end user.

As a distribution we can't force upstream to change, but just filing
bug reports upstream would make us see where they stand on this.

> If you don't want to change the daemon, it could be worked around in our
> build-side code as follows: we could add a new phase to certain build
> systems (or possibly gnu-build-system) that scans each output for
> UTF-16/32 encoded store references that are never referenced in UTF-8.
> If such references exist, a file with an unobtrusive name would be added
> to that output containing those references encoded in UTF-8.  This would
> enable our daemon's existing reference scanner to find all of the
> references.

I agree that that would be nice.  As a first step, even just detecting
problems like that and erroring out would be okay - in order to find them
in the first place.  Right now, it's difficult to detect and so also difficult
to say how wide-spread the problem is.  If the problem is wide-spread enough
my tune could change very quickly.

What you propose is similar to what I did in Java in Guix, only it gives
us even more advantages in the Java case (faster class loading and
eventual non-propagated inputs).

[Message part 2 (application/pgp-signature, inline)]

Send a report that this bug log contains spam.