Subject: [xml2rfc-dev] When is @ascii required?

From: Carsten Bormann <[email-masked]>
Date: Sun, 27 Oct 2019 18:24:03 +0100

I’m not quite sure I understand when I have to provide the ascii=“” attribute.

I can have authors that have umlauts without further ado.
If I have organizations with umlauts, I need to put in ascii=“”, or I get:

t2trg-sworn.xml(20): Error: Found non-ascii content without matching ascii attribute in <organization>: Universit?t Bremen TZI

(Note also that the error message has some mojibake.)

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 27 Oct 2019 18:51:31 +0100

(Looking at the source of the message in preptool.py/check_ascii_text, I also don’t understand this:

                if self.tree.docinfo.encoding.lower() in ['us-ascii', ]:
                    self.die(c, “Found non-ascii content in a document with xml encoding declared as %s” % self.tree.docinfo.encoding)

I don’t understand what the source encoding has to do with this; XML allows me to have beyond-ascii characters in documents the xml encoding of which uses us-ascii (or koi8-r, for that matter).  I think that any access to self.tree.docinfo.encoding at this point in processing is a layer violation.)

Grüße, Carsten

> On Oct 27, 2019, at 18:24, Carsten Bormann <[email-masked]> wrote:
>

---

From: Henrik Levkowetz <[email-masked]>
Date: Sun, 27 Oct 2019 19:40:51 +0100

Hi Carsten,

> t2trg-sworn.xml(20): Error: Found non-ascii content without matching ascii attribute in <organization>: Universit?t Bremen TZI

This is inconsistent.

Based on feedback regarding publication practice regarding ASCII, Latin,
and non-Latin alphabets some time ago, xml2rfc is rendering names (person
and org) with Latin script characters the same way as ASCII, while names
with non-Latin script code points require an additional ascii variant,
which will be rendered in parentheses.

So the grumbling about missing ascii attributes for organization is
inconsistent, and needs to be adjusted.  I'll do so for the next release.

> (Note also that the error message has some mojibake.)

Is this from the web service or on the command line?

	Henrik

---

From: Henrik Levkowetz <[email-masked]>
Date: Sun, 27 Oct 2019 19:50:39 +0100

Hi Carsten,

On 2019-10-27 18:51, Carsten Bormann wrote:
> I don’t understand what the source encoding has to do with this; XML
> allows me to have beyond-ascii characters in documents the xml
> encoding of which uses us-ascii (or koi8-r, for that matter).  I
> think that any access to self.tree.docinfo.encoding at this point in
> processing is a layer violation.)

It may very well be that the test can be improved, but it was triggered
by a hard-to-diagnose failure where an XML file had us-ascii encoding
declared, but contained non-ascii characters.  The RPC didn't manage to
get a handle on what the problem was that triggered the failures, and it
also took me a while to understand what was going on.

	Henrik

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 27 Oct 2019 20:14:25 +0100

>> t2trg-sworn.xml(20): Error: Found non-ascii content without matching ascii attribute in <organization>: Universit?t Bremen TZI

Thanks.  So I’ll continue those experiments with the next xml2rfc version.

(I think that the general direction to allow Latin input but require manual ASCII transliteration for non-Latin is right — most English-speaking people should be able to approximately render Latin outside ßðÐþÞŋƣƦ and maybe İıĸ; there are transliteration libraries for other scripts as well but employing them is probably not a very stable approach.)

> Is this from the web service or on the command line?

Command line.  I notice that I had a Python 2.7 install running; with 3.7 (I know) I get:

t2trg-sworn.xml(20): Error: Found non-ascii content without matching ascii attribute in <organization>: b'Universit?t Bremen TZI’

which is interesting.

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 27 Oct 2019 20:20:45 +0100

On Oct 27, 2019, at 19:50, Henrik Levkowetz <[email-masked]> wrote:
> declared, but contained non-ascii characters. 

Do you remember whether it contained non-ASCII characters themselves (which should already fail in the parser) or character-references to beyond-ASCII characters?  Or maybe entity references?

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Sun, 27 Oct 2019 20:51:46 +0100

On 27.10.2019 19:50, Henrik Levkowetz wrote:
> ...

That should be an error reported at the time of XML parsing...

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Mon, 28 Oct 2019 06:17:49 +0100

> (Note also that the error message has some mojibake.)

A similar issue is
<https://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/443>, reported
two weeks ago.

Best regards, Julian

---

From: Henrik Levkowetz <[email-masked]>
Date: Mon, 28 Oct 2019 15:03:22 +0100

Hi Carsten,

> Do you remember whether it contained non-ASCII characters themselves
> (which should already fail in the parser) or character-references to
> beyond-ASCII characters?  Or maybe entity references?

Oh, it was non-ASCII characters in the input file, and all would have been
well if it had failed cleanly with a reasonable error message in the parser,
but it didn't.

I'll look at relocating this code so as to do a first parse without
expansions of entity references, in order to not block use of such, but
still catch non-ASCII characters in files declared to be us-ascii.

Best regards,

	Henrik

---

From: Julian Reschke <[email-masked]>
Date: Mon, 28 Oct 2019 15:29:25 +0100

> well if it had failed cleanly with a reasonable error message in the parser,

Hm, if you can reproduce this you definitively should raise a bug report
against the XML parser...

Best regards, Julian