Subject: [xml2rfc-dev] Non-ASCII punctuation

From: Jim Schaad <[email-masked]>
Date: Tue, 13 Feb 2018 09:43:41 -0800

I am currently running into problems with non-ASCII punctuation when doing spell checking, so I went to the RFC to see what it said about this issue.  As near as I can tell, there is nothing in the current document that says anything about this.  I believe that the document needs to be updated to address this issue.  I am not sure what I fell about this.  One of the issues in working on this is that the current kramdown converter to xml2rfc v2 language automatically uses the left and right double quotes rather than the vertical double quotes.

For text files, the use of characters such as ‟ (left double quote) is a problem as they are not ASCII.  In HTML, I believe that the general consensus is that the use of left and right double quotes is more readable than single quotes as it is what is generally used in typesetting.   I am not sure that the previous statement is correct or not.  On the third hand, consistency of quotes between the different display formats is probably a nice thing to have.

I believe that my opinion would be that they are permitted, however we need say that we are not going to want to allow some punctuation such as ¡ (inverted exclamation mark) even though they are in the Latin-1 page.  So if we allow extended punctuation, then we need to be clear that not all is going to be allowed except in specific cases (such as quoted text).

Jim

---

From: "Heather Flanagan (RFC Series Editor)" <[email-masked]>
Date: Tue, 13 Feb 2018 09:51:17 -0800

On 2/13/18 9:43 AM, Jim Schaad wrote:
>
> I am currently running into problems with non-ASCII punctuation when
> doing spell checking, so I went to the RFC to see what it said about
> this issue.  As near as I can tell, there is nothing in the current
> document that says anything about this.  I believe that the document
> needs to be updated to address this issue.  I am not sure what I fell
> about this.  One of the issues in working on this is that the current
> kramdown converter to xml2rfc v2 language automatically uses the left
> and right double quotes rather than the vertical double quotes.
>
>  
>
> For text files, the use of characters such as ‟ (left double quote) is
> a problem as they are not ASCII.  In HTML, I believe that the general
> consensus is that the use of left and right double quotes is more
> readable than single quotes as it is what is generally used in
> typesetting.   I am not sure that the previous statement is correct or
> not.  On the third hand, consistency of quotes between the different
> display formats is probably a nice thing to have.
>
>  
>
> I believe that my opinion would be that they are permitted, however we
> need say that we are not going to want to allow some punctuation such
> as ¡ (inverted exclamation mark) even though they are in the Latin-1
> page.  So if we allow extended punctuation, then we need to be clear
> that not all is going to be allowed except in specific cases (such as
> quoted text).
>

I'm doing a bit of homework on this, and found this to be interesting
reading:

https://www.cl.cam.ac.uk/~mgk25/ucs/quotes.html

I am leaning towards straight quotes as unambiguous and consistent. If
someone wants to use other punctuation in Latin-1 such as ¡, that's not
used in English (language of the series) so it would have to be treated
like any other non-ASCII character described in RFC 7997.

-Heather

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 13 Feb 2018 22:15:42 +0100

Hi Heather,

I own a hardbound *book* that is entirely about quotation marks.  It is a fun subject…
For a starter:
https://en.wikipedia.org/wiki/Quotation_mark#Summary_table

Kramdown-rfc generates “smart quotes” (i.e. typographically correct quotes, just as in this email) because, at the time I wrote the code, that setting just was the default, and I didn’t have to change it as xml2rfc happily converts typographic quotes into the typewriter quotes that ASCII offers.  So unless the author changes the settings (search for smart_quotes or 1.0.31 in https://github.com/cabo/kramdown-rfc2629), they get the characters that are called ldquo and rdquo (or lsquo/rsquo) in HTML.  I never changed this because the HTML generated from these XML files just reads better.

I would expect at least some other tools to output XML with typographically correct quotes, as well, so any prep tool for RFCXMLv3 would need to handle them.  By the way, similar considerations apply to other basic typographical characters such as dashes, hyphens, ellipses.

Obviously, RFCs can opt to side with ASCII on the typographic features of the English language even on the XML side, as xml2rfc has been doing for the TXT final form for a long time now.  If you want to go that way, that should not be a submission setting, but a post-preptool distribution setting.  (I personally don’t understand why we would need to do this, as XML is perfectly capable of handling basic typographic features as is HTML.  But it may still be the right thing for formatting to TXT, as it is done now.)

It becomes interesting when people start using non-English variants of quotes (e.g., the German left and right quotes, where the German right quote is identical in many fonts to the English left quote, but we use a lowered left quote).
That is not going to work well with English text.
Guillemets (“angle quotes”) are a boundary case, but probably should not be accepted either (they are rarely used in English, and they have inverse roles in French and Swiss vs. German German typography). 

I’m not going to start talking about thin spaces, but these also need to be handled reasonably.

Grüße, Carsten

PS.: Apple’s spelling dictionary “autocorrects” Guillemets into birds (Guillemots) — even Adobe fell for this malapropism, and Wikipedia outright says “they are not used in the English language”.  Yeah right.

> On Feb 13, 2018, at 18:51, Heather Flanagan (RFC Series Editor) <[email-masked]> wrote:
> 
> I'm doing a bit of homework on this, and found this to be interesting reading:
> 
> I am leaning towards straight quotes as unambiguous and consistent. If someone wants to use other punctuation in Latin-1 such as ¡, that's not used in English (language of the series) so it would have to be treated like any other non-ASCII character described in RFC 7997.
> 
> -Heather
> _______________________________________________
> xml2rfc-dev mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/xml2rfc-dev

---

From: Martin J. Dürst <[email-masked]>
Date: Wed, 14 Feb 2018 15:26:07 +0900

Hi Heather, others,

I was predicting (on this or a related mailing list I think) that this 
issue would turn up sooner or later. It was unavoidable, and easy to 
predict given that already before the 'modern' (incl. non-ASCII) RFCs, 
one occasionally saw Internet Drafts with 'smart' quotes.

On 2018/02/14 06:15, Carsten Bormann wrote:
> Hi Heather,
> 
> I own a hardbound *book* that is entirely about quotation marks.

Can you give us a pointer?

> It is a fun subject…

I think it's a bit more difficult than that. It is important to realize 
that RFCs often contain program text, and program text easily contains a 
few quotes, and these quotes have to be straight quotes, not typographic 
ones. On the other hand, many authors and editors won't use typographic 
quotes, but if the RFC editor wants to keep a consistent style, it would 
want to use typographic quotes in all RFCs where appropriate (i.e. for 
actual quotations, not strings in programs).

Getting that right would in my opinion be worth-while, but it's 
important to understand that it's something that will require a 
significant effort by the RFC production center.

> Guillemets (“angle quotes”) are a boundary case, but probably should not be accepted either (they are rarely used in English, and they have inverse roles in French and Swiss vs. German German typog…

At first approximation, there should be no need for these in RFCs. RFCs 
are English, so they should use English quotes. Citations from French or 
German including quotes should be exceedingly rare.

> I’m not going to start talking about thin spaces, but these also need to be handled reasonably.

And hyphens, and probably a few more things like this.

Regards,   Martin.

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 14 Feb 2018 10:47:33 +0100

On Feb 14, 2018, at 07:26, Martin J. Dürst <[email-masked]> wrote:
> 
> Hi Heather, others,
> 
> I was predicting (on this or a related mailing list I think) that this issue would turn up sooner or later. It was unavoidable, and easy to predict given that already before the 'modern' (incl. non-ASCII) RFCs, one occasionally saw Internet Drafts with 'smart' quotes.
> 

Oh, I'd have to dig that one up.  (I used it in the process of preparing my PhD thesis, so it has been a while.)

> I think it's a bit more difficult than that. It is important to realize that RFCs often contain program text, and program text easily contains a few quotes, and these quotes have to be straight quotes, not typographic ones.

Right, that’s why it is a good idea to support this right from the authoring level.
(If the author used markdown as intended, it all just works out right with very few exceptions.)

> On the other hand, many authors and editors won’t use typographic quotes, but if the RFC editor wants to keep a consistent style, it would want to use typographic quotes in all RFCs where appropriate (i.e. for actual quotations, not strings in programs).

Not sure we need this level of consistency across all RFCs.  If authors want to go for typographic quotes (manually or using tools that handle these for them), let them, but don’t force them on authors that couldn’t care less.  Again, that’s why it is a good thing to support this on the authoring level.

> Getting that right would in my opinion be worth-while, but it’s important to understand that it's something that will require a significant effort by the RFC production center.

(A few scripts can probably be written that will catch almost all errors an author can make here, with a small false-positive rate.)

> At first approximation, there should be no need for these in RFCs. RFCs are English, so they should use English quotes. Citations from French or German including quotes should be exceedingly rare.

Right.  (More so for Chinese, Japanese, Korean, Thai…). So I was focusing on the English usage.
(Oh, and there are at least two of these, with the British often exchanging the role of double and single quotes, starting around the 1960s.  Once more, best handled at the authoring level.)

> And hyphens, and probably a few more things like this.

And dashes, ellipses, …

Grüße, Carsten

---

From: Jim Schaad <[email-masked]>
Date: Wed, 14 Feb 2018 09:17:05 -0800

> -----Original Message-----
> From: xml2rfc-dev [mailto:[email-masked]] On Behalf Of
> Carsten Bormann
> Sent: Wednesday, February 14, 2018 1:48 AM
> To: "Martin J. Dürst" <[email-masked]>
> Cc: [email-masked]
> Subject: Re: [xml2rfc-dev] Non-ASCII punctuation
> 
> > I was predicting (on this or a related mailing list I think) that this issue would
> turn up sooner or later. It was unavoidable, and easy to predict given that
> already before the 'modern' (incl. non-ASCII) RFCs, one occasionally saw
> Internet Drafts with 'smart' quotes.
> >
> > I think it's a bit more difficult than that. It is important to realize that RFCs
> often contain program text, and program text easily contains a few quotes,
> and these quotes have to be straight quotes, not typographic ones.
> 
> > On the other hand, many authors and editors won’t use typographic
> quotes, but if the RFC editor wants to keep a consistent style, it would want
> to use typographic quotes in all RFCs where appropriate (i.e. for actual
> quotations, not strings in programs).
> 
> Not sure we need this level of consistency across all RFCs.  If authors want to
> go for typographic quotes (manually or using tools that handle these for
> them), let them, but don’t force them on authors that couldn’t care less.
> Again, that’s why it is a good thing to support this on the authoring level.
> 
> > Getting that right would in my opinion be worth-while, but it’s important to
> understand that it's something that will require a significant effort by the RFC
> production center.
> 
> (A few scripts can probably be written that will catch almost all errors an
> author can make here, with a small false-positive rate.)
> 
> >> It becomes interesting when people start using non-English variants of
> quotes (e.g., the German left and right quotes, where the German right
> quote is identical in many fonts to the English left quote, but we use a
> lowered left quote).
> >> Guillemets (“angle quotes”) are a boundary case, but probably should not
> be accepted either (they are rarely used in English, and they have inverse
> roles in French and Swiss vs. German German typography).
> >
> > At first approximation, there should be no need for these in RFCs. RFCs are
> English, so they should use English quotes. Citations from French or German
> including quotes should be exceedingly rare.
> 
> (Oh, and there are at least two of these, with the British often exchanging
> the role of double and single quotes, starting around the 1960s.  Once more,
> best handled at the authoring level.)
> 

A good place to start with the list of interesting symbols is the v2 xml2rfc code.  Between it and the DTD it references there is a list of entities that are going to be converted from the Unicode into ASCII text.

Jim