Subject: [xml2rfc] Generating HTML and PDF with Unicode (and diagrams?)

From: Larry Masinter <[email-masked]>
Date: Fri, 9 Mar 2012 16:09:40 -0800

I was wondering if anyone has created a conditional preprocessor for xml2rfc where  alternative content could be supplied for the .html and .html.pdf editions of the document, even if the .txt version remains ASCII only and with ASCII art, the .html version could be rendered with actual unicode characters.

I'm not imagining automatic conversion, but just manually supplying alternative in the .xml source.

---

From: Nico Williams <[email-masked]>
Date: Fri, 9 Mar 2012 18:54:18 -0600

On Fri, Mar 9, 2012 at 6:09 PM, Larry Masinter <[email-masked]> wrote:
> I was wondering if anyone has created a conditional preprocessor for xml2rfc
> where  alternative content could be supplied for the .html and .html.pdf
> editions of the document, even if the .txt version remains ASCII only and
> with ASCII art, the .html version could be rendered with actual unicode
> characters.

I do hope at some point we switch to UTF-8 for .txt...

Nico
--

---

From: Julian Reschke <[email-masked]>
Date: Sat, 10 Mar 2012 02:04:14 +0100

On 2012-03-10 01:09, Larry Masinter wrote:
> I was wondering if anyone has created a conditional preprocessor for
> xml2rfc where alternative content could be supplied for the .html and
> .html.pdf editions of the document, even if the .txt version remains
> ASCII only and with ASCII art, the .html version could be rendered with
> actual unicode characters.
>
> I’m not imagining automatic conversion, but just manually supplying
> alternative in the .xml source.
> ...

I haven't done that, but it would be an interesting experiment. If you 
have a sample in mind, I can put together an XSLT, or even integrate it 
into my existing XSLTs.

Best regards, Julian

---

From: Russ Allbery <[email-masked]>
Date: Fri, 09 Mar 2012 17:24:53 -0800

Nico Williams <[email-masked]> writes:
> Larry Masinter <[email-masked]> wrote:

> I do hope at some point we switch to UTF-8 for .txt...

Amen.  It's quite frustrating to be required to misspell people's names.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>

---

From: Larry Masinter <[email-masked]>
Date: Sat, 10 Mar 2012 07:49:42 -0800

> Amen.  It's quite frustrating to be required to misspell people's names.

I don't think reducing the "frustration" of "misspelling" people's names is a sufficient motivation.

The risks of switching to UTF-8 for .txt have considerable overlap with the security considerations for IRIs (and IDN). The requirement is for global transcriability -- that no matter what your natural language capabilities are, that if you are able to read the technical content of an RFC, you can read and transcribe it.

My only concern at this point is to allow non-normative examples and illustrations to be more readily included.  

I'm puzzling about how extensive to make the choice... whether it's an alternative at the <t> level? Or even to make it only an optional _additional_ section, e.g., the HTML/PDF version would include all of the ASCII text but might also contain additional (XHTML? Or a very restricted subset of XHTML tags? Or SVG diagrams? Images?) markup to be inserted.

Larry

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Russ Allbery
Sent: Friday, March 09, 2012 5:25 PM
To: [email-masked]
Subject: Re: [xml2rfc] Generating HTML and PDF with Unicode (and diagrams?)

>> xml2rfc where  alternative content could be supplied for the .html 
>> and .html.pdf editions of the document, even if the .txt version 
>> remains ASCII only and with ASCII art, the .html version could be 
>> rendered with actual unicode characters.

-- 
Russ Allbery ([email-masked])             <http://www.eyrie.org/~eagle/>
_______________________________________________
xml2rfc mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/xml2rfc

---

From: Julian Reschke <[email-masked]>
Date: Sat, 10 Mar 2012 19:25:36 +0100

On 2012-03-10 16:49, Larry Masinter wrote:
> I'm puzzling about how extensive to make the choice... whether it's an alternative at the<t>  level? Or even to make it only an optional _additional_ section, e.g., the HTML/PDF version would include all of the ASCII text but might also contain additional (XHTML? Or a very restricted subset of XHTML tags? Or SVG diagrams? Images?) markup to be inserted.
>
> Larry
> ...

Maybe we should start small, and look at your concrete use case.

I'll assume it has to do with putting non-ASCII characters into IRI 
examples?

Best regards, Julian

---

From: Russ Allbery <[email-masked]>
Date: Sat, 10 Mar 2012 12:06:05 -0800

> is a sufficient motivation.

I don't really want to get into a long discussion of this, but I do want
to say that I disagree that this is not sufficient motivation.
Attribution in published work is important, and attribution should involve
correctly spelling and representing people's names.  I find it faintly
embarassing to use a publication format that privileges English names over
all of the other contributors around the world.  I realize that this isn't
completely unreasonable given that the publication language is English,
and at the time there was no better alternative, but I think we have a
better alternative now.

But I read http://www-cs-faculty.stanford.edu/~uno/help.html (Non-Latin
Names) at an impressionable age.  :)

---

From: Larry Masinter <[email-masked]>
Date: Sat, 10 Mar 2012 12:26:23 -0800

Sorry, I was trying to have a technical discussion of requirements and use cases, cost and benefit, not a political rant.

> names is a sufficient motivation.

was taken completely out of context. 

You said: "I don't really want to get into a long discussion of this"
but you cannot misquote me and then refuse to discuss it.

>  but I do want to say that I disagree that this is not sufficient motivation.

The context you left out was essential. I said   "Reducing pain authoring", where said pain was caused by misspelling, is not "a sufficient reason" for change Y.
There are several factors in this evaluation:
A: The pain is not eliminated.  It is still necessary to misspell people's names, since there is still a need to prepare an ASCII-only text file.
B: The pain already occurs naturally for all names, since names are sometimes misspelled.
C: There are other related pains that might occur around embarrassment in causing a publication

In your reply, you calibrate your amount of embarrassment ("faintly") against a whole authoring policy for which you are not faintly responsible.

If there are use cases for extending the repertoire of characters allowed in RFC publications, those use cases, and the costs and benefits, need to be examined carefully.

I'm not arguing against a policy of allowing unicode names, I'm arguing that the policy needs to be carefully conditioned so as not to lose important features of the current publication policy.

But it does require you to acknowledge there are some features.

Lest this be called off-topic: 

this question is pretty central to the design of the xml2rfc I proposed of allowing HTML generation to include unicode when TXT generation does not.

---

From: Larry Masinter <[email-masked]>
Date: Sat, 10 Mar 2012 12:34:39 -0800

{{Talking about allowing a conditional-use in XML2RFC where you could supply unicode strings for the .HTML and .PDF rendeirngs}}

> I'll assume it has to do with putting non-ASCII characters into IRI examples?

Yes. There are several examples where accented roman characters I think would be more useful, especially when talking about uncoded and coded forms.
Take the first example of section 3.4.1. 

   For example, the IRI
   "http://r&#xE9;sum&#xE9;.example.org"
   will be converted to
   "http://r%C3%A9sum%C3%A9.example.org".

But in fact there is no way to put "an IRI" in an Internet draft, since  it is a sequence of unicode characters, and to give an example of one in an ascii-only document, you have to use an encoded form.

Since Internet Drafts and RFCs already allow supplying and delivering PDF, we wouldn't have to change IETF policy to get these examples actually legible to those who need to read and interpret them.

---

From: Paul Hoffman <[email-masked]>
Date: Sat, 10 Mar 2012 12:43:43 -0800

> Since Internet Drafts and RFCs already allow supplying and delivering PDF, we wouldn't have to change IETF policy to get these examples actually legible to those who need to read and interpret them.

This is a discussion for rfc-interest, not for xml2rfc. If the RFC Editor allows putting non-ASCII characters in RFCs, it is safe to assume that the xml2rfc tool will be updated to allow that. If the RFC Editor does not allow that, it makes little sense to allow it in Internet Drafts that are intended to become RFCs.

Just because you are punching from the side instead of going through the top, that doesn't change the fact that you are re-opening the can of worms. (I say this as someone who has advocated for what you want in the past, doing so with Internet Drafts, not just questions on mailing lists.)

--Paul Hoffman

---

From: "John Levine" <[email-masked]>
Date: 10 Mar 2012 20:44:51 -0000

>> I do hope at some point we switch to UTF-8 for .txt...

We certainly need to fix the character set problem at some point, but
it seems to me that .txt files with UTF-8 mixed in would cause more
problems than they solve since there's no way to tell from the outside
that there's anything other than ASCII inside.

We're having an informal session on future RFC formats in Paris,
mostly to introduce the new RFC series editor to the swamp.  See you
all there.

R's,
John

---

From: Larry Masinter <[email-masked]>
Date: Sat, 10 Mar 2012 14:00:52 -0800

I'm trying to solve a local problem of making the IRI internet drafts more useful, through a change to (or a pre-processor for) xml2rfc. I do not wish to open the "What formats are allowed for RFCs" can of worms, please stay completely shut. The current policy, however, allows for PDF editions of RFCs. My only question is how to best accomplish, using xml2rfc, this limited design goal.

So no, this is *not* a discussion for rfc-interest, because I do not at all wish to change the current policy, which I think allows everything I'd like to accomplish.

I'm *only* talking about how best to change xml2rfc to be useful in producing submissions which are consistent with existing policy.

Larry
 
-----Original Message-----
From: Paul Hoffman [mailto:[email-masked]] 
Sent: Saturday, March 10, 2012 12:44 PM
To: Larry Masinter
Cc: xml2rfc list; [email-masked]
Subject: Re: [xml2rfc] Generating HTML and PDF with Unicode (and diagrams?)

--Paul Hoffman

---

From: Julian Reschke <[email-masked]>
Date: Sat, 10 Mar 2012 23:09:23 +0100

> I'm *only* talking about how best to change xml2rfc to be useful in producing submissions which are consistent with existing policy.

You could just use rfc2629.xslt and rfc2629toFop.xslt, which should work 
fine with non-ASCII characters.

For "down-conversion" to the all-ASCII format we'd need a simple XSLT 
that strips or converts the examples.

(Stripping will be easier :-)

Best regards, Julian

---

From: Paul Hoffman <[email-masked]>
Date: Sat, 10 Mar 2012 14:18:20 -0800

On Mar 10, 2012, at 2:00 PM, Larry Masinter wrote:

> I do not wish to open the "What formats are allowed for RFCs" can of worms, please stay completely shut. The current policy, however, allows for PDF editions of RFCs.

Those two sentences do not belong together, much less in the same email. "allows for PDF editions of RFCs" *is* part of the can of worms. PDF is a document format. (I find it odd to be saying that to someone who works for Adobe.)

--Paul Hoffman

---

From: Larry Masinter <[email-masked]>
Date: Sat, 10 Mar 2012 14:41:53 -0800

Sorry, I thought RFC 2223 had been updated to allow PDF files as well as Postscript files.  

" PDF is a document format. (I find it odd to be saying that to someone who works for Adobe.)"

I know better than most, and have for longer than I've worked for Adobe. 

   While the primary RFCs is always an ASCII text file, secondary or
   alternative versions of RFC may be provided in PostScript.  This
   decision is motivated by the desire to include diagrams, drawings,
   and such in RFCs.  PostScript documents (on paper only, so far) are
   visually more appealing and have better readability.

I agree that the policy discussion doesn't belong on this mailing list, and I'm trying to keep the topic restricted to
"how to modify xml2rfc so that it is possible to produce documents which meet the guidelines of RFC 2223".

In particular,  how to include
       Diagrams
      Drawings
     "and such" -- interpreting "and such" to include "Examples where presentation of unicode
                            strings in natural examples".

---

From: Larry Masinter <[email-masked]>
Date: Sat, 10 Mar 2012 14:44:34 -0800

> You could just use rfc2629.xslt and rfc2629toFop.xslt, which should work fine with non-ASCII characters.

> For "down-conversion" to the all-ASCII format we'd need a simple XSLT that strips or converts the examples.

> (Stripping will be easier :-)

I'm afraid that I don't believe "stripping" is sufficient. For an example to be useful, it needs to use explicit notation. The ASCII-only version needs a *different* introductory section than the Unicode one, so I need a "alternative selection" syntax and not a "strip one" selection process.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Sun, 11 Mar 2012 15:45:47 +0900

> I'm afraid that I don't believe "stripping" is sufficient. For an example to be useful, it needs to use explicit notation. The ASCII-only version needs a *different* introductory section than the U…

Alternatives can be achieved through stripping by either stripping one 
or another of two pieces. So that's not too difficult. I'll look at this 
technically after the upcoming submission deadline. Ideas for the 
name(s) of the element(s) are appreciated.

Regards,   Martin.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Sun, 11 Mar 2012 16:29:40 +0900

> that there's anything other than ASCII inside.

Sorry, I don't understand this. There's certainly no way to tell from 
the outside of a .txt file that there's anything other than US-ASCII 
inside, but there is also no way to tell that there's anything other 
than UTF-8 inside, and so on. With respect to .txt files, ASCII isn't 
special in any way.

Regards,    Martin.

---

From: Russ Allbery <[email-masked]>
Date: Sat, 10 Mar 2012 23:40:51 -0800

> than UTF-8 inside, and so on. With respect to .txt files, ASCII isn't

There's a transition problem, in that RFC .txt files have always
historically been ASCII.  The maximally-conservative worry is that we have
no idea where that information is embedded and what may be relying on it.

The awkward but mostly backward-compatible thing to do would be to publish
the text UTF-8 version as an additional format for the RFC (.utf8 or
something) and continue to publish a .txt version that's ASCII-only
(however one manages to mangle the UTF-8 characters to do so).  I feel
like that's probably overkill, but I can see the appeal in not breaking
historical guarantees about the format of the RFC .txt files.

---

From: Julian Reschke <[email-masked]>
Date: Sun, 11 Mar 2012 11:50:01 +0100

On 2012-03-11 07:45, "Martin J. Dürst" wrote:
>> I'm afraid that I don't believe "stripping" is sufficient. For an
>> example to be useful, it needs to use explicit notation. The
>> ASCII-only version needs a *different* introductory section than the
>> Unicode one, so I need a "alternative selection" syntax and not a
>> "strip one" selection process.
>

A very pragmatic approach would be to use xsl:when, and command line 
parameter, and let an XSLT processor do the work.

Such as:

<xsl:choose>
   <xsl:when test="for-ascii">
     <t>...</t>
   </xsl:when>
   <xsl:otherwise>...

That being said, the thought of having alternate texts makes me a bit 
uneasy; it would be great if we just could fix the base problem instead 
(the RFC format).

Best regards, Julian

---

From: "Martin J. Dürst" <[email-masked]>
Date: Sun, 11 Mar 2012 21:14:57 +0900

Hello Julian,

> <xsl:when test="for-ascii">

I'm essentially thinking about this, but as a preprocessing step.

> uneasy; it would be great if we just could fix the base problem instead

Yes indeed. But maybe we need an example or two where people can see the 
benefits and the absence of any of the claimed problems.

Regards,    Martin.

---

From: Julian Reschke <[email-masked]>
Date: Sun, 11 Mar 2012 13:21:25 +0100

> I'm essentially thinking about this, but as a preprocessing step.

Yes, that's what I meant as well. Just running XSLT as a preprocessor.

> benefits and the absence of any of the claimed problems.

I think the IRI spec is a very good example, because allowing "ü" would 
make such a big difference :-).

People who think US-ASCII is sufficient really need to have a look at 
RFC 3987, for example.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Sun, 11 Mar 2012 13:27:19 +0100

> Yes, that's what I meant as well. Just running XSLT as a preprocessor.

Like that:

pretest.xml

<section title="foo" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                 xsl:version="1.0">
<xsl:choose>
<xsl:when test="/ascii">
<t>
   Martin Duerst.
</t>
</xsl:when>
<xsl:otherwise>
<t>
   Martin D&#252;rst.
</t>
</xsl:otherwise>
</xsl:choose>
</section>

ascii.xml:

<ascii/>

Run it:

xsltproc pretest.xml ascii.xml

Result:

<?xml version="1.0"?>
<section title="foo"><t>
   Martin Duerst.
</t></section>

---

From: "John R. Levine" <[email-masked]>
Date: 11 Mar 2012 11:32:46 -0400

> Sorry, I don't understand this. There's certainly no way to tell from the 
> outside of a .txt file that there's anything other than US-ASCII inside, but 
> there is also no way to tell that there's anything other than UTF-8 inside, 
> and so on. With respect to .txt files, ASCII isn't special in any way.

All I-Ds and RFC TXT files since the beginning of time have contained only 
ASCII.  (I'm ignoring the early hand-written ones and the few experimental 
Postscript ones.) If you're going to invent a mutant file format that 
contains ASCII mixed with non-ASCII please call it something else.

I entirely agree that we need to do something about a more modern format 
for RFCs, but trying to sneak UTF-8 into the existing text isn't it.

Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. http://jl.ly

---

From: Tony Li <[email-masked]>
Date: Sun, 11 Mar 2012 11:13:19 -0700

On Mar 11, 2012, at 5:21 AM, Julian Reschke wrote:

> 
> People who think US-ASCII is sufficient really need to have a look at RFC 3987, for example.

Guys,

This is not the right list for this unending debate.

Tony

---

From: Julian Reschke <[email-masked]>
Date: Sun, 11 Mar 2012 19:51:55 +0100

On 2012-03-11 16:32, John R. Levine wrote:
> All I-Ds and RFC TXT files since the beginning of time have contained
> only ASCII. (I'm ignoring the early hand-written ones and the few
> experimental Postscript ones.) If you're going to invent a mutant file
> format that contains ASCII mixed with non-ASCII please call it something
> else.

Actually, when we discussed this last time, we found a few RFCs that 
contained ISO-8859-1.

> for RFCs, but trying to sneak UTF-8 into the existing text isn't it.

I believe it would be a huge step forward, and that it would solve some 
of the problems we have nicely.

But let's discuss this in Paris.

Best regards, Julian

---

From: Nico Williams <[email-masked]>
Date: Sun, 11 Mar 2012 16:39:39 -0500

> This is not the right list for this unending debate.

What list should we use for this debate?  But I think this one is the
perfect list for it because its subscribers are clearly intimate with
the relevant issues and details and are already opining.  The main
IETF list is too large an audience at this point, though I agree that
any decision to move to relax the restriction of IETF .txt files to
ASCII should eventually take place on the main IETF list.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Sun, 11 Mar 2012 16:35:50 -0500

> historical guarantees about the format of the RFC .txt files.

I think the fear of putting non-ASCII UTF-8 in .txt is way, *way*
overblown.  What might break?  Not any software or computer system
These .txt files are intended to be consumed by *humans*.  If their
terminals are in a non-UTF-8 locale then the non-ASCII in the .txt
files will display as garbage, and then the user will understand that
something is wrong, and eventually figure out what to do to fix it.
The terminals will *not* crashover non-ASCII UTF-8 encoding, -- if any
did it's a bug that they must be tripping over so frequently that it's
been fixed already.

The obvious retort to our desire for a UTF-8 .txt format is that if
you want UTF-8 then use HTML or PDF.  I reject such a retort on
account of my finding it much easier to read RFCs in text format on a
terminal, in screen, with my favorite regexp-capable pager.

The obvious retort to my preference for plain [UTF-8] text is that
it's just that: a preference, that it's time to move on, or that I
should use a text-based browser.  And my retort to that one is that we
already have the infrastructure for plain text, so supporting this
preference is not hard.

IMO it's time to say that RFC/STD/FYI and Internet-Draft .txt files
are UTF-8.  It's 2012.  It's been a long time (in Internet scale)
since our windowing systems and the terminals we run in them (and
screen readers, and...) learned to render UTF-8.

Nico
--

---

From: Larry Masinter <[email-masked]>
Date: Sun, 11 Mar 2012 14:52:05 -0700

Gosh I'm really sorry to have infected this list with the debate.

I was just asking about conditional XML2RFC to create some easier-to-read PDF files. Please please please stop talking about changing the RFC format or anything else on this mailing list? I'm really really sorry.

Please!

Larry

---

From: "John R. Levine" <[email-masked]>
Date: 11 Mar 2012 21:55:01 -0400

> Actually, when we discussed this last time, we found a few RFCs that 

I just took a look.  I see a little non-ASCII, much of which appears to be 
a mistake. The most recent RFC with non-ASCII text is 2875, which includes 
these lines.  (Since my editor doesn't know what character set it's 
supposed to be, it's likely smashed even though my MUA handles UTF-8):

    TBS: the ^otext"o for computing the SHA-1 HMAC.

should be the "text" for computing the SHA-1 HMAC.

    Signature verification requires CAAEs private key, the CA certificate
    and the generated Certification Request.

should be requires CA's private key

Neither of those look deliberate, probably some junk from MS Word that 
leaked through.

RFC 2708 has similar junk that appears to have leaked:

    assigned IDAEs, there is a limited amount of clear text information
    provided during submission for use by the Job MIB.

should be assigned ID's

One of the few I can find with what deliberate looking non-ASCII is 2557 
which includes:

       E with acute accent becomes 'E.<br>
       E with acute accent becomes &Eacute;.<p>

In that first line the 'E is an accented E in some character set.

> I believe it would be a huge step forward, and that it would solve some of 
> the problems we have nicely.

I think the effort involved for the RFC production center to make their 
tools handle UTF-8 reliably would be disproportionate for the benefit. 
There are a lot of issues beyond non-Roman names, and I think it would be 
better to ask them to upgrade once, not N times.

The BOF is at 1710 on Tuesday.  It's at the same time as marf so I may 
have to run back and forth.

Regards,

---

From: "Martin J. Dürst" <[email-masked]>
Date: Wed, 21 Mar 2012 20:07:09 +0900

Hello Julian, others,

I have started working in this direction. I have used an external 
transform, not inline <xsl:choose>,.... But with XSLT, I hit a rather 
hard wall because author names and the like are in attributes. I think I 
could produce these with inlined <xsl:attribute>s with then contain 
<xsl:choose>, but that's blowing up the source more and more.

Regards,    Martin.

---

From: "Martin J. Dürst" <[email-masked]>
Date: Fri, 23 Mar 2012 15:39:37 +0900

Just for the record, I dealt with the problem I mentioned below by 
adding parallel attributes (e.g. ifullname in parallel to fullname).

Regards,    Martin.