Subject: Re: [TOOLS-DEVELOPMENT] [xml2rfc-dev] Re Updated draft xml2rfc SOW 03

From: Julian Reschke <[email-masked]>
Date: Tue, 11 Jan 2011 10:13:44 +0100

On 11.01.2011 05:07, John R Levine wrote:
> Rather than go through bit by bit, I'll just deal with the nroff and PDF
> parts, because they each have much worse problems than the entire rest
> of the spec. These concerns may seem niggling, but any competent
> contractor would need the answers to these questions before daring to bid.
>
> Before I start, I'm also unclear about what the goal of xml2rfc is. As I
> understand it, it's to help people produce I-Ds, with each I-D
> potentially maturing into an RFC. It's also to help the production house
> produce the RFCs. That's it. So while it makes sense to have HTML output
> to help us old guys with failing vision proof our drafts, it would be
> useless to provide for, say, embedded video since no forseeable RFC will
> allow it.

Agreed. On the other hand, new code should at least be prepared to 
address changes that *will* come at some point of time, namely extending 
the set of allowed characters (maybe just for contact information and 
examples).

> I went back and looked at the nroff output of the current xml2rfc, which
> I should have done ages ago. What it produces is not "normal" nroff.
> Rather, it is a highly stylized subset of nroff that will reproduce the
> paginated plain text version when run through GNU nroff using whatever
> macro package the production house uses. It is unlikely to be of any use
> for anything other than what the production house does with it, hand
> tweaking the output format. The current RFP text is both too specific,
> with details of how it turns various XML elements into nroff, and misses
> huge parts like the fact that before it generates any nroff at all, it
> lays out and paginates the whole document and inserts a table of
> contents with page numbers.

The same applies to the Index. Note that if manual changes in the NROFF 
output cause page breaks to move, the TOC and the Index need to be 
updated manually. Is this really something we want the Production Center 
to do in the future as well???

> It is a poor use of our money to reproduce this kludge, rather than
> working with the production house to put the necessary tweakage into
> xml2rfc itself, but if we insist on doing so, the only meaningful spec
> is that its output match xml2rfc's text output when fed through GNU
> nroff and the macro package.

Exactly.

> Even though the PDF bit is less than a sentence, it has three major
> problems that vastly increase the scope of work:
>
> a) "proportional text"
>
> b) "non-ASCII artwork"
>
> c) "PDF"
>
> The current xml2pdf has one layout engine that produces the plain text
> and nroff output. Its job is very simple since all characters are the
> same size, there's the same number of characters on each line, and the
> same number of lines on each page. Unpaginated text is a trivial tweak
> to the paginated text. Conceptually at least, it is just the paginated
> text with the headers, footers, and page numbers taken out.
>
> The HTML stuff doesn't use a layout engine since it is pretty much
> translated element by element, and the layout happens in your web
> browser when it renders the HTML.
>
> But if xml2rfc has to generate "proportional text", it needs a new and
> much more complex layout engine. The immediate question is in what
> typeface(s) is the text? It needs to know what typeface it's generating,
> so it can look up the width of every character to do line filling. If
> you want the lines to look good, you also need to know the height of
> each character and which ones have ascenders and descenders. This is a
> whole lot of extra work, and I think an unwarranted extra feature to add
> since it's not useful in real RFCs which are fixed pitch text.

It's a can of worms, and it doesn't even address the multiple possible 
PDF output formats (PDF/a? Reflowable)?

Can we *please* clarify what's the use case for the PDF option?

> I don't know what "non-ASCII artwork" means, but it's both too much work
> and inappropriate. Does it mean EPS files? A line drawing sublanguage
> liek GRAP? Embedded PNG files? I don't know, and I fear nobody else does
> either. Whatever it is, again, since you can't put non-ASCII artwork in
> an RFC, I don't see any point to this for generating I-D's and RFCs
> other than making an end run back into the RFC format wars.

I'm interested in making progress in this area, but right now certainly 
is not the moment for it :-)

> And finally, producing even the simplest PDF directly from xml2rfc will
> vastly increase the scope of work. Currently it reads plain ASCII input,
> and produces plain ASCII output. (HTML is coded in ASCII, of course.)

Note that the xml2rfc HTML output already includes a few non-ASCII 
characters, such as nbsp and cooler quote characters (although probably 
encoded in ASCII through character entities).

> PDFs are very much not ASCII. They're also rather difficult to produce
> from python. I am not a python whiz, but poking around looking for
> python libraries to write PDFs, the most likely ones I see are a
> commercial package called ReportLab with a limited open source version
> that might or might not be adequate, and an open source package called
> Haru. Both involve downloading and compiling a C library and interfacing
> it to your Python interpreter. That seems like an awful lot of baggage
> for a marginal feature.
>
> If I had to do this, I would do it in perl rather than python, simply
> because it has adequate native PDF libraries. But my advice remains
> Don't Do That.

+1

> So anyway:
>
> For nroff, take out all the existing language and replace it with
> something like "produce nroff output that will recreate the paginated
> text when used with GNU nroff and the production center's macro package."
>
> For PDF, take it all out and if people insist, replace with "provide a
> way to postprocess formatted text into PDF using standard open source
> tools".
> Feel free to add a similar sentence for epub.

I'd even drop the PDF thingy completely.

Best regards, Julian

---

From: Russ Housley <[email-masked]>
Date: Tue, 11 Jan 2011 15:50:03 -0500

You have convinced me to drop the HTML output.

I think we need the PDF to support non-ASCII art.  Is there any other way forward on that front?

Russ

On Jan 11, 2011, at 4:13 AM, Julian Reschke wrote:

> Agreed. On the other hand, new code should at least be prepared to address changes that *will* come at some point of time, namely extending the set of allowed characters (maybe just for contact information and examples).
> 
> The same applies to the Index. Note that if manual changes in the NROFF output cause page breaks to move, the TOC and the Index need to be updated manually. Is this really something we want the Production Center to do in the future as well???
> 
> It's a can of worms, and it doesn't even address the multiple possible PDF output formats (PDF/a? Reflowable)?
> 
> I'm interested in making progress in this area, but right now certainly is not the moment for it :-)
> 
> Note that the xml2rfc HTML output already includes a few non-ASCII characters, such as nbsp and cooler quote characters (although probably encoded in ASCII through character entities).
> 
> _______________________________________________
> TOOLS-DEVELOPMENT mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tools-development

---

From: "John R Levine" <[email-masked]>
Date: 10 Jan 2011 23:07:45 -0500

Rather than go through bit by bit, I'll just deal with the nroff and PDF 
parts, because they each have much worse problems than the entire rest of 
the spec.  These concerns may seem niggling, but any competent contractor 
would need the answers to these questions before daring to bid.

Before I start, I'm also unclear about what the goal of xml2rfc is.  As I 
understand it, it's to help people produce I-Ds, with each I-D potentially 
maturing into an RFC.  It's also to help the production house produce the 
RFCs.  That's it.  So while it makes sense to have HTML output to help us 
old guys with failing vision proof our drafts, it would be useless to 
provide for, say, embedded video since no forseeable RFC will allow it.

I went back and looked at the nroff output of the current xml2rfc, which I 
should have done ages ago.  What it produces is not "normal" nroff. 
Rather, it is a highly stylized subset of nroff that will reproduce the 
paginated plain text version when run through GNU nroff using whatever 
macro package the production house uses.  It is unlikely to be of any use 
for anything other than what the production house does with it, hand 
tweaking the output format.  The current RFP text is both too specific, 
with details of how it turns various XML elements into nroff, and misses 
huge parts like the fact that before it generates any nroff at all, it 
lays out and paginates the whole document and inserts a table of contents 
with page numbers.

It is a poor use of our money to reproduce this kludge, rather than 
working with the production house to put the necessary tweakage into 
xml2rfc itself, but if we insist on doing so, the only meaningful spec is 
that its output match xml2rfc's text output when fed through GNU nroff and 
the macro package.

a) "proportional text"

b) "non-ASCII artwork"

c) "PDF"

The current xml2pdf has one layout engine that produces the plain text and 
nroff output.  Its job is very simple since all characters are the same 
size, there's the same number of characters on each line, and the same 
number of lines on each page. Unpaginated text is a trivial tweak to the 
paginated text.  Conceptually at least, it is just the paginated text with 
the headers, footers, and page numbers taken out.

The HTML stuff doesn't use a layout engine since it is pretty much 
translated element by element, and the layout happens in your web browser 
when it renders the HTML.

But if xml2rfc has to generate "proportional text", it needs a new and 
much more complex layout engine.  The immediate question is in what 
typeface(s) is the text?  It needs to know what typeface it's generating, 
so it can look up the width of every character to do line filling.  If you 
want the lines to look good, you also need to know the height of each 
character and which ones have ascenders and descenders.  This is a whole 
lot of extra work, and I think an unwarranted extra feature to add since 
it's not useful in real RFCs which are fixed pitch text.

I don't know what "non-ASCII artwork" means, but it's both too much work 
and inappropriate.  Does it mean EPS files?  A line drawing sublanguage 
liek GRAP?  Embedded PNG files?  I don't know, and I fear nobody else does 
either.  Whatever it is, again, since you can't put non-ASCII artwork in 
an RFC, I don't see any point to this for generating I-D's and RFCs other 
than making an end run back into the RFC format wars.

And finally, producing even the simplest PDF directly from xml2rfc will 
vastly increase the scope of work.  Currently it reads plain ASCII input, 
and produces plain ASCII output.  (HTML is coded in ASCII, of course.) 
PDFs are very much not ASCII.  They're also rather difficult to produce 
from python.  I am not a python whiz, but poking around looking for python 
libraries to write PDFs, the most likely ones I see are a commercial 
package called ReportLab with a limited open source version that might or 
might not be adequate, and an open source package called Haru.  Both 
involve downloading and compiling a C library and interfacing it to your 
Python interpreter.  That seems like an awful lot of baggage for a 
marginal feature.

If I had to do this, I would do it in perl rather than python, simply 
because it has adequate native PDF libraries.  But my advice remains Don't 
Do That.

So anyway:

For nroff, take out all the existing language and replace it with 
something like "produce nroff output that will recreate the paginated text 
when used with GNU nroff and the production center's macro package."

For PDF, take it all out and if people insist, replace with "provide a way 
to postprocess formatted text into PDF using standard open source tools".
Feel free to add a similar sentence for epub.

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
"I dropped the toothpaste", said Tom, crestfallenly.

---

From: "John R Levine" <[email-masked]>
Date: 11 Jan 2011 16:30:25 -0500

> You have convinced me to drop the HTML output.

That's unfortunate, since it is both useful and not an enormous amount of 
work.  Please put it back in.  If the new xml2rfc doesn't have it, I'm 
sure I won't be the only one nursing along the old tcl code so I can get 
HTML versions to proof.

> way forward on that front?

Since nobody appears to understand what "non-ASCII art" means, the only 
sensible way forward is to delete the references to PDF and to non-ASCII 
art.

Non-ASCII art could be anything from encapsulated Postscript to JPEG scans 
of oil on canvas.  No competent bidder would agree to do it, since it's an 
unknowable amount of work.  And whatever it is, it'd have to be removed 
and replaced with old fashioned ASCII art before turning an I-D into an 
ASCII RFC, so it's not useful for a document intending to be an RFC.

I realize that at some future time we may finally settle on some RFC 
formats other than plain ASCII.  Once we know what they are, it would make 
sense to have xml2rfc generate them.  Perhaps we should add a requirement 
that the bidder be willing to bid on further work once we figure out what 
we want.

R's,
John

Regards,

---

From: Russ Housley <[email-masked]>
Date: Tue, 11 Jan 2011 17:08:39 -0500

John:

> That's unfortunate, since it is both useful and not an enormous amount of work.  Please put it back in.  If the new xml2rfc doesn't have it, I'm sure I won't be the only one nursing along the old tcl code so I can get HTML versions to proof.

I guess I am not understanding you at all.  I thought this action was giving you half of what you wanted.  I'll put it back.

> Since nobody appears to understand what "non-ASCII art" means, the only sensible way forward is to delete the references to PDF and to non-ASCII art.
> 
> Non-ASCII art could be anything from encapsulated Postscript to JPEG scans of oil on canvas.  No competent bidder would agree to do it, since it's an unknowable amount of work.  And whatever it is, it'd have to be removed and replaced with old fashioned ASCII art before turning an I-D into an ASCII RFC, so it's not useful for a document intending to be an RFC.
> 
> I realize that at some future time we may finally settle on some RFC formats other than plain ASCII.  Once we know what they are, it would make sense to have xml2rfc generate them.  Perhaps we should add a requirement that the bidder be willing to bid on further work once we figure out what we want.

There are politically important RFCs where the PDF version is the one that matters.

Here is one: http://www.rfc-editor.org/rfc/rfc5317.pdf

How do we handle this type of document?

Russ

---

From: Tony Hansen <[email-masked]>
Date: Tue, 11 Jan 2011 17:30:28 -0500

Huh?? I would have thought you would have come to the exact opposite 
conclusion.

The current xml2rfc generates HTML -- and people use it all the time. 
The only real question is: which flavor? At a minimum, the SoW should 
call for the new translator to generate equivalent HTML as to what is 
currently generated.

     Tony

On 1/11/2011 3:50 PM, Russ Housley wrote:
> xml2rfc-dev mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/xml2rfc-dev

---

From: Tony Hansen <[email-masked]>
Date: Tue, 11 Jan 2011 19:02:35 -0500

> How do we handle this type of document?

Ah, this is a really useful question.

Taking a look at that particular example, I see that it's the standard 
text output converted to PDF, merged with a series of slides that were 
produced by another package and converted to PDF.

Let's look at a few other examples:

rfc 4128: standard text output converted to PDF, followed by a series of 
PDF pages generated by a totally different program

A number were created using a totally different package, such as TeX. I 
think rfc 1119 is an example of that.

But now for one of the most interesting ones: rfc 5598.

This one was generated from xml2rfc generating HTML, and then the HTML 
was converted to PDF. Note that <artwork/> takes a URI attribute, 
specifying an image file to display when generating HTML. All of the 
fancy figures and charts that you find in there were done this way. I 
think Dave tweaked the HTML a bit to generate the page headers using 
CSS, but overall it was stock xml2rfc generating HTML.

     Tony

---

From: Julian Reschke <[email-masked]>
Date: Wed, 12 Jan 2011 01:54:11 +0100

> You have convinced me to drop the HTML output.

I did? How so? I think we need HTML output. I actually think it's the 
most important output format.

The only reason not to have it in the SoW is that we already have a 
perfectly working and well-maintained implementation.

> I think we need the PDF to support non-ASCII art.  Is there any other way forward on that front?

Hm, no. As John said.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Wed, 12 Jan 2011 01:55:40 +0100

On 11.01.2011 22:30, John R Levine wrote:
> ...
> That's unfortunate, since it is both useful and not an enormous amount
> of work. Please put it back in. If the new xml2rfc doesn't have it, I'm
> ...

Well, you wouldn't need to. rfc2629.xslt does that for you. It also 
produces ePub and PDF, if you want.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Wed, 12 Jan 2011 01:57:55 +0100

On 12.01.2011 01:04, John R Levine wrote:
> ...
> RFC 5598 also has an PDF version, which Dave appears to have created
> using Apache FOP, an open source package that can produce PDF from XML
> input using XSL-FO and other TLAs I don't entirely understand. It uses a
> variety of variable pitch fonts and styles, and includes some box and
> line diagrams that appear to have been rendered into bitmaps and
> embedded in the document.
> ...

-> 
<http://greenbytes.de/tech/webdav/rfc2629xslt/rfc2629xslt.html#output.pdf.fop> 
(maybe with a few tweaks from Dave).

> ...
> By the way, I ran both rfc5317.pdf and rfc5598.pdf through a validator
> for the archival PDF/A standard. Neither was even close to validating,
> which means they are at some risk of bit rot as PDF software evolves in
> the future. Producing archival PDFs is not simple.
> ...

I think Apache FOP is going to do PDF/A in some point in the future (you 
know, volunteers needed etc.).

Otherwise, +1 to what you said.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Wed, 12 Jan 2011 02:19:58 +0100

> CSS, but overall it was stock xml2rfc generating HTML.

Actually, this was one with rfc2629toFO.xslt, and that being processed 
by Apache FOP. No HTML in between.

More examples at <http://greenbytes.de/tech/webdav/> (scroll down to 
"RFCs").

Best regards, Julian

---

From: Tony Hansen <[email-masked]>
Date: Tue, 11 Jan 2011 20:45:29 -0500

> More examples at <http://greenbytes.de/tech/webdav/> (scroll down to 

Ok, that's fine. The main point though is that it was done using the 
xml2rfc language.

Does the rewrite of xml2rfc need to replicate rfc2629toFO.xslt or 
rfc2629.xslt? I don't think so.

I do think it's important to eventually improve the web form to be able 
to generate output using those alternate formats.

But that's a matter for the web site, not the base converter.

     Tony

---

From: Julian Reschke <[email-masked]>
Date: Wed, 12 Jan 2011 02:54:10 +0100

> rfc2629.xslt? I don't think so.

I don't think so. Both are under BSD license. I plan to maintain both. 
t. The PDF conversion currently is mainly restricted by what Apache FOP 
does, so there's little I can do to make it better.

+1

Best regards, Julian

---

From: "John R Levine" <[email-masked]>
Date: 11 Jan 2011 19:04:09 -0500

> How do we handle this type of document?

According to the PDF comments in RFC 5317, it was produced in Microsoft 
Word and postprocessed through a component of MacOS called Quartz that 
produces the PDF.

That particular document is ten pages of fixed pitch text that might as 
well have been produced by running the ASCII version through enscript, 
with a 58 page printout of a Powerpoint deck inserted between the 8th and 
9th pages.  There's a variety of ways one could have spliced that set of 
pages together, none of which I would dream of trying to build into a 
simple tool like xml2rfc.

RFC 5598 also has an PDF version, which Dave appears to have created using 
Apache FOP, an open source package that can produce PDF from XML input 
using XSL-FO and other TLAs I don't entirely understand.  It uses a 
variety of variable pitch fonts and styles, and includes some box and line 
diagrams that appear to have been rendered into bitmaps and embedded in 
the document.

Unless the spec is going to be "do everything that MS Word can do on a 
Mac", this is not a problem that we should try to solve.  After the bunch 
of PDFs a decade ago, I see approximately one PDF RFC per year. I'd 
suggest at that level, the production house can limp along with whatever 
ad-hoc approach to PDF they've been using, and we should concentrate on 
getting them a version of xml2rfc that works for the other 400 RFCs 
they'll be handling every year.

By the way, I ran both rfc5317.pdf and rfc5598.pdf through a validator for 
the archival PDF/A standard. Neither was even close to validating, which 
means they are at some risk of bit rot as PDF software evolves in the 
future.  Producing archival PDFs is not simple.

Eventually I think it'll probably be a good idea to allow a profile of 
PDF/A as an archival format, but this is not the place to try to do it.

Regards,

---

From: Russ Housley <[email-masked]>
Date: Mon, 17 Jan 2011 17:09:51 -0500

Please review.

Russ

---

From: "John R. Levine" <[email-masked]>
Date: 17 Jan 2011 23:17:59 -0500

> Please review.

That does look better.  In the section on nroff output, somewhere around 
the top of page 5 you really should say that the nroff output has to 
be paginated, and include the table of contents and, if defined, 
the index.  (It has to be paginated to know what page numbers to put in 
the TOC and index.)

If the contractor is steeped in RFC folklore, they'll know that, but if 
they aren't, it'll be quite a surprise, even if they've read RFC 2223.

Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. http://jl.ly

---

From: Tony Hansen <[email-masked]>
Date: Mon, 17 Jan 2011 23:32:49 -0500

Much improved. This version separates out the requirements, 
enhancements, and issues much more clearly.

Thank you Russ.

     Tony