Subject: [xml2rfc-dev] Fwd: Migration to ietf.org or rfc-editor.org

From: Alice Hagens <[email-masked]>
Date: Wed, 29 Sep 2010 10:26:27 -0400

[Forwarding because held by moderator.]

Begin forwarded message:

> From: Sandy Ginoza <[email-masked]>
> Date: September 29, 2010 10:22:36 AM EDT
> To: [email-masked]
> Cc: RSE <[email-masked]>, RFC Editor <[email-masked]>
> Subject: Re: [xml2rfc-dev] Migration to ietf.org or rfc-editor.org
> 
> Greetings All,
> 
> In reviewing this thread on the xml2rfc-dev list, we're sending replies to various discussion points.  Below is the summary of what we think we've seen so far.  Please see comments below for more detail.
> 
> Summary
> 
> Short term actions:
> 
> 1. update xml2rfc to handle Status of This Memo text and updated header format
> 
> 2. RFC Editor continues to create and archive .nroff files 
> 
> 
> Long term community discussion and RSE consideration:
> 
> 1. is there agreement to abandon .nroff files as the official archival source file? 
> 
> 2. is .txt or .html output more important?  i.e., focus on producing printable docs vs online docs?
> 
> Thanks,
> Sandy and Alice
> 
> ---------------
> 
> Julian Reschke wrote:
> 
>> So how *should* the mid-term look like?
>> 
>> 
>> - We do have a somewhat working TCL code base, which is hard to maintain, and lacks people understanding TCL. It also suffers from a prehistoric non-conforming XML-parser-like thingy being part of it.
>> - There is an XSLT implementation that transforms conforming source code to (X)HTML. This one has been maintained by me for many years, and will continue to be maintained.
>> Formatting to TXT is tricky, and involves understanding the XML format, the boilerplate issues, and also line-breaking, page-breaking and table formatting.
>> For everything *except* the TXT-specific there is working XSLT code. I'd recommend writing a post-processor that takes its XHTML output, and reformats that as plain text. To make this work well, we might have to augment the XHTML with a few extensions (hard-wired class names maybe), but I think that's superior to redoing the first part from scratch.
>> If I would need to do that, I'd probably use Java. But any language that can parse an XHTML document and which has decent string processing should do.
> 
> Please note that the RFC Editor continues to archive .nroff files.  We understand that NROFF was chosen because it is a stable and simple source format for all RFCs.  Currently all RFCs have an archived NROFF file; if the new xml2rfc does not generate NROFF, then there will not be a uniform type of source file archived for all RFCs (since XML files have not been used to create approximately 50% of the I-Ds that are approved for publication as RFCs).
> 
> 
> Julian Reschke wrote:
>>> Are we really saying: XSLT ->  XHMTL ->  NROFF ->  TXT
>> 
>> I think what we *should* do is stop at XHTML :-).
>> 
>> There's little what *we* can do to change how the RFC Production Center works, maybe except forcing them to stop wasting time & money on micro-optimizations.
>> 
> 
> Could you provide examples? Perhaps you are referring to past development of the rfcedstyle PI?
> 
> From our point of view, the current requests for updates to handle Status of This Memo text and headers per RFC 5741 and to insert page breaks in decent locations are essential to creating the final text output.
> 
> 
>> *If* a tool would be written that re-uses the existing XSLT code, it could be an internal processing step, by default invisible. The user wouldn't even know it happened.
> 
> Meaning the nroff as an internal processing step, but available to the RFC Editor for archival purposes?
> 
> 
> Julian Reschke wrote: 
> 
>> Producing PDF is a more or less (*) solved problem already, see
>> <http://greenbytes.de/tech/webdav/rfc2629xslt/rfc2629xslt.html#output.pdf> for instructions, and <http://greenbytes.de/tech/webdav/rfc2616.pdf> for an example.
>> That being said: I'm not sure that we should invest time and money to
>> produce PDF..., good HTML seems to be way more important to me. 
> 
> The RFC Editor began to (and will continue to) post PDF files was so users could print the files easily. We agree that the HTML versions (created by rfcmarkup) that print nicely fill this need, but believe it's still useful to provide PDF files.
> 
> 
> John Levine wrote:
> 
>> It's hard to say whether it's easier to intuit the necessary semantics from the XML or the XHTML. If we get someone to rewrite xml2rfc, I think an important goal needs to be that it can produce output that the RFC editor can use without post editing. If it can't, they'll probably still want it to produce nroff.
> 
> Correct, if we cannot get the text correct without post-editing, we will tweak the .nroff file to get the final output.  We have been lucky that xml2rfc creates .nroff files, as we would have experienced a significant slowdown in production with the multiple boilerplate changes and the yet-to-be-updated header format.  Also, without an .nroff file, the archived source file (.xml) would not have matched the published document.
> 
> 
> Julian Reschke wrote: 
> 
>> Indeed. The problem is that the RFC production center apparently likes the ability to micromanage vertical space and line breaks. That is a problem. I'm not convinced that they'd switch to any tool that gives them less control than nroff does today. 
> 
> Controlling vertical space and line breaks is necessary to get nicely formatted documents and avoid bad page and line breaks.  We would be willing to test and switch to a newer tool if the tool could easily provide a means to insert page and line breaks.  However, this discussion ignores the question of whether we're moving away from the historical archiving of .nroff files, as mentioned above.  NROFF files are the official source files of RFCs; the text files can be recreated by the .nroff files verbatim.  If the thought here is to eliminate .nroff as the official source file, this portion of the discussion should probably be moved to the rfc-interest list.  
> 
> 
> John Levine wrote:
> 
>> Based on the experiments Paul H and I did when we were applying to be the RFC Ed, it shouldn't be hard to make the tool smart enough to get the spacing right on its own, although it would take a long time for the RFC Ed to believe that. 
> 
> Again, we would be willing to test new tools available to us.  We would also be willing to switch tools if community consensus is reached that .nroff is obsolete.
> 
> 
> Julian Reshke wrote:
> 
>> That being said; printing is totally overrated, and we really should focus on what matters today (reflowable text, accessibility, display on ebooks come to mind). 
> 
> This seems to be a topic that requires community input.
> 
> As an fyi, we still receive requests for text files via email.  There are RFC readers that do not have such easy access to the online versions of documents nor easy access to e-readers.
> 
> 
> Russ Housley wrote:
>> I would like to see the boilerplate updated as soon as possible.  Does
>> anyone have a feel for the level of effort to do it? 
> 
> We agree; we would like to see xml2rfc updated to handle the new header requirements and Status of This Memo text.
> 
> 
> Russ Housley wrote:
> 
>> Well, the RFC Editor and the tools site both offer PDF.  I do not know
>> if the two sites use the same tools to make the PDF files.
> 
> We use an in-house script to create the .pdf files; we believe Henrik uses his own script.
> 
> 
> Julian Reschke wrote:
> 
>> As far as I recall, all tools that generate PDF today do that based on the TXT version, so if we change the way the TXT gets generated this will have zero effect.
> 
> This is correct afaik.
> 
> 
> Julian Reschke wrote:
> 
>> That being said, producing PDF from the TXT version is pretty pointless, except to make it possible to print. 
> 
> Right - as mentioned, exactly the reason why the RFC Editor began to produce the .pdf files was for printing purposes.
> 
> 
> Tony Hansen wrote:
> 
>> No one's mentioned the generation of nroff -- I have problems gauging how important that is still. I think it's going to be mandatory until we get enough richness in the XML2RFC language for the RFC editor to be happy to eliminate nroff as their intermediary language.
> 
> Eliminating the use of .nroff seems to require discussion with the RSE, RSAG, and community.
> (Rather than "intermediary", nroff is currently the archival source format.) 
> 
> 
> John Levine wrote: 
> 
>> I'd rather that the rewritten version not produce nroff, but instead we have our rewriters engage the RFC Ed and find out what they actually do with nroff and have the rewritten tool do it, too. I'd be surprised if it were much more than forcing page breaks, inserting or deleting blank lines, and perhaps line breaks. If we can get them to do that, as we go forward we'll have definitive XML source for RFCs which I hope people agree would be a good idea. 
> 
> We would be happy to work with the developer to define the formatting parameters.  Typically, the post-xml2rfc updates would be minimal.  However, with the headers and boilerplate changes, the post-xml2rfc updates has been significant for every document published.  In the future, it would be nice to coordinate tool updates with the required changes.

---

From: Sandy Ginoza <[email-masked]>
Date: Wed, 29 Sep 2010 07:22:36 -0700

Greetings All,

Summary

Short term actions:

Thanks,
Sandy and Alice

---------------

This is correct afaik.

---

From: Julian Reschke <[email-masked]>
Date: Wed, 29 Sep 2010 16:41:36 +0200

Hi Sandy,

thanks for the feedback and welcome to the list...

> 2. RFC Editor continues to create and archive .nroff files

Note that xml2rfc (experimental) already handles the updated header 
format, with the caveat that the balancing of the columns was 
...tweaked, because the publication stream can now get very wide.

It would be interesting to know whether the current solution (making the 
left-hand-side wider) is acceptable.

> 1. is there agreement to abandon .nroff files as the official archival source file?

Who can decide this?

> 2. is .txt or .html output more important?  i.e., focus on producing printable docs vs online docs?

HTML prints better than TXT already.

> ...
> ...

Also compact/subcompact. In my latest RFC, this affected the 
indentations for Normative and Informative References. Also, I recall 
that a line break was moved somewhere else.

> Meaning the nroff as an internal processing step, but available to the RFC Editor for archival purposes?

No, the tool I'm thinking of would produce either (X)HTML, TXT or NROFF, 
and whatever it does so to get there would be hidden from the regular 
user (this was just to clarify that usually, people wouldn't need to 
know what's going on internally).

> Controlling vertical space and line breaks is necessary to get nicely formatted documents and avoid bad page and line breaks.  We would be willing to test and switch to a newer tool if the tool cou…

I think this is part of the problem. Few people read RFCs in paginated 
form. We are optimizing for a minority.

> As an fyi, we still receive requests for text files via email.  There are RFC readers that do not have such easy access to the online versions of documents nor easy access to e-readers.

People having email, but not a web browser?

Best regards, Julian

---

From: Glenn Kowack <[email-masked]>
Date: Wed, 29 Sep 2010 13:25:18 -0400

Sandy et al,
   As this discussion develops, let's be sure review this internally within the RFC Editor, say at our next meeting, before this goes too far along toward looking like a decision.  It may be appropriate to carry this to the RSAG.

More below.

> 1. is there agreement to abandon .nroff files as the official archival source file?

Phrasing a long-term discussion as "is there agreement to abandon?" presupposes moving in this direction.  Was that intended and do you really think we're that far along?

thanks,
Glenn
__

---

From: Glenn Kowack <[email-masked]>
Date: Wed, 29 Sep 2010 13:50:05 -0400

On Sep 29, 2010, at 10:41 AM, Julian Reschke wrote:

> Note that xml2rfc (experimental) already handles the updated header format, with the caveat that the balancing of the columns was ...tweaked, because the publication stream can now get very wide.
> 
> It would be interesting to know whether the current solution (making the left-hand-side wider) is acceptable.
> 

Very important question.  Not just who, but how.  Sandy and Alice put the right tone on this: long-term community discussion and RSE consideration. Nroff has been around so long, we should be both very deliberate and be sure the community has a chance to review and be comfortable with this.  If we were to go in this direction (which is still hypothetical), we'd need to do something like this, after further discussion:

  1) RFC Editor (PC staff and RSE) should make a summary assessment and recommendation after RSAG input.  This should be transparent so the community can provide input.  An RSE committee (like the Citations committee) is probably overkill, but if required, then we'd set one up.  We would invite community members to participate.
  2) Sign-off by the RSE after full RSAG review.  (I would expect their broad support for this, or it's back to the drawing board.)
  3) Posting for community review and feedback.
  4) If it looks like a consensus is reached, then implement the changes and defined, and call it out on the RFC-Editor site and current practice.

The IAB will be in the loop in case they want to review process or consensus.
The RSE should lead this process to be sure all important inputs and voices are heard.  Executing these steps might be 'lighter' than all these steps imply; we'll make them as light or heavy as warranted.

Thoughts?

regards,
Glenn
___

> Also compact/subcompact. In my latest RFC, this affected the indentations for Normative and Informative References. Also, I recall that a line break was moved somewhere else.
> 
> No, the tool I'm thinking of would produce either (X)HTML, TXT or NROFF, and whatever it does so to get there would be hidden from the regular user (this was just to clarify that usually, people wouldn't need to know what's going on internally).
> 
>> Julian Reschke wrote:
>> 
> I think this is part of the problem. Few people read RFCs in paginated form. We are optimizing for a minority.
> 
>> Julian Reshke wrote:
>>

---

From: Julian Reschke <[email-masked]>
Date: Wed, 29 Sep 2010 21:36:01 +0200

> The RSE should lead this process to be sure all important inputs and voices are heard.  Executing these steps might be 'lighter' than all these steps imply; we'll make them as light or heavy as war…

This sounds like an extremely expensive process for deciding on 
something that should be a matter specific to the RFC production center.

Was there ever a community consensus that NROFF has any official role?

Best regards, Julian

---

From: Glenn Kowack <[email-masked]>
Date: Wed, 29 Sep 2010 15:51:54 -0400

On Sep 29, 2010, at 3:36 PM, Julian Reschke wrote:

> This sounds like an extremely expensive process for deciding on something that should be a matter specific to the RFC production center.
> 

Julian,
     Good point - the process is potentially weighty.  However, the test of how heavy to make the process is to see if anyone cares.  If there's no community interest or disagreement, then it goes very quickly and we can drop particular steps.  If not, we make the process correspondingly extensive and deliberate.  So, the right (short?) way forward is available within my "soup to nuts" superset process.  That said, what does anyone on this thread think about nroff?  Does anyone care?

Re past community consensus about NROFF's role, I wasn't here for much of that and would again like to here what anyone knows about the history.

thanks,
Glenn

---

From: Tony Hansen <[email-masked]>
Date: Wed, 29 Sep 2010 16:14:06 -0400

When talking about nroff, you have to be *very* specific as to which 
role you're referring to.

1) There are people who happily use nroff today to *generate* I-D's to 
give to the RFC Editor. I'm co-author on one doc where my co-author 
prefers it. This use is not affected by what we're discussing.

2) He should be able to send the nroff input to the RFC Editor, although 
I don't know that the RFC Editor has ever accepted nroff source files 
from random users. Sandy? Alice? If nroff source *is* accepted from 
users, this use should not be not affected by what we're discussing.

3) The RFC Editor stores a couple of source forms of the RFC as part of 
its archive. This has traditionally been the nroff source, but in recent 
years they've stored the XML2RFC source plus the tweaked version of the 
nroff. For RFCs that were not written in XML, I believe they also 
convert the text version into nroff and then back again in order to have 
the nroff version available.

It's only use case #3 that is being discussed here -- the use of nroff 
by the RFC Editor as a required-internal archive format.

     Tony

---

From: Sandy Ginoza <[email-masked]>
Date: Wed, 29 Sep 2010 13:51:45 -0700

Thanks for providing the clarification Tony!  Please see comments in-line.

On Sep 29, 2010, at 1:14 PM, Tony Hansen wrote:

> When talking about nroff, you have to be *very* specific as to which role you're referring to.
> 
> 1) There are people who happily use nroff today to *generate* I-D's to give to the RFC Editor. I'm co-author on one doc where my co-author prefers it. This use is not affected by what we're discussing.

> 2) He should be able to send the nroff input to the RFC Editor, although I don't know that the RFC Editor has ever accepted nroff source files from random users. Sandy? Alice? If nroff source *is* accepted from users, this use should not be not affected by what we're discussing.

Yes, we accept .nroff files from authors, and we would continue to accept/use author submitted .nroff files as starting points for the editing process.

> 3) The RFC Editor stores a couple of source forms of the RFC as part of its archive. This has traditionally been the nroff source, but in recent years they've stored the XML2RFC source plus the tweaked version of the nroff. For RFCs that were not written in XML, I believe they also convert the text version into nroff and then back again in order to have the nroff version available.
> 
> It's only use case #3 that is being discussed here -- the use of nroff by the RFC Editor as a required-internal archive format.

If an .xml file is submitted, we store the .xml and .nroff source files, but .nroff if considered the official archival source file.

Below is a summary of the conversion process for the various starting points of the editing process.

A) NROFF --> TXT
B) TXT --> NROFF --> TXT
C) XML --> NROFF --> TXT

As Tony indicated, cases A and B would continue without change; the list is only discussing case 3.  

Moving away from having an .nroff source file in the near future seems like a bad idea, as the output looks very different depending on whether you are using an .xml or .nroff source file.

Thanks,
Sandy

---

From: Tony Hansen <[email-masked]>
Date: Wed, 29 Sep 2010 17:16:43 -0400

Thanks for the confirmation Sandy.

This is going on a slight tangent to the other questions, but is 
directly relevant to the question of the evolution of the xml2rfc processor.

I'd really like to understand better the differences you encounter using 
xml2rfc -> text vs xml2rfc -> nroff -> text.

Also, do you do any post processing on the nroff output? Do you need to 
use post processing filters that eliminate backspaces or do other odd 
things?

     Tony

On 9/29/2010 4:51 PM, Sandy Ginoza wrote:
> Moving away from having an .nroff source file in the near future seems 
> like a bad idea, as the output looks very different depending on 
> whether you are using an .xml or .nroff source file.