Subject: [TOOLS-DEVELOPMENT] Draft xml2rfc Statement of Work

From: Russ Housley <[email-masked]>
Date: Mon, 3 Jan 2011 11:09:21 -0500

The IAOC is getting ready for find a contractor to enhance xml2rfc.  Here is the current draft SOW.  You comments are appreciated.

There are a few open questions.

1.  Should improvement be made to the DTD?  If so, what are they?

2.  Is is appropriate for us to require that any code developed for the project be written in python?  If not, why not?

Russ

---

From: Julian Reschke <[email-masked]>
Date: Mon, 03 Jan 2011 22:22:39 +0100

> 1.  Should improvement be made to the DTD?  If so, what are they?

The DTD needs to evolve. However, I'm not sure that we should combine 
replacement of the tool with this activity; it should involve the right 
people (such as the xml2rfc mailing list), has different requirements on 
timing, and should be on-going (as opposed to the replacement of the 
tool which will not be an ongoing activity, right?).

But since you asked:

- there should a way to put paragraph breaks into lists without having 
to abuse <vspace>; see 
<http://greenbytes.de/tech/webdav/rfc2629xslt/rfc2629xslt.html#ext.element.lt> 
(this mirrors a change that Bill Fenner was working on a long time ago)

- the ability to use <xref> elements inside <artwork>

- more expressive power in <xref> elements, for instance for marking up 
section numbers, see 
<http://greenbytes.de/tech/webdav/rfc2629xslt/rfc2629xslt.html#ext-rfc2629.xref>

> 2.  Is is appropriate for us to require that any code developed for the project be written in python?  If not, why not?

I don't think it's appropriate. You could spell out general expectations 
on the programming language, and also signal that you have a preference.

*If* you do, you'll have to be clear on what it means exactly. No other 
code allowed?

Now to the SoW:

> ...
> This Statement of Work for the xml2rfc tool can be accomplished in several ways. One
> approach would gather several existing programs, including nroff or groff, and develop
> software that handles the xml2rfc input format and the calls upon the existing programs
> as needed to complete the overall task. Another approach is to develop a single program
> that handles the xml2rfc input format and does all of the processing as well. Both
> approaches have pros and cons. The first approach will require less development time for
> the basic tool, but the installation will be more complex. The second approach will
> require more development time for the basic tool, and the installation will be more
> straightforward.
> Each bid must describe the development approach that will be used, include the list of
> existing programs that will be used, and describe how the xml2rfc tools will be installed
> on an Apache Web Server, Windows, Mac, and Linux. A bidder may provide one or
> more bid, one for each approach that they are interested in proposing.

This should also include discussion about what it means to build on 
existing tools (will they be forked?), and what it means not to (how do 
you prevent implementations of RFC2629 processors from becoming 
incompatible?).

> The remainder of this document describes the xml2rfc tool requirements. It covers input,
> output, checking, xml2rfc current processing, xml2rfc enhancements, nroff translation,
> other elements that do not have simple nroff translations, citation library cache
> processing, and the ability to augment the citation library with personal entries.
> 1. Input
> XML as described in the current xml2rfc DTD.
 > ...

You'll need to be more precise about what the "current" DTD is. Note 
that the current one (as of Nov 2010) isn't shipping but includes 
changes we did for the 2010 boilerplate.

Also, xml2rfc.tcl is using a non-conforming XML parser. If we want 
evolution, we need to clarify that non-wellformed XML doesn't need to be 
processed, and that all XML input that represents the same Infoset needs 
to be supported.

> Sample file: sample_input.xml
> This file is a modified version of a template by Elwyn Davies
> (http://tools.ietf.org/tools/templates/draft-davies-template-bare.txt). It was used as a
> starting point because it shows various features of xml2rfc.

I'd recommend to use a more complex sample, such as the source for 
<http://greenbytes.de/tech/webdav/rfc2629xslt/testcase.html>

> 2. Output
> The new xml2rfc tool will have six output modes:
> 1. nroff: Uses a subset of nroff commands (primarily .nf, .fi, .in, .ti., .ne, .bp, .ce.
> additional commands are used for the header as detailed below);

I think we should have a separate discussion about why this is a 
desirable feature, given the fact that whatever nroff can produce needs 
separate post-processing anyway (TOC and Index page numbers, form feeds).

My understanding is that there are two reasons:

1) The RFC Editor uses nroff as archival format, and

2) The RFC Production Center edits the nroff output of xml2rfc to 
workaround shortcomings, like for avoiding widows and orphans.

Re 1): we should find out what needs to be done to make XML an 
acceptable archival format, and

Re 2): we should fix these shortcomings, so that the additional step 
isn't needed in the first place.

> 2. paginated plain text;
> 3. paginated PDF;

Paginated PDF is too vague. Proportional or monospace? With in-PDF 
links? Metadata? Anchors? Functional links in TOC and index? Which PDF 
version? Support for non-ASCII art artwork? PDF/a support? Reflowable?

> 4. unpaginated plain text, without headers, footers, and page breaks;
> 5. HTML (unpaginated); and

This should include:

- compliance (which HTML dialect?)
- CSS paged media support
- XHTML support

> 6. complete XML with reference elements when the citation library has been used.

Indeed.

BTW: I think epub would be much more useful than PDF.

> ...
> 3. Checking
> Currently if the PI strict="yes" is used, then xml2rfc tries to enforce IDnits conventions
> and DTD validity. However, since there is a stand-alone I-D nits checker available at
> http://www.ietf.org/tools/idnits/, there is no reason to incorporate the same checking into
> the xml2rfc tool.
 > ...

That's vague. I can see why IDNits checking is orthogonal, but checking 
for DTD validity and other constraints not expressed in the DTD should 
be required.

Of course, if we allow the tool to reject DTD-invalid documents (do 
we?), the switch becomes less important.

> 4. xml2rfc processing
> xml2rfc does lots of things that are not explicitly in the XML file, including:
> • Numbering the sections and subsections (any sections appearing after the
> references section receive letters: Appendix A, B, etc.)
> • Creates the Table of Contents
> • Various formatting
> xml2rfc automatically (without explicit commands in the XML file) performs the
> following formatting:
> • Adds 2 spaces after every period, except a period after a single letter (e.g. "This is
> Example A.[one space]") because it is interpreted as an initial.
> • Formats the reference entries consistent
> with
> current
> preferred
> usage,
> assuming the elements and attributes of the reference element have been used
> appropriately. Note: the RFC Editor strongly encourages use of the citation
> libraries.

Who is maintaining these libraries? (Asking because there's currently an 
non-addressed problem with these). IMHO, they should be generated 
directly from the RFC database and the internet drafts database.

> xml2rfc automatically (without explicit commands in the XML file) performs the
> following formatting of the nroff output:
> • Inserts \ before each \ (backslash).
> 3
> • Inserts /& at the beginning of a line that begins with a period (.)
> • Inserts \% before every hyphenated word. This is not necessary and has no
> practical effect in the plain text output because the word is still broken across the
> line in the nroff. For example, compare all the unnecessary \% in
> current_output.nroff versus sample_output.nroff. Since we have the ability to
> explicitly insert \% (using &nbhy;) when needed (see the example with "EX-bit"
> in the sample_output.nroff), we don't need these extraneous and ineffectual \%
> inserted before every string containing hyphen.
> • Inserts \ before each ' (single apostrophe). These backslashes are only truly
> necessary before a single apostrophe that begins a new line or that is preceded by
> space characters. However, the extraneous escaping of every single apostrophe
> does not have a negative effect.
> • Inserts .in 4 (or higher, based on the length of the section number) before section
> titles. (See description below.)
> • Inserts \0 when figures, tables, and sections are named by a number.
> For example, outputs Figure\01 and Section\05. This is not necessary;
> \0 should simply be a space.
> • Similarly, in the references section, it inserts \0 between each date element's
> month and year attributes, and between each seriesInfo element's name and value
> attributes. This is not necessary; \0 should simply be a space.
> With these improvements, the xml2rfc output is adjusted as appropriate.
> XML input:
> <reference anchor="RFC2119">
> <front>
> <title>Key words for use in RFCs to Indicate
> Requirement Levels</title>
> <author initials="S" surname="Bradner">
> <organization />
> </author>
> <date month="March" year="1997" />
> </front>
> <seriesInfo name="BCP" value="14" />
> <seriesInfo name="RFC" value="2119" />
> </reference>
> The current xml2rfc produces this nroff output:
> [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
> Requirement Levels", BCP\014, RFC\02119, March\01997.
> 4
> The preferred nroff output is:
> [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
> Requirement Levels", BCP 14, RFC 2119, March 1997.
> The plain text output is:
> [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
> Requirement Levels", BCP 14, RFC 2119, March 1997.

This is very interesting, but seems to be micro-managing things. For 
instance, an NROFF output option should be allowed to always use MS 
macros (although the current output does not).

Also, I'd recommend that changes in xml2rfc's nroff output should simply 
be tested in the xml2rfc code base.

> 5. xml2rfc enhancements
> Improved Error Messages
> The improved xml2rfc tool must look for common errors in the input and report them to
> the user in a manner that aids error correction. These errors include, at a minimum,
> missing </t> and </list> closings, bad entity references, and improper order of an
> ordered element.

This could be non-trivial if an off-the-shelf XML parser is used.

I would *strongly* recommend to clarify that *correct* handling of XML 
input is of higher priority than this, otherwise people might be tempted 
to come up with yet another broken "almost XML" parser.

> Ability to Select Reference Element Anchor
> Add the ability to use a reference element from the citation library, but assign it an
> anchor other than the one provided. This allows the author to use a nickname (such as
> [ABNF]) for the anchor instead of an RFC number or I-D string (such as [RFC5234]).

This is a request for a new feature that should be discussed and agreed 
upon as part of updating the vocabulary (I recommend the xml2rfc mailing 
list for that). (the mailing list archives are likely to contain 
discussions about this already).

> For example, while recognizing that these would be different if using ENTITYs instead
> of PIs to pull from the citation libraries:

The PIs should be deprecated; they replicate a feature already supported 
by XML.

> <?rfc include="reference.RFC.5234.xml" my_anchor="ABNF" ?>
> <?rfc include="reference.I-D.ietf-mmusic-ice.xml" my_anchor="ICE"
> ?>
> Allow Hyphen as a Bullet
> Add the ability to use a hyphen as a symbol to make lists like:
> - apples
> - oranges
> - bananas
> This should be done by adding a PI to set the order that symbols are used, such as:
> <?rfc list-symbols="- o * +"?>

You need to clarify which output formats this is supposed to affect; 
depending on the format, there maybe no control about the actual glyph.

> Check for Newer Version
> When installed on a personal machine, the program shall include a feature that checks for
> the availability of a newer version of xml2rfc. If an update is available, simple and clear
> instructions for fetching and installing the newer version shall be provided to the user.
> 5
> Improved Documentation
> Clear documentation on how to use xml2rfc shall be provided. The documentation shall
> include all built-in options, and how to deal with error conditions.
> Clear documentation for installation on each platform shall be provided.
> 6. nroff translation
> XML nroff
> all text in <t> tags will be wrapped and
> indented 3 spaces (not including lists or
> artwork)
> .fi
> .in 3
> <artwork> </artwork> .nf / .fi
> <?rfc needLines="X" ?> .ne X
> <?rfc needLines="100" ?>
> (this is a hack to get a page break; it doesn't
> always work. perhaps there could be a new
> command created to insert a page break)

Not convinced. The right way is to allow the vocabulary to express what 
not to break, or to suggest good places for breaks. Forcing page breaks 
in a specific place only works for a very specific fixed output format, 
not for more complex formats as (printed) HTML or PDF.

> .bp
> <vspace blankLines="0" /> or <vspace /> .br
> <figure align="center"> .ce X where X is the number of lines of
> the total figure (includes the text in the
> figure's title, preamble, and postamble
> elements, if any.)
> Note: currently this is handled oddly by
> xml2rfc. See .ce 8192 in
> current_output.nroff.
> <texttable> [no simple translation. .nf and the data
> of the table formatted using the current
> logic.]

Why not a proper nroff tbl?

> <![CDATA[ ... ]]> [no simple translation. used inside of
> artwork element to prevent characters
> from being interpreted as XML. Should
> continue to function as it does
> currently.]

CDATA is a serialization aspect of XML and has nothing to do with the 
vocabulary. XML CDATA sections simply need to be handled as specified by 
the XML spec.

> ...
> An odd case occurs occasionally when one (or more) of the references has an anchor that
> is a very long string (over some max number of characters). The behavior differs
> depending on this PI:
> rfcedstyle="no": xml2rfc inserts a line break so the reference entry starts on a new line
> underneath the anchor. Normative and Informative References may have different
> indentation.
> rfcedstyle="yes": xml2rfc keeps increasing the indent for all entries to accommodate this
> very long anchor. This is problematic because most entries are followed by spaces to
> match, and all entries are moved right because of one long anchor.
> Ideally, when rfcedstyle="yes" or "no", the behavior would be:
> • very long anchors (over a larger number of max characters than currently used)
> cause a line break to be inserted.
> • both reference sections (Normative and Informative) use the same amount of
> indentation.

Ideally, we just pick one of these formats and use it consistently.

> ...
> Titles
> Currently there is no way to explicitly insert a line break into a title. Line breaks can only
> be controlled by inserting &nbsp; in the spaces of the title. The new xml2rfc tool shall
> allow authors to insert a <vspace /> into the title element, which will cause .br in the
> nroff output.

Seems like a bad idea to me. What we already have makes more sense to me 
(specify where not to break), as it will do the right thing for 
different output widths.

> 8. Citation library cache processing
> There are citation libraries available from http://xml.resource.org/. The xml2rfc tools
> will cache these locally. If xml2rfc determines that the local cache is more than 24 hours
> old, it will attempt to fetch the most recent citation libraries. Stale citation libraries may
> be used to facilitate extended off-line use of xml2rfc. Further, the xml2rfc tool will allow
> authors to easily configure a different location for fetching the citation libraries.

As far as I know, these libraries are currently unmaintained (meaning: 
the scripts producing them). At some point we need to address that.

> 9. Personal citation library additions
> The xml2rfc tools must be able to support additional personal citations. The personal
> citations must allow an author to reference documents that are not in the citation library,
> including Internet-Drafts that is have not been posted yet.

That's a standard feature of XML (entity inclusion). If you're looking 
for more (like a UI etc), it should be stated clearly. If the feature 
inherent in XML isn't sufficient, we should define something properly 
instead of relying on this old PI hack.

Best regards, Julian

---

From: Lars Eggert <[email-masked]>
Date: Tue, 4 Jan 2011 13:02:32 +0200

> - the ability to use <xref> elements inside <artwork>

and <abstract>

Lars

---

From: Julian Reschke <[email-masked]>
Date: Tue, 04 Jan 2011 13:02:58 +0100

On 04.01.2011 12:02, Lars Eggert wrote:
>> - the ability to use<xref>  elements inside<artwork>
>
> and<abstract>

Um?

<xref> is allowed in <abstract>, although it's discouraged -- we may 
want to discuss whether that makes sense for symbolic references though.

Best regards, Julian

---

From: Robert Sparks <[email-masked]>
Date: Fri, 7 Jan 2011 12:42:16 -0600

One (hopefully not too long delayed) question/suggestion concerning:

8. Citation library cache processing
There are citation libraries available from http://xml.resource.org/. The xml2rfc tools will cache these locally. If xml2rfc determines that the local cache is more than 24 hours old, it will attempt to fetch the most recent citation libraries. Stale citation libraries may be used to facilitate extended off-line use of xml2rfc. Further, the xml2rfc tool will allow authors to easily configure a different location for fetching the citation libraries.

As someone who runs xml2rfc on a local machine,  I wouldn't mind the tool fetching and updating a cache of the citation libraries 
itself, but I would be quite upset if it broke or worked more slowly because it couldn't reach xml.resource.org (while I'm on a plane).

It would work as well for me if it just told me the cache was getting stale and gave me instructions on how to update it.

If, for some reason, someone wanted to run against an older version of the citation library, this requirement would make doing so
extraordinarily difficult. Allowing this  updating to be turned off, or allowing the user to point to an alternate copy (perhaps as part of 9) 
would work around that.

And finally, we should capture that the URL http://xml.resource.org/ be configuration, not hardcoded data.

RjS

On Jan 3, 2011, at 10:09 AM, Russ Housley wrote:

> <xml2rfc-sow-d02.pdf>_______________________________________________
> TOOLS-DEVELOPMENT mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tools-development

---

From: Russ Housley <[email-masked]>
Date: Fri, 7 Jan 2011 20:51:40 -0500

Good point.  I'll update this.

Russ

---

From: "John R. Levine" <[email-masked]>
Date: 3 Jan 2011 20:31:37 -0500

> replacement of the tool with this activity;

Agreed.  I'd just say in the RFP that they need to be prepared for the DTD 
to change, and perhaps quote some sort of maintenance price.

>> 2.  Is is appropriate for us to require that any code developed for the 
>> project be written in python?  If not, why not?

Heck, no.  There's nothing wrong with python, but whatever is good about 
python is equally good about perl, ruby, maybe java, and probably other 
languages that don't immediately occur to me.  RFP should say that the 
proposal should specify what implementation language they plan to use, and 
it has to be something that runs without undue pain on the platforms we 
care about.

>> One approach would gather several existing programs, including nroff or 
>> groff, and develop software that handles the xml2rfc input format and 
>> the calls upon the existing programs as needed to complete the overall 
>> task. Another approach is to develop a single program that handles the 
>> xml2rfc input format and does all of the processing as well. Both 
>> approaches have pros and cons.

I dunno about you but I hate the second plan.  Depending on what I'm doing 
I might or might not want to run stuff through nroff, etc.  It's also not 
hard to imagine a setup where you'd want it to create the nroff, even 
though you don't happen to have nroff installed locally.

Agree with the comments about the output formats.  It's too many, and 
they're underspecified.

>> Add the ability to use a reference element from the citation library, but 
>> assign it an anchor other than the one provided.

> This is a request for a new feature that should be discussed and agreed upon 
> as part of updating the vocabulary

Indeed.  I'd prefer it not do that, since I find it easier to read RFCs if 
they don't all have their own clever nicknames for everything.

Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. http://jl.ly

---

From: Tony Hansen <[email-masked]>
Date: Sun, 09 Jan 2011 20:58:01 -0500

(I'll try not to repeat other people's voiced concerns, but might 
inadvertently.)

First off, separate out the concerns between the web page that is used 
to provide a service for users and the tool that converts from xml2rfc 
input format to the end format. The two are *not* the same, and the SOW 
should only be concerned with the latter. The fact that their code is 
currently in the same svn repository is historical and need not be 
continued.

Let me again note that there *are* differences between versions of 
nroff. Of particular note is that the default one for at least some 
versions of linux (a shell front end to groff) purposely disables 
various things because, as the code notes, it's "intended for use with 
man". As it exists, it can *not* be used to generate tables of content 
or indices that are subsequently incorporated through a second/third 
pass. It also generates non-ASCII output.

A single sample input file is insufficient for being the only test of 
record. It's fine for a sample to illustrate a few points, but there a 
number of XML directives and processing instructions (PIs) that need to 
be considered. Most of them are not mentioned in the SOW at all, but 
need to continue working. We have a fairly extensive corpus we use 
currently; being able to pass all of the ones that exist should be 
considered mandatory, except for those in the corpus that are intended 
to generate old flavors of output such as generated prior to certain 
dates and whatnot.

The desired outputs are not well defined. For example, what does 
"paginated PDF" mean? I *think* you mean "a PDF representation of 
paginated plain text". Is that right?

Which HTML representations do you want? Comparable to the one currently 
generated by xml2rfc?

I think what you want is output comparable to the five output formats 
(text, nroff, html, unpaginated text, and expanded xml) generated by the 
current xml2rfc, plus a PDF representation of paginated plain text.

I think the organization of this document is somewhat awkward. For 
example, section 4 mixes and alternates between descriptions and 
criticisms of things it does for all output formats and things it does 
only for nroff. Some of the things listed seem like they're criticisms 
and include specific suggested changes, while others don't have specific 
changes. Section 6 also discusses nroff translation at another level, 
without any indication of if things are changes from the current output 
or not, so it's kind of hard to reconcile. Would the current validation 
corpus work with this or not?

The whole set of "without explicit commands in the XML file" notes in 
section 4 is kind of strange: xml2rfc is *supposed* to generate output 
without explicit commands in the XML file, and some things are stylistic 
choices.

I think sections 4 and 6 would be better recast as

     xml2rfc processing
         all formats
         nroff-specific
     criticisms
         things that could be done better in all formats
         nroff-specific

Where are the criticisms are coming from? I have some of my own, and 
disagree with some of the ones listed.

For example, a stylistic choice was made by Marshall to always keep the 
words RFC, Figure and Section and the ilk on the same line as the 
numbers attached to them in the nroff output. Discussion about 
outputting \0 instead of a space should not be done without discussion 
of the underlying stylistic choice. Is this stylistic choice something 
that the RFC editor disagrees with? Who thinks that this would be an 
improvement? What's the rationale for it?

I think you meant \& instead of /& in section 4.

The description of \ processing is slightly off; sometimes it outputs 4 
backslashes. However the processing of \ in titles seems to have 
problems at times. I think it would be better if \ were output as \e, 
which always generates a backslash.

In section 5, some of the suggested improvements have already been 
implemented in recent versions of xml2rfc, but differently. For example, 
"Allow Hyphen as a bullet" was implemented as

<?rfc text-list-symbols="-o*+"?>

FYI: The default is currently "o*+-", contrary to what it says in section 6.

In section 5, the description of "select reference element anchor" is 
unclear to me. Are you referring to what is printed, as in

     [ABNF]

or what is used as the value of the target= attribute within the <xref> tag?

For section 7, note that the current DTD has this definition for consensus:

           consensus   (no|yes)           #IMPLIED

Section 7 is once again mixing together the expected output and the 
particular nroff that is currently generated.

In the section 7 description of titles, you're asking for a DTD change, 
and it should be described as such. It's not

For section 8, I would expect that any such options would be tunable, as 
not everyone would desire such features.

Section 9 is poorly defined. I'd say that people can do that right now, 
but I don't know of anyone that does so. So I must be misunderstanding 
what is being asked.

     Tony Hansen
     [email-masked]

---

From: Tony Hansen <[email-masked]>
Date: Sun, 09 Jan 2011 22:30:36 -0500

> 1.  Should improvement be made to the DTD?  If so, what are they?
Yes, but not at the same time a new tools is being written.

> 2.  Is is appropriate for us to require that any code developed for the project be written in python?  If not, why not

No, it's too constrictive. However, it *should* be a language that is in 
the top-10 list of widely used languages. It also needs to be a language 
for which there is either native support or easily downloadable support 
on each of Windows, Linux, Solaris and Mac. (My suggested list.) Such 
language support should also not require gigabytes of space to install.

     Tony Hansen
     [email-masked]

---

From: "John R. Levine" <[email-masked]>
Date: 9 Jan 2011 23:10:39 -0500

> Let me again note that there *are* differences between versions of nroff.

I'm rather unclear on what the goal of producing nroff output is.  If it 
is just to support the current RFC production process, we should say so 
and specify that it works with whatever version of nroff and macros the 
production house uses.  If it's for something else, we need to define the 
something else, but as a user of nroff for over 35 years, let me reiterate 
that it's time to let nroff die.

I think we also agree that a medium term goal is to make this tool produce 
RFCs directly, so (not necessarily right now) we need to identify whatever 
it is that xml2rfc+nroff does that xml2rfc needs to do, and provide it.

> The desired outputs are not well defined. For example, what does "paginated 
> PDF" mean? I *think* you mean "a PDF representation of paginated plain text". 
> Is that right?

Whatever it means, I think we should drop PDF output.  If it just means a 
version of the paginated ASCII text that one can open in Acrobat rather 
than Notepad, there are perfectly adequate open source programs like 
enscript and a2ps to do that, and I see no reason to reinvent them.  If it 
means something more like the result of running the nroff version through 
troff, or printing the HTML pages, that opens the hideous can of worms 
that is non-ASCII RFC formats, which needs to be slammed back shut.

> backslashes. However the processing of \ in titles seems to have problems at 
> times. I think it would be better if \ were output as \e, which always 
> generates a backslash.

In general, I hope we agree that we don't want to slavishly reproduce the 
bugs in the current code.

Regards,

---

From: Julian Reschke <[email-masked]>
Date: Mon, 10 Jan 2011 16:17:22 +0100

On 10.01.2011 05:10, John R. Levine wrote:
> something else, but as a user of nroff for over 35 years, let me
> reiterate that it's time to let nroff die.
> ...

+1

I believe this is an attempt to solve the wrong problem.

I'm also afraid that a huge investment in code that generates NROFF, and 
may depend on NROFF for TXT generation, may paint us into a corner if we 
actually make progress on publication format changes later on 
(hopefully, when we have a new RSE).

> enscript and a2ps to do that, and I see no reason to reinvent them. If
> it means something more like the result of running the nroff version
> through troff, or printing the HTML pages, that opens the hideous can of
> worms that is non-ASCII RFC formats, which needs to be slammed back shut.
> ...

At least it's a can of worms we shouldn't open right now :-)

>> The description of \ processing is slightly off; sometimes it outputs
>> 4 backslashes. However the processing of \ in titles seems to have
> In general, I hope we agree that we don't want to slavishly reproduce
> the bugs in the current code.
> ...

I believe that's uncontroversial. However, it may not be clear what's a 
bug. xml2rfc does a few funny things that *I* consider bugs, but others 
might consider a feature.

Best regards, Julian

---

From: Russ Housley <[email-masked]>
Date: Mon, 10 Jan 2011 10:43:41 -0500

Julian:

I do not know what change to make to the SOW in response to this comment.

Russ

On Jan 3, 2011, at 4:22 PM, Julian Reschke wrote:

> The PIs should be deprecated; they replicate a feature already supported by XML.

---

From: Russ Housley <[email-masked]>
Date: Mon, 10 Jan 2011 10:50:43 -0500

Julian:

I am looking for an author to have their own personal extensions to the citation library, in the same way that a person can add words to a spell check dictionary.  Something that is embedded in the XML file is already supported.

Russ

On Jan 3, 2011, at 4:22 PM, Julian Reschke wrote:

> That's a standard feature of XML (entity inclusion). If you're looking for more (like a UI etc), it should be stated clearly. If the feature inherent in XML isn't sufficient, we should define something properly instead of relying on this old PI hack.

---

From: Russ Housley <[email-masked]>
Date: Mon, 10 Jan 2011 10:56:54 -0500

Everyone that commented said that we should not require python.  Fine; I get it.  However, I see no easy way to quantify the point that Tony is making, so my plan is to put this in the SOW:

The python language is preferred for software development. Proposals that use other languages will be accepted, but the proposal must explain why a language other than python is desirable for the tasks proposed.

This should highlight the language being used so that we can evaluate the choice that is being made.

Russ

On Jan 9, 2011, at 10:30 PM, Tony Hansen wrote:

> No, it's too constrictive. However, it *should* be a language that is in the top-10 list of widely used languages. It also needs to be a language for which there is either native support or easily downloadable support on each of Windows, Linux, Solaris and Mac. (My suggested list.) Such language support should also not require gigabytes of space to install.

---

From: Julian Reschke <[email-masked]>
Date: Mon, 10 Jan 2011 17:19:18 +0100

On 10.01.2011 16:53, Russ Housley wrote:
> John:
>
> There is a constituency for all of these.  And, Julian is advocating for the addition of epub.  I do not see any useful way to trim the list without a very long discussion with the community. Frankly, no is not the time for that discussion.  I do not want to take any energy away from the RSE discussion.
> ...

Clarifying: what I'm said is that I personally believe that ePub would 
be more useful then PDF. I'd be fine with doing neither right now.

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Mon, 10 Jan 2011 17:20:17 +0100

> The python language is preferred for software development. Proposals that use other languages will be accepted, but the proposal must explain why a language other than python is desirable for the t…

Such as... "designed for processing XML input"? :-)

---

From: Julian Reschke <[email-masked]>
Date: Mon, 10 Jan 2011 17:22:34 +0100

> I do not know what change to make to the SOW in response to this comment.

The SoW shouldn't mention the PI-based syntax for reference inclusion at 
all (except may be clarifying that this syntax is deprecated).

Best regards, Julian

---

From: Julian Reschke <[email-masked]>
Date: Mon, 10 Jan 2011 17:23:51 +0100

> I am looking for an author to have their own personal extensions to the citation library, in the same way that a person can add words to a spell check dictionary.  Something that is embedded in the…

As I said; this is working already and is an inherent XML feature. Just 
declare the entity with a local (relative URI), and reference it later on.

Best regards, Julian

---

From: Russ Housley <[email-masked]>
Date: Mon, 10 Jan 2011 10:53:18 -0500

John:

Russ

On Jan 3, 2011, at 8:31 PM, John R. Levine wrote:

> Agree with the comments about the output formats.  It's too many, and they're underspecified.

---

From: "John R. Levine" <[email-masked]>
Date: 10 Jan 2011 12:04:35 -0500

> There is a constituency for all of these.

With specific reference to PDF, I don't have any objection to producing 
PDFs, but I think it'd be a poor use of anyone's time to make them other 
than by running the text through one of the existing tools, and we should 
say that, perhaps mentioningthat the web site should provide a button to 
do that.

Again, for nroff, we need to say what we mean.  If it's support existing 
production tools, say that.  If it's (perish forbid) support umpteen dusty 
versions of nroff that someone might want to use, say that.

Regards,

---

From: Russ Housley <[email-masked]>
Date: Mon, 10 Jan 2011 15:21:24 -0500

Julian:

We want the PI-based syntax to continue to work, there are many files using it.  So, we are really saying that new features should not be based on PI-based syntax.

Russ

On Jan 10, 2011, at 11:22 AM, Julian Reschke wrote:

> The SoW shouldn't mention the PI-based syntax for reference inclusion at all (except may be clarifying that this syntax is deprecated).
> 
> Best regards, Julian

---

From: Tony Hansen <[email-masked]>
Date: Mon, 10 Jan 2011 15:26:01 -0500

A comment I forgot to mention before:

The comments about the nroff generation of \% are incorrect. The 
definition of \% says:

(Nroff/Troff User's Manual, section 13):

     A hyphenation indicator character may be imbedded in a word to 
specify desired hyphenation points, or may be prepended to suppress 
hyphenation.
     \% is defined as the default hyphenation indicator character.

Note the last part of that sentence that defines what prepending \% means.

I just ran nroff on a couple different linux boxes and a solaris box, 
and they all treated \% properly.

Where is this failing?

     Tony

---

From: Julian Reschke <[email-masked]>
Date: Mon, 10 Jan 2011 21:26:47 +0100

> We want the PI-based syntax to continue to work, there are many files using it.  So, we are really saying that new features should not be based on PI-based syntax.

Sounds good to me.

---

From: Russ Housley <[email-masked]>
Date: Mon, 10 Jan 2011 15:42:13 -0500

I defer to Alice to respond to this one.

Russ

On Jan 10, 2011, at 3:26 PM, Tony Hansen wrote:

> The comments about the nroff generation of \% are incorrect. The definition of \% says:
> 
>    A hyphenation indicator character may be imbedded in a word to specify desired hyphenation points, or may be prepended to suppress hyphenation.
> I just ran nroff on a couple different linux boxes and a solaris box, and they all treated \% properly.
> 
> Where is this failing?
> 
>    Tony

---

From: Tony Hansen <[email-masked]>
Date: Mon, 10 Jan 2011 16:33:25 -0500

> I believe this is an attempt to solve the wrong problem.

I think there's a mistaken impression that

     *) that nroff is capable of doing much more than it really can.
     *) the nroff code being generated by the current xml2rfc requires 
nroff to do much more work than it really does.
     *) that xml2rfc can be seriously simplified by removing the text 
and html generation code from it.

Note *ALL* of the following are done inside of the current xml2rfc:

     pagination calculations
     indentation calculations
     line length and word wrapping calculations (including the 
characters that are going to fit on each line)
     table of contents processing
     table processing
     artwork processing

The current xml2rfc does multiple passes through the input in order to 
do all of this processing. All of these things are calculated *prior* to 
generating the nroff code.

Nroff then *replicates* what xml2rfc has already decided the text needs 
to look like, based on the nroff codes it's fed.

If all of this processing were taken out of xml2rfc, that is, if 
xml2rfc's logic were lobotomized to not do things like table of content 
processing or table processing, just saying that that processing could 
be taken over and done within nroff is totally wrong. Nroff just doesn't 
have the ability to do lots of those things -- it has to be bolted onto 
the side.

For example, the table of content processing cannot be done inside of 
nroff itself. You have to use macros that write to stderr or separate 
external files (which not all nroff's support) and an external process 
that runs nroff multiple times. Note also that there are common versions 
of nroff that can NOT be used for table of content processing because 
they don't allow multiple output streams to work.

Another example is tables: there are no nroff macros that generate 
tables. Table processing was bolted on to the side of nroff a long time 
ago through the tbl macros; tbl itself is a separate program that does 
all of the table processing and generates simplified nroff code as its 
output to replicate what the tbl processor determined the table should 
look like. Once again, nroff is relegated to replicating what tbl has 
alreay decided that the text needs to look like.

However, here's a kicker: the RFC editor does not permit the use of 
things like tbl macros, or other post processing tools. They don't even 
like it when someone writes their own macros. So the nroff you want out 
of the pipeline must be able to be used as input into the RFC editor 
process.  It cannot be generating TBL macros, and must have already 
figured out what the table of contents looks like.

     Tony Hansen

---

From: Alice Hagens <[email-masked]>
Date: Mon, 10 Jan 2011 17:14:44 -0500

The \%s seem extraneous. In the nroff output \% is placed before every word that contains a hyphen. It is placed before a hyphenated word, even if that word is broken across a line.  A couple examples from nroff output:

o  Integrity checking to counter modifications to a message on a \%hop-
\%by-hop basis.

For the case of making reservations across the provider backbone, we
observe that BGP/MPLS VPNs do not create any \%per-customer forwarding
state in the P (provider core) routers.  Thus, in order to make
reservations on behalf of \%customer-specified flows, it is clearly
necessary to make some sort of aggregated reservation from \%PE-PE and
then map individual, \%customer-specific reservations onto an aggregate
reservation.  That is similar to the problem tackled in [RFC3175] and
[RFC4804], with the additional complications of handling \%customer-                            
specific addressing associated with BGP/MPLS VPNs.

Perhaps this placement of \% throughout the nroff output was based on a misinterpretation of the RFC Editor's policy regarding added hyphenation. Hyphens may (and do) appear at the end of a line because hyphenated words may be broken across a line.  Added hyphenation is what is not 
allowed, as described in http://www.rfc-editor.org/rfc-style-guide/rfc-style:

   *  An RFC must not contain:  ...
      -  (added) hyphenation at right margin.

         Do not use hyphenation at the right margin to split existing
         words that do not "naturally" contain hyphens (e.g., "Inter-
         net").  However, hyphenated words (e.g., "Internet-
         Draft") may be split at the hyphen across successive lines.
For the RFC Editor's purposes, I believe &nbhy; is sufficient for preventing the line break in specific cases, e.g., preventing "8-bit" from being broken across a line. (Currently, for that case, we could either insert &nbhy; in the XML input, or go into the nroff output and remove the line break after the "\%8-".)

Hope this clarifies.

Thanks,
Alice

---

From: Tony Hansen <[email-masked]>
Date: Mon, 10 Jan 2011 17:22:32 -0500

First off, differentiate between what is offered via the web service and 
what is part of the xml2rfc xml->text/html/nroff converter.

Then recognize that the xml2rfc web service can certainly add support 
for both epub and pdf without any change to the xml2rfc converter.

For those that want to do the same on their local machine, we can 
provide pointers to tools that do the same.

     Tony Hansen

---

From: Robert Sparks <[email-masked]>
Date: Mon, 10 Jan 2011 16:34:14 -0600

This is an important distinction. I think we should work to make it obvious wherever we can.

RjS

On Jan 10, 2011, at 4:22 PM, Tony Hansen wrote:

> First off, differentiate between what is offered via the web service and what is part of the xml2rfc xml->text/html/nroff converter.
> 
> Then recognize that the xml2rfc web service can certainly add support for both epub and pdf without any change to the xml2rfc converter.
> 
> For those that want to do the same on their local machine, we can provide pointers to tools that do the same.
> 
>    Tony Hansen
> 
> _______________________________________________

---

From: Tony Hansen <[email-masked]>
Date: Thu, 13 Jan 2011 18:12:56 -0500

The following comments of mine from Draft 2 have not yet been addressed 
in Draft 3.

Right now the SoW is a jumble, bouncing backing back and forth between 
descriptions of what xml2rfc does currently, bug fixes, stylistic 
changes, and outright enhancements to the language.

For the things that can be considered bugs, why aren't they filed as 
issues in trac? An SoW seems to be totally the wrong place to list them. 
It would be better to require that

	the new xml2rfc generate output comparable to version XXX of
	xml2rfc.tcl, plus fixes to the various issues that have been
	filed in trac, plus the enhancements specified in section YYY.

Then remove everything from the SoW that is a fix for a bug. The SoW 
would then be much more readable and manageable.

If you insist on keeping everything in here,
I think sections 4 through 7 would be better recast as

     xml2rfc processing
         all formats
         nroff-specific
     criticisms
     enhancements
	all formats
	nroff-specific

More specific points follow.

	Tony

A single sample input file is insufficient for being the only test of 
record. It's fine for a sample to illustrate a few points, but there a 
number of XML directives and processing instructions (PIs) that need to 
be considered. Most of them are not mentioned in the SoW at all, but 
need to continue working. We have a fairly extensive corpus we use 
currently; being able to pass all of the ones that exist should be 
considered mandatory, except for those in the corpus that are intended 
to generate old flavors of boilerplate.

I had comments before about the output modes. This comment remains, but 
has evolved: What does "paginated PDF with support for proportional text 
and non-ASCII artwork" mean? Are you referring to the text version of 
the RFC rendered as PDF (as generated from enscript)? But that doesn't 
have proportional text nor non-ASCII artwork. Where'd the proportional 
text requirement come from? If you really want proportional text, you 
could get that using the nroff output and passing it through troff. But 
it wouldn't look very good. You'd be better off generating the HTML 
output and then passing that output through something like wkhtmltopdf. 
That is, it doesn't necessarily belong in xml2rfc.

Which HTML representations do you want? It should say "Comparable to the 
one currently generated by xml2rfc". We can discuss later possible other 
outputs.

The description of /& in section 4 is still wrong -- it should say \&.

The description of \ processing is still wrong; sometimes it outputs 4 
backslashes to get through the -ms macros processing. (Changing to using 
\e would be an enhancement.)

In section 5, some of the suggested improvements have already been 
implemented in recent versions of xml2rfc, but differently.

For example, remove the section "Allow Hyphen as a bullet" because it 
was already implemented as

The description of <list style="symbols"> is still wrong in section 6. 
The default is currently "o*+-".

In section 5, the description of "select reference element anchor" is an 
enhancement to the language. The responders to the SoW should not be 
responsible for designing how to respond to this paragraph.

Remove the footnote in section about consensus -- it's already in the 
current DTD.

The section 7 description of titles is asking for a DTD change. It's an 
enhancement.

In section 9, the reference to the apache web server should not be part 
of this SoW. Move that paragraph to a footnote saying

	This enhancement will also require that the web server support
	allow a user to provide their personal citations library
	along with the current-provided XML file containing the
	document.

     Tony Hansen
     [email-masked]

---

From: Julian Reschke <[email-masked]>
Date: Fri, 14 Jan 2011 10:10:07 +0100

Indeed.

> It would be better to require that

I just added <http://trac.tools.ietf.org/tools/xml2rfc/trac/ticket/21> 
for the reference renaming feature request.

> to generate old flavors of boilerplate.

I'd like to include those as well.

> one currently generated by xml2rfc". We can discuss later possible other

I think a new tool should do better :-)

> along with the current-provided XML file containing the

To me it's not clear why this is a requirement at all. Submitting 
multiple files will be even more painful than submitting a single one, 
and the SoW already requires that the local installation should be 
painless, and has a self-contained XML output as one of the required 
output formats.

Best regards, Julian