Subject: [TOOLS-DEVELOPMENT] Upcoming tools based on the rfc format redesign

From: Robert Sparks <[email-masked]>
Date: Mon, 17 Mar 2014 10:58:26 -0500

All -

Heather and I spent a few minutes in London brainstorming again on the
impact of the new rfc format and the first early steps to getting there
will be on tools. Here is a high-level list of what we've identified.

I've tried to write in the description in terms of the problem to solve
rather than how we will solve it. (Many of these are likely addressable
by extending existing tools, but that's not where I would like us to
focus at the moment).

Do you see any impacts missing from this list?

* We need to be able to validate and repair a document in the new xml format

     * validating against the new grammar definition
     * ensuring all UTF-8 sequences are valid
     * ensuring unicode literals (U+xxxx) are valid codepoints
     * verifying a match when both are given for a particular character
       (we have an ongoing discussion about markup to help with this)
     * This needs to provide good diagnostic output when a document fails
       to validate so the rfced and authors can repair any problems
     * This should run as both a web service and on individual's own
       computers

* We need to move several functions that the editor currently applies to
   text documents to work with the xml document

     * spell checking
     * duplicate word detection
     * identification of unprintable characters (may be addressed by
       ensuring all UTF-8 sequences are valid and allowed).
     * perform validity checks on chunks of formal languages in the
       document (where we use Bill's ABNF checker, SMICng, and pyang currently)

* We need to be able to determine whether provided SVG is valid and
   satisfies our profile

* We need to be able to render the xml document in several presentation
   formats

     * HTML
     * Text
     * PDF
     * ePub

     The format design team is currently working on internet-drafts that
     provide requirements for the format of the HTML, Text, and PDF
     representations.

     That team has deferred definition of the ePub format until later,
     and recommends waiting until later to tool it.

     It is not yet clear that the HTML document will define the CSS used
     for the HTML served by the RFC Editor. If it is not, Heather would like
     help defining it.

* We will need to ensure that the announcement tools deal properly with
   author names containing non-ascii UTF-8

* We will need to ensure that the tools that extract author names from
   documents and add them to the datatracker are gathering information from
   the new fields correctly

* We need diff tools that will work with the canonical xml and the
   presentation formats

* We need a tool that will automate to the extent possible conversion of
   a document submitted in I-D text form into the canonical xml format.

We also noted the need to develop training:

* Initial training on the tools above for the RFC Editor staff
* Standing training/tutorials for the community

The next steps, once we've verified this list isn't missing major items,
will be to add requirement details to each list item. (As noted above,
some of these are underway in I-D form already).

The TMC will gather those requirements into Statements of Work for
development.

Thanks for helping ensure this is on target,

RjS

---

From: "Scott O. Bradner" <[email-masked]>
Date: Mon, 17 Mar 2014 12:45:34 -0400

quite a list - looks good to me

one question (I have not been following the rfcformat discussions in detail)
what about tables of content? are they going to be discouraged or auto generated?

since the same table will not work in different formats

Scott

On Mar 17, 2014, at 11:58 AM, Robert Sparks <[email-masked]> wrote:

> _______________________________________________
> TOOLS-DEVELOPMENT mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tools-development

---

From: "Heather Flanagan (RFC Series Editor)" <[email-masked]>
Date: Mon, 17 Mar 2014 09:53:22 -0700

> what about tables of content? are they going to be discouraged or auto generated?

They are going to be auto-generated and required.  This is one of the
areas where Style Guide and new format requirements meet.

-Heather

---

From: "Scott O. Bradner" <[email-masked]>
Date: Mon, 17 Mar 2014 12:53:47 -0400

great - thanks

Scott

---

From: Henrik Levkowetz <[email-masked]>
Date: Tue, 18 Mar 2014 12:11:47 +0100

Hi,

Some comments inline, trying to stay away from the 'how' part (where I
also would have comments ;) .  Where nothing else is said, infer an Ack :

>      * validating against the new grammar definition

Ack, this is correct; I'll just note that with the current tools it's almost
a no-op since they validate directly from the plugged in grammar definition
using generic library code for xml validation.

>    text documents to work with the xml document

I'm not convinced that this is correct, if taken to mean that the following
functions need to be implemented in tools that operate _directly_ on the xml.
Why would it not be acceptable for the tools below to operate on one of the
defined distribution formats (maybe unpaginated text), and use them with a
wrapper for on-the-fly conversion from xml to the relevant distribution format?

> * We need diff tools that will work with the canonical xml and the

I think this is a bigger bite than we need to take.  Diff tools for PDF in
particular is problematic.  Again, why is a diff which wraps a conversion
from xml to unpaginated text insufficient?

>    a document submitted in I-D text form into the canonical xml format.

How strong a _need_ is this?  Is it stronger than desire?  My guess is that
it's driven by the number of text-only drafts which arrive for publication,
and the amount of effort required by staff to mark a draft up as xml.

(FWIW, I've started thinking of adapting my htmlization script to produce
xml instead, but have no idea of how well that will work, and I'm still not
sure it's the best approach, given that we want a new tool to do this.)

The strength of the need is of course related to the amount of work it is
reasonable to put into the creation of such a tool.

> Thanks for helping ensure this is on target,

Best regards,

	Henrik

---

From: Julian Reschke <[email-masked]>
Date: Tue, 18 Mar 2014 12:36:13 +0100

> using generic library code for xml validation.

But keep in mind that the plan is to use RNG instead of DTD for v3.

>>       * ensuring all UTF-8 sequences are valid

The XML parser will tell us.

>>       * ensuring unicode literals (U+xxxx) are valid codepoints

Um. Why?

>>         (we have an ongoing discussion about markup to help with this)

...and a proposal from Joe H. for a tool that doesn't need explicit markup.

>>         ensuring all UTF-8 sequences are valid and allowed).

Define "unprintable".

>>     documents and add them to the datatracker are gathering information from

Both of these should just use the XML source.

> from xml to unpaginated text insufficient?

+1. Proper HTML diffing will be hard. To review changes, text diffs 
should work just fine.

> reasonable to put into the creation of such a tool.

In my experience, detecting paragraphs and titles is feasible, while 
anything beyond that quickly becomes very tricky.

> ...

Best regards, Julian

---

From: Henrik Levkowetz <[email-masked]>
Date: Tue, 18 Mar 2014 12:59:32 +0100

Hi Julian, all,

> But keep in mind that the plan is to use RNG instead of DTD for v3.

I assume that will work with your tools; lxml, the library that xml2rfc v2
uses to validate and parse, can also validate against a Relax NG schema, so
that's OK.

> ...and a proposal from Joe H. for a tool that doesn't need explicit markup.

I won't believe in that tool being able to handle the unbounded number of
variations which plain-text-draft-writing-authors will come up with, until I
actually see it working.

Dealing with the variations people come up with to specify authors and refer to
sections in plain running text, in the author-extraction and htmlization tools,
I believe that taking the path of not requiring explicit surrounding markup (as
proposed) in the archival format will lead to a host of pain.

Also, that approach doesn't handle the problem of strictly identifying which
unicode usages in plain text are permissible according to the new rules.

> Both of these should just use the XML source.

Agreed.  It's still code that needs to be written and tested, though.

> anything beyond that quickly becomes very tricky.

The htmlization tool does a bit more, and does it _mostly_ correctly.  And yes,
there's a lot of experience and heuristics captured in that tool, so I agree.

Best regards,

	Henrik

---

From: Julian Reschke <[email-masked]>
Date: Tue, 18 Mar 2014 13:25:49 +0100

> uses to validate and parse, can also validate against a Relax NG schema, so

Good to know!

> unicode usages in plain text are permissible according to the new rules.

Yes, I do agree that explicit markup for this would be good.

> there's a lot of experience and heuristics captured in that tool, so I agree.

Well, that tool for instance doesn't detect lists.

Best regards, Julian

---

From: Henrik Levkowetz <[email-masked]>
Date: Tue, 18 Mar 2014 13:34:33 +0100

> Well, that tool for instance doesn't detect lists.

True.  During the IETF-89 week and after I've been giving detection of artwork
and other non-paragraph text some thought; I'm sure there are a myriad of possible
cases there.  Which comes back to how much effort it's reasonable to put into a
conversion tool; how much final manual adjustment is acceptable?

Best regards,

	Henrik

---

From: Robert Sparks <[email-masked]>
Date: Tue, 18 Mar 2014 09:18:44 -0500

trimming heavily

> wrapper for on-the-fly conversion from xml to the relevant distribution format?
The XML is the resulting document. It _IS_ the distribution format.
The spelling corrections have to be made in the XML document, not
on the way to the presentation formats.
What you describe would require the rfc editor to look at suggested spelling
changes in a derived format and manually carry them back into the XML.
Why shouldn't the tool work directly on the XML instead?

>>       * duplicate word detection
Same
>>         document (where we use Bill's ABNF checker, SMICng, and pyang currently)
I am expecting the  XML to have markup better identifying the chunks to 
extract to feed to these validators.
It may be that the first level of effort is to get to a workflow similar 
to the current one by converting the XML to text and then running these 
validation tools on the text.
But we should look at whether using the information in the XML makes the 
extraction better enough to move to it right away rather than later.

> from xml to unpaginated text insufficient?
This is worth a careful discussion.
We may not need pdf or html diffs. HTMLized diffs of text (like what rfcdiff
can produce now) may cover the vast majority of what the rfc editor and the
document editors will need. But I think there are going to be cases where
the differences to highlight in the XML attributes, or in something that 
gets
rendered away (like SVG), were looking at diffs in the XML
RjS

---

From: Robert Sparks <[email-masked]>
Date: Tue, 18 Mar 2014 09:22:08 -0500

trimming heavily

>>>         ensuring all UTF-8 sequences are valid and allowed).
Yes, that bothered me too. The parenthetical comment was my first attempt
to figure out what that really means in this context.
Would it make sense to you if the task were replaced by
* Ensure the document is well-formed UTF-8 and all sequences are allowed?
(Heather would have to provide an explicit list of allowed or disallowed 
characters).

RjS

---

From: Robert Sparks <[email-masked]>
Date: Tue, 18 Mar 2014 09:30:14 -0500

trimming heavily

> Both of these should just use the XML source.
There are details to work through here that go beyond what the input 
source is.

Right now, we have only one version of a name to look at when figuring 
out what Person records to
associate with an RFC Document as authors.

When an RFC author has both a name listed with an ascii variant, do we 
only match against the ascii
variant, or do we use the preferred name?

Not asking for that design to happen here - just pointing out that it 
will have to happen, and we
have more than one place in the workflow that will have to be inspected 
to make sure it's not making
an assumption that is invalidated by having the name available in two forms.

RjS

---

From: Julian Reschke <[email-masked]>
Date: Tue, 18 Mar 2014 15:36:25 +0100

> an assumption that is invalidated by having the name available in two

Are you saying that there's a workflow that assumes that the full author 
name somehow uniquely identifies an author?

Best regards, Julian

---

From: Robert Sparks <[email-masked]>
Date: Tue, 18 Mar 2014 09:49:51 -0500

On 3/18/14, 9:36 AM, Julian Reschke wrote:
> Are you saying that there's a workflow that assumes that the full 
> author name somehow uniquely identifies an author?
Yes, but the identification isn't necessarily limited to just name, and 
what actually gets stored right now is an Email object. (And it happens 
to internet-drafts, not just rfcs).

A typical result:

In [29]: Document.objects.get(docalias__name='rfc7024').authors.all()
Out[29]: [<Email: [email-masked]>, <Email: [email-masked]>, <Email: 
[email-masked]>, <Email: [email-masked]>, <Email: 
[email-masked]>, <Email: [email-masked]>]

---

From: Julian Reschke <[email-masked]>
Date: Tue, 18 Mar 2014 15:50:37 +0100

> extraction better enough to move to it right away rather than later.

FWIW, I spell-check XML using aspell (using the HTML option).

Best regards, Julian

---

From: Henrik Levkowetz <[email-masked]>
Date: Tue, 18 Mar 2014 16:13:32 +0100

Hi Robert,

> The XML is the resulting document. It _IS_ the distribution format.

No.  XML is the archival format, and vice versa.  Already in
draft-rfc-format-flanagan 'distribution format' was used to explain
'publication format'.  But that's of course not the issue here; how to apply
spell checking is the issue.

> on the way to the presentation formats.

Agreed.  I didn't necessarily see it as meaning that the checking had to be
made on the xml file explicitly.

> Why shouldn't the tool work directly on the XML instead?

No reason, unless we have to custom-write one.  Then it becomes a question of
effort vs. benefit.  Where I think your statement and mine differs is how
strongly expressed the need to operate on the xml directly is; I also think
it's _desirable_ to have tools which lets you correct the xml directly.

> I am expecting the XML to have markup better identifying the chunks
> to extract to feed to these validators. It may be that the first
> level of effort is to get to a workflow similar to the current one by
> converting the XML to text and then running these validation tools on
> the text. But we should look at whether using the information in the
> XML makes the extraction better enough to move to it right away
> rather than later.

Ok, makes sense.

> gets rendered away (like SVG), were looking at diffs in the XML

Agreed.

	Henrik

---

From: Tony Hansen <[email-masked]>
Date: Tue, 18 Mar 2014 11:18:16 -0400

> * We need to be able to determine whether provided SVG is valid and

It would be useful if there were simple ways to extract portions of the 
document, such as "extract all of the ABNF", "extract all of the code", 
etc. This would make it much easier to write tools such as the above.

>   a document submitted in I-D text form into the canonical xml format.

There also need to be tools to convert from the V2 XML to the V3 XML 
format, where there are differences.

     Tony Hansen

---

From: Julian Reschke <[email-masked]>
Date: Tue, 18 Mar 2014 16:55:07 +0100

> etc. This would make it much easier to write tools such as the above.

<http://greenbytes.de/tech/webdav/rfc2629xslt/rfc2629xslt.html#extract-artwork>

Best regards, Julian

---

From: Tony Hansen <[email-masked]>
Date: Tue, 18 Mar 2014 13:34:55 -0400

> conversion tool; how much final manual adjustment is acceptable?

One advantage of doing the detection at the XML semantic level is that 
you have builtin detection of artwork and other non-paragraph text.

     Tony