Subject: [TOOLS-DEVELOPMENT] Preview release of Text Submission Converter, id2xml

From: Megan Ferguson <[email-masked]>
Date: Wed, 12 Jul 2017 19:28:36 -0700

Hi Henrik,

Files: non-xml2rfc-generated files generally — status check
Version: 1.0.3

This mail is comprised of a few queries about this type of file generally as well as a summary 
of the manual updates we have been making in text files to get id2xml to parse and represent 
as much of the text accurately as possible.  We appreciate whatever feedback you may have on 
the following.

1) Would it be possible to use the citation tags [RFC…] and [I-D….] in the references section 
as a trigger to automatically pull those from the citation library?  (So that if the entry itself 
is poor, it doesn’t really matter…).  I believe you previously said this information was pulled from
the seriesInfo, which makes sense as taking a look at (for example):

https://www.rfc-editor.org/rfc/v3test/draft-ietf-trill-directory-assist-mechanisms-12v3.xml, 

we see a full reference entry for [ARPND] aka draft-ietf-trill-arp-optimization in the references 
section but we also see:

<!ENTITY I-D.ietf-trill-arp-optimization SYSTEM "https://xml2rfc.ietf.org/public/rfc/bibxml3/
reference.I-D.draft-ietf-trill-arp-optimization.xml”>

at the top of the xml file.

The same question for other references that live in the citation library (e.g., other SDOs like 
W3C in bibxml6).  

[Generally, updating the citation tag only would be much less time intensive than updating the info.  
And having the correct input would be desirable (vs. re-adding the reference to the xml) so that 
references aren’t errantly removed.]

2) The SoTM text trigger seems to be quite sensitive.  The text appearing in the following files 
is not far off that generated by xml2rfc, but even the slight variations cause this text to be 
unrecognized.  Even copying in the text from https://www.ietf.org/ietf-ftp/1id-guidelines.html#anchor7 
gives an error as it is single spaced (and disclaimer that it does contain some typos…).

draft-ietf-mpls-app-aware-tldp-09
draft-ietf-pals-status-reduction-05
draft-ietf-ippm-6man-pdm-option-13

3) Here is a list of the (current) manual changes we are making in order to make id2xml parse with the 
current version.  Please let me know if I have mischaracterized any functionality or if any of these 
items can be resolved using the tool in some manner I am unaware of.

Header updates:

-Remove any blank lines between top left 'Key word: text’ entries
-Update to use first initial instead of full first name (or you get "Warning: This author is listed 
in the Authors’ Addresses section, but was not found  on the first page: and the authors section will 
be absent from the XML generated)

Boilerplate updates:

-Replace Copyright and SoTM text to ensure exactly matches output from xml2rfc
-Ensure use of “Copyright License” as a title exactly

List format updates:

-Add blank lines between list items to fix numbering
-change (1) to 1 
-fix indentation with - (dash) to all be inline indentation-wise
-fix indentation generally — if things are not aligned, they don’t work
-update + or -iv or anything not xml2rfc-compliant as a list marker

Section header updates:
-  Change Appendix A: to be Appendix A. or just Appendix A

Reference entry updates: 

-Change A. Nonymous to Nonymous, A. 
-Review comma use as missing commas will cause a reference to be missed
-Add double quotes around titles
-Date updates:
	-Change February 10 2016 to 10 February 2016 -- what about commas etc.?
	-Change Summer 1996 to a specific month
	
Authors’ Addresses updates:

-Add a blank line before email addresses (temporary?)
-Remove any number from this section heading
-Add URI: before any entry (same for email?)
-Spacing
	-Need to make sure there is whitespace where needed 
        (e.g., title with no blank line before the figure)
	- Need to review in text output from the xml because sometimes errant
-Review for mismatch between authors in header and Addresses section

Misc. updates:

-Review for defined in “section” x and similar

4) Here is a pointer to a diff file between an original and one including the manual edits we made in order to get 
the document to parse (a file we have previously discussed, just an example):

https://www.rfc-editor.org/rfc/v3test/draft-ietf-trill-directory-assist-mechanisms-12v3preedits-rfcdiff.html 

Here is a pointer to a diff between the original and the text created from the id2xml output of our 
manually updated original (another example):

https://www.rfc-editor.org/rfc/v3test/draft-ietf-trill-directory-assist-mechanisms-12v3-rfcdiff.html 

5) The following is a list of checks we are making once the file parses using xml2rfc.  The amount of time this takes 
is very dependent on the number of lists, figures, and references per document in addition to how clean the indentation
was in the original.

Lists:
-Review numbering of lists
-Review correct implementation of list elements
-Ensure no figures have been turned into lists errantly

Figures:
-Review figures for alignment and missing text

Spacing:
-Review indentation of text around a colon followed by two spaces
-Review whitespace generally

References:
-Include references that could not be generated through text fixes previously

Authors:
-Review all information from original is included in the file (no missing email addresses)

Thank you.

Megan

---

From: Henrik Levkowetz <[email-masked]>
Date: Thu, 13 Jul 2017 17:47:27 +0200

Hi Megan,

On 2017-07-13 04:28, Megan Ferguson wrote:
> This mail is comprised of a few queries about this type of file 
> generally as well as a summary of the manual updates we have been
> making in text files to get id2xml to parse and represent as much of
> the text accurately as possible. We appreciate whatever feedback you
> may have on the following.
> 
> 1) Would it be possible to use the citation tags [RFC…] and [I-D….] 
> in the references section as a trigger to automatically pull those
> from the citation library? (So that if the entry itself is poor, it
> doesn’t really matter…). I believe you previously said this
> information was pulled from the seriesInfo, which makes sense as
> taking a look at (for example):
> 
> we see a full reference entry for [ARPND] aka 
> draft-ietf-trill-arp-optimization in the references section but we
> also see:
> 

Yes.  I'd prefer to make this happen if you specify a switch, maybe
something like --use-citation-tags, since this requires the user to
be aware of the effects, and having inspected and possibly fixed the
tags (something which may not be the case for general use in the
community).  But I can see the advantage to you, and will add this
in the next release.

> The same question for other references that live in the citation 
> library (e.g., other SDOs like W3C in bibxml6).

To a limited extent; if the tags are sufficiently regular, as the
tool won't be looking up individual tags while working, but will be
relying on recognising certain patterns in the tag.

> [Generally, updating the citation tag only would be much less time 
> intensive than updating the info. And having the correct input would
> be desirable (vs. re-adding the reference to the xml) so that 

Ack.

> 2) The SoTM text trigger seems to be quite sensitive. The text 
> appearing in the following files is not far off that generated by
> xml2rfc, but even the slight variations cause this text to be 
> unrecognized. Even copying in the text from 
> https://www.ietf.org/ietf-ftp/1id-guidelines.html#anchor7 gives an
> error as it is single spaced (and disclaimer that it does contain
> some typos…).

Normalising space before doing the processing is something I've
thought of doing.  I'll add that in the next release.

I can accommodate a number of acceptable variations on the boilerplate
text.  I think one way forward here is also to tighten the text that
idnits will accept, as part of the upcoming idnits rewrite.  Once the
new idnits is in production you should see less of this problem.

> 3) Here is a list of the (current) manual changes we are making in 
> order to make id2xml parse with the current version. Please let me
> know if I have mischaracterized any functionality or if any of these 

Will do.  I also have at least one suggestion below of how the tool
might be modified to ease the work.  It would be valuable if you could
indicate which of these are most work-intensive, and most worthy of
attention to lessen the manual work.

> -Update to use first initial instead of full first name (or you get 
>  "Warning: This author is listed in the Authors’ Addresses section,
>  but was not found on the first page: and the authors section will be
>  absent from the XML generated)

Right.

> -Ensure use of “Copyright License” as a title exactly

Yes. If you have commonly occurring variations on the Copyright License
title I can include those in the next release.

> -Add blank lines between list items to fix numbering

Even better, "1."

> -fix indentation with - (dash) to all be inline indentation-wise

This applies to all bullet forms -- indentation after the bullet line
needs to match the start of text column on the bullet line.

> -update + or -iv or anything not xml2rfc-compliant as a list marker

Ack.

> -  Change Appendix A: to be Appendix A. or just Appendix A

Yes.  If Appendix A: is common, I can add recognition of that.

> 	-Change Summer 1996 to a specific month

Right.  And no commas in the date string.

> -Add a blank line before email addresses (temporary?)

Yes, that should not be needed with the next release, unless the
email address is preceeded by a street address.  A blank line is
needed to terminate a street/postal address.

> -Remove any number from this section heading

I don't think that's needed any more.

> -Add URI: before any entry (same for email?)

True for URIs, yes.  And Email: or EMail: before email.

> -Review for mismatch between authors in header and Addresses section

The tool should warn if there's any author mismatch.

> 4) Here is a pointer to a diff file between an original and one 
> including the manual edits we made in order to get the document to
> parse (a file we have previously discussed, just an example):
> 

This matches pretty much what I'd expect.  I don't think you need to
add the "8.  References" line if you give the two references sections
appropriate section numbers (8. and 9.)

> https://www.rfc-editor.org/rfc/v3test/draft-ietf-trill-directory-assist-mechanisms-12v3-rfcdiff.html 

This also looks pretty much like I'd expect.  There is one place I'm
surprised at the lack of proper list handling, where the output after
the tool gives two sequential list items which both are numbere "1.",
in section 2.5.  Did you add blank lines between paragraphs and list,
and between list items there?

The sublist which is numbered 2.a, 2.b, etc. won't be properly
recognized as a numbered list unless you change to 2.1, 2.2, etc.
Right now I suspect it gets treated like one or more hanging lists.

> -Ensure no figures have been turned into lists errantly

Makes sense.

> -Review whitespace generally

Seems reasonable

> -Include references that could not be generated through text fixes previously

Ack.  This should be eased by the --use-citation-tags switch.

> -Review all information from original is included in the file (no missing email addresses)

Ack.

Best regards,

	Henrik

---

From: Megan Ferguson <[email-masked]>
Date: Fri, 14 Jul 2017 09:48:34 -0700

Hi Henrik.

Inline below with MF.

Thanks so much for your time!

Megan

> community).  But I can see the advantage to you, and will add this

MF - That sounds like it would be really helpful if it’s possible.
> relying on recognising certain patterns in the tag.
MF - Right.  Assuming this might involve us updating to the known tags.

> new idnits is in production you should see less of this problem.

MF - That sounds like a viable workaround.  The versions I see in the files below 
are all very similar and we speculate they come from either an outdated
template.  I’d have to do more research on this to get you a list of some of the most 
common forms if that is desirable instead of idnits.
>> absent from the XML generated)
MF - The header updates can be tedious depending on how many authors there are, what 
information is missing from the lefthand side, etc.
> title I can include those in the next release.
MF - I did a quick scan of our original files received from present back to RFCs in 
the 7000s.  Looks like these are the top hits:

Copyright
Copyright Notice
Copyright Notice and License
Copyright and License Notice
Copyright, Disclaimer, and Additional IPR Provisions
Copyright and IPR Provisions
Copyright Statement

> Even better, "1.”
MF - ack.
> Yes.  If Appendix A: is common, I can add recognition of that.
MF - Yes.  That would be good as this is quite common.  
> Right.  And no commas in the date string.

MF - Added to my list.  Updating the reference entries in text is time-intensive as 
they appear in such variant ways in non-xml2rfc-generated files.  The citation tag 
suggestion above would help mitigate some of this.
> I don't think that's needed any more.
MF - Removing from my list.

> True for URIs, yes.  And Email: or EMail: before email.
MF - and e-mail/E-mail too?
> The tool should warn if there's any author mismatch.
MF- Right.
> appropriate section numbers (8. and 9.)

MF - Ok, good to know.
> and between list items there?

MF - The spacing is off there, and I didn’t take time to fix it (assume we would 
probably handle something like this in xml anyway).
> Right now I suspect it gets treated like one or more hanging lists.
MF - Yes.
>> -Ensure no figures have been turned into lists errantly
MF - This is time heavy when the document is full of lists (like an extensive terms list 
that is full of indentation problems would be a major cleanup for us).
>> -Review figures for alignment and missing text

MF - Again, if the document had several figures that were misappropriated as lists, this 
can get time-intensive for us to recopy in as artwork.  Thinking of specifically documents
that have several appendices full of figures describing things in text.  So far, these have
been translating pretty well, but some limitations still exist.  Here is my most recent example 
(from draft-ietf-sidr-bgpsec-algs-18) where the double spacing in the middle throw it off:

Original:
A.3.  BGPsec IPv4

   BGPSec IPv4 Update from AS(65536) to AS(65537):
   ===============================================
   Binary Form of BGPSec Update (TCP-DUMP):

   FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF 
   01 03 02 00 00 00 EC 40  01 01 02 80 04 04 00 00 
   00 00 80 0E 0D 00 01 01  04 C6 33 64 64 00 18 C0 
   00 02 90 1E 00 CD 00 0E  01 00 00 01 00 00 01 00 
   00 00 FB F0 00 BF 01 47  F2 3B F1 AB 2F 8A 9D 26 
   86 4E BB D8 DF 27 11 C7  44 06 EC 00 48 30 46 02 
   21 00 EF D4 8B 2A AC B6  A8 FD 11 40 DD 9C D4 5E 
   81 D6 9D 2C 87 7B 56 AA  F9 91 C3 4D 0E A8 4E AF 
   37 16 02 21 00 90 F2 C1  29 AB B2 F3 9B 6A 07 96 
   3B D5 55 A8 7A B2 B7 33  3B 7B 91 F1 66 8F D8 61 
   8C 83 FA C3 F1 AB 4D 91  0F 55 CA E7 1A 21 5E F3 
   CA FE 3A CC 45 B5 EE C1  54 00 48 30 46 02 21 00 
   EF D4 8B 2A AC B6 A8 FD  11 40 DD 9C D4 5E 81 D6 
   9D 2C 87 7B 56 AA F9 91  C3 4D 0E A8 4E AF 37 16 
   02 21 00 8E 21 F6 0E 44  C6 06 6C 8B 8A 95 A3 C0 
   9D 3A D4 37 95 85 A2 D7  28 EE AD 07 A1 7E D7 AA 
   05 5E CA 

Text output from id2xml:

A.3.  BGPsec IPv4

   BGPSec IPv4 Update from AS(65536) to AS(65537):
   ===============================================
   Binary Form of BGPSec Update (TCP-DUMP):

   FF FF FF FF FF FF FF FF  FF FF FF FF FF FF FF FF
                            01 03 02 00 00 00 EC 40 01 01 02 80 04 04 00
                            00 00 00 80 0E 0D 00 01 01 04 C6 33 64 64 00
                            18 C0 00 02 90 1E 00 CD 00 0E 01 00 00 01 00
                            00 01 00 00 00 FB F0 00 BF 01 47 F2 3B F1 AB
                            2F 8A 9D 26 86 4E BB D8 DF 27 11 C7 44 06 EC
                            00 48 30 46 02 21 00 EF D4 8B 2A AC B6 A8 FD
                            11 40 DD 9C D4 5E 81 D6 9D 2C 87 7B 56 AA F9
                            91 C3 4D 0E A8 4E AF 37 16 02 21 00 90 F2 C1
                            29 AB B2 F3 9B 6A 07 96 3B D5 55 A8 7A B2 B7
                            33 3B 7B 91 F1 66 8F D8 61 8C 83 FA C3 F1 AB
                            4D 91 0F 55 CA E7 1A 21 5E F3 CA FE 3A CC 45
                            B5 EE C1 54 00 48 30 46 02 21 00 EF D4 8B 2A
                            AC B6 A8 FD 11 40 DD 9C D4 5E 81 D6 9D 2C 87
                            7B 56 AA F9 91 C3 4D 0E A8 4E AF 37 16 02 21
                            00 8E 21 F6 0E 44 C6 06 6C 8B 8A 95 A3 C0 9D
                            3A D4 37 95 85 A2 D7 28 EE AD 07 A1 7E D7 AA
                            05 5E CA

> Ack.  This should be eased by the --use-citation-tags switch.
MF - Exactly!