Subject: Re: [TOOLS-DEVELOPMENT] Preview release of Text Submission Converter, id2xml

From: Megan Ferguson <[email-masked]>
Date: Tue, 6 Jun 2017 17:17:03 -0700

Hi Henrik,

Input file: draft-ietf-sidr-bgpsec-protocol-23
Version: id2xml 1.0.0
Issues: Figures/Tables and References 

1) Figures & Tables

Note - the related figures triggered a warning as the example that follows (i.e., Figures 3-7), 
so perhaps these issues are already known.

draft-ietf-sidr-bgpsec-protocol-23test.txt(422): Warning: Unexpected title: expected 'Table ...’, 
found 'Figure 4: Secure_Path format.'.  This looks like a texttable that has been entered as a figure.  The generated XML will need adjustment.

a) Extra space added in figures.  

Original text:

            +-----------------------------------------------+
            | Secure_Path Length                 (2 octets) |
            +-----------------------------------------------+
            | One or More Secure_Path Segments   (variable) |
            +-----------------------------------------------+

                      Figure 4: Secure_Path format.

id2xml text output version:

            +-----------------------------------------------+
            | Secure_Path Length                 (2 octets) |
            +-----------------------------------------------+
            | One or More Secure_Path Segments   (variable) |
            |                                               |
            +-----------------------------------------------+

                      Figure 4: Secure_Path format.

b) Perhaps because of the error mentioned, the figure numbers were changed.

This also occurred with Figure 8, which was renumbered as Figure 4, Figure 9 which 
became Figure 5, Figure 10 which became Figure 6, and Figure 11 which became Figure 7.

Original:

    +------------------------------------------------------+
    | pCount         (1 octet)                             |
    +------------------------------------------------------+
    | Confed_Segment flag (1 bit) |  Unassigned (7 bits)   | (Flags)
    +------------------------------------------------------+
    | AS Number      (4 octets)                            |
    +------------------------------------------------------+

                  Figure 5: Secure_Path Segment format.

id2xml output version:

    +------------------------------------------------------+
    | pCount         (1 octet)                             |
    +------------------------------------------------------+
    | Confed_Segment flag (1 bit) |  Unassigned (7 bits)   | (Flags)
    +------------------------------------------------------+
    | AS Number      (4 octets)                            |
    +------------------------------------------------------+

                  Figure 3: Secure_Path Segment format.

An example in the xml created by id2xml:

	<figure anchor="figure-5" title="Secure_Path Segment format."><artwork><![CDATA[
 +------------------------------------------------------+
 | pCount         (1 octet)                             |
 +------------------------------------------------------+
 | Confed_Segment flag (1 bit) |  Unassigned (7 bits)   | (Flags)
 +------------------------------------------------------+
 | AS Number      (4 octets)                            |
 +------------------------------------------------------+
]]></artwork>
	</figure>
	<t>
  The AS Number (in Figure 5) is the AS number of the BGP speaker that
  added this Secure_Path Segment to the BGPsec_Path attribute.  (See
  <xref target="section-4"/> for more information on populating this field.)</t>

c) We see that the “Figures” that were made into texttables did not use a title element, but created the following.  Additionally, we do not see any alignment designation for the text (e.g., <ttcol align='center’>text</ttcol>).

id2xml XML:

<t>
  Figure 3 provides the specification of the format for the BGPsec_Path
  attribute.</t>

	<texttable style="full"><ttcol> Secure_Path                             (variable)</ttcol>
	<c>Sequence of one or two Signature_Blocks (variable)</c>
	<c></c>
	</texttable>
	<t><list hangIndent="15" style="hanging"><t>
                 Figure 3: BGPsec_Path attribute format.</t>
	</list>
	</t>

2) We note that references to sections and figures within the document did not use <xref> elements consistently.  For example:

id2xml in xml file:

<t>
If the update message was received from a peer that is not
expected to set pCount=0 (see <xref target="section-4.2"/>
and Section 4.3) then
check to ensure that the pCount field in the most-recently added
Secure_Path Segment is not equal to zero.  (Note: See router
configuration guidance related to this in Section 7.2.)</t>

With figure mentions not using them at all (at least in this file).

3) A few missed reference entries (creating empty reference elements) and xref elements in the running text (this issue was previously reported).  The output XML file wouldn't validate because of the errors. 

ERROR: Unable to validate the XML document: draft-ietf-sidr-bgpsec-protocol-23test.xml
draft-ietf-sidr-bgpsec-protocol-23test.xml: Line 2002: Element front content does not follow the DTD, expecting (title , author+ , date , area* , workgroup* , keyword* , abstract? , note*), got nothing.
draft-ietf-sidr-bgpsec-protocol-23test.xml: Line 2006: Element front content does not follow the DTD, expecting (title , author+ , date , area* , workgroup* , keyword* , abstract? , note*), got nothing.
draft-ietf-sidr-bgpsec-protocol-23test.xml: Line 2025: Element front content does not follow the DTD, expecting (title , author+ , date , area* , workgroup* , keyword* , abstract? , note*), got nothing.

Thanks!

Megan

---

From: Henrik Levkowetz <[email-masked]>
Date: Tue, 13 Jun 2017 14:18:17 +0200

Hi Megan,

> a) Extra space added in figures.  

Right.  This is a bug, an empty table cell.  Will fix.

> b) Perhaps because of the error mentioned, the figure numbers were changed.

Yes.  So, there are related issues here.

The basic question is how to handle things which can be marked up as tables.
id2xml takes the approach that if it can be marked up as tables, it will be,
because even if it makes no difference if it's done as tables or as figures
in the text output, it will make a lot of difference in the quality of html
output.  So if it is recognized as a table, the generated xml will be for a
texttable, rather than for a figure.

When it turns out that the draft numbers it as a figure, you have 2 options:
either change the markup, removing the texttable markup and adding in figure
artwork instead, which will preserve the figure numbering, or, what I'd
recommend in most cases, accept the change from Figure to Table, and adjust
the title accordingly.  This will in most cases give the best html output.

<snip additional examples>

> c) We see that the “Figures” that were made into texttables did not
> use a title element, but created the following. Additionally, we do
> not see any alignment designation for the text (e.g., <ttcol
> align='center’>text</ttcol>).

Yes.  For tables which have a Figure title, but get texttable markup,
no title is created.  I could change that, so that I recognize the Figure:
as an indication of a title line, also for texttables.  That would probably
remove some manual work.  Will fix.

> 2) We note that references to sections and figures within the
> document did not use <xref> elements consistently. For example:
> 
> id2xml in xml file:
> 
> <t>

Aha. Ok.  That's a bug in the pattern that recognizes Section references.
Will fix.

> With figure mentions not using them at all (at least in this file).

Right.  I can add that, but there's an issue here, with the potential
re-numbering of Table: and Figure: items.  I'll see what I can do.

> 3) A few missed reference entries (creating empty reference elements)
> and xref elements in the running text (this issue was previously
> reported). The output XML file wouldn't validate because of the
> errors.
> 
> 

Aha. Ok, I'll dig into that.  Looks like a clear bug.

Thank you for the feedback!

Best regards,

	Henrik

---

From: Henrik Levkowetz <[email-masked]>
Date: Tue, 13 Jun 2017 14:26:16 +0200

So, regarding this:

> Aha. Ok, I'll dig into that.  Looks like a clear bug.

When id2xml fails to parse a reference, it inserts a <reference/> element
with only a <front/>, which will not convert.  This is so that you don't
unintentionally end up with missing references in the converted XML.  The
solution is generally to adjust the reference text in the input file to
match the format you would expect; I think that id2xml is now at a point
where it will recognize pretty much all reference text entries which have
the format used in published RFCs.

I'm going to look at the 3 instances in draft-ietf-sidr-bgpsec-protocol-23
and see if additional patterns make sense, though.

If you would like a different approach than leaving empty <front/> elements
in the file, I'd be happy to adjust things the way you want them; just let
me know what you'd like.

Best regards,

	Henrik

---

From: Henrik Levkowetz <[email-masked]>
Date: Tue, 13 Jun 2017 14:39:05 +0200

Returning to this issue again:

> and see if additional patterns make sense, though.

Changing "February 10, 2016" to "10 February 2016" lets the [Borchert]
reference entry be parsed.  I would prefer not to add another pattern to
handle this, as commas generally separate the reference parts, and adding
patterns for individual parts that contain commas are bound to lead to
complications.

For the other 2 entries ([FIPS186-4] and [SP800-90A]) I've now added a new
pattern which will convert them.

	Henrik