Subject: Re: [TOOLS-DEVELOPMENT] Preview release of Text Submission Converter, id2xml

From: Megan Ferguson <[email-masked]>
Date: Fri, 19 May 2017 12:55:42 -0700

Hi Henrik,

Some notes on our initial test pass.  Please let me know if you would like any further information on any of the points below.

Disclaimer: We understand that the editors will need to clean up the XML when it’s been converted from .txt to .xml.  Please note that we are reporting oddities without trying to figure out which ones could be fixed (i.e., automated).  

Thank you.

Megan

PIs, etc.
——

1) We were surprised to see symrefs set to no as most reference entries we see use the RFC number as citation tag.

2) We see the encoding as utf-8.  We were under the impression that ascii would be the default unless utf-8 characters were included in the file, which they were not in our test cases.

Author elements
————————

3) We see several differences in the use of the elements in the author’s postal address.

v2:
    <author fullname="Les Ginsberg" initials="L" surname="Ginsberg">
      <organization>Cisco Systems</organization>

      <address>
        <postal>
          <street>821 Alder Drive</street>
          <city>Milpitas</city>
          <code>95035</code>
          <region>CA</region>
          <country>USA</country>
        </postal>

        <email>[email-masked]</email>
      </address>
    </author>

id2xml:

    <author fullname="Les Ginsberg" surname="Ginsberg" initials="L. ">
      <organization>Cisco Systems</organization>
      <address>
        <postal>
          <street>821 Alder Drive</street>
          <street>Milpitas, CA  95035</street>
          <street>USA</street>
        </postal>
        <email>[email-masked]</email>
      </address>
    </author>

Artwork tags vs. lists 
———————
4) The following list was converted to a hanging list, which really changed the output.  
The author-submitted version had this list in artwork.

Original text:

   An MI-RTR MUST NOT support [RFC5120] multi-topology within a non-zero
   instance when any non-zero ITID is supported.  The following TLVs
   MUST NOT be sent in an LSP associated with a non-zero instance which
   supports a non-zero ITID and such an LSP MUST be ignored when
   received:

    TLV 222 - MT IS Neighbors
    TLV 235 - MT IP Reachability
    TLV 237 - MT IPv6 Reachability

id2xml text output:

   An MI-RTR MUST NOT support [RFC5120] multi-topology within a non-zero
   instance when any non-zero ITID is supported.  The following TLVs
   MUST NOT be sent in an LSP associated with a non-zero instance which
   supports a non-zero ITID and such an LSP MUST be ignored when
   received:

   TLV 222 - MT IS Neighbors

      TLV 235 - MT IP Reachability TLV 237 - MT IPv6 Reachability

In v2 (as submitted by authors):

      <figure>
        <artwork>
<![CDATA[ 

 TLV 222 - MT IS Neighbors
 TLV 235 - MT IP Reachability
 TLV 237 - MT IPv6 Reachability

]]></artwork>
      </figure>

In id2xml:

      <t>
        <list style="hanging"><t hangText="TLV 222 - MT IS Neighbors"/>
<t>    TLV 235 - MT IP Reachability
    TLV 237 - MT IPv6 Reachability</t></list>
      </t>

Another related example:

In the original text:

   Per [RFC6822], IANA has registered two EUI-48 multicast addresses
   from the IANA-managed EUI address space as specified in [RFC7042].
   The addresses are as follows:

      01-00-5E-90-00-02 AllL1MI-ISs
      01-00-5E-90-00-03 AllL2MI-ISs

   All references to [RFC6822] in the IS-IS TLV Codepoints registry
   should be replaced by references to this document.

In the id2xml text output:

   Per [RFC6822], IANA has registered two EUI-48 multicast addresses
   from the IANA-managed EUI address space as specified in [RFC7042].
   The addresses are as follows:

   01-00-5E-90-00-02 AllL1MI-ISs

      01-00-5E-90-00-03 AllL2MI-ISs

   All references to [RFC6822] in the IS-IS TLV Codepoints registry
   should be replaced by references to this document.

In v2 (as submitted by authors)

      <t><figure>
          <artwork>
<![CDATA[   01-00-5E-90-00-02 AllL1MI-ISs 

   01-00-5E-90-00-03 AllL2MI-ISs 

]]></artwork>
        </figure></t>

      <t><figure>
          <artwork><![CDATA[   01-00-5E-90-00-02 AllL1MI-ISs                             
   01-00-5E-90-00-03 AllL2MI-ISs                                                         
]]></artwork>
        </figure></t>

In id2xml:

      <t>
        <list style="hanging"><t hangText="01-00-5E-90-00-02 AllL1MI-ISs"/>
<t>      01-00-5E-90-00-03 AllL2MI-ISs</t></list>
      </t>

5) The text following this figure was originally included in an <artwork> tag and appeared as slightly indented in the text.

original text:

     Type:   7
     Length: 2 - 254
     Value:
                                            No. of octets
                 +-------------------------+
                 | IID (0 - 65535)         |     2
                 +-------------------------+
                 | Supported ITID          |     2
                 +-------------------------+
                 :                         :
                 +-------------------------+
                 | Supported ITID          |     2
                 +-------------------------+

     When the IID = 0, the list of supported ITIDs MUST NOT be present.

     An IID-TLV with IID = 0 MUST NOT appear in an SNP or LSP. When
     the TLV appears (with a non-zero IID) in an SNP or LSP, exactly
     one ITID MUST be present indicating the instance-specific topology
     with which the PDU is associated. If no ITIDs or multiple ITIDs are
     present or the IID is zero, then the PDU MUST be ignored.

     When the IID is non-zero and the TLV appears in an IIH, the set
     of ITIDs supported on the circuit over which the IIH is sent is
     included. There MUST be at least one ITID present.

     ITID #0 is reserved for a specific use case as described later
     in this document. ITID #0 MUST NOT be supported in combination
     with any non-zero ITID. If multiple ITIDs are advertised in
     an IIH and one of the ITIDs is #0 then the PDU MUST be ignored.

     Multiple IID-TLVs MAY appear in IIHs. If multiple IID-TLVs are
     present and the IID value in all IID-TLVs is not the same, then
     the PDU MUST be ignored.

   A single IID-TLV will support advertisement of up to 126 ITIDs.  If
   multiple IID-TLVs are present in an IIH PDU, the supported set of
   ITIDs is the union of all ITIDs present in all IID-TLVs.

id2xml text output:

     Type:   7
     Length: 2 - 254
     Value:
                                            No. of octets
                 +-------------------------+
                 | IID (0 - 65535)         |     2
                 +-------------------------+
                 | Supported ITID          |     2
                 +-------------------------+
                 :                         :
                 +-------------------------+
                 | Supported ITID          |     2
                 +-------------------------+

   When the IID = 0, the list of supported ITIDs MUST NOT be present.

   An IID-TLV with IID = 0 MUST NOT appear in an SNP or LSP.  When

      the TLV appears (with a non-zero IID) in an SNP or LSP, exactly
      one ITID MUST be present indicating the instance-specific topology
      with which the PDU is associated.  If no ITIDs or multiple ITIDs
      are present or the IID is zero, then the PDU MUST be ignored.

   When the IID is non-zero and the TLV appears in an IIH, the set

      of ITIDs supported on the circuit over which the IIH is sent is
      included.  There MUST be at least one ITID present.

   ITID #0 is reserved for a specific use case as described later

      in this document.  ITID #0 MUST NOT be supported in combination
      with any non-zero ITID.  If multiple ITIDs are advertised in an
      IIH and one of the ITIDs is #0 then the PDU MUST be ignored.

   Multiple IID-TLVs MAY appear in IIHs.  If multiple IID-TLVs are

      present and the IID value in all IID-TLVs is not the same, then
      the PDU MUST be ignored.

   A single IID-TLV will support advertisement of up to 126 ITIDs.  If
   multiple IID-TLVs are present in an IIH PDU, the supported set of
   ITIDs is the union of all ITIDs present in all IID-TLVs.

In v2:

   <figure>
          <artwork><![CDATA[  Type:   7 
  Length: 2 - 254 
  Value:  
                                         No. of octets
              +-------------------------+
              | IID (0 - 65535)         |     2
              +-------------------------+
              | Supported ITID          |     2
              +-------------------------+
              :                         :
              +-------------------------+
              | Supported ITID          |     2
              +-------------------------+

  When the IID = 0, the list of supported ITIDs MUST NOT be present.

  An IID-TLV with IID = 0 MUST NOT appear in an SNP or LSP. When 
  the TLV appears (with a non-zero IID) in an SNP or LSP, exactly 
  one ITID MUST be present indicating the instance-specific topology
  with which the PDU is associated. If no ITIDs or multiple ITIDs are
  present or the IID is zero, then the PDU MUST be ignored.

  When the IID is non-zero and the TLV appears in an IIH, the set 
  of ITIDs supported on the circuit over which the IIH is sent is 
  included. There MUST be at least one ITID present. 

  ITID #0 is reserved for a specific use case as described later
  in this document. ITID #0 MUST NOT be supported in combination
  with any non-zero ITID. If multiple ITIDs are advertised in
  an IIH and one of the ITIDs is #0 then the PDU MUST be ignored.

  Multiple IID-TLVs MAY appear in IIHs. If multiple IID-TLVs are
  present and the IID value in all IID-TLVs is not the same, then
  the PDU MUST be ignored.
]]></artwork>
        </figure>
    <t>A single IID-TLV will support advertisement of up to 126 ITIDs. If
       multiple IID-TLVs are present in an IIH PDU, the supported set of
       ITIDs is the union of all ITIDs present in all IID-TLVs.</t>

id2xml:

        </t>
        <figure>
          <artwork><![CDATA[
  Type:   7
  Length: 2 - 254
  Value:
                                         No. of octets
              +-------------------------+
              | IID (0 - 65535)         |     2
              +-------------------------+
              | Supported ITID          |     2
              +-------------------------+
              :                         :
              +-------------------------+
              | Supported ITID          |     2
              +-------------------------+
]]></artwork>
          <postamble>     When the IID = 0, the list of supported ITIDs MUST NOT be present.</postamble>
        </figure>
        <t>
          <list style="hanging"><t hangText="An IID-TLV with IID = 0 MUST NOT appear in an SNP or LSP. When"/>
<t>     the TLV appears (with a non-zero IID) in an SNP or LSP, exactly
     one ITID MUST be present indicating the instance-specific topology
     with which the PDU is associated. If no ITIDs or multiple ITIDs are
     present or the IID is zero, then the PDU MUST be ignored.</t><t hangText="When the IID is non-zero and the TLV appears in an IIH, the set"/>
<t>     of ITIDs supported on the circuit over which the IIH is sent is
     included. There MUST be at least one ITID present.</t><t hangText="ITID #0 is reserved for a specific use case as described later"/>
<t>     in this document. ITID #0 MUST NOT be supported in combination
     with any non-zero ITID. If multiple ITIDs are advertised in
     an IIH and one of the ITIDs is #0 then the PDU MUST be ignored.</t><t hangText="Multiple IID-TLVs MAY appear in IIHs. If multiple IID-TLVs are"/>
<t>     present and the IID value in all IID-TLVs is not the same, then
     the PDU MUST be ignored.</t></list>
        </t>
        <t>
   A single IID-TLV will support advertisement of up to 126 ITIDs.  If
   multiple IID-TLVs are present in an IIH PDU, the supported set of
   ITIDs is the union of all ITIDs present in all IID-TLVs.</t>

Citations throughout text
—————————

6) It appears that citations were not converted to xref tags throughout.  For example:

v2:
    <section anchor="Security" title="Security Considerations">
      <t>
Security concerns for IS-IS are addressed in [ISO10589, [RFC5304],

      and [RFC5310].</t>
    </section>

id2xml:
    <section title="Security Considerations" anchor="section-7.">
      <t>
   Security concerns for IS-IS are addressed in [ISO10589, [RFC5304], and [RFC5310].</t>
    </section>

Reference Entries in the References section
————————
7) We note that <reference> elements were not used and the references section was created using a <section> element instead of <references> and appears in the <middle> as opposed to <back>.  

Is it possible to make these be includes (e.g., <?rfc include='reference.RFC.XXXX’?>)?  That way, the RPC will know the entries are being pulled from the citation library and contain the correct information.  We understand that v3 will have to contain the expanded references.  

in v2:

  </middle>

  <back>
    <references title="Normative References">
      <?rfc include="reference.RFC.2119"?>

      <?rfc include='reference.RFC.5304'?>

      <?rfc include='reference.RFC.5120'?>

      <?rfc include='reference.RFC.5303'?>

      <?rfc include='reference.RFC.5306'?>

      <?rfc include='reference.RFC.5310'?>

      <?rfc include='reference.RFC.6213'?>

      <?rfc include='reference.RFC.6232'?>

      <?rfc include='reference.RFC.6233'?>

      <?rfc include="reference.RFC.6822"?>

      <?rfc include="reference.RFC.6823"?>

      <?rfc ?>

      <?rfc ?>

      <reference anchor="ISO10589">
        <front>
          <title>Intermediate system to Intermediate system intra-domain
          routeing information exchange protocol for use in conjunction with
          the protocol for providing the connectionless-mode Network Service
          (ISO 8473), ISO/IEC 10589:2002, Second Edition.</title>

          <author fullname="ISO &quot;International Organization for Standardization&quot;"/>

          <date month="Nov" year="2002"/>
        </front>
      </reference>
    </references>

in id2xml:

    <section title="References" anchor="section-9.">
      <section title="Normative References" anchor="section-9.1.">
        <t>
          <list style="hanging" hangIndent="11"><t hangText="[ISO10589]"/>
<t>              "Intermediate system to Intermediate system intra-domain routeing information exchange protocol for use in conjunction with the protocol for providing the connectionless-mode Network Service (ISO 8473), ISO/IEC 10589:2002, Second Edition.", Nov 2002.</t><t hangText="[RFC2119]">Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119,
              DOI 10.17487/RFC2119, March 1997,
              <eref target="http://www.rfc-editor.org/info/rfc2119"/>.</t><t hangText="[RFC5120]">Przygienda, T., Shen, N., and N. Sheth, "M-ISIS: Multi Topology (MT) Routing in Intermediate System to Intermediate Systems (IS-ISs)", RFC 5120,
              DOI 10.17487/RFC5120, February 2008,
              <eref target="http://www.rfc-editor.org/info/rfc5120"/>.</t><t hangText="[RFC5303]">Katz, D., Saluja, R., and D. Eastlake 3rd, "Three-Way Handshake for IS-IS Point-to-Point Adjacencies", RFC 5303,
              DOI 10.17487/RFC5303, October 2008,
              <eref target="http://www.rfc-editor.org/info/rfc5303"/>.</t><t hangText="[RFC5304]">Li, T. and R. Atkinson, "IS-IS Cryptographic Authentication", RFC 5304, DOI 10.17487/RFC5304, October
              2008, <eref target="http://www.rfc-editor.org/info/rfc5304"/>.</t><t hangText="[RFC5306]">Shand, M. and L. Ginsberg, "Restart Signaling for IS-IS",
              RFC 5306, DOI 10.17487/RFC5306, October 2008,
              <eref target="http://www.rfc-editor.org/info/rfc5306"/>.</t><t hangText="[RFC5310]">Bhatia, M., Manral, V., Li, T., Atkinson, R., White, R.,
              and M. Fanto, "IS-IS Generic Cryptographic Authentication", RFC 5310, DOI 10.17487/RFC5310, February
              2009, <eref target="http://www.rfc-editor.org/info/rfc5310"/>.</t><t hangText="[RFC6213]">Hopps, C. and L. Ginsberg, "IS-IS BFD-Enabled TLV",
              RFC 6213, DOI 10.17487/RFC6213, April 2011,
              <eref target="http://www.rfc-editor.org/info/rfc6213"/>.</t><t hangText="[RFC6232]">Wei, F., Qin, Y., Li, Z., Li, T., and J. Dong, "Purge Originator Identification TLV for IS-IS", RFC 6232,
              DOI 10.17487/RFC6232, May 2011,
              <eref target="http://www.rfc-editor.org/info/rfc6232"/>.</t><t hangText="[RFC6233]">Li, T. and L. Ginsberg, "IS-IS Registry Extension for Purges", RFC 6233, DOI 10.17487/RFC6233, May 2011,
              <eref target="http://www.rfc-editor.org/info/rfc6233"/>.</t></list>
        </t>
        <figure>
          <artwork><![CDATA[
[RFC6822]  Previdi, S., Ed., Ginsberg, L., Shand, M., Roy, A., and D.
           Ward, "IS-IS Multi-Instance", RFC 6822,
           DOI 10.17487/RFC6822, December 2012,
           <http://www.rfc-editor.org/info/rfc6822>.
]]></artwork>
        </figure>
        <t>
          <list style="hanging" hangIndent="11">
            <t hangText="[RFC6823]">Ginsberg, L., Previdi, S., and M. Shand, "Advertising Generic Information in IS-IS", RFC 6823,
              DOI 10.17487/RFC6823, December 2012,
              <eref target="http://www.rfc-editor.org/info/rfc6823"/>.</t>
          </list>
        </t>
      </section>
      <section title="Informative References" anchor="section-9.2.">
        <t>
          <list style="hanging" hangIndent="11">
            <t hangText="[RFC5309]">Shen, N., Ed. and A. Zinin, Ed., "Point-to-Point Operation over LAN in Link State Routing Protocols", RFC 5309,
              DOI 10.17487/RFC5309, October 2008,
              <eref target="http://www.rfc-editor.org/info/rfc5309"/>.</t>
            <t hangText="[RFC7042]">Eastlake 3rd, D. and J. Abley, "IANA Considerations and IETF Protocol and Documentation Usage for IEEE 802 Parameters", BCP 141, RFC 7042, DOI 10.17487/RFC7042,
              October 2013, <eref target="http://www.rfc-editor.org/info/rfc7042"/>.</t>
          </list>
        </t>
      </section>
    </section>
  </middle>
  <back>

8) In v2, there is currently a setting to keep DOIs, draft strings, and URLs on the same line (if possible for the latter).

9) We generally see a single space after the author initials instead of two spaces there.

original text:

   [RFC5310]  Bhatia, M., Manral, V., Li, T., Atkinson, R., White, R.,
              and M. Fanto, "IS-IS Generic Cryptographic
              Authentication", RFC 5310, DOI 10.17487/RFC5310, February
              2009, <http://www.rfc-editor.org/info/rfc5310>.

id2xml text output:

   [RFC5310]  Bhatia, M., Manral, V., Li, T., Atkinson, R., White, R.,
              and M.  Fanto, "IS-IS Generic Cryptographic
              Authentication", RFC 5310, DOI 10.17487/RFC5310, February
              2009, <http://www.rfc-editor.org/info/rfc5310>.

---

From: Julian Reschke <[email-masked]>
Date: Fri, 19 May 2017 22:26:13 +0200

On 2017-05-19 21:55, Megan Ferguson wrote:
> ...
> ...

+1

> 2) We see the encoding as utf-8.  We were under the impression that ascii would be the default unless utf-8 characters were included in the file, which they were not in our test cases.

How does that matter if the input is indeed all ASCII?

> ...
> ...

Why? Includes are not part of the xml2rfc vocabulary. Inline <reference> 
entries or external entity references 
(<https://greenbytes.de/tech/webdav/rfc7749.html#including.files>) seem 
like a better choice.

> ...

Best regards, Julian

---

From: Henrik Levkowetz <[email-masked]>
Date: Fri, 19 May 2017 23:20:25 +0200

Hi Megan,

On 2017-05-19 21:55, Megan Ferguson wrote:
> Hi Henrik,
> 
> Some notes on our initial test pass. Please let me know if you would
> like any further information on any of the points below.

In general, I'd very much appreciate the relevant draft name when you
point out an issue; that will let me use that in testing and fixing
the issue.  As you'll see below, I believe that many of the issues you
mention below are fixed in the latest release (1.0.0-rc1), but there
are some cases below where the draft name would be good to have.

> Disclaimer: We understand that the editors will need to clean up the
> XML when it’s been converted from .txt to .xml. Please note that we
> are reporting oddities without trying to figure out which ones could
> be fixed (i.e., automated).

Ack, understood.  Thank you for the work in testing and writing up the
issues :-)

More inline:

> Thank you.
> 
> Megan
> 
> PIs, etc.
> ——
> 
> 1) We were surprised to see symrefs set to no as most reference
> entries we see use the RFC number as citation tag.

The generated xml should reflect the symrefs setting appropriate for
the input.  If the input file uses symrefs="no", the xml will try to
detect that; and likewise for symrefs="yes".  Could you send me the
input source which gave symrefs="no" when you did not expect it, please?

> 2) We see the encoding as utf-8. We were under the impression that
> ascii would be the default unless utf-8 characters were included in
> the file, which they were not in our test cases.

utf-8 is the default encoding for XML; I have not made any effort to
set a different encoding based on detecting ascii or not.  Does this
make a difference for you?  If so, could you give more details so I
understand better?

In the past, using encoding='ascii' would make sense, to enforce the
ascii-only policy; however going forward, I expect utf-8 will be the
norm, and once utf-8 becomes permitted in some elements, it's probably
better to consistently use encoding='utf-8' in the xml files, rather
than have different settings depending on content.  However, I am not
the user here, and don't know all the wrinkles involved ,:-)

>         <email>[email-masked]</email>

Yes.  The v2 address elements will go away when we hit v3, which doesn't try
to squeeze addresses into a US-centric format.  (For an example of how 
completely different from the US address concept some other addresses are, see
https://en.wikipedia.org/wiki/Japanese_addressing_system)

Trying to map an arbitrary address into a format which uses the v2 fields
correctly would be a major undertaking, and would be abandoned within a year.
I've chosen not to do so in id2xml, leaving the proper matching to a human.

>       TLV 235 - MT IP Reachability TLV 237 - MT IPv6 Reachability

Yes.  A definite flaw.  This is much improved in version 1.0.0-rc1, which
I just released.  There will, however, still be cases where id2xml tries
to make lists out of something that originally was artwork.  This is for
good reasons: Artwork is the last fallback if nothing else will fit, because
it doesn't map well to html elements -- it's just a fixed-width-font blob.

I'll snip the other examples you give -- please try v1.0.0-rc1 on the
relevant documents, and see if it does better.  I believe it will.  If not,
please let me know the document names I should use to reproduce the issue.

> 5) The text following this figure was originally included in an
> <artwork> tag and appeared as slightly indented in the text.
> 
> original text:

...

>                                             No. of octets

( <<snip>> a number of similar cases )

Yes.  This was a recurring phenomenon for some documents in 0.9.3.
I believe it will be better handled by 1.0.0-rc1.  Could you check?

If not, could you please provide the name of the input document, so
I can add it to my test suite and work in improving the handling?

> 6) It appears that citations were not converted to xref tags

This should also be much improved in 1.0.0-rc1.  Still missing is
conversion of citations within tables; I've found that the chances
of messing up there are higher.  I'll have a look at addressing that
case, too.

> 7) We note that <reference> elements were not used and the references
> section was created using a <section> element instead of <references>
> and appears in the <middle> as opposed to <back>.

The detection of reference sections was very mixed in version 0.9.3.  If
the reference sections were numbered, they would be missed, if they were
not, they would be processed.  Version 1.0.0-rc1 attempts to recognize also
numbered reference sections, and generate both inline <reference/> entries
and entity definitions at the top.

> Is it possible to make these be includes (e.g., 
> <?rfc include='reference.RFC.XXXX’?>)? That way, the RPC will know the
> entries are being pulled from the citation library and contain the
> correct information. We understand that v3 will have to contain the
> expanded references.

It is possible, but as you indicate, the <?rfc include=""?> construct is
deprecated.  The best (I believe) alternative when working with both v2
and v3 xml sources is to use entity definitions, and have those expanded by
xml2rfc (in v2) and by the preptool (in v3) during final processing.  In
order to make it easy to use entities, I insert entity definitions at the
start of the generated xml both in 0.9.3 and in 1.0.0-rc1.

The choice of inserting both entity definitions at the start, and expanded
xml refereces in the <references/> sections was done to make it easier for
you to choose one or the other.  If you'd rather I inserted the appropriate
entity at the point of reference, I could do so.  You would then instead
of:

  <back>

see:

  <back>
      &RFC2119;
      &RFC5304;
      ...

and it would pull the reference entries from the citation library in the
same way as with the <?rfc include ?> pi.

> 8) In v2, there is currently a setting to keep DOIs, draft strings,
> and URLs on the same line (if possible for the latter).

Ok; which setting are you thinking of?  I can certainly insert it if
desired.

>               2009, <http://www.rfc-editor.org/info/rfc5310>.

There was an issue in 0.9.2 where extra spaces were not stripped from the
initials string; that was fixed in 0.9.3, and should also be fixed in 
1.0.0-rc1.  Please let me know if you find that it's not the case.

Thank you so much for the feedback!

Best regards,

	Henrik

---

From: Sandy Ginoza <[email-masked]>
Date: Fri, 19 May 2017 14:43:37 -0700

Hi Henrik

> On May 19, 2017, at 2:20 PM, Henrik Levkowetz <[email-masked]> wrote:
> 

Ah, this may have been my misunderstanding.  I thought the .txt files would be output differently depending, perhaps, on how encoding was set in the XML file.  Is it correct that ALL .txt files will have a BOM, regardless of whether the encoding is set to UTF-8 (I was thinking it was only to be applied to those .txt files that contained UTF-8 to let the reader know the file is not strictly ASCII)?  

Thanks!

Sandy

---

From: Henrik Levkowetz <[email-masked]>
Date: Sat, 20 May 2017 11:31:47 +0200

Hi Sandy,

On 2017-05-19 23:43, Sandy Ginoza wrote:
> Ah, this may have been my misunderstanding. I thought the .txt files
> would be output differently depending, perhaps, on how encoding was
> set in the XML file.

Aha. No, but in a limited sense.  More below.

> Is it correct that ALL .txt files will have a
> BOM, regardless of whether the encoding is set to UTF-8 (I was
> thinking it was only to be applied to those .txt files that contained
> UTF-8 to let the reader know the file is not strictly ASCII)?> 

The important thing here is that ASCII is a subset of UTF-8.

Given that, it is equally correct to give an XML file with pure ASCII
content the encoding='utf-8' and to give it the encoding='ascii', with
no change in the rest of the bytes in the file.  And the same goes for
the output; you can say that a file with pure ascii content is encoded
as UTF-8, and be perfectly right; there is no difference in the encoding
between ASCII and UTF-8 for pure-ASCII content.

So returning briefly to the 'No' above: If the XML content is pure ASCII,
the output will be the same irrespective of whether the file states
"encoding='ascii'" or "encoding='utf-8'".  If the XML content is non-ASCII
UTF-8, and the file states "encoding='ascii'", the file is invalid.

For the XML files, I'd go with encoding='utf-8' consistently.  For the
generated text files, my preference would be to only tag them with a
BOM in contexts where no other way of signalling the encoding exists.
But that may complicate your toolchain too much.  So the choice is then
between giving all .txt files a BOM, or giving only .txt files with non-
ASCII content a BOM.  Both will result in valid files, but the BOM will
be completely unnecessary and unhelpful if the content is pure ASCII.

So I would go with the concept you stated in your question above, of
applying a BOM only to the text files which have non-ASCII content.

You should be able to identify .txt output with non-ascii content by use
of the 'file' command, e.g., "$ file --brief --mime-encoding rfcNNNN.txt"
-- this should output the encoding, in our case either 'utf-8' or 'us-ascii'.

This turned out somewhat long-winded, but I hope it helps.

Best regards,

	Henrik

---

From: "HANSEN, TONY L" <[email-masked]>
Date: Sun, 21 May 2017 01:52:10 +0000

Let’s just say that I’m not a fan of BOM on UTF-8 files, ever, and encoding=’ascii’ should just disappear.

Some would say that’s too extreme a stance, and others that it’s just right.

	Tony

On 5/20/17, 5:31 AM, "TOOLS-DEVELOPMENT on behalf of Henrik Levkowetz" <[email-masked] on behalf of [email-masked]> wrote:

    Hi Sandy,

    

    

    

    Best regards,
    
    	Henrik

---

From: Henrik Levkowetz <[email-masked]>
Date: Sun, 21 May 2017 10:56:38 +0200

> encoding=’ascii’ should just disappear.

Oh, I agree.  And I've voiced that sentiment about BOM in the context of
utf-8 text RFCs before.  But the decision isn't mine, and a couple of years
ago a survey showed enough programs of note which couldn't deal with utf-8
files if they didn't have a BOM, that a decision was made to add BOMs.

> Some would say that’s too extreme a stance, and others that it’s just

To me it sounds just right :-)

	Henrik

> _______________________________________________
> TOOLS-DEVELOPMENT mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tools-development
>

---

From: Julian Reschke <[email-masked]>
Date: Sun, 21 May 2017 13:11:58 +0200

> To me it sounds just right :-)

Just throwing in pros and cons:

- avoiding a BOM on documents that do not need it avoids clutter that 
some software might get confused by

- having the BOM on all documents avoids a special-case, and might cause 
people to switch to software that handles the BOM correctly

Best regards, Julian

---

From: Megan Ferguson <[email-masked]>
Date: Fri, 26 May 2017 11:49:00 -0700

Hi Henrik,

Thank you for your reply and explanations.  Inline below with MF.

Megan

Begin forwarded message:

> From: Henrik Levkowetz <[email-masked]>
> Subject: Re: [TOOLS-DEVELOPMENT] Preview release of Text Submission Converter, id2xml
> Date: May 19, 2017 at 2:20:25 PM PDT
> To: Megan Ferguson <[email-masked]>
> Cc: [email-masked]
> 
> Hi Megan,
> 

MF- Ack.  I used draft-ietf-isis-mi-bis-03 originally.  I have rerun this same file with the newer version and found 
most of the items previously discussed to be resolved.

I also tested on draft-ietf-rtgwg-yang-key-chain-24.

Latest Release Notes:
---------------------

1) Missing references:

When running id2xml (v1.0.0-rc1) on draft-ietf-isis-mi-bis-03, I received the following warning:

id2xml draft-ietf-isis-mi-bis-03test.txt
Converting 'draft-ietf-isis-mi-bis-03test.txt'

draft-ietf-isis-mi-bis-03test.txt(660): Warning: Failed parsing a reference:
   [ISO10589]
              "Intermediate system to Intermediate system intra-domain
              routeing information exchange protocol for use in
              conjunction with the protocol for providing the
              connectionless-mode Network Service (ISO 8473), ISO/IEC
              10589:2002, Second Edition.", Nov 2002.
Written to 'draft-ietf-isis-mi-bis-03test.xml’

With the reference being removed but the reference element still included (causing v2 not to parse until the empty 
element was removed).

Note that I also got a similar error running draft-ietf-rtgwg-yang-key-chain-24:

id2xml draft-ietf-rtgwg-yang-key-chain-24test.txt
Converting 'draft-ietf-rtgwg-yang-key-chain-24test.txt'

draft-ietf-rtgwg-yang-key-chain-24test.txt(983): Warning: Failed parsing a reference:
   [Dobb96b]  Dobbertin, H., "The Status of MD5 After a Recent Attack",
              CryptoBytes Vol. 2, No. 2, Summer 1996.

*This second instance was more interesting to me based on the fact that the document contains a [Dobb96a] reference 
that is used very similarly, but was picked up just fine.

2) I was curious about how a reference to a BCP that included more than one RFC would be handled (for example, BCP 9 below).  

In text, it would appear as:

   [BCP9]     Bradner, S., "The Internet Standards Process -- Revision
              3", BCP 9, RFC 2026, October 1996.

              Kolkman, O., Bradner, S., and S. Turner, "Characterization
              of Proposed Standards", BCP 9, RFC 7127, January 2014.

              Dusseault, L. and R. Sparks, "Guidance on Interoperation
              and Implementation Reports for Advancement to Draft
              Standard", BCP 9, RFC 5657, September 2009.

              Housley, R., Crocker, D., and E. Burger, "Reducing the
              Standards Track to Two Maturity Levels", BCP 9, RFC 6410,
              October 2011.

              Resnick, P., "Retirement of the "Internet Official
              Protocol Standards" Summary Document", BCP 9, RFC 7100,
              December 2013.

              Dawkins, S., "Increasing the Number of Area Directors in
              an IETF Area", BCP 9, RFC 7475, March 2015.

              <http://www.rfc-editor.org/info/bcp9>

I don’t have a draft in queue to test this on, so I just manually added a reference to BCP 9 and corresponding citation 
to draft-ietf-rtgwg-yang-key-chain-24.txt. Below is the resulting XML from id2xml:

<reference anchor="BCP9"><front><title>The Internet Standards Process -- Revision 3</title>
	<author fullname="S. Bradner" initials="S." surname="Bradner"/>
	<date month="October" year="1996"/>
	</front>
	<seriesInfo name="BCP" value="9"/>
	<seriesInfo name="RFC" value="2026"/>
	</reference>
	<reference><front/>
	</reference>
	<reference><front/>
	</reference>
	<reference><front/>
	</reference>
	<reference><front/>
	</reference>
	<reference><front/>
	</reference>
	<reference><front/>
	</reference>

*Note - references to multi-RFC BCPs and STDs are rare and require manual post-xml2rfcv2 updates with our current tools. 
The current possible workaround in xml is to include an <annotation> element.  However, I *believe* only one such element is allowed.

3) There are XML snippets in the appendices of draft-ietf-rtgwg-yang-key-chain-24.  id2xml put the first two them into lists, 
which broke the alignment (the original XML by the authors used <artwork>), but the third snippet in the last appendix was put into <artwork>.

Original text:

Appendix A.  Examples

A.1.  Simple Key Chain with Always Valid Single Key

   <?xml version="1.0" encoding="utf-8"?>
   <data xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
     <key-chains xmlns="urn:ietf:params:xml:ns:yang:ietf-key-chain">
       <key-chain>
         <name>keychain-no-end-time</name>
         <description>
           A key chain with a single key that is always valid for
           transmission and reception.
         </description>
         <key>
           <key-id>100</key-id>
           <lifetime>
             <send-accept-lifetime>
               <always/>
             </send-accept-lifetime>
           </lifetime>
           <crypto-algorithm>hmac-sha-256</crypto-algorithm>
           <key-string>
             <keystring>keystring_in_ascii_100</keystring>
           </key-string>
         </key>
       </key-chain>
     </key-chains>
   </data>

id2xml text:
Appendix A.  Examples

A.1.  Simple Key Chain with Always Valid Single Key

   <?xml version="1.0" encoding="utf-8"?>
     <data xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"> <key-chains
     xmlns="urn:ietf:params:xml:ns:yang:ietf-key-chain"> <key-chain>
     <name>keychain-no-end-time</name> <description> A key chain with a
     single key that is always valid for transmission and reception.

       </description> <key> <key-id>100</key-id> <lifetime> <send-
       accept-lifetime> <always/>

          </send-accept-lifetime>

          </lifetime>
            <crypto-algorithm>hmac-sha-256</crypto-algorithm> <key-
            string> <keystring>keystring_in_ascii_100</keystring>

          </key-string>

       </key>

          </key-chain>

          </key-chains>

   </data>

[snip]
> input source which gave symrefs="no" when you did not expect it, please?

MF  - This was in the XML output from draft-ietf-isis-mi-bis-03.txt: it appears to be set to “yes” in the current version of id2xml.

[snip]

> please let me know the document names I should use to reproduce the issue.
MF - This is greatly improved in the current version when running this same file.
> I believe it will be better handled by 1.0.0-rc1.  Could you check?
MF - Yes, this seems to be resolved.

> of messing up there are higher.  I'll have a look at addressing that
MF - Yes it is.
> and entity definitions at the top.

MF - This seems to be the case when rerunning draft-ietf-isis-mi-bis-03.  
(Please note the 2 exceptions I encountered above.)

> same way as with the <?rfc include ?> pi.
MF - We believe it is more desirable for our purposes to have the entity definitions at the start and the &RFC2119;-style calls in the References section.  
> Ok; which setting are you thinking of?  I can certainly insert it if

MF - Sorry - I don’t know of a specific setting.  I simply remember them implementing it into one of the v2 versions. 
> 1.0.0-rc1.  Please let me know if you find that it's not the case.
MF - This is resolved.

---

From: Sandy Ginoza <[email-masked]>
Date: Fri, 26 May 2017 12:18:04 -0700

Hi All,

xml v2 does not handle the formatting for these types of reference entries, and I believe the goal for id2xml is to convert .txt to .xml v2 (so this will not be addressed via id2xml) — though, please correct me if I’m mistaken.  If not, we will continue to manually update these entries post xml2rfc. 

However, I am curious about where the reference information to create the XML entry is being pulled from.  Does it XMLize the text entry or does it pull info from somewhere else? 

Thanks,
Sandy

> On May 26, 2017, at 11:49 AM, Megan Ferguson <[email-masked]> wrote:
> 
>              <http://www.rfc-editor.org/info/bcp9 <http://www.rfc-editor.org/info/bcp9>>
> 
>

---

From: Henrik Levkowetz <[email-masked]>
Date: Fri, 26 May 2017 22:08:09 +0200

On 2017-05-26 21:18, Sandy Ginoza wrote:
> Hi All,
> 
> xml v2 does not handle the formatting for these types of reference
> entries, and I believe the goal for id2xml is to convert .txt to .xml
> v2 (so this will not be addressed via id2xml) — though, please
> correct me if I’m mistaken. If not, we will continue to manually
> update these entries post xml2rfc.

For version 1.x of id2xml this is right.  The intention is to produce a
version 2.x of id2xml which will produce vocabulary v3 xml as part of the
current work.  At that time, entries like the one below should be
correctly parsed into xml (v3) by id2xml.

> However, I am curious about where the reference information to create
> the XML entry is being pulled from. Does it XMLize the text entry or
> does it pull info from somewhere else?

That would be interesting to know, yes.

	Henrik

---

From: Henrik Levkowetz <[email-masked]>
Date: Fri, 26 May 2017 22:27:21 +0200

Hi Megan,

On 2017-05-26 20:49, Megan Ferguson wrote:
> With the reference being removed but the reference element still
> included (causing v2 not to parse until the empty element was
> removed).

Yes, this is the current behaviour with a reference that cannot be parsed.

If you are happy with the text in the input document (i.e., it looks the way
you would want to see it in an RFC, then please let me know, and I'll see if
I can adjust the set of regexes used to parse reference text.  In the case
above, it seems to me that series information has incorrectly been made part
of the document title.  I would expect the reference to be correctly parsed
if you break out the series info the way you'd like to see it in a published
document.

>               CryptoBytes Vol. 2, No. 2, Summer 1996.

Yes.  There's nothing in the current rule set for reference text which knows
how to handle 'Summer 1996'.

> *This second instance was more interesting to me based on the fact
> that the document contains a [Dobb96a] reference that is used very
> similarly, but was picked up just fine.
> 
> 2) I was curious about how a reference to a BCP that included more
> than one RFC would be handled (for example, BCP 9 below).

[snip]

Sandy posted a comment on this, which was to-the-point.

> 3) There are XML snippets in the appendices of
> draft-ietf-rtgwg-yang-key-chain-24. id2xml put the first two them
> into lists, which broke the alignment (the original XML by the
> authors used <artwork>), but the third snippet in the last appendix
> was put into <artwork>.

Right.  This is something I also found meanwhile.  I've enhanced the
regex which identifies code snippets to recognise xml and relatives;
this should be handled correctly in 2.0.0rc3.

[snipping a number of points where you confirm that 1.0.0-rc1 fixed
the issue -- thank you, good to get the confirmation]

> MF - We believe it is more desirable for our purposes to have the
> entity definitions at the start and the &RFC2119;-style calls in the
> References section.

Ok, I will remove the generated reference xml and instead insert the
reference entities in the next release.

> MF - Sorry - I don’t know of a specific setting.  I simply remember them implementing it into one of the v2 versions. 

Ok.  FWIW, when xml2rfc processes what id2xml currently generates, it
tries to keep DOIs etc. on one line.  That's actually a source of
differences with the input documents in the test suite, in several cases,
because they don't always try as hard to keep the DOIs etc. on the same
line ,:-)

>> 1.0.0-rc1.  Please let me know if you find that it's not the case.

Ok, splendid.

Thank you again for the feedback!

Best regards,

	Henrik