Subject: [TOOLS-DEVELOPMENT] Preview release of Text Submission Converter, id2xml

From: Henrik Levkowetz <[email-masked]>
Date: Mon, 15 May 2017 12:59:49 +0200

Hi,

I've just uploaded the third preview release (0.9.2) of 'id2xml', the
text-submission converter tool:

  https://pypi.python.org/pypi/id2xml

You should be able to install it with:

  $ pip install id2xml

All feedback is very welcome.

From the README file:

This tool, 'id2xml', is intended for use by the RFC-Editor staff, in order to
produce a first xml2rfc-compatible XML version from text-only Internet-Draft
submissions.

id2xml may also be useful for Internet-Draft authors who wish to start working
on a new version of an older draft or RFC, for which no xml2rfc-compatible XML
source is available.

The releases of the 0.9.x release series are preview releases, with a number
of known deficiencies.  They are released in order that potential users may
provide feedback on the most desired improvements ahead of a 1.0.0 release.

Version 0.9.0 can process the drafts specified in the development Statement of
Work to XML files acceptable to xml2rfc, and can also process a number of
other test files to acceptable XML.  However, adding new drafts to the test
set still reveals weaknesses in many cases, so trouble-free processing of any
arbitrary draft should not be expected from the 0.9.x series.

The XML produced follows RFC 7749 [1]_ in version 0.9.x and 1.x of the tool, and will follow RFC 7991 [2]_ in version 2.x, which will be released once
support is available to process XML sources which follows the RFC 7991
vocabulary.

.. [1] Reschke, J., "The "xml2rfc" Version 2 Vocabulary", RFC 7749, DOI
   10.17487/RFC7749, February 2016, <http://www.rfc-editor.org/info/rfc7749>.

.. [2] Hoffman, P., "The "xml2rfc" Version 3 Vocabulary", RFC 7991, DOI
   10.17487/RFC7991, December 2016, http://www.rfc-editor.org/info/rfc7991>.

Best regards,

	Henrik

---

From: Russ Housley <[email-masked]>
Date: Mon, 15 May 2017 15:53:00 -0400

Henrik:

After  $ pip install id2xml, what is the command line to produce a .xml form a .txt that contains an RFC or Internet-Draft?

Russ

> On May 15, 2017, at 6:59 AM, Henrik Levkowetz <[email-masked]> wrote:
> 
> Hi,
> 
> _______________________________________________
> TOOLS-DEVELOPMENT mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tools-development

---

From: Henrik Levkowetz <[email-masked]>
Date: Mon, 15 May 2017 22:11:19 +0200

Hi Russ,

> After  $ pip install id2xml, what is the command line to produce a .xml form a .txt that contains an RFC or Internet-Draft?

Something like

 $ id2xml draft-old-foo-bar.txt -o draft-new-foo-bar.xml

or

 $ id2xml draft-foo-bar.txt -p ~/workdir/

should do it.

On first invocation, id2xml will produce an .xml file also if you don't
give an explicit output file or output path, if there isn't already an
xml file with the inferred name; but to avoid possibly overwriting an
original xml source, it won't do any overwrites unless given an explicit
target name or path.

If $ id2xml --help doesn't work, there's probably an issue with your default
pip and your default python not agreeing on installation directories.
Write me off-list if you run into indications that that's the case.

Best regards,

	Henrik

---

From: Robert Sparks <[email-masked]>
Date: Mon, 15 May 2017 16:15:47 -0400

> After  $ pip install id2xml, what is the command line to produce a .xml form a .txt that contains an RFC or Internet-Draft?
id2xml draft-whatever.txt

id2xml --help shows a lot of options

---

From: Sandy Ginoza <[email-masked]>
Date: Mon, 15 May 2017 13:38:42 -0700

Hi all,

Please note that I’ve added Megan Ferguson to the thread, as she is the lead for testing v3-related tools for the RPC.  

We haven’t been able to successfully run id2xml yet.  Matt installed the updated version for us this morning to see if that helped, but we’re still getting the same errors and we’re unable to view output of "id2xml —help"

Are we missing something?  Some of the error messages are below.

Thanks!
Sandy

> > id2xml draft-ietf-rtgwg-yang-key-chain-24test.txt
> Traceback (most recent call last):
>  File "/usr/bin/id2xml", line 9, in <module>
>    load_entry_point('id2xml==0.9.1', 'console_scripts', 'id2xml')()
>  File "/usr/lib/python3.4/site-packages/pkg_resources/__init__.py", line 558, in load_entry_point
>    return get_distribution(dist).load_entry_point(group, name)
>  File "/usr/lib/python3.4/site-packages/pkg_resources/__init__.py", line 2682, in load_entry_point
>    return ep.load()
>  File "/usr/lib/python3.4/site-packages/pkg_resources/__init__.py", line 2355, in load
>    return self.resolve()
>  File "/usr/lib/python3.4/site-packages/pkg_resources/__init__.py", line 2361, in resolve
>    module = __import__(self.module_name, fromlist=['__name__'], level=0)
>  File "/usr/lib/python3.4/site-packages/id2xml/parser.py", line 43, in <module>
>    from __init__ import __version__
> ImportError: No module named ‘__init__'

> id2xml draft-ietf-sidr-rpki-oob-setup-09.txt > draft-ietf-sidr-rpki-oob-setup-test.xml
Traceback (most recent call last):
  File "/usr/bin/id2xml", line 9, in <module>
    load_entry_point('id2xml==0.9.1', 'console_scripts', 'id2xml')()
  File "/usr/lib/python3.4/site-packages/pkg_resources/__init__.py", line 558, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python3.4/site-packages/pkg_resources/__init__.py", line 2682, in load_entry_point
    return ep.load()
  File "/usr/lib/python3.4/site-packages/pkg_resources/__init__.py", line 2355, in load
    return self.resolve()
  File "/usr/lib/python3.4/site-packages/pkg_resources/__init__.py", line 2361, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/lib/python3.4/site-packages/id2xml/parser.py", line 43, in <module>
    from __init__ import __version__
ImportError: No module named ‘__init__'

> id2xml draft-ietf-sidr-rpki-oob-setup-09.txt | draft-ietf-sidr-rpki-oob-setup-test.xml
If 'draft-ietf-sidr-rpki-oob-setup-test.xml' is not a typo you can use command-not-found to lookup the package that contains it, like this:
    cnf draft-ietf-sidr-rpki-oob-setup-test.xml
Traceback (most recent call last):
  File "/usr/bin/id2xml", line 9, in <module>
    load_entry_point('id2xml==0.9.2', 'console_scripts', 'id2xml')()
  File "/usr/lib/python3.4/site-packages/pkg_resources/__init__.py", line 558, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python3.4/site-packages/pkg_resources/__init__.py", line 2682, in load_entry_point
    return ep.load()
  File "/usr/lib/python3.4/site-packages/pkg_resources/__init__.py", line 2355, in load
    return self.resolve()
  File "/usr/lib/python3.4/site-packages/pkg_resources/__init__.py", line 2361, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/lib/python3.4/site-packages/id2xml/parser.py", line 44, in <module>
    from __init__ import __version__
ImportError: No module named ‘__init__'

> id2xml --help
Traceback (most recent call last):
  File "/usr/bin/id2xml", line 9, in <module>
    load_entry_point('id2xml==0.9.2', 'console_scripts', 'id2xml')()
  File "/usr/lib/python3.4/site-packages/pkg_resources/__init__.py", line 558, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python3.4/site-packages/pkg_resources/__init__.py", line 2682, in load_entry_point
    return ep.load()
  File "/usr/lib/python3.4/site-packages/pkg_resources/__init__.py", line 2355, in load
    return self.resolve()
  File "/usr/lib/python3.4/site-packages/pkg_resources/__init__.py", line 2361, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/lib/python3.4/site-packages/id2xml/parser.py", line 44, in <module>
    from __init__ import __version__
ImportError: No module named '__init__'

> On May 15, 2017, at 1:15 PM, Robert Sparks <[email-masked]> wrote:
> 
> 
>

---

From: Henrik Levkowetz <[email-masked]>
Date: Mon, 15 May 2017 23:02:19 +0200

Hi Sandy,

Responding off list.

Best,

	Henrik

---

From: Julian Reschke <[email-masked]>
Date: Tue, 16 May 2017 07:19:28 +0200

On 2017-05-15 12:59, Henrik Levkowetz wrote:
> ...

Looks promising.

Did a test run with rfc7230.txt, and re-transformed it with 
rfc2629.xslt. Warnings/errors:

> WARNING: @initials 'R. ': did you mean 'R.'? (at line 11)
> WARNING: @initials 'J. ': did you mean 'J.'? (at line 23)
> WARNING: front matter mentions RFC 2616 for which there is no reference element (at line 9)
> WARNING: front matter mentions RFC 2818 for which there is no reference element (at line 9)
> WARNING: @initials 'R. ': did you mean 'R.'? (at line 11)
> WARNING: @initials 'J. ': did you mean 'J.'? (at line 23)
> ERROR: no XSLT template for element 'list' (at line 1602)
> ERROR: no XSLT template for element 'list' (at line 1605)
> ERROR: no XSLT template for element 'list' (at line 1602)
> ERROR: no XSLT template for element 'list' (at line 1605)
> WARNING: excessive whitespace in street: 'San Jose, CA  95110' (at line 16)
> WARNING: excessive whitespace in street: 'Muenster, NW  48155' (at line 28)

The errors about "list" is because there's no style and no enclosing 
list (should be style='empty').

Best regards, Julian

PS: will look at the output in detail soonish...

---

From: Russ Housley <[email-masked]>
Date: Tue, 16 May 2017 12:05:53 -0400

Henrik:

I used draft-ietf-curdle-cms-eddsa-signatures-05.txt as the input, and I got a few surprises.  I used xml2rfc to generate a text file from the .xml that id2xml produced

First, the heading on the title page.

ORIGINAL

Internet-Draft                                                R. Housley
Intended status: Standards Track                          Vigil Security
Expires: 11 October 2017                                   11 April 2017

   Use of EdDSA Signatures in the Cryptographic Message Syntax (CMS)
            <draft-ietf-curdle-cms-eddsa-signatures-05.txt>

NEW

Internet-Draft                                                R. Housley
Internet-Draft                                            Vigil Security
Expires: October 13, 2017                                 April 11, 2017

   Use of EdDSA Signatures in the Cryptographic Message Syntax (CMS)
             draft-ietf-curdle-cms-eddsa-signatures-05.txt

Second, the top of each page.  This can probably be resolved by only using the Month and Year, but the result is a bit odd.

ORIGINAL

Internet-Draft                                             11 April 2017

NEW

Internet-Draft                     11                         April 2017

Third, the NOTE format.

ORIGINAL

      NOTE: Either id-sha512 or id-shake256 is used as part to the
      private key signing operation.  However, the private key signing
      operation does not take a message digest computed with one of
      these algorithms as an input.

NEW

   NOTE: Either id-sha512 or id-shake256 is used as part to the

      private key signing operation.  However, the private key signing
      operation does not take a message digest computed with one of
      these algorithms as an input.

Fourth, and indented quote.

ORIGINAL

   The EdDSA specification [EDDSA] includes the following warning.  It
   deserves highlighting, especially when signed-data is used without
   signed attributes and the content to be signed might be quite large:

      PureEdDSA requires two passes over the input.  Many existing APIs,
      protocols, and environments assume digital signature algorithms
      only need one pass over the input, and may have API or bandwidth
      concerns supporting anything else.

NEW

   The EdDSA specification [EDDSA] includes the following warning.  It
   deserves highlighting, especially when signed-data is used without
   signed attributes and the content to be signed might be quite large:

   PureEdDSA requires two passes over the input.  Many existing APIs,

      protocols, and environments assume digital signature algorithms
      only need one pass over the input, and may have API or bandwidth
      concerns supporting anything else.

Thanks for the good work.  I’m sure it will get better and better.

Russ

---

From: Henrik Levkowetz <[email-masked]>
Date: Wed, 17 May 2017 01:38:49 +0200

Hi Russ,

On 2017-05-16 18:05, Russ Housley wrote:
> Henrik:
> 
> I used draft-ietf-curdle-cms-eddsa-signatures-05.txt as the input,
> and I got a few surprises. I used xml2rfc to generate a text file
> from the .xml that id2xml produced
> 

Yes.  xml2rfc wants to put either a stream identifier, a WG name, or the
traditional 'Network Working Group' as the first word of the lefhand
column.  It took 'Internet-Draft' to be the WG name; I've now told it
to not do that (which results in 'Network Working Group' at the top
left, and also in recognising the intended status).

The date formatting is also as given by xml2rfc; we could change xml2rfc
to produce the format you used if people would be happier with that for
drafts.  (It's more natural and readable for me, but I don't know if
it's better in an English-language style sense.)

The angle brackets around the filename are also given by xml2rfc.

> Second, the top of each page. This can probably be resolved by only
> using the Month and Year, but the result is a bit odd.
> 
> ORIGINAL
> 

Ugh.  This is a bug.  It should not be using '11' as short title.
Now fixed in upcoming 0.9.4

>       these algorithms as an input.

Yes.  This is a recurring flaw.  I'll see what I can do with it (but
it's not as easy as it might seem, as I have to juggle list styles
of nested lists to produce evenly indented paragraphs, and in this
case I think the heuristics took the 'NOTE:' part as the start of a
hanging list.

>       concerns supporting anything else.

Yes.  Same flaw as the third.  I'll give this some attention.

> Thanks for the good work.  I’m sure it will get better and better.

Thank you for the feedback!

	Henrik

---

From: Henrik Levkowetz <[email-masked]>
Date: Fri, 19 May 2017 01:05:26 +0200

Hi Julian,

All of the issues below which are due to the xml generation should be
fixed in 0.9.3.  I'm getting closer to releasing a 1.0.0-rc1; there are
a lot of improvements since 0.9.3, so you may want to wait with looking
at the generated xml in detail until that one is out.

Best regards,

	Henrikma