Subject: [xml2rfc-dev] xml2rfc v2 to v3 conversions, TOC and index entries

From: John C Klensin <[email-masked]>
Date: Sun, 24 Oct 2021 13:44:55 -0400

Hi.

The working version of rfc5321bis,
draft-ietf-emailcore-rfc5321bis-05, is still causing
difficulties with xml2rfc v3 and the conversation machinery.
While it complies correctly with the online converter at
https://xml2rfc.tools.ietf.org/ (which was used to create the
version posted a few minutes ago), an attempt to convert it to
v3 using
https://xml2rfc.tools.ietf.org/experimental.html#xml2rfcv2tov3
produces a message indicating that the document is empty.

This has been mentioned before, but the v2 processor is still
ignoring "toc='include'" attributes on fourth-level section
elements (e.g., sections with numbers like 4.1.1.1 do not appear
in the table of contents) a bug that was identified and reported
when RFC 5321 was published in October 2008.  In addition, that
v2 processor produces page numbers in the index for the text
version and nothing but the names of the index entries (i.e., no
target) for the PDF one.  Attempting to figure out what would
happen if the document were converted to v3 is what led to the
error noted above.

I don't think the EMAILCORE WG is likely to be finished with
that document in the next month or two, but I would oppose
actual publication of an RFC based on this document without a
proper and complete TOC and Index.  I hope these issues can be
sorted out before they become a problem for the RPC.

best,
   john

---

From: Julian Reschke <[email-masked]>
Date: Sun, 24 Oct 2021 19:57:20 +0200

> produces a message indicating that the document is empty.

I'll be happy to have a look if you can post the source file somewhere
(or mail it to me directly).

> when RFC 5321 was published in October 2008.  In addition, that

There's is no "toc" attribute in V2. (in V2, the maxdepth for the Toc
can be set using the "docdepth" processing instruction)

> ...

Best regards, Julian

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 24 Oct 2021 20:53:56 +0200

Hi John,

if you want to continue editing that document in XML form, you will minimize the pain by converting it from v2 to v3 as soon as possible and never looking back.
(There is also unrelated advice in this mail to stop using tools.ietf.org.)

xml2rfc --v2v3 oldxml.xml

➔ oldxml.v2v3.xml

This file is going to be your new source.

If you don’t have xml2rfc (which is quite easy to install), sending your file to somebody who has xml2rfc installed is the easiest way to get the update.  You only have to do this conversion once, because you will be editing the v3 xml after that.

> On 2021-10-24, at 19:44, John C Klensin <[email-masked]> wrote:
> 
> Hi.
> 
> https://xml2rfc.tools.ietf.org/

We are in a transition away from this site; I would expect pain when continuing to use it.

> (which was used to create the

This kind of pain.

> This has been mentioned before, but the v2 processor is still

I would strongly advise to stop using that processor.

Your RFC will be published as RFCXMLv3, with text and html/pdf renderings of that XML format alongside, and it will have to stay in the envelope of what is possible with these formats and tools.  You are much better off seeing early in development what that envelope is, than getting a potentially destructive v3 conversion only at publishing time.

> […]
> produces page numbers in the index for the text
> target) for the PDF one.  

Consider yourself lucky that you still get pages and page numbers in the TXT version as long as it is an Internet-Draft.  Since RFC 8650, there have been no page numbers in the TOC for final RFCs.

> proper and complete TOC and Index.  

You will not get a "proper and complete" TOC and Index.
(I think we do share a definition what a “proper and complete” TOC/Index is, but that is not what the RFCXML process that has been in force since RFC 8650 will give you.)

> I hope these issues can be

Maybe if the document is delayed enough to be published under RFCXMLv4 rules…

Grüße, Carsten

---

From: "John R. Levine" <[email-masked]>
Date: 24 Oct 2021 15:58:35 -0400

Can you send me the XML so I can take a look?

We now have a programmer working on xml2rfc who is pretty good at fixing 
bugs.

> sorted out before they become a problem for the RPC.

Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

---

From: John C Klensin <[email-masked]>
Date: Sun, 24 Oct 2021 16:07:05 -0400

Carsten,

Noted but please look at this from the standpoint of an
overworked author with little time to spend on the IETF and
personal priorities that put getting this standards track
document out rather than working on the technology.  This is a
long and complex document whose source still contains
annotations from the development of RFC 2821.  The source of
some of its elements was in nroff or manually converted from
plain text, and the source history in XML goes back to xml2rfc
v1, with RFC 5321 being the result of a v1->v2 conversion.    

I understand the difficulties associated with using the "tools"
version(s), but, having just checked, the official
recommendations still point to it.  Moreover, it is much easier
(for me, at least) to use the online/web version than a locally
compiled one.  I also note that
<https://www.rfc-editor.org/materials/FAQ-xml2rfcv3.html> points
to the tools version so, if the preferred policy is to
discourage its use, there is work to be done.   

Perhaps things have changed, but, while I tried to get the
current version of xml2rfc installed on my Windows machine some
months ago, I ran into enough problems that I gave up (it is,
IMO, irrelevant whether those problems were due to Windows, my
lack or skill and/or patience, other applications that it messed
up, or the code itself).  I do have a copy running on one of my
FreeBSD machines, but it is not convenient for me to use.  

The last time I tried the conversion exercise -- probably in the
IETF 110 timeframe-- it appeared to work but an attempt to
compile the result didn't, leaving a need for human translation
of semantics and then hand-editing a source document that,
including comments, annotations, and tracking information, is
just a hair short of 7000 lines long.  As I said, long and
complicated document. Based on tests run at the time on short
files written natively in v3, v3 was not supporting features
that the document needs [2] whether they could be faked by
manually rewriting the source or not.  I hope the conversion
machinery, and the v3 compiler itself, have improved
considerably since then, but don't really have time to run the
tests.

So, for better or worse, the pain-minimizing strategy for me and
my personal pain threshold is to continue with v2.  If the
public v2 processor goes away, I'll probably use my own copy,
figuring out whether making the Windows version work or moving
files to and from the FreeBSD system is less painful.  If I'm
told that I cannot submit v2 files to the RFC Editor any more,
I'll submit the ASCII, possibly manually edited to include some
or all of the fourth-level TOC entries, to the RFC Editor and
let the RPC cope (exactly the problem (for them) that I've been
trying to avoid). 

As to the TOC, I was not aware that RFC 8650 updated RFC 7322,
nor that there was community consensus that the various
renderings from authoritative v3 source were acceptable.
Indeed, I recall a few statements that, since the source was
authoritative, those renderings should be considered transitory.
In any event, if we need to have that discussion, it should
probably be moved to the main IETF discussion list and/or the
RFC Future one.

Finally, if the message you are trying to deliver is that, if I
encounter further problems with v2 and/or the tools environment,
I should not bother reporting them, just say so -- that would be
fairly easy and painless for me to do.

best,
   john

[1] Follow the "Internet Standards" tab from <www.ietf.org> to
"Internet-Drafts", click on it to get to
<https://www.ietf.org/standards/ids/> , and then note the
"xml2rfc" link under "Submit an Internet-Draft".

[2] That includes both the index (and I have been suggesting
since long before the v3 project started that, consistent with
the Style Guide, it should be producing references to section
numbers not page numbers) and control over TOC nesting but also
how some of the syntax productions were constructed.

--On Sunday, October 24, 2021 20:53 +0200 Carsten Bormann
<[email-masked]> wrote:

> Hi John,
> 
> if you want to continue editing that document in XML form, you
> will minimize the pain by converting it from v2 to v3 as soon
> as possible and never looking back. (There is also unrelated
> advice in this mail to stop using tools.ietf.org.)
> 
> If you don't have xml2rfc (which is quite easy to install),
> sending your file to somebody who has xml2rfc installed is the
> easiest way to get the update.  You only have to do this
> conversion once, because you will be editing the v3 xml after
> that.
> 
> Your RFC will be published as RFCXMLv3, with text and html/pdf
> renderings of that XML format alongside, and it will have to
> stay in the envelope of what is possible with these formats
> and tools.  You are much better off seeing early in
> development what that envelope is, than getting a potentially
> destructive v3 conversion only at publishing time.
> 
>> […]
>> version and nothing but the names of the index entries (i.e.,
>> no target) for the PDF one.  
> 
> Consider yourself lucky that you still get pages and page
> numbers in the TXT version as long as it is an Internet-Draft.
> Since RFC 8650, there have been no page numbers in the TOC for
> final RFCs.
> 
> (I think we do share a definition what a "proper and
> complete" TOC/Index is, but that is not what the RFCXML
> process that has been in force since RFC 8650 will give you.)
>

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 24 Oct 2021 23:12:06 +0200

On 2021-10-24, at 22:07, John C Klensin <[email-masked]> wrote:
> document out rather than working on the technology.  

Hi John,

you can imagine that having started one widely used I-D/RFC authoring tool, I do converse with people that fit this description, just now and then :-)

> This is a

Sure.  You are doing the equivalent of editing a document that has RFC 2022 codes to switch between different encodings in the middle of the document, and mixes EUC-JP with KOI-8R with abandon.
Please do convert to the equivalent of UTF-8…

> recommendations still point to it.  

By now, you should be aware how much you can buy for an “official recommendation” (I’m not even sure anyone more deeply embedded in the IETF gears would agree to this term).

> Moreover, it is much easier
> compiled one.  

The first time, yes.  The hundredth time, no.

> I also note that

Yes, there is.

> FreeBSD machines, but it is not convenient for me to use.  

We need to get better in solving problems of this kind.
Clearly, we need a help desk for authors so people don’t waste lots of time on dead ends.
(And, yes, author-tools.ietf.org has fallbacks.  
Just don’t use the older ones that are now partially mired in cobwebs.)

> considerably since then, but don't really have time to run the

The conversion was running very well at IETF110 time.
You need to spend time with someone who can diagnose the problems you were running into.
This is not a bicycle you can fix yourself, this is a car that occasionally needs a garage.

> my personal pain threshold is to continue with v2.  

… and that is exactly what I’m trying to save you from.

> If the

(You can; they’ll convert them to v3 as the first thing.)

> let the RPC cope (exactly the problem (for them) that I've been

That is indeed a sensible strategy (I know other people who still use nroff).
Just don’t be surprised when little of the thoughts that went into your version of the document survive the translation into RFCXMLv3.

> As to the TOC, I was not aware that RFC 8650 updated RFC 7322,

It is the first RFC that was published under v3 rules (which you can see from the fact that the TOC doesn’t have page numbers).

> renderings from authoritative v3 source were acceptable.

Certainly not, but it’s what we have.

> probably be moved to the main IETF discussion list and/or the

Any such discussion should be on RFC-interest, please.

xml2rfc just is a piece of software that tries to implement the rules (and fortunately, fixed some of them due to the willpower of Henrik, when he was still contributing a lot).

> fairly easy and painless for me to do.

Most people on xml2rfc-dev do know that there are more problems with xml2rfc.tools.ietf.org than we can fix (the latter number is not significantly different from zero, but sometimes it is possible).  Please do stop using it (except where the new tools aren’t in place, such as for some kinds of reference bibxml).

The part of the tools environment that is being maintained: Please do report problems, preferably in a way that we can help.

Problem: V2 is history.
The tribulations that authors that start from v2 documents have to go through are *not*.
But the answer will not be to make v2 better, but to improve v3 and some of the tools.

> "xml2rfc" link under "Submit an Internet-Draft”.

Yes.  This *is* being worked on, but occasionally reminding people that this doesn’t have to take a year is useful.

> how some of the syntax productions were constructed.

The v3 project has happened, and since late 2019 (RFC8650) we are now in the process of counting he casualties and plastering over the wounds.  Again, if what you are trying to do doesn’t fit in the v3 envelope, you want to know this now.  So you can:

(1) manage to do what none of us has managed to do very much: fix v3
(2) go ahead and create a useful document in the envelope possible by v3.

(Please don’t stop attempting (1), but please do recognize the timelines for that are sometimes longer than those for updating RFC821ter will be.)

(In case you wonder: I’m just the messenger here, and my point is that there are ways to waste a lot of time with the undesirable aspects of the situation, which I’m occasionally trying to help people out of.)

Grüße, Carsten

---

From: Julian Reschke <[email-masked]>
Date: Mon, 25 Oct 2021 08:30:11 +0200

First of all; see Carten's reply.

Other than that:

> FreeBSD machines, but it is not convenient for me to use.

I just happened to install xml2rfc on Windows last week. Thr process is:

- install Python3 (I use the Cygwin version)
- install pip
- "pip install xml2rfc" (this required a few dev packages for Cygwin
present: python, libxml2, libxslt)

> how some of the syntax productions were constructed.

1) The index issue has been fixed a few months ago
(<https://trac.ietf.org/trac/xml2rfc/ticket/418>).

2) ToC nesting depth can be controlled in both v2 and v3, but
differently (v3 uses attributes on the root element, where v2 used
processing instructions). Inclusion into the ToC can be conrolled in v3
using
<https://greenbytes.de/tech/webdav/rfc7991.html#element.section.attribute.toc>).
This works for me; if it does not work for you, it would be good to
report that issue.

3) Regarding the final point: details, please.

Best regards, Julian

---

From: "Martin Thomson" <[email-masked]>
Date: Mon, 25 Oct 2021 17:47:54 +1100

> present: python, libxml2, libxslt)

I would not recommend using Cygwin to anyone today.  If you need a *nix environment, you can install WSL [1].

However, you can install python for windows[2] and get a windows version.  From there you can open a command line prompt (win+R then type "cmd") and install xml2rfc with "pip install --user xml2rfc".

I had to go looking for the xml2rfc.exe file that this creates, which was at "C:\Users\<me>\AppData\Roaming\Python\Python36\Scripts" (I've an old Python version, yours might be different).  Adding that directory to your path (win then type "path" and you should see a suggestion for "Edit the system environment variables"), might help.  

Of course, you can just find xml2rfc.exe and drag a .xml file onto the .exe to get a .txt file if you just want to see how it works.

[1] https://docs.microsoft.com/en-us/windows/wsl/about
[2] https://www.python.org/getit/windows/

---

From: Julian Reschke <[email-masked]>
Date: Mon, 25 Oct 2021 09:23:25 +0200

> I would not recommend using Cygwin to anyone today.  If you need a *nix environment, you can install WSL [1].

I'm using that as well, but did not want to complicate things more :-).

Best regards, Julian