Subject: Re: [xml2rfc-dev] xml2rfc GUI alpha

From: Henrik Levkowetz <[email-masked]>
Date: Mon, 15 Aug 2011 14:39:02 +0200

Hi Mike,

On 2011-08-11 01:48 Mike Biglan said:

> 
> USAGE
> 
> Documents are added to the queue by clicking 'Add File(s)' or going
> to File > Add File(s) in the menu. This will populate the list on the
> left.

Having tried this out, I'm not sure that
 - The idea of a list and queue of sources is helpful, as compared with
   handling a single document source at a time
 - The layout of the source and target input elements are logical.  The
   output directory field in particular is situated before the input
   source field, which seems backward to me.

For the first one, one idea might be to look at other GUIs which
does similar processing of source to target files; one group of such
which comes to mind is expanders/viewers of compressed files.  There
is often no document window at all until one has chosen a source file,
and batch/queued processing is a special option if available at all.
Settings are handled by a separate window, rather than as part of
the main window (which currently has the 'Output directory' setting).

My suggestions would be:

  * Get rid of the queue and the queue window, but show the
    path and name of the source file above the tab pane.
  * Get rid of the output directory field in the main window.
  * Have only one way of choosing the source file, i.e., from the
    file menu
  * By default, the output files go in the same directory as the
    source file.  Sending output somewhere else can be handled by
    in the Preferences window.
  * Having one area for conversion settings right below the menu
    is OK, I guess, even if I would rather have had it as option
    menu toggles.
  * One tab pane should be enough, rather than having two as is
    done now, if one of the tabs show the source with errors indicated
    inline.  Then you could 
	1) show the source in the xml tab when a source has been
	   chosen,
	2) show processing progress in the output pane when 'convert'
	   has been pressed
 	3) show output in output tabs (those could be created as
	   output becomes available, rather than always being
	   present, *if that's easy*) when available, or go back
	   to an editable source pane for error fixes.
    If error indication is only available in the output pane,
    then I guess we need to keep the two tab panes.

> To preview any of these documents, double click the name or
> icon of the document and its converted format will be displayed in
> the main view, unless there were any errors in conversion, in which
> case the errors are printed to the text output in the bottom pane,
> and a highlighted line of XML is opened in the main view containing
> the error. For working documents, clicking the other tabs in the main
> view (HTML, Nroff...) will show you the preview for that format.

None of the tabs shows any content when I run the current SVN HEAD
under Debian Linux, using the procedure you sent on the 12th.  The
files in the designated output directory looks fine, on the other
hand.

> Clicking the 'Convert' button on the upper right corner of the screen
> will convert ALL of the documents in the document queue to the
> formats enabled under 'Formats', and output them to the directory
> specified under 'Output directory', using any additional settings
> specified under 'Settings'.
> 
> 
> APPLICATION DETAILS
> 
> Xml2rfc-gui bundles an internal snapshot of xml2rfc along with it;
> this means it does not invoke the *system* installed xml2rfc. The
> xml2rfc-gui package will operate exactly the same regardless of
> whether xml2rfc is installed or the specifics of the version.
> However, since python imports are resolved dynamically, I suppose it
> is technically possible if you were to remove the xml2rfc binaries
> from the xml2rfc-gui package that the xml2rfc import would then
> resolve to the system version, but this would be difficult to do
> without breaking the application and a fairly unusual use-case.

This makes some sense, although it may also surprise some people,
especially on Linux.  I don't think this is enough to change the
approach, though.

> The preferences window (found by going to 'Options' in the menu, then
> 'Preferences') allows you to set the Cache and Citation Library
> directories, along with change various appearance and behavior
> settings of the application.

Right.

> Any GUI elements which are able to invoke the worker thread (document
> queue, convert button) are disabled during processing, to keep the
> application thread-safe. I believe on all operating systems it just
> shows them as grayed-out.> 

Ok.

> CURRENT WORK IN PROGRESS
> 
> - Safely halting execution during batch processing
> - Creating About and Help menu pages
> - Version checking and updating. Let us know if you don't think this
>   is needed.

Version checking will be helpful; having people do the download and
re-install themselves will be acceptable.

> - Generating and displaying a summary report after running a batch
>   conversion, optionally saving to a log file or shown to the user.

See above; if we need a batch option and it shows output in an output
tab, and that output can be copied, I think it's enough.

> - The templates directory is bundled in the same directory as the
>   application.  It may be more appropriate to go in a hidden location,
>   if desired, which can be accomplished in the installer.

Ok.

> We appreciate any feedback you have about the application thus far --
> and talk with you soon,

I hope the feedback above is helpful; I expect other people on the
xml2rfc-dev list will also have feedback.

Best regards,

	Henrik

---

From: Mike Biglan <[email-masked]>
Date: Tue, 16 Aug 2011 22:29:56 -0700

Hi Henrik and All,

We've prepared major updates to the GUI application based on your suggestions Henrik. All these are committed to SVN. In addition, here are the OSX and Windows builds of this version (though not updates on the deb packages yet):
  - Mac: http://minornine.com/xml2rfc/xml2rfc-gui-0.6.0-osx.zip
  - Windows: http://minornine.com/xml2rfc/xml2rfc-gui-0.6.0-win32.zip

Specific updates to GUI App:
  
  - Batch queue removed
  - Source document is added from File > Open Source
  - Various new options in preferences and menus
  - Stderr/stdout views collapsed into one view
  - Output tabs are created/destroyed as needed
  - Multiple lines will now be highlighted for errors
  - Experimental support for clicking error 'links' in the console to jump to an error
  - Some persistence added to application temporary settings
  - Application takes an input file as an argument now
  - Version bumped to 0.6.0

> Having tried this out, I'm not sure that the idea of a list and queue of sources is helpful, as compared with handling a single document source at a time

Our old code could do the batch processing mode and that is still in the SVN history. Perhaps this would be more appropriate as a separate branch in the repository. We spent a fair amount of time putting that together and found it very helpful in our (batch) testing. If anyone else on this list has interest in this, please let us know.

One other note. At this point we opted not to display errors 'inline' because that seems to conflict with being able to edit the XML directly in the application. Please let us know if you'd like us to change this.

Hope this is helpful -- and appreciate all the feedback, 

Mike

---

From: Henrik Levkowetz <[email-masked]>
Date: Wed, 17 Aug 2011 13:03:24 +0200

Hi Mike,

>   - Windows: http://minornine.com/xml2rfc/xml2rfc-gui-0.6.0-win32.zip

That's fast turnaround.  Thank you.

>   - Version bumped to 0.6.0

The new version looks good to me so far (some comments below, though).
Nice to see the Mac version, too :-)

Trying this one out, there's one thing I'd like differently, related to
the references library.  I'm used to being able to give a set of paths
for this, which makes it possible for me to have all regular references
in the bibxml*/ directories, while document-specific references are placed
in a local ./references/ directory.  This begs two questions:

Does the GUI version honor any environment variable which could be set
to a list of directories (in addition to the preferences setting)?

Would it be possible to make the preferences setting editable as a
text box (in addition to, or as a complement to the current file find
dialog mechanism) and make it accept a list of places to look for a
reference file, rather than just one?

>> Having tried this out, I'm not sure that the idea of a list and
>> queue of sources is helpful, as compared with handling a single
>> document source at a time
> 
> 
> Our old code could do the batch processing mode and that is still in
> the SVN history. Perhaps this would be more appropriate as a separate
> branch in the repository. We spent a fair amount of time putting that
> together and found it very helpful in our (batch) testing.

Right, that makes sense and was worthwhile.

> If anyone
> else on this list has interest in this, please let us know.

If it's fairly easy to accomplish, maybe you could keep the functionality
even if the visible UI support is different, by permitting the selection
of multiple files in File Open.  You can't see the individual output files
by selecting a specific entry in the queue anymore, but the output window
could still reflect how things have gone.  Up to you, really.

> One other note. At this point we opted not to display errors 'inline'
> because that seems to conflict with being able to edit the XML
> directly in the application.

Ah.  Yes, that makes sense.

> Please let us know if you'd like us to
> change this.

Works for me.

> Hope this is helpful -- and appreciate all the feedback,

Thank you, Mike.

In addition to the feedback on the new version given above, I'll try to
exercise it some more and give additional feedback if needed.

Best regards,

	Henrik

---

From: Henrik Levkowetz <[email-masked]>
Date: Wed, 17 Aug 2011 13:15:28 +0200

Hi Mike, Josh,

Some more feedback on the references path setting in the GUI app (on OS X).
At first, I could not make that work for me.  Here's the reason and
background:

The online references library is organized as 5 directories under a
common root:

	<<references path>>/
		bibxml/			# contains RFC references
		bibxml2/		# contains ANSI and other Std refs
		bibxml3/		# contains draft references
		bibxml4/		# contains W3C references
		bibxml5/		# contains 3GPP references

When I set the references path in the GUI app to the root of this
subtree, it didn't find my references at all; while setting it to
the specific bibxml directory which contained one of the references
I used resulted in the relevant file being found and read.  I think
we need to support recursive search for reference files so that people
can use the layout of the online library show above locally, too.

Best regards,

	Henrik

---

From: Josh Bothun <[email-masked]>
Date: Thu, 18 Aug 2011 11:05:20 -0700

On Aug 17, 2011, at 4:03 AM, Henrik Levkowetz wrote:

> ...

Yes, right now the GUI application will look at the environment variable $XML_LIBRARY, defaulting
to /var/lib/xml2rfc if the variable is not set or the path is invalid.  (Is there a more appropriate
default here?).  Unless the user explicitly sets a new directory, in which case the application will use that.
Pressing 'reset' on that setting or 'restore to defaults' will reset the directory to $XML_LIBRARY.

$XML_LIBRARY is treated as a single path, however both the CLI and GUI could be easily modified to 
parse it as a list of delimited directories (i believe a colon delimiter is the UNIX convention but comma, semicolon would work)

> reference file, rather than just one?

Sure, we can make this into a list you can add/subtract from, using either
direct text input or by clicking a browse/add button to get a file dialog.  The libraries
would then be consulted in list order, so perhaps a move up/down function would 
be appropriate.

-Josh

---

From: Josh Bothun <[email-masked]>
Date: Thu, 18 Aug 2011 11:14:24 -0700

Hi Henrik,

> can use the layout of the online library show above locally, too.

Agreed, I wasn't sure how to handle that.  We'll go ahead and implement recursive search for a referenced file.

One question is how to handle a reference that explicitly gives one or more directories, such as "bibxml/reference.RFC.XXXX.xml" -- should this be treated as a full path and left alone, or searched recursively as well?  And if so, should only the basename be searched recursively, or the full relative path?

-Josh

---

From: Henrik Levkowetz <[email-masked]>
Date: Thu, 18 Aug 2011 21:46:34 +0200

Hi Josh,

On 2011-08-18 20:05 Josh Bothun said the following:
>>> - Source document is added from File>  Open Source
> Yes, right now the GUI application will look at the environment
> variable $XML_LIBRARY, defaulting to /var/lib/xml2rfc if the variable
> is not set or the path is invalid. (Is there a more appropriate
> default here?).

On Linux, /usr/share/xml2rfc if installed by a system installer, otherwise
/usr/local/share/xml2rfc.

I'd prefer the environment variable to reflect the program -- 'XML_LIBRARY'
is much too generic.  Please make it 'XML2RFC_LIBRARY'.

> Unless the user explicitly sets a new directory, in
> which case the application will use that. Pressing 'reset' on that
> setting or 'restore to defaults' will reset the directory to
> $XML_LIBRARY.

Ok.

> $XML_LIBRARY is treated as a single path, however both the CLI and
> GUI could be easily modified to parse it as a list of delimited
> directories (i believe a colon delimiter is the UNIX convention but
> comma, semicolon would work)

Please use colon under Un*x, semicolon under Windows.

> would then be consulted in list order, so perhaps a move up/down function would

Up/down seems quite fancy; text box editing to re-order should suffice.

Best regards,

	Henrik

---

From: Henrik Levkowetz <[email-masked]>
Date: Thu, 18 Aug 2011 21:51:51 +0200

Hi Josh,

On 2011-08-18 20:14 Josh Bothun said the following:
> Agreed, I wasn't sure how to handle that. We'll go ahead and
> implement recursive search for a referenced file.

Excellent.

> One question is how to handle a reference that explicitly gives one
> or more directories, such as "bibxml/reference.RFC.XXXX.xml" --
> should this be treated as a full path and left alone, or searched
> recursively as well? And if so, should only the basename be searched
> recursively, or the full relative path?

If the location contains a path component, there's no recursive search,
I think.  Otherwise it becomes too magic.  If the path is relative it
refers to the references_path root (or roots, if a list of paths is
specified through environment variable or otherwise); if the path is
absolute it is used as-is.  If it's absolute and not found, the warning
could suggest that a reference with no path component, which will be
searched for in on all defined library paths, would be less prone to
break.

Best,

	Henrik

---

From: Tony Hansen <[email-masked]>
Date: Thu, 18 Aug 2011 16:57:59 -0400

> parse it as a list of delimited directories (i believe a colon delimiter is the UNIX convention but comma, semicolon would work)

The problems I've been having have also been due to the references library.

(For purposes of exposition, I'm going to refer to the existing tcl 
version as xml2rfc1, and the new one as xml2rfc2.)

This is one area where we absolutely must be compatible with xml2rfc1 to 
begin with.

The value of $XML_LIBRARY needs to be treated as a path. For unix/mac 
systems, it needs to use ":" as the delimiter. For Windows systems, it 
needs to use ";" as the delimiter. That's it -- nothing else. A trailing 
delimiter, leading delimiter or two delimiters next to each other 
indicates the "current directory" is to be looked in at that point. I 
suggest that "current directory" should mean the "directory of the 
document being converted".

You say that it defaults to /var/lib/xml2rfc, whereas it appears to 
instead default to a path of /var/lib/xml2rfc PLUS the directory of the 
document: For example, here is a warning message I got when my document 
was in the My Documents/ietf directory.

     WARNING: Include file not found: C:/Documents and 
Settings/Administrator/My
     Documents/ietf\reference.RFC.2119.xml

If I put the reference.RFC.2119.xml file in *either* place, xml2rfc2 
finds the reference file.

BTW, these WARNING messages are not clickable, whereas the subsequent 
but useless IDREF attribute message *is* clickable.

xml2rfc1 does not do any sort of depth-first search -- I'm not convinced 
we should do that in xml2rfc2 to begin with.

For what it's worth, I don't like that it's writing the converted files 
in the same place where it's looking for the references. It would be 
better to write the converted files in the same directory as the source.

> would then be consulted in list order, so perhaps a move up/down function would

After you make the preferences a path, I suggest you use 
"/var/lib/xml2rfc:" as the default setting. (Yes, that's a trailing 
delimiter there.)

Some other suggestions:

Until there's a "save file" option, the XML page should not be modifiable.

When the file *has* been modified, there should be an indication (like a 
"*" by "XML" in the tab) to visibly indicate that it has been modified. 
Saving the file of course would clear that "*".

> One question is how to handle a reference that explicitly gives one or more directories, such as "bibxml/reference.RFC.XXXX.xml" -- should this be treated as a full path and left alone, or searched…

I think xml2rfc1 ignores the partial path. But it also does *not* do a 
recursive search -- instead it expects each of the bibxml* directories 
to be listed separately.

On 8/18/2011 2:26 PM, Josh Bothun wrote:
> Hi Henrik,
> On Aug 17, 2011, at 4:32 AM, Henrik Levkowetz wrote:
>> Hi Mike, Josh,
>>
>> Possibly there's some information missing related to the comments/questions
>> quoted below.
>>
>> There is an online repository of references available in 5 subdirectories
>> of a specific URL (as I mentioned in a previous email without going into
>> the location of it, as I didn't realize that this might not be known).
>> Well, the URL is this, with the listed subdirectories:
>>
>>    http://xml.resource.org/public/rfc/
>> So given that there is one canonical URL for the online reference library,
>> it should be possible to try to pull also a reference not specified by
>> a http:// path from the internet and cache it locally, which is what I think
>> Tony was hoping for.
> We can indeed support this, but I have a couple of questions about the specifics of the behavior.  Namely,
>
> 1. Should the application look for the reference in online library before, after, or instead of looking in local library? ($XML_LIBRARY or otherwise explicitly set in the GUI)

How about treating "network" as another option in the list of 
directories to search?

Note: xml2rfc1 always searches the network *last*.

> 2. Is the behavior for handling<!ENTITY>  and<?rfc include>  instructions identical, or are there differing rules for each?

 From what I can tell in xml2rfc1, <!ENTITY SYSTEM>, <!ENTITY PUBLIC> 
and <?rfc include> are treated identically as far as searching for the file.

>> If you put in support for doing this, it would probably be a good idea to
>> make the online library url configurable in the Preferences, too.
> Certainly, it will likely be grouped under the 'References' tab.

Hope these comments are useful.

     Tony

---

From: Henrik Levkowetz <[email-masked]>
Date: Thu, 18 Aug 2011 23:19:06 +0200

Hi Tony,

A few points of discussion, inline.  Some text snipped:

On 2011-08-18 22:57 Tony Hansen said the following:

...

> The value of $XML_LIBRARY needs to be treated as a path.

And if xml2rfc1 understood $XML_LIBRARY, we need to continue to do so.
I'd still prefer that the docs suggested that people use $XML2RFC_LIBRARY
going forward, I still think $XML_LIBRARY is much to generic.

> For unix/mac

Earlier I suggested that the appropriate default be /usr/share/xml2rfc,
rather than /var/lib/xml2rfc -- this is the correct place according to
the Filesystem Hierarchy Standard (http://www.pathname.com/fhs/) so we
should try to make it so if at all possible.

> we should do that in xml2rfc2 to begin with.

Hmm.  I'd prefer to have that -- I've been annoyed by the lack of it
before -- but if you feel strongly about this then let's skip it for
now.

> better to write the converted files in the same directory as the source.

?!?  I haven't seen this behaviour -- my recent tests saw it put the
converted file with the source.

...

> "/var/lib/xml2rfc:" as the default setting. (Yes, that's a trailing

See above -- I'd suggest '/usr/share/xml2rfc:' in that case.

> Until there's a "save file" option, the XML page should not be modifiable.

Yes, I thought about the 'save' option, too.  Agreed.

> Saving the file of course would clear that "*".

Good point.

Best regards,

	Henrik

>    From what I can tell in xml2rfc1,<!ENTITY SYSTEM>,<!ENTITY PUBLIC>
> and<?rfc include>  are treated identically as far as searching for the file.
>
> _______________________________________________
> xml2rfc-dev mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/xml2rfc-dev
>

---

From: Mike Biglan <[email-masked]>
Date: Wed, 24 Aug 2011 10:29:53 -0700

Hi All,

Some updates from our modifications from last week below for both the command-line tool and the GUI applications.

Command Line Tool - xml2rfc version 2.2:
=======

1) IREF rendering improved

2) External reference resolution has been overhauled and is significantly more robust. Recent comments from Tony/Henrik have been taken into consideration. Let us know if you'd like us to provide a separate email with pseudocode description of the resolution algorithm -- it can be found in a docstring of cli/xml2rfc/parser.py at line 100. 

In a nutshell:
- Absolute paths are treated as-is.
- Non-absolute paths are searched in the following order: (a) Local locations (b) Network location, automatically caching (c) Source directory
	Where "Local locations" is a list of directories in $XML_LIBRARY, defaulting to '/usr/share/xml2rfc:', and "Network Location" is "http://xml.resource.org/public/rfc/", both "Local locations" and "Network Location" are modifiable in the GUI, but have the same defaults as the CLI.
- Filenames with no path component are NOT searched for recursively, but I am ready to implement this if needed.
	
3) Due to the complexity of how references are resolved, they get their own exception layer. The application will now properly halt on unresolved references, and error messages should be much clearer now.  If --verbose was used, xml2rfc will tell you all the locations it looked for a reference before failing.  I believe this should greatly improve our ability to troubleshoot some of the problems through community testing, and cross-platform testing.

GUI - xml2rfc-gui version 0.7.1:
=======

1) Editing/Saving of XML source file implemented, as long as file is not read-only.  A modified document will contain an asterisk (*) next to its title in the editor

2) Configurable list of local libraries and network library in Preferences > References.

3) Brought up to speed with new reference resolution / errors in core library

4) Minor cosmetic tweaks

Here are new permalinks to the built application:

Windows: http://minornine.com/xml2rfc/xml2rfc-gui-0.7.1-win32.zip
OS X: http://minornine.com/xml2rfc/xml2rfc-gui-0.7.1-osx.zip
Debian: http://minornine.com/xml2rfc/xml2rfc-gui-0.7.1-debian.zip

DEBIAN PACKAGE NOTE: It is installed as "dpkg -i <file>.deb".   If there are any dependency issues, running sudo apt-get -f install will resolve them.  In a real repository, such as installing xml2rfc-gui from apt-get, these dependencies would be resolved automatically.  Although the file is currently labeled python-xml2rfc-gui.deb, the application is invoked as "xml2rfc-gui"

We'll go back through the latest email thread and directly respond to any unresolved sub-thread.

Thanks for the input and talk with you all soon,

Mike

---

From: Tony Hansen <[email-masked]>
Date: Wed, 24 Aug 2011 15:07:09 -0400

Thanks Mike.

more below

         Tony

> 	Where "Local locations" is a list of directories in $XML_LIBRARY, defaulting to '/usr/share/xml2rfc:', and "Network Location" is "http://xml.resource.org/public/rfc/", both "Local locations" and "…

I like how this is shaping up. Is there a command line option that 
controls whether the network is consulted?

> 3) Due to the complexity of how references are resolved, they get their own exception layer. The application will now properly halt on unresolved references, and error messages should be much clear…

I like the --verbose option.

> 2) Configurable list of local libraries and network library in Preferences>  References.
>

These all sound good.

I like what I see!! A definite improvement.

Some feedback.

1) when run from the command line, it should put itself in the background.

2) we need a nice logo for it. :-)

3) I made some changes to the XML, clicked on Convert, and it silently 
threw the changes away. Some possible options:

     a)    automatically save the changes

     b)    prompt whether to save the changes or discard the changes 
before converting

I like (b) a bit better.

4) Where it displays resolving a reference that was retrieved over the 
net, I see:

     Resolving include... 
\var\cache\xml2rfc\_HTTP_CACHE\reference.RFC.2049.xml

I would have expected to see something like

     Resolving include... 
http://xml.resource.org/public/rfc/bibxml/reference.RFC.2049.xml
     Caching in \var\cache\xml2rfc\_HTTP_CACHE\reference.RFC.2049.xml

5) Errors resolving includes don't have a pointer back into the document.

6) I changed a </references> to </referencesx> and got the expected 
error message:

     Line 682: expected '>', line 682, column 7

However, the actual error was on line 680 instead of 682.

Keep up the good work!

     Tony

> DEBIAN PACKAGE NOTE: It is installed as "dpkg -i<file>.deb".   If there are any dependency issues, running sudo apt-get -f install will resolve them.  In a real repository, such as installing xml2rfc-gui from apt-get, these dependencies would be resolved automatically.  Although the file is currently labeled python-xml2rfc-gui.deb, the application is invoked as "xml2rfc-gui"
>
>