Subject: [netmod] Comments on draft-lhotka-netmod-yang-markup-00

From: Robert Wilton <[email-masked]>
Date: Fri, 17 Mar 2017 12:30:04 +0000

Hi Lada,

I've had a quick scan of your YANG markup extension draft and I have a 
few comments:

Allowing description, and similar descriptive statements, to use 
something other than text seems like it could be useful in some cases.

I'm not sure that allowing the statements to use any text-like media 
type is a good idea, this could increase the burden on tool makers if 
each module author chooses their own preferred format.

Instead, I think that it might be better to restrict it to a very small 
set of media types, that could be extended in future.  I would think 
that initially just allowing plain text and one particular flavour of 
markdown would be a reasonable starting point.

I think that the only formats that should be allowed are those that are 
still readily readable as plain text, so that tools that don't want to 
parse the formatted text can still sensibly display the descriptive 
statements.  I.e. I don't think that it would be helpful to allow things 
like text/xml since it isn't easy to read.

Allowing this extension on particular descriptive statements may also be 
helpful.  It seems plausible that the vast majority of these statements 
in a module might just be written in plain text with just a few of them 
using more advanced formatting like markdown.

Finally, I have a concern that if more structured formatting in the 
comments is used then would that encourage model writers to produce more 
verbose comments, and if so that might possibly reduce the readability 
of the modules.  Although, I guess ultimately one has to trust the model 
writers to do the right thing.

Thanks,
Rob

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 17 Mar 2017 13:55:03 +0100

Hi Rob,

thank you for reading the draft.

> On 17 Mar 2017, at 13:30, Robert Wilton <[email-masked]> wrote:
> 
> Hi Lada,
> 
> I've had a quick scan of your YANG markup extension draft and I have a few comments:
> 
> Allowing description, and similar descriptive statements, to use something other than text seems like it could be useful in some cases.
> 
> I'm not sure that allowing the statements to use any text-like media type is a good idea, this could increase the burden on tool makers if each module author chooses their own preferred format.
> 
> Instead, I think that it might be better to restrict it to a very small set of media types, that could be extended in future.  I would think that initially just allowing plain text and one particular flavour of markdown would be a reasonable starting point.

I agree although I am not sure that we can easily find an agreement on the particular flavour. So maybe we can leave it open and let the "market" decide.

> 
> I think that the only formats that should be allowed are those that are still readily readable as plain text, so that tools that don't want to parse the formatted text can still sensibly display the descriptive statements.  I.e. I don't think that it would be helpful to allow things like text/xml since it isn't easy to read.

Sure, and the draft says:

   It is RECOMMENDED to use only media types representing "lightweight"
   markup that is easy to read even in the unprocessed source form, such
   as "text/markdown".

> 
> Allowing this extension on particular descriptive statements may also be helpful.  It seems plausible that the vast majority of these statements in a module might just be written in plain text with just a few of them using more advanced formatting like markdown.

Yes, I was thinking about this. On the other hand, plain text is typically compatible with any markup format, so this would probably be useful only if somebody wanted to use multiple different markup formats in the same module, which sounds crazy. But we can discuss it.

> 
> Finally, I have a concern that if more structured formatting in the comments is used then would that encourage model writers to produce more verbose comments, and if so that might possibly reduce the readability of the modules.  Although, I guess ultimately one has to trust the model writers to do the right thing.

Well, this draft doesn't permit anything above what module writers can do already now - the descriptions etc. have to be valid YANG strings as before. The only new thing is a piece of metadata that may be helpful for some tools but that can also be safely ignored.

Thanks, Lada

--
Ladislav Lhotka, CZ.NIC Labs
PGP Key ID: 0xB8F92B08A9F76C67

---

From: William Lupton <[email-masked]>
Date: Fri, 17 Mar 2017 13:13:24 +0000

Lada, Rob, all,

I would support use of markdown, with the principle that descriptions (etc) should remain readable as plain text. Indeed many of the published NETMOD YANG modules have descriptions that look as though they would already render quite well if regarded as markdown (blank lines as paragraph separators, bulleted and numbered lists etc).

Note that GitHub have recently published https://githubengineering.com/a-formal-spec-for-github-markdown, which might be worth looking at or even adopting? From a quick perusal, it doesn’t seem to include any GH specifics (e.g for referencing issues or pull requests), which actually surprised me a little.

Maybe this goes too far, but I would also consider conventions allowing validation of references to enums, bits, nodes etc within descriptions. Obviously this is potentially a slippery slope and potentially (if additional markup is needed) a threat to readability, but also a potential gain (can be validated and so increases quality, can become hyperlinks when rendered, etc).

William

PS, We support this sort of thing in TR-069 data model description strings. Showing our age, the markup is mediawiki-like. For example:
* If the ACS sets the value of this parameter to {{enum|Requested}}, the CPE MUST initiate the corresponding diagnostic test. When writing, the only allowed values are {{enum|Requested}} and {{enum|Canceled}}…
* Identifier of the class of product for which the serial number applies.  That is, for a given manufacturer, this  parameter is used to identify the product or class of product over which the {{param|SerialNumber}} parameter is unique.

> On 17 Mar 2017, at 12:55, Ladislav Lhotka <[email-masked]> wrote:
> 
> Hi Rob,
>

---

From: Robert Wilton <[email-masked]>
Date: Fri, 17 Mar 2017 14:11:47 +0000

> I agree although I am not sure that we can easily find an agreement on the particular flavour. So maybe we can leave it open and let the "market" decide.
I think that would be a mistake.  How would device/tools vendors know 
which versions to spend time implementing?

>     markup that is easy to read even in the unprocessed source form, such
I was proposing REQUIRED instead of RECOMMENDED.

> Yes, I was thinking about this. On the other hand, plain text is typically compatible with any markup format, so this would probably be useful only if somebody wanted to use multiple different mark…
I was only thinking of a mix of some plain text descriptions and some 
using markup.

Tooling may choose to format raw text differently from markup text (e.g. 
converting lines into a paragraph).  Also a markup processor would be 
extra work, and may not warn or error on the structure of a plain text 
description that doesn't conform to the markup rules.

Thanks,
Rob

> .

---

From: Robert Wilton <[email-masked]>
Date: Fri, 17 Mar 2017 14:16:09 +0000

Hi William,

> Note that GitHub have recently published https://githubengineering.com/a-formal-spec-for-github-markdown, which might be worth looking at or even adopting? From a quick perusal, it doesn’t seem to…
The spec that this is based on seems to be very long though! E.g. 
http://spec.commonmark.org/0.27/, 115 pages!

> Maybe this goes too far, but I would also consider conventions allowing validation of references to enums, bits, nodes etc within descriptions. Obviously this is potentially a slippery slope and po…
A reasonable idea.

Thanks,
Rob

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 17 Mar 2017 15:24:11 +0100

> On 17 Mar 2017, at 14:13, William Lupton <[email-masked]> wrote:
> 
> Lada, Rob, all,
> 

Right, actually my itch that I am trying to scratch with this proposal is the use of descriptions from existing modules in HTML, e.g. in tooltips, and parsing lists is tricky because module authors use different whitespace conventions.

> Note that GitHub have recently published https://githubengineering.com/a-formal-spec-for-github-markdown, which might be worth looking at or even adopting? From a quick perusal, it doesn’t seem to…

May be there are two levels: one is YANG the language where I would prefer not to prescribe a particular markup format, and another is 6087bis that may do it for IETF documents.

> Maybe this goes too far, but I would also consider conventions allowing validation of references to enums, bits, nodes etc within descriptions. Obviously this is potentially a slippery slope and po…

In fact, it would be quite useful to be able to include such references in custom error messages. Schematron allows for it, and this is why with DSDL validation messages generated by Schematron are often more helpful than those coming from RELAX NG.

Lada

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 17 Mar 2017 16:08:42 +0100

> On 17 Mar 2017, at 15:11, Robert Wilton <[email-masked]> wrote:
> 
> 
> 
> I think that would be a mistake.  How would device/tools vendors know which versions to spend time implementing?

I would agree to have a fixed choice in 6087bis, but there may be niche communities or special cases that need something else, so YANG IMO shouldn't preclude it a priori.

> I was proposing REQUIRED instead of RECOMMENDED. 

I usually write my modules in the YIN format with a very restricted set of HTML tags that are used in the text arguments:

https://gitlab.labs.nic.cz/labs/yang-tools/wikis/editing_yang#yin-schema-extensions

It would be useful to indicate "text/html" media type, and a YIN->YANG convertor could translate it e.g. to "text/markdown".

> I was only thinking of a mix of some plain text descriptions and some using markup.
> 
> Tooling may choose to format raw text differently from markup text (e.g. converting lines into a paragraph).  Also a markup processor would be extra work, and may not warn or error on the structure of a plain text description that doesn't conform to the markup rules.

Actually, the design philosophy behind markdown is that there is no formal concept of validity and so processors should never complain.

On the other hand, it is conceivable that some descriptions may use a more specialized format. For example, we could use a media type (variant) for the top-level "contact" statements that we include in modules, and tools could then more easily extract this metadata.

So maybe we could really permit the "text-media-type" extension anywhere and apply some natural scoping rules.

Lada

---

From: Robert Wilton <[email-masked]>
Date: Fri, 17 Mar 2017 17:15:24 +0000

> I would agree to have a fixed choice in 6087bis, but there may be niche communities or special cases that need something else, so YANG IMO shouldn't preclude it a priori.
I think that this idea works quite well as a YANG extension, and that 
would be a better path than baking it into YANG at this time.

I'm still not convinced that allowing an arbitrary encoding is a good idea.

> https://gitlab.labs.nic.cz/labs/yang-tools/wikis/editing_yang#yin-schema-extensions
OK.  I had assumed that nobody was using YIN ;-)
> It would be useful to indicate "text/html" media type, and a YIN->YANG convertor could translate it e.g. to "text/markdown".
Isn't that just part of the YIN conversion to YANG. I.e. by the time the 
file is YANG it would be text/markdown.

> Actually, the design philosophy behind markdown is that there is no formal concept of validity and so processors should never complain.
But how does an implementation know what it needs to support so that it 
works with all descriptions?  I think that there would need to be a 
based specification to be useful.
> On the other hand, it is conceivable that some descriptions may use a more specialized format. For example, we could use a media type (variant) for the top-level "contact" statements that we includ…
I'm not sure ... this is just sounding like extra work/complexity, and 
YANG is already pretty complex!

Thanks,
Rob

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 20 Mar 2017 13:39:04 +0100

Robert Wilton <[email-masked]> writes:

>>>>> I'm not sure that allowing the statements to use any text-like
>>>>> media type is a good idea, this could increase the burden on tool
>>>>> makers if each module author chooses their own preferred format.
>>>>>

The burden on tool makers wouldn't be much less if authors use only
lightweight markups - for example, markdown and reStructuredText would
still need two parsers.

> OK.  I had assumed that nobody was using YIN ;-)

I do, because the nxml mode of Emacs is so great.

> file is YANG it would be text/markdown.

That's how I do it now, but I assume modules in YIN syntax could
possibly exist on their own. For example, if they are intended for HTML
rendering, or including in formats like xml2rfc, then YIN could make
sense.

A YIN -> YANG conversion tool would change "text/html" e.g. to
"text/markdown".

Of course, it can be argued that one could use markdown right away in YIN
descriptions but it it often handier to have everything in XML - validators
(such as Emacs) or XSL processors can then also effectively work with
contents of descriptions.

> YANG is already pretty complex!

In any case, tools can always leave the text as it is. I will include
this possibility in open issues. 

Lada