Subject: [netmod] yanglint and implemented versus imported YANG modules

From: William Lupton <[email-masked]>
Date: Mon, 6 Mar 2017 16:44:52 +0000

All,

This message arose from a [email-masked] <mailto:[email-masked]> “draft-ietf-pim-igmp-mld-yang-02.txt: YANG compilation isuse” (sic) thread <https://www.ietf.org/mail-archive/web/yang-multicast/current/threads.html#00232> initiated by Benoit.

I thought it would be useful for NETMOD to see the part of the discussion that relates to implemented versus imported YANG modules.
Benoit Claise reported this warning:
warn: Schema node "ietf-ip:ipv4" not found (/ietf-interfaces:interfaces/ietf-interfaces:interface[ietf-interfaces:name = current()]/ietf-ip:ipv4)
Radek Krejčí replied:
These warnings are printed because in yanglint, until explicitly stated, the imported modules (such as ietf-interfaces and ietf-ip), are supposed to be only imported, not implemented. The data nodes in imported schemas are not available, which is the reason of these warnings.
William Lupton (that’s me!) asked / commented:
Why are the complaints only about ip:ipv4 (etc) and not about if:interfaces (etc), which are also referenced in the must statements?
This makes it hard for an automated tool (such as Benoit’s) because it needs to know which other YANG files to process in addition to the “file of interest”.
Radek Krejčí replied:
According to RFC 7950, sec 5.6.6 (3rd paragraph) [ED: 5.6.5?], when an implemented module augments another module (ietf-interfaces), the augmented module MUST be also implemented. So libyang automatically changes the augmented module from imported to the implemented. The same rule applies also in case of referring a module in path (leafref) and by deviating a module. But it does not apply when a module data is used in must or when conditions. That's the reason why it complains just about ietf-ip and not about ietf-interfaces.
YANG actually does not provide a way to specify that a particular import is also expected to be implemented. Therefore, libyang needs some help with setting modules implemented - all the explicitly loaded modules are supposed to be implemented, if the module is just implicitly loaded from the search directory and user did not expressed that it is supposed to be implemented, it is kept only imported to provide groupings or type definitions
Benoit Claise asked (referring to my reference to automated tools):
Would it be possible to improve the warning (and the related test, by testing implemented instead of import), basically telling that the module itself is fine?

I’m interested to know that NETMOD thinks about this distinction between implemented versus imported (in the absence of any instance documents). I guess my (maybe naive) view is that if all I’m doing is checking for errors in my YANG model then I don’t care about this. If my YANG is good I want to see no warnings or errors, and if it’s bad then I want to be told this (and why).

Thanks,
William

---

From: Robert Wilton <[email-masked]>
Date: Mon, 6 Mar 2017 16:59:36 +0000

Hi William,

I think that what yanglint is doing here is sane, i.e. I think that its 
interpretation/split between imported vs implemented modules is 
supported by the YANG RFC.

However, for validation purposes it seems that it would be useful if 
yanglint had an option to assume that all imported modules are 
implicitly implemented without requiring them to be explicitly specified.

Thanks,
Rob

On 06/03/2017 16:44, William Lupton wrote:
> All,
>
> This message arose from a [email-masked] 
> <mailto:[email-masked]> “draft-ietf-pim-igmp-mld-yang-02.txt: 
> YANG compilation isuse” (sic) thread 
> <https://www.ietf.org/mail-archive/web/yang-multicast/current/threads.html#00232> initiated 
> by Benoit.
>
> I thought it would be useful for NETMOD to see the part of the 
> discussion that relates to implemented versus imported YANG modules.
>
>  1. Benoit Claise reported this warning:
>       * warn: Schema node "ietf-ip:ipv4" not found
>         (/ietf-interfaces:interfaces/ietf-interfaces:interface[ietf-interfaces:name
>         = current()]/ietf-ip:ipv4)
>  2. Radek Krejčí replied:
>       * These warnings are printed because in yanglint, until
>         explicitly stated, the imported modules (such as
>         ietf-interfaces and ietf-ip), are supposed to be only
>         imported, not implemented. The data nodes in imported schemas
>         are not available, which is the reason of these warnings.
>  3. William Lupton (that’s me!) asked / commented:
>       * Why are the complaints only about ip:ipv4 (etc) and not about
>         if:interfaces (etc), which are also referenced in the must
>         statements?
>       * This makes it hard for an automated tool (such as Benoit’s)
>         because it needs to know which other YANG files to process in
>         addition to the “file of interest”.
>  4. Radek Krejčí replied:
>       * According to RFC 7950, sec 5.6.6 (3rd paragraph) [ED: 5.6.5?],
>         when an implemented module augments another module
>         (ietf-interfaces), the augmented module MUST be also
>         implemented. So libyang automatically changes the augmented
>         module from imported to the implemented. The same rule applies
>         also in case of referring a module in path (leafref) and
>         by deviating a module. But it does not apply when a module
>         data is used in must or when conditions. That's the reason why
>         it complains just about ietf-ip and not about ietf-interfaces.
>       * YANG actually does not provide a way to specify that a
>         particular import is also expected to be implemented.
>         Therefore, libyang needs some help with setting modules
>         implemented - all the explicitly loaded modules are supposed
>         to be implemented, if the module is just implicitly loaded
>         from the search directory and user did not expressed that it
>         is supposed to be implemented, it is kept only imported to
>         provide groupings or type definitions
>  5. Benoit Claise asked (referring to my reference to automated tools):
>       * Would it be possible to improve the warning (and the related
>         test, by testing implemented instead of import), basically
>         telling that the module itself is fine?
>
>
> I’m interested to know that NETMOD thinks about this distinction 
> between implemented versus imported (in the absence of any instance 
> documents). I guess my (maybe naive) view is that if all I’m doing is 
> checking for errors in my YANG model then I don’t care about this. If 
> my YANG is good I want to see no warnings or errors, and if it’s bad 
> then I want to be told this (and why).
>
> Thanks,
> William
>
>
> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod

---

From: William Lupton <[email-masked]>
Date: Tue, 7 Mar 2017 09:00:03 +0000

Could this be the default in the first of these two cases?

Usage:
    yanglint [options] [-f { yang | yin | tree }] <file>...
        Validates the YANG module in <file>, and all its dependencies.

    yanglint [options] [-f { xml | json }] <schema>... <file>...
        Validates the YANG modeled data in <file> according to the <schema>.

> On 6 Mar 2017, at 16:59, Robert Wilton <[email-masked]> wrote:
> 
> Hi William,
> 
> I think that what yanglint is doing here is sane, i.e. I think that its interpretation/split between imported vs implemented modules is supported by the YANG RFC.
> 
> However, for validation purposes it seems that it would be useful if yanglint had an option to assume that all imported modules are implicitly implemented without requiring them to be explicitly specified.
> 
> Thanks,
> Rob
> 
>> [email-masked] <mailto:[email-masked]>
>> https://www.ietf.org/mailman/listinfo/netmod <https://www.ietf.org/mailman/listinfo/netmod>
>

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 07 Mar 2017 10:30:51 +0100

> implicitly implemented without requiring them to be explicitly

This will fail if a module just wants to use a grouping or typedef from
an imported module but not data nodes that may also be there. 

It is exactly the problem that I mentioned in the discussion about
NETMOD charter: we need a way to specify a complete data model. In my
YANG/I-D development environment [1], a hello XML file is used for this
purpose.

Lada

[1] https://github.com/llhotka/YANG-I-D

-- 
Ladislav Lhotka, CZ.NIC Labs
PGP Key ID: 0xB8F92B08A9F76C67

---

From: Radek Krejčí <[email-masked]>
Date: Tue, 7 Mar 2017 10:49:19 +0100

Hi,
I think that the default approach should be to expect that, until explicitly stated, the imported module is just imported, not implemented. But it is fine to me to add an option to force the implemented flag on all the modules. Benoit can then use the flag in his scripts.

However, I checked RFC 7950 again, and I would like to have a feedback from the NETMOD group to the section 7.1.5, that states that
"the importing module may:
...
   o  use any node in the imported module's schema tree in "must",
      "path", and "when" statements, or as the target node in "augment"
      and "deviation" statements."

I'm interested in "must" and "when". I'm not sure why it is mentioned here, because, in contrast to "path", "augment" and "deviation",
"when" and "must" contain XPath expression, so actually even not defined schema node can be used in it (and this is the reason why yanglint does not complain with error, but just with warning). Of course, the result is then always false (ok, depending on the rest of the expression, but it simply does not depend on the data presence). And this is the reason to have it at least as warning, because it is usually not the original intention of the author.

Regards,
Radek

Dne 7.3.2017 v 10:00 William Lupton napsal(a):
>> On 6 Mar 2017, at 16:59, Robert Wilton <[email-masked] <mailto:[email-masked]>> wrote:
>>
>> Hi William,
>>
>>>       * warn: Schema node "ietf-ip:ipv4" not found (/ietf-interfaces:interfaces/ietf-interfaces:interface[ietf-interfaces:name = current()]/ietf-ip:ipv4)
>>>  2. Radek Krejčí replied:
>>>       * These warnings are printed because in yanglint, until explicitly stated, the imported modules (such as ietf-interfaces and ietf-ip), are supposed to be only imported, not implemented. The data nodes in imported schemas are not available, which is the reason of these warnings.
>>>       * Why are the complaints only about ip:ipv4 (etc) and not about if:interfaces (etc), which are also referenced in the must statements?
>>>       * This makes it hard for an automated tool (such as Benoit’s) because it needs to know which other YANG files to process in addition to the “file of interest”.
>>>  4. Radek Krejčí replied:
>>>       * According to RFC 7950, sec 5.6.6 (3rd paragraph) [ED: 5.6.5?], when an implemented module augments another module (ietf-interfaces), the augmented module MUST be also implemented. So libyang automatically changes the augmented module from imported to the implemented. The same rule applies also in case of referring a module in path (leafref) and by deviating a module. But it does not apply when a module data is used in must or when conditions. That's the reason why it complains just about ietf-ip and not about ietf-interfaces.
>>>       * YANG actually does not provide a way to specify that a particular import is also expected to be implemented. Therefore, libyang needs some help with setting modules implemented - all the explicitly loaded modules are supposed to be implemented, if the module is just implicitly loaded from the search directory and user did not expressed that it is supposed to be implemented, it is kept only imported to provide groupings or type definitions
>>>       * Would it be possible to improve the warning (and the related test, by testing implemented instead of import), basically telling that the module itself is fine?
>>>
>>>

---

From: Radek Krejčí <[email-masked]>
Date: Tue, 7 Mar 2017 10:54:39 +0100

Hi Lada,

> an imported module but not data nodes that may also be there. 

but does it affect the validation of the module?

> [1] https://github.com/llhotka/YANG-I-D

we have this feature in TODO for yanglint, but I'm afraid that it does not solve the issue - even now the script can read some additional file with the specification which modules are expected to be loaded before the module being validated (i.e. which imported module is supposed to be implemented). The root of the issue is that this information is not part of the importing module itself.

Radek

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 07 Mar 2017 12:53:39 +0100

> but does it affect the validation of the module?

Potentially it could - for example, the imported module may have some
default content with "must" statements. Practically, it shouldn't be a
problem most of the time.

> we have this feature in TODO for yanglint, but I'm afraid that it does

Now it's better to use yang-library, as Yangson does (but maybe this is
your plan, right?).

> not solve the issue - even now the script can read some additional
> file with the specification which modules are expected to be loaded
> before the module being validated (i.e. which imported module is
> supposed to be implemented). The root of the issue is that this
> information is not part of the importing module itself.

This would be one option, otherwise this information can also be
included in the document separately as metadata - validation
instructions. This could BTW also solve the issue of what modules are
supposed to be validated.

Lada

---

From: Ladislav Lhotka <[email-masked]>
Date: Tue, 07 Mar 2017 13:05:19 +0100

>       and "deviation" statements."

Strictly speaking, this is incorrect. XPath expressions do not refer to
schema nodes but rather to nodes/nodesets in an instance data tree.

>
> I'm interested in "must" and "when". I'm not sure why it is mentioned
> here, because, in contrast to "path", "augment" and "deviation",

Actually, the argument of "path" is also a (simplified) XPath expression.

> "when" and "must" contain XPath expression, so actually even not
> defined schema node can be used in it (and this is the reason why
> yanglint does not complain with error, but just with warning). Of
> course, the result is then always false (ok, depending on the rest of
> the expression, but it simply does not depend on the data
> presence). And this is the reason to have it at least as warning,
> because it is usually not the original intention of the author.

Agreed, Lada

---

From: William Lupton <[email-masked]>
Date: Tue, 7 Mar 2017 14:35:45 +0000

Inline. Thanks, W.

> On 7 Mar 2017, at 09:49, Radek Krejčí <[email-masked]> wrote:
> 
> Hi,
>      and "deviation" statements.”

So this was the situation in the original message, right? This warning was being output:
* warn: Schema node "ietf-ip:ipv4" not found (/ietf-interfaces:interfaces/ietf-interfaces:interface[ietf-interfaces:name = current()]/ietf-ip:ipv4)

…not because "ietf-ip:ipv4” doesn’t exist, but because the ietf-ip module was imported rather than implemented.

I think you are saying that (per RFC 7950 section 7.1.5) “ietf-ip:ipv4” should have been found, in which case there wouldn’t have been a warning. Is that correct?

---

From: Radek Krejčí <[email-masked]>
Date: Tue, 7 Mar 2017 16:12:41 +0100

> I think you are saying that (per RFC 7950 section 7.1.5) “ietf-ip:ipv4” should have been found, in which case there wouldn’t have been a warning. Is that correct?

I'm saying that I don't understand why the "must" and "when" are mentioned in 7.1.5, because they actually can refer to even undefined nodes - it is not supposed to be an error. However, there will be a warning as yanglint's hint to the user that something is weird - not necessarily with the module itself, but maybe with a way it is used in server. yanglint's warnings are not printed as YANG specification warnings (I don't know what should be such warnings - specification just says that something is valid or not), warnings are printed because there can be a serious issue with using the module for the data instances - when/must condition will always result to false and it's probably not what you expect.

BTW, yanglint (in devel branch) is already able to make all the imported modules implemented (option -i). But that does not help much. Respectively, it helps in case of ietf-igmp-mld module. But there are other IETF modules/drafts that fail with this option because they need to import multiple revisions of the same module (not directly, but because some of the imported modules imports different revision of some module). And this is not possible in case you want all the revisions/imports to be implemented - only single revision of a module can be implemented.

Regards,
Radek