Subject: [netmod] leafref to lists that contain system-controlled entries

From: "Sterne, Jason (Nokia - CA/Ottawa)" <[email-masked]>
Date: Fri, 13 Oct 2017 18:43:45 +0000

Hi all,

There are a few threads on the mailing list that touch on the concept of system-controlled resources (mostly list entries):

https://mailarchive.ietf.org/arch/msg/netmod/3fTSHIh_MfHzmuDCoicAGiXA2E0
https://mailarchive.ietf.org/arch/msg/netmod/KIsSgKByQWpqYzA4i6Bwc8fuH3w
https://mailarchive.ietf.org/arch/msg/netmod/mjLJdiYErtNG41dJ5bJ5ji07cz0

A few drafts & RFCs also refer to the concept:
https://tools.ietf.org/html/draft-ietf-netmod-revised-datastores-04
https://tools.ietf.org/html/rfc7223

Several vendor implementations have list entries (instance data) that are populated by the server and can be referenced (leafref) from other places in the configuration.  These system entries are useful pre-created policies, interfaces, etc that can then be used (and referred-to) by operators in their explicit configuration.

If those entries are only expected to exist in the <operational> datastore, then in theory any references to them in user created configuration will cause a validation problem in the candidate/running (missing leafref target).

One solution discussed in the mailing lists is to change every reference to lists that could contain a system created entry to a "require-instance false" leafref.  But then some useful validation is lost.  In many cases the model is more correctly "require-instance true" but the set of targets includes the system create entries.

Another solution discussed is to have the system created entries appear in the <intended> datastore (as part of template/expansion).  That would make validation pass on the intended datastore, but then the candidate/running/startup datastores would not be valid (would be missing leafref targets if any part of the config refers to system created entries).  THis sounds similar to the problem that has been discussed in the past about the fact that templates (in the running) basically mean the running/candidate aren't necessarily valid (until after template expansion, which means only the intended would be valid).

Another approach could be to actually have those system created entries show up in running/candidate.  That would ensure that references to those entries are valid.  But if the whole concept of templates just cause the running/candidate to not be valid anyways maybe we wouldn't worry about the invalid aspect of references to system created list entries ?

Rgds,
Jason

---

From: Martin Bjorklund <[email-masked]>
Date: Mon, 16 Oct 2017 08:55:59 +0200 (CEST)

"Sterne, Jason (Nokia - CA/Ottawa)" <[email-masked]> wrote:
> Hi all,
> 
> There are a few threads on the mailing list that touch on the concept
> of system-controlled resources (mostly list entries):
> 
> Several vendor implementations have list entries (instance data) that
> are populated by the server and can be referenced (leafref) from other
> places in the configuration.  These system entries are useful
> pre-created policies, interfaces, etc that can then be used (and
> referred-to) by operators in their explicit configuration.
> 
> If those entries are only expected to exist in the <operational>
> datastore, then in theory any references to them in user created
> configuration will cause a validation problem in the candidate/running
> (missing leafref target).
> 
> One solution discussed in the mailing lists is to change every
> reference to lists that could contain a system created entry to a
> "require-instance false" leafref.  But then some useful validation is
> lost.  In many cases the model is more correctly "require-instance
> true" but the set of targets includes the system create entries.
> 
> Another solution discussed is to have the system created entries
> appear in the <intended> datastore (as part of template/expansion).
> That would make validation pass on the intended datastore, but then
> the candidate/running/startup datastores would not be valid (would be
> missing leafref targets if any part of the config refers to system
> created entries).  THis sounds similar to the problem that has been
> discussed in the past about the fact that templates (in the running)
> basically mean the running/candidate aren't necessarily valid (until
> after template expansion, which means only the intended would be
> valid).
> 
> Another approach could be to actually have those system created
> entries show up in running/candidate.  That would ensure that
> references to those entries are valid.

This works if the "system created entries" behave just like any other
entry, i.e., a client (with proper access rigths) can modify and
delete them.

If this is not the case, the next question is what happens if the
client creates its own entry with the same name as a system created
entry?  Is this illegal, or will the user created entry override the
system created entry?

Another question to think about is what happpens with user config that
refers to such a system created entry if a new software image is
loaded where the system created entry is no longer present?

/martin

---

From: Robert Wilton <[email-masked]>
Date: Mon, 16 Oct 2017 11:39:35 +0100

Hi Jason,

On 13/10/2017 19:43, Sterne, Jason (Nokia - CA/Ottawa) wrote:
> “require-instance false” leafref. But then some useful validation is 
> lost.  In many cases the model is more correctly “require-instance 
> true” but the set of targets includes the system create entries.
>

I agree that this is not a good general solution for system created 
configuration that is always expected to exist on the device because 
some of the useful validation is lost.

But I think that this solution does work well were the system created 
entries are truly dynamic in nature, e.g. it seems to work well for the 
topology YANG module where the topology may be explicitly configured, 
but would more normally be learned dynamically from protocol 
interactions, or perhaps be configured via a dynamic configuration protocol.

> basically mean the running/candidate aren’t necessarily valid (until 
> after template expansion, which means only the intended would be valid).
>

I think that this solution is OK, but not necessarily ideal.

As you say, it means that <running>/<candidate>/<startup> may not be 
valid, which I see as quite a big down side.  Longer term, I think that 
it would be a good aim to allow the configuration to be validated off 
the device, if a client desired to do so.

> references to those entries are valid. But if the whole concept of 
> templates just cause the running/candidate to not be valid anyways 
> maybe we wouldn’t worry about the invalid aspect of references to 
> system created list entries ?
>

I know that quite a few implementations do this today, but I'm generally 
not a fan of the system modifying <running>.  It seems that an overall 
architecture is much cleaner if <running> has a single source of truth 
and hence can be exclusively controlled by the client.

But I also like the approach where the client (rather than the device) 
explicitly writes these default entries into the configuration, if they 
are referenced elsewhere by the configuration, to make the configuration 
"complete".  E.g. if part of the configuration references the loopback0 
interface then also explicitly add the necessary loopback0 configuration 
to instantiate the "loopback0" interface.  When this configuration is 
pushed to the device (i.e. using merge or replace operation semantics) 
then the system should silently accept/ignore the explicit configuration 
to create the loopback0 interface if it already exists on the system.

At the moment, IETF, and other SDOs are busy defining standard YANG 
models, but for those models to end up being truly generic they also 
need to have consistency about which bits of configuration are always 
expected to implicitly exist on the device.  E.g. considering the 
example above of configuration referencing loopback0: if some systems 
automatically create a loopback0 interface and others do not, then a 
generic configuration needs to handle both scenarios.

If IETF standardizes YANG configuration templates, then perhaps it would 
be good to investigate whether some of these "useful default system 
properties" could instead be embodied into one or more standard device 
templates?  These templates could then be explicitly referenced in the 
<running> configuration.  This may allow <running> to be small, but 
still allow it to be "complete" and able to be validated off the box.

Thanks,
Rob

> Rgds,
>
> Jason
>
>
>
> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod

---

From: "Sterne, Jason (Nokia - CA/Ottawa)" <[email-masked]>
Date: Mon, 23 Oct 2017 23:46:16 +0000

Thanks Martin.  Pls see below.
Jason

> -----Original Message-----
> From: Martin Bjorklund [mailto:[email-masked]]
> Sent: Monday, October 16, 2017 2:56
> To: Sterne, Jason (Nokia - CA/Ottawa) <[email-masked]>
> Cc: [email-masked]
> Subject: Re: [netmod] leafref to lists that contain system-controlled entries
> 
...snip...
> This works if the "system created entries" behave just like any other entry,
> i.e., a client (with proper access rigths) can modify and delete them.

[>>JTS] I think there are 2 classes of these "system created entries":
1) deletable entries
2) non-deletable entries

For deletable entries we had old discussions about those a few years ago on the list where it was proposed these are modelled as "default startup datastore" entries (i.e. instantiated by the system at startup time if there is no startup datastore present).  They can then be deleted or modified by a client (and then a subsequent startup would not have those entries).

I'm more concerned here with non-deletable entries.  The entries themselves would always be present (or perhaps only be present if some other part of the config is actually referencing them).   Some types of entrie may be modifiable (i.e. their leafs or other descendants can be changed) and some may not be modifiable.

My responses below are based on non-deletable entries.

> 
> If this is not the case, the next question is what happens if the client creates
> its own entry with the same name as a system created entry?  Is this illegal,
> or will the user created entry override the system created entry?

[>>JTS] I think creation of an entry by a client would just be silently accepted (and wouldn't affect what a client sees in a <get-config> since it was already present).  Similarly a deletion of an entry would be silently accepted but the entry would still be there in a <get-config> response.
As mentioned above, some entries may have modifiable leafs and descendants.  Some may not (would return an error if attempting to set the value of a leaf/descendant).

> refers to such a system created entry if a new software image is loaded
> where the system created entry is no longer present?

[>>JTS] A few options:
1) only use this for entries that are never really expected to be obsoleted (which may actually be a pretty likely case from what I've seen of these types of entries)
2) have the startup config/datastore contain these entries (so they would be created as explicit user entries if they don't exist as system created in the future)
3) have upgrade code that accepts references to "old" entries and auto-convert them to some new entry (assuming one exists)

---

From: "Sterne, Jason (Nokia - CA/Ottawa)" <[email-masked]>
Date: Mon, 23 Oct 2017 23:58:27 +0000

Thanks Rob.  Please see below.
Jason

From: Robert Wilton [mailto:[email-masked]]
Sent: Monday, October 16, 2017 6:40
To: Sterne, Jason (Nokia - CA/Ottawa) <[email-masked]>; [email-masked]
Subject: Re: [netmod] leafref to lists that contain system-controlled entries

Hi Jason,

I agree that this is not a good general solution for system created configuration that is always expected to exist on the device because some of the useful validation is lost.

But I think that this solution does work well were the system created entries are truly dynamic in nature, e.g. it seems to work well for the topology YANG module where the topology may be explicitly configured, but would more normally be learned dynamically from protocol interactions, or perhaps be configured via a dynamic configuration protocol.

[>>JTS] OK - I think I follow you here.  You're saying that if there are references to truly dynamic entries, then since those entries will come and go (vs static bootup-time system entries that are always there), references to them will more likely be "require-instance false".   But that does mean the system has to allow references to non-existent entries, and you lose validation. You also risk errors like referencing a name that is just 1 char different from what you really wanted but the system can't tell you that you got it wrong.

As you say, it means that <running>/<candidate>/<startup> may not be valid, which I see as quite a big down side.  Longer term, I think that it would be a good aim to allow the configuration to be validated off the device, if a client desired to do so.

I know that quite a few implementations do this today, but I'm generally not a fan of the system modifying <running>.  It seems that an overall architecture is much cleaner if <running> has a single source of truth and hence can be exclusively controlled by the client.

But I also like the approach where the client (rather than the device) explicitly writes these default entries into the configuration, if they are referenced elsewhere by the configuration, to make the configuration "complete".  E.g. if part of the configuration references the loopback0 interface then also explicitly add the necessary loopback0 configuration to instantiate the "loopback0" interface.  When this configuration is pushed to the device (i.e. using merge or replace operation semantics) then the system should silently accept/ignore the explicit configuration to create the loopback0 interface if it already exists on the system.

[>>JTS] Yes - that is another option and I like it.  If I follow you correctly, the server would never return these system entries in a <get-config> unless the client/operator had already explicitly 'created' them.
So the operator has the option to make the system entries visible or not.

I think the server should still accept references to the system entries even if the client/operator hasn't explicitly created them.  The whole point is that those system entries are there and waiting for operators to use them from the start (without *having* to explicitly create or define them).   In that case the references would be 'dangling' (unresolved) as far as an offline validation is concerned (but a client could select to fix that by explicitly defining any entries they want to reference).

At the moment, IETF, and other SDOs are busy defining standard YANG models, but for those models to end up being truly generic they also need to have consistency about which bits of configuration are always expected to implicitly exist on the device.  E.g. considering the example above of configuration referencing loopback0: if some systems automatically create a loopback0 interface and others do not, then a generic configuration needs to handle both scenarios.

If IETF standardizes YANG configuration templates, then perhaps it would be good to investigate whether some of these "useful default system properties" could instead be embodied into one or more standard device templates?  These templates could then be explicitly referenced in the <running> configuration.  This may allow <running> to be small, but still allow it to be "complete" and able to be validated off the box.
[>>JTS] I'm not clear on this approach.
How would the templates be referenced in the <running> ?  Can you give me an example ? (is this different than a direct reference to a system created entry that I am talking about ?)
How would this allow <running> to be small ? Do the templates contain the full definition of the system created entries ?

Thanks,
Rob

Rgds,
Jason

netmod mailing list

[email-masked]<mailto:[email-masked]>

---

From: Robert Wilton <[email-masked]>
Date: Tue, 24 Oct 2017 13:26:24 +0100

Hi Jason,

Please see further comments inline ...

On 24/10/2017 00:58, Sterne, Jason (Nokia - CA/Ottawa) wrote:
> *From:*Robert Wilton [mailto:[email-masked]]
> *Sent:* Monday, October 16, 2017 6:40
> *To:* Sterne, Jason (Nokia - CA/Ottawa) <[email-masked]>; 
> [email-masked]
> *Subject:* Re: [netmod] leafref to lists that contain 
> system-controlled entries
>
> Hi Jason,
>
>     There are a few threads on the mailing list that touch on the
>     concept of system-controlled resources (mostly list entries):
>
>     Several vendor implementations have list entries (instance data)
>     that are populated by the server and can be referenced (leafref)
>     from other places in the configuration.  These system entries are
>     useful pre-created policies, interfaces, etc that can then be used
>     (and referred-to) by operators in their explicit configuration.
>
>     configuration will cause a validation problem in the
>     candidate/running (missing leafref target).
>
>     “require-instance false” leafref.  But then some useful validation
>     is lost. In many cases the model is more correctly
>     “require-instance true” but the set of targets includes the system
>     create entries.
>
>
> entries are truly dynamic in nature, e.g. it seems to work well for 
> the topology YANG module where the topology may be explicitly 
> configured, but would more normally be learned dynamically from 
> protocol interactions, or perhaps be configured via a dynamic 
> configuration protocol.
>
> *//*
>
> */[>>JTS] OK – I think I follow you here.  You’re saying that if there 
> are references to truly dynamic entries, then since those entries will 
> come and go (vs static bootup-time system entries that are always 
> there), references to them will more likely be “require-instance 
> false”.   But that does mean the system has to allow references to 
> non-existent entries, and you lose validation. You also risk errors 
> like referencing a name that is just 1 char different from what you 
> really wanted but the system can’t tell you that you got it wrong./*
>

Yes.  I don't think that you can solve this during existing YANG 
datastore validation, as it is defined today.

But I also think that NETCONF/YANG is potentially missing an RPC that is 
a step beyond validation, but before actually applying configuration.

YANG and the NMDA datastores draft makes it clear that both <running> 
and <intended> are always valid datastores. This means that the 
validation rules for these datastores really should not depend on the 
current hardware in a device if that hardware could be removed or 
change.  Otherwise, if you allow validation to depend on the current 
hardware capabilities, then if someone pulls out a linecard, that would 
cause a previously valid configuration to immediately become invalid, 
violating the rule that <running>/<intended> are always valid.

I think that a potential solution to this problem, is that a new NETCONF 
RPC could be defined that is a "<should-successfully-apply>" 
operation.   Processing during this new RPC would be able to check 
against current system resources, current hardware capabilities, etc, 
and would be designed to indicate whether the system expects (but does 
not guarantee) that the configuration would completely apply 
successfully without any errors or unapplied configuration.  
Configuration datastores would not have to always conform to this 
constraint, so that if an operator changed or removed a linecard, the 
configuration would still be "valid" (as per NMDA and RFC7950 rules) but 
would fail a subsequent "should-successfully-apply" check.

> *//*
>     appear in the <intended> datastore (as part of
>     template/expansion).  That would make validation pass on the
>     intended datastore, but then the candidate/running/startup
>     datastores would not be valid (would be missing leafref targets if
>     any part of the config refers to system created entries).  THis
>     sounds similar to the problem that has been discussed in the past
>     about the fact that templates (in the running) basically mean the
>     running/candidate aren’t necessarily valid (until after template
>     expansion, which means only the intended would be valid).
>
>
> valid, which I see as quite a big down side.  Longer term, I think 
> that it would be a good aim to allow the configuration to be validated 
> off the device, if a client desired to do so.
>
>
>
>     references to those entries are valid.  But if the whole concept
>     of templates just cause the running/candidate to not be valid
>     anyways maybe we wouldn’t worry about the invalid aspect of
>     references to system created list entries ?
>
>
> I know that quite a few implementations do this today, but I'm 
> generally not a fan of the system modifying <running>.  It seems that 
> an overall architecture is much cleaner if <running> has a single 
> source of truth and hence can be exclusively controlled by the client.
>
> explicitly writes these default entries into the configuration, if 
> they are referenced elsewhere by the configuration, to make the 
> configuration "complete".  E.g. if part of the configuration 
> references the loopback0 interface then also explicitly add the 
> necessary loopback0 configuration to instantiate the "loopback0" 
> interface. When this configuration is pushed to the device (i.e. using 
> merge or replace operation semantics) then the system should silently 
> accept/ignore the explicit configuration to create the loopback0 
> interface if it already exists on the system.
>
> *//*
>
> */[>>JTS] Yes – that is another option and I like it.  If I follow you 
> correctly, the server would never return these system entries in a 
> <get-config> unless the client/operator had already explicitly 
> ‘created’ them. /*
>
> */So the operator has the option to make the system entries visible or 
> not./*
>
Yes, exactly.

/*
*/
>
> *//*
>
> *//*
>
> */I think the server should still accept references to the system 
> entries even if the client/operator hasn’t explicitly created them.  
> The whole point is that those system entries are there and waiting for 
> operators to use them from the start (without *having* to explicitly 
> create or define them).   In that case the references would be 
> ‘dangling’ (unresolved) as far as an offline validation is concerned 
> (but a client could select to fix that by explicitly defining any 
> entries they want to reference)./*
>
I think that this is OK.

Effectively, I see that being like a static system provided template for 
configuration that is merged with <running> to form <intended>, which is 
then validated.

> *//*
> If IETF standardizes YANG configuration templates, then perhaps it 
> would be good to investigate whether some of these "useful default 
> system properties" could instead be embodied into one or more standard 
> device templates?  These templates could then be explicitly referenced 
> in the <running> configuration.  This may allow <running> to be small, 
> but still allow it to be "complete" and able to be validated off the box.
>
> */[>>JTS] I’m not clear on this approach. /*
>
OK, so this is just an idea:

1) Assume a YANG extension is defined to allow templates to be defined.
2) Further, assume that there is a way to store, and uniquely name some 
of those templates in a standard place.

So, perhaps IETF could define a template like this, which is stored as a 
well defined place:

<template>
   <name>ietf-basic-router-v1</name>
   <config>
     <interfaces>
       <interface>
         <name>Loopback0</name>
         <type>ianaift:loopback</type>
       </interface>
       <interface>
         <name>Null0</name>
         <type>ianaift:null</type>
       </interface>
     </interfaces>
   </config>
</template>

Now, when it comes to the configuration file for your device, it could 
look like this:
   <config>
<use-remote-template>http://yang-templates.ietf.org/ietf-basic-router-v1</use-remote-template>

     ... rest of config as normal.
   </config>

So, the expanded configuration would include the explicit configuration 
for Loopback0 and Null0 interfaces, but that would be pulled via use of 
a remote template (the contents of which is probably already cached on 
the device).

> *//*
>
> */How would the templates be referenced in the <running> ?  Can you 
> give me an example ? (is this different than a direct reference to a 
> system created entry that I am talking about ?)/*
>
The above is just an idea.  Normally I would expect configuration 
templates to be defined in the running configuration.  But here I was 
considering the idea that a template is predefined in some way, and then 
referenced, so that it doesn't have to be provided inline in the running 
configuration.

> *//*
>
> */How would this allow <running> to be small ? Do the templates 
> contain the full definition of the system created entries ?/*
>
Yes, I was assuming that the template could contain what might normally 
be represented as system created entries today.  Standard templates 
could either be defined by SDOs, vendors, or operators, as long as there 
is a way to reference them.

Thanks,
Rob

> *//*
>     [email-masked] <mailto:[email-masked]>
>

---

From: "Sterne, Jason (Nokia - CA/Ottawa)" <[email-masked]>
Date: Fri, 27 Oct 2017 17:46:17 +0000

Hi Rob,
Please see inline.
Rgds,
Jason

From: Robert Wilton [mailto:[email-masked]]
Sent: Tuesday, October 24, 2017 8:26
To: Sterne, Jason (Nokia - CA/Ottawa) <[email-masked]>; [email-masked]
Subject: Re: [netmod] leafref to lists that contain system-controlled entries

Hi Jason,

From: Robert Wilton [mailto:[email-masked]]
Sent: Monday, October 16, 2017 6:40
To: Sterne, Jason (Nokia - CA/Ottawa) <[email-masked]><mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [netmod] leafref to lists that contain system-controlled entries

Hi Jason,

Yes.  I don't think that you can solve this during existing YANG datastore validation, as it is defined today.

But I also think that NETCONF/YANG is potentially missing an RPC that is a step beyond validation, but before actually applying configuration.

YANG and the NMDA datastores draft makes it clear that both <running> and <intended> are always valid datastores. This means that the validation rules for these datastores really should not depend on the current hardware in a device if that hardware could be removed or change.  Otherwise, if you allow validation to depend on the current hardware capabilities, then if someone pulls out a linecard, that would cause a previously valid configuration to immediately become invalid, violating the rule that <running>/<intended> are always valid.
[>>JTS] In implementations that support templating today, the <running> is not always valid.  It is only upon expansion of the templates that you end up with a valid datastore (i.e. in the <intended>).  The template may provide missing mandatory parms, provide the actual targets of leafrefs, etc.
  I can't see a way to actually make <running> or <candidate> valid with the concept of templates  (unless we define validity as after a temporary expansion in some post-processed version of the <running> - but that would just be the <intended>).
  This problem of validity isn't specific to dynamic behavior (e.g. hardware coming & going), or to system provisioned objects.  It is a problem just with static config as well.

I think that a potential solution to this problem, is that a new NETCONF RPC could be defined that is a "<should-successfully-apply>" operation.   Processing during this new RPC would be able to check against current system resources, current hardware capabilities, etc, and would be designed to indicate whether the system expects (but does not guarantee) that the configuration would completely apply successfully without any errors or unapplied configuration.
[>>JTS] "completely apply successfully" does not necessarily mean the candidate or running are "valid" against the YANG model.
Configuration datastores would not have to always conform to this constraint, so that if an operator changed or removed a linecard, the configuration would still be "valid" (as per NMDA and RFC7950 rules) but would fail a subsequent "should-successfully-apply" check.
[>>JTS] I see it the other way around.  With template support, a <running> datastore will usually pass the "should-succesfully-apply" criteria but won't be 'valid'.

Effectively, I see that being like a static system provided template for configuration that is merged with <running> to form <intended>, which is then validated.
[>>JTS] Yes.  But that means the <running> or <candidate> is not valid (but would pass your <should-succesfully-apply> check) , but the <intended> is valid.

1) Assume a YANG extension is defined to allow templates to be defined.
2) Further, assume that there is a way to store, and uniquely name some of those templates in a standard place.

So, perhaps IETF could define a template like this, which is stored as a well defined place:

<template>

Now, when it comes to the configuration file for your device, it could look like this:
  <config>
    <use-remote-template>http://yang-templates.ietf.org/ietf-basic-router-v1</use-remote-template>

So, the expanded configuration would include the explicit configuration for Loopback0 and Null0 interfaces, but that would be pulled via use of a remote template (the contents of which is probably already cached on the device).
[>>JTS] Sorry for harping on this point -> but this still means the candidate/running are not valid.  It is only post-expansion that a datastore becomes valid against the YANG model.   I'm not saying this "use-remote-template" idea doesn't have merit, but it still leaves us with the problem of invalid running/candidate.

It is also causing me to think that perhaps the transition from running->intended could also include the addition of the system provisioned objects (if they haven't been explicitly configured by the client as discussed above) as part of that 'expansion'.

How would the templates be referenced in the <running> ?  Can you give me an example ? (is this different than a direct reference to a system created entry that I am talking about ?)
The above is just an idea.  Normally I would expect configuration templates to be defined in the running configuration.  But here I was considering the idea that a template is predefined in some way, and then referenced, so that it doesn't have to be provided inline in the running configuration.

How would this allow <running> to be small ? Do the templates contain the full definition of the system created entries ?
Yes, I was assuming that the template could contain what might normally be represented as system created entries today.  Standard templates could either be defined by SDOs, vendors, or operators, as long as there is a way to reference them.

Thanks,
Rob

Thanks,
Rob

Rgds,
Jason

netmod mailing list