Subject: [secdir] secdir last call review of draft-ietf-bmwg-ipv6-tran-tech-benchmarking-07.txt

From: Taylor Yu <[email-masked]>
Date: Sun, 07 May 2017 19:57:38 +0000

I have reviewed this document as part of the security directorate's
ongoing effort to review all IETF documents being processed by the
IESG.  These comments were written primarily for the benefit of the
security area directors.  Document editors and WG chairs should treat
these comments just like any other last call comments.

Summary: Ready with nits

The "SHOULD" in the following sentence doesn't seem like a valid RFC
2119 keyword usage to me.

   "Any implications for network security arising
   from the DUT/SUT SHOULD be identical in the lab and in production
   networks."

Please consider replacing it with lowercase "should".  (I read it as
predicting a correlation between the network security properties of the
DUT in the lab environment and its behavior in a production environment,
not as a guideline for implementors.)

Comments:

I'm not sure if you would consider this to be in scope, but might it be
useful to instrument implementations being benchmarked with runtime
error or anomaly detection?  (This would be in addition to the
uninstrumented "black-box" measurements.)  This could lead to detecting
security-relevant bounds checking or memory management errors induced by
aggressive benchmarking workloads, possibly identifying vulnerabilities
early enough to fix them before they're exploited.

Some kinds of instrumentation could have a substantial performance
impact, so it might be best to start testing well below the limits of
uninstrumented performance of the devices/systems under test.

Editorial:

Section 13 (Security Considerations) uses "SUT" without a prior
expansion.  Presumably it means "System Under Test" or "Software Under
Test"?

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Sun, 7 May 2017 20:41:52 +0000

Hi Taylor,

> not as a guideline for implementors.)

This *is* a guideline to implementors, who are part of the intended
audience. We don't want testers to waste time benchmarking 
implementations that are for the lab-only; it is recommended
to test the systems intended for production (and such testing
will be safer in the isolated lab, of course).

Also, if implementations have run-time error instrumentation,
so be it, but collecting this info is normally beyond the scope
of the blackbox lab texting of external phenomena.

hope this helps,
Al

> -----Original Message-----
> From: Taylor Yu [mailto:[email-masked]]
> Sent: Sunday, May 07, 2017 3:58 PM
> To: [email-masked]; [email-masked]; draft-ietf-bmwg-ipv6-tran-tech-
> [email-masked]
> Subject: secdir last call review of draft-ietf-bmwg-ipv6-tran-tech-
> benchmarking-07.txt
>

---

From: Taylor Yu <[email-masked]>
Date: Mon, 08 May 2017 01:34:02 +0000

> will be safer in the isolated lab, of course).

I'm sorry, I guess I misinterpreted, then.  Do you mean that the
implementations tested in the benchmarking lab should not materially
differ from ones intended for production?  As in the tested
implementations should be have neither stronger nor weaker security
properties than their deployed counterparts?  I might be able to suggest
improved wording if I understand your intentions correctly.

> of the blackbox lab texting of external phenomena.

I think there is an opportunity here to use instrumented versions of
implementations to detect security-relevant errors and anomalies as they
operate near their performance limits.  I also concede that sort of work
might be outside the scope of this document.  (I can envision a related
document that would cover such security-related topics.)

---

From: Marius Georgescu <[email-masked]>
Date: Tue, 9 May 2017 11:38:04 +0300

Hello Taylor,

Thank you very much for reviewing the document.
Please find my comments inline.

> On May 8, 2017, at 4:34 AM, Taylor Yu <[email-masked]> wrote:
> 
Indeed, we meant that the implementations tested in the lab should not differ from the ones intended for production.

> document that would cover such security-related topics.)
This sounds like an interesting benchmarking subject. However, I think it's outside the scope of this document and should remain that way.
Maybe a specific document dedicated to this subject is a better idea.

Best regards,
Marius

---

From: Taylor Yu <[email-masked]>
Date: Thu, 11 May 2017 22:55:19 +0000

> Indeed, we meant that the implementations tested in the lab should not
> differ from the ones intended for production.

In that case, what do you think of the following change?

OLD

   Further, benchmarking is performed on a "black-box" basis, relying
   solely on measurements observable external to the DUT/SUT. Special
   capabilities SHOULD NOT exist in the DUT/SUT specifically for
   benchmarking purposes. Any implications for network security arising
   from the DUT/SUT SHOULD be identical in the lab and in production
   networks.

NEW

   Further, benchmarking is performed on a "black-box" basis, relying
   solely on measurements observable external to the DUT.  Special
   capabilities SHOULD NOT exist in the DUT specifically for
   benchmarking purposes.  Testers and implementors SHOULD ensure that
   the DUT has identical security properties to its production version.
   For example, any security hardening or instrumentation present on the
   DUT SHOULD also be present in the production version, and vice versa.

Best regards,
-Taylor

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Thu, 11 May 2017 23:21:10 +0000

Hi Taylor,
please see in-line reply, thanks,
Al

> Sent: Thursday, May 11, 2017 6:55 PM
> To: Marius Georgescu
> Cc: MORTON, ALFRED C (AL); [email-masked]; [email-masked]; draft-ietf-
> [email-masked]
> Subject: Re: secdir last call review of draft-ietf-bmwg-ipv6-tran-tech-
> benchmarking-07.txt
> 
[ACM] 
This new sentence has redundant effect with second sentence (unchanged).
Furthermore, Testing personnel have no means to ensure the identical 
properties, and implementers may not yet have a production version available
(Benchmarking often takes place with very early products, whose
non-tested aspects may change before reaching production).

>    DUT SHOULD also be present in the production version, and vice versa.
[ACM] 
Again, this cannot be ensured when it is not a tested feature, 
and there may be *many* (later) production releases that differ 
from the DUT. The example goes too far.

At this point, I feel it is appropriate to point out that BMWG has 
used the original wording of the paragraphs in this section for years
(~10), and their intent has been sufficiently clear to the community.

---

From: Marius Georgescu <[email-masked]>
Date: Thu, 25 May 2017 14:55:46 +0300

Hello Taylor,

Please see my answer inline.
> On May 12, 2017, at 2:21 AM, MORTON, ALFRED C (AL) <[email-masked]> wrote:
> 
> Hi Taylor,
>> From: Taylor Yu [mailto:[email-masked] <mailto:[email-masked]>]
>> Cc: MORTON, ALFRED C (AL); [email-masked] <mailto:[email-masked]>; [email-masked] <mailto:[email-masked]>; draft-ietf-
>> [email-masked] <mailto:[email-masked]>

[MG] I agree with Al, I personally don't see how the new text is improving the old.
I understand that it is important to consider the security features as a potential parameter when testing, and that it should be isolated.
However, I find the current text clear enough.
Would you please elaborate on why it isn't from your perspective.

Best regards,
Marius