Subject: Re: [PMOL] draft-ietf-pmol-metrics-framework-08.txt posted
From: Al Morton <[email-masked]>
To: Vinayak Hegde <[email-masked]>, Benoit Claise <[email-masked]>
Cc: "Miguel A. Garcia" <[email-masked]>, Leeyoung <[email-masked]>, [email-masked], "[email-masked]" <[email-masked]>
Date: Sun, 30 Jan 2011 09:10:04 -0500
List-Id: Performance Metrics at Other Layers <pmol.ietf.org>

Hi Vin,

You raise some interesting informational points.

Unfortunately, both the Working Group and IETF-wide
Last Calls ended some time ago, and Benoit has published the
version that the IESG will consider at their next meeting.

Perhaps Benoit can see if there is any commonality with
IESG comments, and incorporate some of your thoughts
if possible.

regards,
Al

At 01:50 PM 1/29/2011, Vinayak Hegde wrote:
>Hi Benoit,
>
>The RFC certainly looks good and does covers a lot of ground. I would
>like to add some more information to the RFC. My comments follow:
>...<stuff deleted>...
>Regards
>Vinayak
>
>On Sat, Jan 29, 2011 at 7:34 PM, Benoit Claise <[email-masked]> wrote:
> > Al,
> >
> > Yes indeed. My mistake.
> >
> > Regards, Benoit
> >>
> >> That's IETF Last Call, n'est pas?
> >>
> >> At 05:45 AM 1/28/2011, Benoit Claise wrote:
> >>>
> >>> Dear all,
> >>>
> >>> Now that the WG Last Call is over, I submitted a new version of the PMOL
> >>> framework.
> >>> http://www.ietf.org/id/draft-ietf-pmol-metrics-framework-08.txt
> >>>
> >>> It includes the feedback from: sec-dir, gen-art, rtg-dir, and ops-dir
> >>>
> >>> Regards, Benoit.
> >>>

Date: Sun, 30 Jan 2011 00:20:29 +0530
From: Vinayak Hegde <[email-masked]>
To: Benoit Claise <[email-masked]>
Cc: "Miguel A. Garcia" <[email-masked]>, Leeyoung <[email-masked]>, [email-masked], "[email-masked]" <[email-masked]>
Subject: Re: [PMOL] draft-ietf-pmol-metrics-framework-08.txt posted

Hi Benoit,

The RFC certainly looks good and does covers a lot of ground. I would
like to add some more information to the RFC. My comments follow:

In section 2.2

You might want to add that we might need to invite experts based on
the protocol metrics that RFC intends to address (for example HTTP,
SIP, SCTP, SFTP etc). You will need to invite experts to address the
nuances of the each protocol which can influence the metrics.

In Section 5.1

A couple more additions here - Two more factors that can cause
variation (mostly for live traffic) is

1. Sampling criteria
Sampling can be done randomly (probabilistic) or you can look at every
nth connection or every nth packet. The sampling criteria might
introduce some bias in the measurements.

2. Period (and duration) of measurement
Some live traffic can have patterns eg. measurement of FTP/ HTTP
latency during 24 hours. The traffic is generally high during the
mid-day and low at the middle of the night. There is similar
variations during the week depending on the type of traffic being
measured. Furthermore economic incentives such as pricing plans (no
cap accounting during night for example) can have bearing on the final
measurement results. Care needs to be taken to account for a complete
"pattern cycle" for such measurements.

In section 5.3.1

Another type of metric that can be useful is when the time spans might
be overlapping. Many such metrics are those that are based on moving
windows. A good example is streaming protocols / applications that
rely on buffering (where you need to look at moving windows to measure
QoE). Such measurements can be useful to model the QoE when there is
burst loss. Since all packets are not created equal the same amount of
packets can have different impacts when they occur in a pattern within
a moving window (such as a glitch in video or clicks/gaps in audio).

In Section 5.4.2

(iii) Method of Measurement or Calculation
Some methods of calculation might require discarding some data
collected (due to outliers) so as to make the measurement parameters
meaningful. Example is burstable billing which is pretty common in
ISPs (see http://en.wikipedia.org/wiki/Burstable_billing for more
details). In this after sorting the 5-min samples, the top 5
percentile data is discarded.

Please add -

(vii) Characteristics of underlying traffic

Characteristics of underlying traffic on which measurement is done is
important as noted above (due to difference in traffic patterns). Also
in video/audio some packets might be more important as compared to
others based on the codec and buffering pattern of the application.

Regards
Vinayak

On Sat, Jan 29, 2011 at 7:34 PM, Benoit Claise <[email-masked]> wrote:
> Al,
>
> Yes indeed. My mistake.
>
> Regards, Benoit
>>
>> That's IETF Last Call, n'est pas?
>>
>> At 05:45 AM 1/28/2011, Benoit Claise wrote:
>>>
>>> Dear all,
>>>
>>> Now that the WG Last Call is over, I submitted a new version of the PMOL
>>> framework.
>>> http://www.ietf.org/id/draft-ietf-pmol-metrics-framework-08.txt
>>>
>>> It includes the feedback from: sec-dir, gen-art, rtg-dir, and ops-dir
>>>
>>> Regards, Benoit.
>>>
>>>
>>> _______________________________________________
>>> PMOL mailing list
>>> [email-masked]
>>> https://www.ietf.org/mailman/listinfo/pmol
>>
>> _______________________________________________
>> PMOL mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/pmol
>
> _______________________________________________
> PMOL mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/pmol
>

Date: Sat, 29 Jan 2011 15:04:17 +0100
From: Benoit Claise <[email-masked]>
To: Al Morton <[email-masked]>
Cc: "Miguel A. Garcia" <[email-masked]>, Leeyoung <[email-masked]>, [email-masked], "[email-masked]" <[email-masked]>
Subject: Re: [PMOL] draft-ietf-pmol-metrics-framework-08.txt posted

Al,

Yes indeed. My mistake.

Regards, Benoit
> That's IETF Last Call, n'est pas?
>
> At 05:45 AM 1/28/2011, Benoit Claise wrote:
>> Dear all,
>>
>> Now that the WG Last Call is over, I submitted a new version of the 
>> PMOL framework.
>> http://www.ietf.org/id/draft-ietf-pmol-metrics-framework-08.txt
>>
>> It includes the feedback from: sec-dir, gen-art, rtg-dir, and ops-dir
>>
>> Regards, Benoit.
>>
>>
>> _______________________________________________
>> PMOL mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/pmol
>
> _______________________________________________
> PMOL mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/pmol

Date: Fri, 28 Jan 2011 15:56:17 -0500
To: Benoit Claise <[email-masked]>, "[email-masked]" <[email-masked]>
From: Al Morton <[email-masked]>
Cc: "Miguel A. Garcia" <[email-masked]>, Leeyoung <[email-masked]>, [email-masked]
Subject: Re: [PMOL] draft-ietf-pmol-metrics-framework-08.txt posted

That's IETF Last Call, n'est pas?

At 05:45 AM 1/28/2011, Benoit Claise wrote:
>Dear all,
>
>Now that the WG Last Call is over, I submitted a new version of the 
>PMOL framework.
>http://www.ietf.org/id/draft-ietf-pmol-metrics-framework-08.txt
>
>It includes the feedback from: sec-dir, gen-art, rtg-dir, and ops-dir
>
>Regards, Benoit.
>
>
>_______________________________________________
>PMOL mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/pmol

Date: Fri, 28 Jan 2011 11:45:24 +0100
From: Benoit Claise <[email-masked]>
To: "[email-masked]" <[email-masked]>
Cc: "Miguel A. Garcia" <[email-masked]>, Leeyoung <[email-masked]>, [email-masked]
Subject: [PMOL] draft-ietf-pmol-metrics-framework-08.txt posted

Dear all,

Now that the WG Last Call is over, I submitted a new version of the PMOL 
framework.
http://www.ietf.org/id/draft-ietf-pmol-metrics-framework-08.txt

It includes the feedback from: sec-dir, gen-art, rtg-dir, and ops-dir

Regards, Benoit.

From: [email-masked]
To: [email-masked]
Date: Fri, 28 Jan 2011 02:45:01 -0800
Cc: [email-masked]
Subject: [PMOL] I-D Action:draft-ietf-pmol-metrics-framework-08.txt

--NextPart

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the Performance Metrics for Other Layers Working Group of the IETF.

	Title           : Guidelines for Considering New Performance Metric Development
	Author(s)       : A. Clark, B. Claise
	Filename        : draft-ietf-pmol-metrics-framework-08.txt
	Pages           : 22
	Date            : 2011-01-28

This document describes a framework and a process for developing
Performance Metrics of protocols and applications transported over
IETF-specified protocols, and that can be used to characterize
traffic on live networks and services.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-pmol-metrics-framework-08.txt

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

--NextPart
Content-Type: Message/External-body;
	name="draft-ietf-pmol-metrics-framework-08.txt";
	site="ftp.ietf.org"; access-type="anon-ftp";
	directory="internet-drafts"

Content-Type: text/plain
Content-ID: <[email-masked]>

--NextPart--

Content-class: urn:content-classes:message
Date: Tue, 25 Jan 2011 11:24:21 +0100
From: "Romascanu, Dan (Dan)" <[email-masked]>
To: <[email-masked]>
Subject: [PMOL] FW: Gen-ART review of draft-ietf-pmol-metrics-framework-07.txt

=20

-----Original Message-----
From: Miguel A. Garcia [mailto:[email-masked]]=20
Sent: Tuesday, January 25, 2011 11:55 AM
To: [email-masked]; [email-masked]; Al Morton; Romascanu,
Dan (Dan)
Cc: General Area Review Team
Subject: Gen-ART review of draft-ietf-pmol-metrics-framework-07.txt

I have been selected as the General Area Review Team (Gen-ART) reviewer
for this draft. For background on Gen-ART, please see the FAQ at
http://wiki.tools.ietf.org/area/gen/trac/wiki/GenArtfaq

Please resolve these comments along with any other comments you may
receive.

Document: draft-ietf-pmol-metrics-framework-07.txt
Reviewer: Miguel Garcia <[email-masked]> Review Date:
25-Jan-2011 IETF LC End Date: 27-Jan-2011

Summary: The document is ready for publication as a BCP.=09

Major issues: none

Minor issues: none

Nits/editorial comments:

- Abstract: the word "over" is duplicated.

- Section 1, first paragraph. Missing dot closing the paragraph.

- Section 3, first paragraph. I recommend to use the present tense in
the last sentence of the paragraph:

   "As such, this document does not defined any performance metrics".

- Section 5.2, first paragraph.

   s/of an networking/of a networking
        ^^

- Section 5.4.3, last paragraph. Add a formal informative reference to
RFC 3550.

- Section 5.4.5, forth paragraph. Add a formal informative reference to
RFC 3611.

- Section 5.4.5, 7th paragraph. Add a formal informative reference to
RFC 3611.

/Miguel

--=20
Miguel A. Garcia
+34-91-339-3608
Ericsson Spain

Content-class: urn:content-classes:message
Date: Fri, 21 Jan 2011 21:11:44 +0100
From: "Romascanu, Dan (Dan)" <[email-masked]>
To: <[email-masked]>
Subject: [PMOL] FW: RFC 6076 on Basic Telephony SIP End-to-End PerformanceMetrics

Congratulations and thanks to Daryl for the extraordinary tenacity and
resilience, thanks to Al and to the whole working group for contributing
and supporting this work.=20

Regards,

Dan

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
[email-masked]
Sent: Thursday, January 20, 2011 7:52 PM
To: [email-masked]; [email-masked]
Cc: [email-masked]; [email-masked]
Subject: [PMOL] RFC 6076 on Basic Telephony SIP End-to-End
PerformanceMetrics

A new Request for Comments is now available in online RFC libraries.

       =20
        RFC 6076

        Title:      Basic Telephony SIP End-to-End Performance=20
                    Metrics=20
        Author:     D. Malas, A. Morton
        Status:     Standards Track
        Stream:     IETF
        Date:       January 2011
        Mailbox:    [email-masked],=20
                    [email-masked]
        Pages:      27
        Characters: 61559
        Updates/Obsoletes/SeeAlso:   None

        I-D Tag:    draft-ietf-pmol-sip-perf-metrics-07.txt

        URL:        http://www.rfc-editor.org/rfc/rfc6076.txt

This document defines a set of metrics and their usage to evaluate the
performance of end-to-end Session Initiation Protocol (SIP) for
telephony services in both production and testing environments.  The
purpose of this document is to combine a standard set of common metrics,
allowing interoperable performance measurements, easing the comparison
of industry implementations.  [STANDARDS-TRACK]

This document is a product of the Performance Metrics for Other Layers
Working Group of the IETF.

This is now a Proposed Standard Protocol.

STANDARDS TRACK: This document specifies an Internet standards track
protocol for the Internet community,and requests discussion and
suggestions for improvements.  Please refer to the current edition of
the Internet Official Protocol Standards (STD 1) for the standardization
state and status of this protocol.  Distribution of this memo is
unlimited.

This announcement is sent to the IETF-Announce and rfc-dist lists.
To subscribe or unsubscribe, see
  http://www.ietf.org/mailman/listinfo/ietf-announce
  http://mailman.rfc-editor.org/mailman/listinfo/rfc-dist

For searching the RFC series, see
http://www.rfc-editor.org/rfcsearch.html.
For downloading RFCs, see http://www.rfc-editor.org/rfc.html.

Requests for special distribution should be addressed to either the
author of the RFC in question, or to [email-masked].  Unless
specifically noted otherwise on the RFC itself, all RFCs are for
unlimited distribution.

The RFC Editor Team
Association Management Solutions, LLC

_______________________________________________
PMOL mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/pmol

To: [email-masked], [email-masked]
From: [email-masked]
Date: Thu, 20 Jan 2011 09:51:35 -0800 (PST)
Cc: [email-masked], [email-masked]
Subject: [PMOL] RFC 6076 on Basic Telephony SIP End-to-End Performance Metrics

A new Request for Comments is now available in online RFC libraries.

        RFC 6076

        Title:      Basic Telephony SIP End-to-End Performance 
                    Metrics 
        Author:     D. Malas, A. Morton
        Status:     Standards Track
        Stream:     IETF
        Date:       January 2011
        Mailbox:    [email-masked], 
                    [email-masked]
        Pages:      27
        Characters: 61559
        Updates/Obsoletes/SeeAlso:   None

        I-D Tag:    draft-ietf-pmol-sip-perf-metrics-07.txt

        URL:        http://www.rfc-editor.org/rfc/rfc6076.txt

This document defines a set of metrics and their usage to evaluate
the performance of end-to-end Session Initiation Protocol (SIP) for
telephony services in both production and testing environments.  The
purpose of this document is to combine a standard set of common
metrics, allowing interoperable performance measurements, easing the
comparison of industry implementations.  [STANDARDS-TRACK]

This document is a product of the Performance Metrics for Other Layers Working Group of the IETF.

This is now a Proposed Standard Protocol.

STANDARDS TRACK: This document specifies an Internet standards track
protocol for the Internet community,and requests discussion and suggestions
for improvements.  Please refer to the current edition of the Internet
Official Protocol Standards (STD 1) for the standardization state and
status of this protocol.  Distribution of this memo is unlimited.

This announcement is sent to the IETF-Announce and rfc-dist lists.
To subscribe or unsubscribe, see
  http://www.ietf.org/mailman/listinfo/ietf-announce
  http://mailman.rfc-editor.org/mailman/listinfo/rfc-dist

For searching the RFC series, see http://www.rfc-editor.org/rfcsearch.html.
For downloading RFCs, see http://www.rfc-editor.org/rfc.html.

Requests for special distribution should be addressed to either the
author of the RFC in question, or to [email-masked].  Unless
specifically noted otherwise on the RFC itself, all RFCs are for
unlimited distribution.

The RFC Editor Team
Association Management Solutions, LLC

From: The IESG <[email-masked]>
To: IETF-Announce <[email-masked]>
Date: Thu, 13 Jan 2011 11:13:09 -0800
Cc: [email-masked]
Subject: [PMOL] Last Call: <draft-ietf-pmol-metrics-framework-07.txt> (Guidelines for	Considering New Performance Metric Development) to BCP

The IESG has received a request from the Performance Metrics for Other
Layers WG (pmol) to consider the following document:
- 'Guidelines for Considering New Performance Metric Development'
  <draft-ietf-pmol-metrics-framework-07.txt> as a BCP

The IESG plans to make a decision in the next few weeks, and solicits
final comments on this action. Please send substantive comments to the
[email-masked] mailing lists by 2011-01-27. Exceptionally, comments may be
sent to [email-masked] instead. In either case, please retain the
beginning of the Subject line to allow automated sorting.

The file can be obtained via
http://datatracker.ietf.org/doc/draft-ietf-pmol-metrics-framework/

IESG discussion can be tracked via
http://datatracker.ietf.org/doc/draft-ietf-pmol-metrics-framework/

No IPR declarations have been submitted directly on this I-D.

From: [email-masked]
To: [email-masked]
Date: Thu, 13 Jan 2011 09:45:01 -0800
Cc: [email-masked]
Subject: [PMOL] I-D Action:draft-ietf-pmol-metrics-framework-07.txt

--NextPart

A New Internet-Draft is available from the on-line Internet-Drafts directories.
This draft is a work item of the Performance Metrics for Other Layers Working Group of the IETF.

	Title           : Guidelines for Considering New Performance Metric Development
	Author(s)       : A. Clark, B. Claise
	Filename        : draft-ietf-pmol-metrics-framework-07.txt
	Pages           : 22
	Date            : 2011-01-13

This document describes a framework and a process for developing
performance metrics of protocols and applications transported over
over IETF-specified protocols, and that can be used to characterize
traffic on live networks and services.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-ietf-pmol-metrics-framework-07.txt

Internet-Drafts are also available by anonymous FTP at:
ftp://ftp.ietf.org/internet-drafts/

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

--NextPart
Content-Type: Message/External-body;
	name="draft-ietf-pmol-metrics-framework-07.txt";
	site="ftp.ietf.org"; access-type="anon-ftp";
	directory="internet-drafts"

Content-Type: text/plain
Content-ID: <[email-masked]>

--NextPart--

Content-class: urn:content-classes:message
Date: Thu, 13 Jan 2011 16:49:47 +0100
From: "Romascanu, Dan (Dan)" <[email-masked]>
To: "Benoit Claise" <[email-masked]>
Cc: [email-masked]
Subject: Re: [PMOL] AD review of draft-ietf-pmol-metrics-framework-06

This is a multi-part message in MIME format.

------_=_NextPart_001_01CBB339.836D75C9
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Thanks. As soon as your revised version is submitted I will issue the
IETF Last Call.=20
=20
To the question at E9 - I believe that the answer is Yes - because this
paragraph emphasizes the need of an MET-DIR (provisional name) review
same as OPS-DIR, SEC-DIR - etc. at the time of an IETF Last Call (the
latest).=20
=20
Dan
=20

________________________________

	From: Benoit Claise [mailto:[email-masked]]=20
	Sent: Thursday, January 13, 2011 2:45 PM
	To: Romascanu, Dan (Dan)
	Cc: [email-masked]
	Subject: Re: [PMOL] AD review of
draft-ietf-pmol-metrics-framework-06
=09
=09
	Thank you Dan for the detailed review.
	See inline.
	If you agree with the changes, I'll post a new draft version
right away.
=09

		Please find below the AD review of
draft-ietf-pmol-metrics-framework-06.
		This document is in good shape. There is however at
least one key issue
		which I suggest to be clarified in a revised version of
the document
		before submitting it to IETF Last Call.=20
	=09
		Meta-issue:
	=09
		I believe that the document should make the
recommendation that the
		Performance Metrics Entity more clear. For this purpose
I would
		recommend a change on the following lines in Section
1.1:=20
	=09
		OLD:=20
	=09
		   This document refers to a Performance Metrics Entity,
which will
		   become a directorate, whose goal is to coordinate the
Performance
		   Metric development at the IETF.
	=09
		NEW:=20
	=09
		   This document refers to a Performance Metrics Entity,
whose goal is=20
		   to advice and support the Performance Metric
development at the IETF.
		   A recommendation about the Performance Metrics Entity
is made in=20
		   Section 6.6.=20

	Done.
=09

		And NEW:=20
	=09
		Section 6.6
	=09
		    This document recommends that the Performance
Metrics Entity be=20
		    implemented as a directorate in one of the IETF
Areas, providing
		advice
		    and support as described in this document to all
areas in the IETF.=20

	Done. I titled this section 6.6  "Recommendations"
=09
=09
=09

		Editorial Issues:=20
	=09
		E1. I suggest to add at the end of Section 1.1 a
paragraph that lists
		the intended audience for this document including but
maybe not
		restricted to IETF participants who write performance
metrics documents
		in the IETF, reviewers of such documents and members of
the Performance
		Metrics Entity.=20

	      "The intended audience for this document includes, but is
not
	     restricted to, IETF participants who write performance
metrics documents
	     in the IETF, reviewers of such documents, and members of
the Performance
	     Metrics Entity. "
=09

	=09
		E2. A couple of bullets are duplicated in Section 5.1
(or I cannot see
		the differences)

	Done.
=09

		E3. In Section 5.2, (ii) - s/QoS G1000 [G1000]/QoS
[G1000]/

	Done.
=09

		E4. In Section 5.3.2 - provide a reference for the G.107
R Factor

	Done.
=09

		E5. Expand BOF, NAT at first occurrences

	Done.
=09

		E6. Section 5.7 - ALL is not a keyword, do not
capitalize it

	Done.
=09

		E7. Section 6.1 - 'Proposals SHOULD be vetted by the
corresponding
		protocol development WG ...' - I think that we should be
more clear here
		about what 'vetted' means. In most cases this means a
charter item
		and/or a deliverable in the WG plans

	That makes sense since we're proposing a directorate.
	I added "Proposals SHOULD be deliverables of the corresponding
protocol development WG charters. As such,... " in the text below
=09
	OLD:
=09
	   Proposals SHOULD be prepared as Internet Drafts, describing
the
	   Performance Metric and conforming to the qualifications above
as much
	   as possible.
	   Proposals SHOULD be vetted by the corresponding protocol
development
	   WG prior to discussion by the Performance Metrics Entity.
This
	   aspect of the process includes an assessment of the need for
the
	   Performance Metric proposed and assessment of the support for
their
	   development in IETF.
	NEW:
=09
=09
	   Proposals SHOULD be prepared as Internet Drafts, describing
the
	   Performance Metric and conforming to the qualifications above
as much
	   as possible.
=09
	   Proposals SHOULD be deliverables of the corresponding
protocol development
	   WG charters. As such, the Proposals SHOULD be vetted by that=20
	   WG prior to discussion by the Performance Metrics Entity.
This
	   aspect of the process includes an assessment of the need for
the
	   Performance Metric proposed and assessment of the support for
their
	   development in IETF.

		E8. There is some overlap between the 4th and 6th
paragraphs in this
		section when it comes to relating to other SDOs

	Proposal: a combination of the 4th and 6th paragraphs
=09
	OLD:
=09
	   Proposals SHOULD include an assessment of interaction and/or
overlap
	   with work in other Standards Development Organizations.
=09
	   Proposals SHOULD survey the existing standards work in the
area and
	   identify additional expertise that might be consulted, or
possible
	   overlap with other standards development orgs.
=09
	NEW:
	   Proposals SHOULD include an assessment of interaction and/or
overlap
	   with work in other Standards Development Organizations.
Proposals SHOULD=20
	   identify additional expertise that might be consulted.
=09

		E9. In section 6.4 I think that we need to mention that
a formal review
		is RECOMMENDED by the time the document is reviewed by
the Area
		Directors, or an IETF Last Call is being conducted -
same as expert
		reviews are being performed by other directorates

	Do we actually need to say this, as we wrote=20
=09
	   "Proposals SHOULD be deliverables of the corresponding
protocol development
	   WG charters"

	Regards, Benoit.
=09

		Thanks and Regards,
	=09
		Dan
	=09
	=09
		_______________________________________________
		PMOL mailing list
		[email-masked]
		https://www.ietf.org/mailman/listinfo/pmol

------_=_NextPart_001_01CBB339.836D75C9
Content-Type: text/html;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD><TITLE></TITLE>
<META content=3D"text/html; charset=3Dus-ascii" =
http-equiv=3DContent-Type>
<META name=3DGENERATOR content=3D"MSHTML 8.00.6001.18999"></HEAD>
<BODY bgColor=3D#ffffff text=3D#000000>
<DIV><SPAN class=3D792164715-13012011><FONT color=3D#0000ff size=3D2=20
face=3DArial>Thanks. As soon as your revised version is submitted I will =
issue the=20
IETF Last Call. </FONT></SPAN></DIV>
<DIV><SPAN class=3D792164715-13012011><FONT color=3D#0000ff size=3D2=20
face=3DArial></FONT></SPAN>&nbsp;</DIV>
<DIV><SPAN class=3D792164715-13012011><FONT color=3D#0000ff size=3D2 =
face=3DArial>To the=20
question at E9 - I believe that the answer is Yes - because this =
paragraph=20
emphasizes the need of an MET-DIR (provisional name) review same as =
OPS-DIR,=20
SEC-DIR - etc. at the time of an IETF Last Call (the latest).=20
</FONT></SPAN></DIV>
<DIV><SPAN class=3D792164715-13012011><FONT color=3D#0000ff size=3D2=20
face=3DArial></FONT></SPAN>&nbsp;</DIV>
<DIV><SPAN class=3D792164715-13012011><FONT color=3D#0000ff size=3D2=20
face=3DArial>Dan</FONT></SPAN></DIV>
<DIV><SPAN class=3D792164715-13012011></SPAN>&nbsp;</DIV><BR>
<BLOCKQUOTE=20
style=3D"BORDER-LEFT: #0000ff 2px solid; PADDING-LEFT: 5px; MARGIN-LEFT: =
5px; MARGIN-RIGHT: 0px">
  <DIV dir=3Dltr lang=3Den-us class=3DOutlookMessageHeader align=3Dleft>
  <HR tabIndex=3D-1>
  <FONT size=3D2 face=3DTahoma><B>From:</B> Benoit Claise =
[mailto:[email-masked]]=20
  <BR><B>Sent:</B> Thursday, January 13, 2011 2:45 PM<BR><B>To:</B> =
Romascanu,=20
  Dan (Dan)<BR><B>Cc:</B> [email-masked]<BR><B>Subject:</B> Re: [PMOL] AD =
review=20
  of draft-ietf-pmol-metrics-framework-06<BR></FONT><BR></DIV>
  <DIV></DIV>Thank you Dan for the detailed review.<BR>See inline.<BR>If =
you=20
  agree with the changes, I'll post a new draft version right away.<BR>
  <BLOCKQUOTE=20
  =
cite=3Dmid:[email-masked]=
aya.com=20
  type=3D"cite"><PRE wrap=3D"">Please find below the AD review of =
draft-ietf-pmol-metrics-framework-06.
This document is in good shape. There is however at least one key issue
which I suggest to be clarified in a revised version of the document
before submitting it to IETF Last Call.=20

Meta-issue:

I believe that the document should make the recommendation that the
Performance Metrics Entity more clear. For this purpose I would
recommend a change on the following lines in Section 1.1:=20

OLD:=20

   This document refers to a Performance Metrics Entity, which will
   become a directorate, whose goal is to coordinate the Performance
   Metric development at the IETF.

NEW:=20

   This document refers to a Performance Metrics Entity, whose goal is=20
   to advice and support the Performance Metric development at the IETF.
   A recommendation about the Performance Metrics Entity is made in=20
   Section 6.6.=20
</PRE></BLOCKQUOTE>Done.<BR>
  <BLOCKQUOTE=20
  =
cite=3Dmid:[email-masked]=
aya.com=20
  type=3D"cite"><PRE wrap=3D"">And NEW:=20

Section 6.6

    This document recommends that the Performance Metrics Entity be=20
    implemented as a directorate in one of the IETF Areas, providing
advice
    and support as described in this document to all areas in the IETF.=20
</PRE></BLOCKQUOTE>Done. I titled this section 6.6&nbsp;=20
  "Recommendations"<BR><BR><BR>
  <BLOCKQUOTE=20
  =
cite=3Dmid:[email-masked]=
aya.com=20
  type=3D"cite"><PRE wrap=3D"">Editorial Issues:=20

E1. I suggest to add at the end of Section 1.1 a paragraph that lists
the intended audience for this document including but maybe not
restricted to IETF participants who write performance metrics documents
in the IETF, reviewers of such documents and members of the Performance
Metrics Entity.=20
</PRE></BLOCKQUOTE>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; "The intended audience =
for=20
  this document includes, but is not<BR>&nbsp;&nbsp;&nbsp;&nbsp; =
restricted to,=20
  IETF participants who write performance metrics=20
  documents<BR>&nbsp;&nbsp;&nbsp;&nbsp; in the IETF, reviewers of such=20
  documents, and members of the Performance<BR>&nbsp;&nbsp;&nbsp;&nbsp; =
Metrics=20
  Entity. "<BR>
  <BLOCKQUOTE=20
  =
cite=3Dmid:[email-masked]=
aya.com=20
  type=3D"cite"><PRE wrap=3D"">
E2. A couple of bullets are duplicated in Section 5.1 (or I cannot see
the differences)
</PRE></BLOCKQUOTE>Done.<BR>
  <BLOCKQUOTE=20
  =
cite=3Dmid:[email-masked]=
aya.com=20
  type=3D"cite"><PRE wrap=3D"">E3. In Section 5.2, (ii) - s/QoS G1000 =
[G1000]/QoS [G1000]/
</PRE></BLOCKQUOTE>Done.<BR>
  <BLOCKQUOTE=20
  =
cite=3Dmid:[email-masked]=
aya.com=20
  type=3D"cite"><PRE wrap=3D"">E4. In Section 5.3.2 - provide a =
reference for the G.107 R Factor
</PRE></BLOCKQUOTE>Done.<BR>
  <BLOCKQUOTE=20
  =
cite=3Dmid:[email-masked]=
aya.com=20
  type=3D"cite"><PRE wrap=3D"">E5. Expand BOF, NAT at first occurrences
</PRE></BLOCKQUOTE>Done.<BR>
  <BLOCKQUOTE=20
  =
cite=3Dmid:[email-masked]=
aya.com=20
  type=3D"cite"><PRE wrap=3D"">E6. Section 5.7 - ALL is not a keyword, =
do not capitalize it
</PRE></BLOCKQUOTE>Done.<BR>
  <BLOCKQUOTE=20
  =
cite=3Dmid:[email-masked]=
aya.com=20
  type=3D"cite"><PRE wrap=3D"">E7. Section 6.1 - 'Proposals SHOULD be =
vetted by the corresponding
protocol development WG ...' - I think that we should be more clear here
about what 'vetted' means. In most cases this means a charter item
and/or a deliverable in the WG plans
</PRE></BLOCKQUOTE><BR>That makes sense since we're proposing a=20
  directorate.<BR>I added "Proposals SHOULD be deliverables of the =
corresponding=20
  protocol development WG charters. As such,... " in the text=20
  below<BR><BR>OLD:<BR><PRE>   Proposals SHOULD be prepared as Internet =
Drafts, describing the
   Performance Metric and conforming to the qualifications above as much
   as possible.
</PRE><PRE>   Proposals SHOULD be vetted by the corresponding protocol =
development
   WG prior to discussion by the Performance Metrics Entity.  This
   aspect of the process includes an assessment of the need for the
   Performance Metric proposed and assessment of the support for their
   development in IETF.
</PRE>NEW:<BR><BR><PRE>   Proposals SHOULD be prepared as Internet =
Drafts, describing the
   Performance Metric and conforming to the qualifications above as much
   as possible.

   <U>Proposals SHOULD be deliverables of the corresponding protocol =
development
   WG charters. As such, </U>the Proposals SHOULD be vetted <U>by that=20
   WG</U> prior to discussion by the Performance Metrics Entity.  This
   aspect of the process includes an assessment of the need for the
   Performance Metric proposed and assessment of the support for their
   development in IETF.
</PRE><BR>
  <BLOCKQUOTE=20
  =
cite=3Dmid:[email-masked]=
aya.com=20
  type=3D"cite"><PRE wrap=3D"">E8. There is some overlap between the 4th =
and 6th paragraphs in this
section when it comes to relating to other SDOs
</PRE></BLOCKQUOTE>Proposal: a combination of the 4th and 6th=20
  paragraphs<BR><BR>OLD:<BR><PRE>   Proposals SHOULD include an =
assessment of interaction and/or overlap
   with work in other Standards Development Organizations.

   Proposals SHOULD survey the existing standards work in the area and
   identify additional expertise that might be consulted, or possible
   overlap with other standards development orgs.

NEW:
   Proposals SHOULD include an assessment of interaction and/or overlap
   with work in other Standards Development Organizations.  Proposals =
SHOULD=20
   identify additional expertise that might be consulted.

</PRE>
  <BLOCKQUOTE=20
  =
cite=3Dmid:[email-masked]=
aya.com=20
  type=3D"cite"><PRE wrap=3D"">E9. In section 6.4 I think that we need =
to mention that a formal review
is RECOMMENDED by the time the document is reviewed by the Area
Directors, or an IETF Last Call is being conducted - same as expert
reviews are being performed by other directorates
</PRE></BLOCKQUOTE>Do we actually need to say this, as we wrote =
<BR><PRE>   "Proposals SHOULD be deliverables of the corresponding =
protocol development
   WG charters"</PRE><BR>Regards, Benoit.<BR>
  <BLOCKQUOTE=20
  =
cite=3Dmid:[email-masked]=
aya.com=20
  type=3D"cite"><PRE wrap=3D"">Thanks and Regards,

Dan

_______________________________________________
PMOL mailing list
<A class=3Dmoz-txt-link-abbreviated =
href=3D"mailto:[email-masked]">[email-masked]</A>
<A class=3Dmoz-txt-link-freetext =
href=3D"https://www.ietf.org/mailman/listinfo/pmol">https://www.ietf.org/=
mailman/listinfo/pmol</A>
</PRE></BLOCKQUOTE><BR></BLOCKQUOTE></BODY></HTML>

------_=_NextPart_001_01CBB339.836D75C9--

Date: Thu, 13 Jan 2011 13:44:36 +0100
From: Benoit Claise <[email-masked]>
To: "Romascanu, Dan (Dan)" <[email-masked]>
Cc: [email-masked]
Subject: Re: [PMOL] AD review of draft-ietf-pmol-metrics-framework-06

This is a multi-part message in MIME format.
--------------010600070907040005080106
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Thank you Dan for the detailed review.
See inline.
If you agree with the changes, I'll post a new draft version right away.
> Please find below the AD review of draft-ietf-pmol-metrics-framework-06.
> This document is in good shape. There is however at least one key issue
> which I suggest to be clarified in a revised version of the document
> before submitting it to IETF Last Call.
>
> Meta-issue:
>
> I believe that the document should make the recommendation that the
> Performance Metrics Entity more clear. For this purpose I would
> recommend a change on the following lines in Section 1.1:
>
> OLD:
>
>     This document refers to a Performance Metrics Entity, which will
>     become a directorate, whose goal is to coordinate the Performance
>     Metric development at the IETF.
>
> NEW:
>
>     This document refers to a Performance Metrics Entity, whose goal is
>     to advice and support the Performance Metric development at the IETF.
>     A recommendation about the Performance Metrics Entity is made in
>     Section 6.6.
Done.
> And NEW:
>
> Section 6.6
>
>      This document recommends that the Performance Metrics Entity be
>      implemented as a directorate in one of the IETF Areas, providing
> advice
>      and support as described in this document to all areas in the IETF.
Done. I titled this section 6.6  "Recommendations"

> Editorial Issues:
>
> E1. I suggest to add at the end of Section 1.1 a paragraph that lists
> the intended audience for this document including but maybe not
> restricted to IETF participants who write performance metrics documents
> in the IETF, reviewers of such documents and members of the Performance
> Metrics Entity.
       "The intended audience for this document includes, but is not
      restricted to, IETF participants who write performance metrics 
documents
      in the IETF, reviewers of such documents, and members of the 
Performance
      Metrics Entity. "
>
> E2. A couple of bullets are duplicated in Section 5.1 (or I cannot see
> the differences)
Done.
> E3. In Section 5.2, (ii) - s/QoS G1000 [G1000]/QoS [G1000]/
Done.
> E4. In Section 5.3.2 - provide a reference for the G.107 R Factor
Done.
> E5. Expand BOF, NAT at first occurrences
Done.
> E6. Section 5.7 - ALL is not a keyword, do not capitalize it
Done.
> E7. Section 6.1 - 'Proposals SHOULD be vetted by the corresponding
> protocol development WG ...' - I think that we should be more clear here
> about what 'vetted' means. In most cases this means a charter item
> and/or a deliverable in the WG plans

That makes sense since we're proposing a directorate.
I added "Proposals SHOULD be deliverables of the corresponding protocol 
development WG charters. As such,... " in the text below

OLD:

    Proposals SHOULD be prepared as Internet Drafts, describing the
    Performance Metric and conforming to the qualifications above as much
    as possible.

    Proposals SHOULD be vetted by the corresponding protocol development
    WG prior to discussion by the Performance Metrics Entity.  This
    aspect of the process includes an assessment of the need for the
    Performance Metric proposed and assessment of the support for their
    development in IETF.

NEW:

    Proposals SHOULD be prepared as Internet Drafts, describing the
    Performance Metric and conforming to the qualifications above as much
    as possible.

    _Proposals SHOULD be deliverables of the corresponding protocol development
    WG charters. As such,_the Proposals SHOULD be vetted_by that
    WG_  prior to discussion by the Performance Metrics Entity.  This
    aspect of the process includes an assessment of the need for the
    Performance Metric proposed and assessment of the support for their
    development in IETF.

> E8. There is some overlap between the 4th and 6th paragraphs in this
> section when it comes to relating to other SDOs
Proposal: a combination of the 4th and 6th paragraphs

OLD:

    Proposals SHOULD include an assessment of interaction and/or overlap
    with work in other Standards Development Organizations.

    Proposals SHOULD survey the existing standards work in the area and
    identify additional expertise that might be consulted, or possible
    overlap with other standards development orgs.

NEW:
    Proposals SHOULD include an assessment of interaction and/or overlap
    with work in other Standards Development Organizations.  Proposals SHOULD
    identify additional expertise that might be consulted.

> E9. In section 6.4 I think that we need to mention that a formal review
> is RECOMMENDED by the time the document is reviewed by the Area
> Directors, or an IETF Last Call is being conducted - same as expert
> reviews are being performed by other directorates
Do we actually need to say this, as we wrote

    "Proposals SHOULD be deliverables of the corresponding protocol development
    WG charters"

Regards, Benoit.
> Thanks and Regards,
>
> Dan
>
>
> _______________________________________________
> PMOL mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/pmol

--------------010600070907040005080106
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
  <head>
    <meta content="text/html; charset=ISO-8859-1"
      http-equiv="Content-Type">
    <title></title>
  </head>
  <body bgcolor="#ffffff" text="#000000">
    Thank you Dan for the detailed review.<br>
    See inline.<br>
    If you agree with the changes, I'll post a new draft version right
    away.<br>
    <blockquote
cite="mid:[email-masked]"
      type="cite">
      <pre wrap="">Please find below the AD review of draft-ietf-pmol-metrics-framework-06.
This document is in good shape. There is however at least one key issue
which I suggest to be clarified in a revised version of the document
before submitting it to IETF Last Call. 

Meta-issue:

I believe that the document should make the recommendation that the
Performance Metrics Entity more clear. For this purpose I would
recommend a change on the following lines in Section 1.1: 

OLD: 

   This document refers to a Performance Metrics Entity, which will
   become a directorate, whose goal is to coordinate the Performance
   Metric development at the IETF.

NEW: 

   This document refers to a Performance Metrics Entity, whose goal is 
   to advice and support the Performance Metric development at the IETF.
   A recommendation about the Performance Metrics Entity is made in 
   Section 6.6. 
</pre>
    </blockquote>
    Done.<br>
    <blockquote
cite="mid:[email-masked]"
      type="cite">
      <pre wrap="">
And NEW: 

Section 6.6

    This document recommends that the Performance Metrics Entity be 
    implemented as a directorate in one of the IETF Areas, providing
advice
    and support as described in this document to all areas in the IETF. 
</pre>
    </blockquote>
    Done. I titled this section 6.6&nbsp; "Recommendations"<br>
    <br>
    <br>
    <blockquote
cite="mid:[email-masked]"
      type="cite">
      <pre wrap="">Editorial Issues: 

E1. I suggest to add at the end of Section 1.1 a paragraph that lists
the intended audience for this document including but maybe not
restricted to IETF participants who write performance metrics documents
in the IETF, reviewers of such documents and members of the Performance
Metrics Entity. 
</pre>
    </blockquote>
    &nbsp;&nbsp;&nbsp;&nbsp;&nbsp; "The intended audience for this document includes, but is not<br>
    &nbsp;&nbsp;&nbsp;&nbsp; restricted to, IETF participants who write performance metrics
    documents<br>
    &nbsp;&nbsp;&nbsp;&nbsp; in the IETF, reviewers of such documents, and members of the
    Performance<br>
    &nbsp;&nbsp;&nbsp;&nbsp; Metrics Entity. "<br>
    <blockquote
cite="mid:[email-masked]"
      type="cite">
      <pre wrap="">

E2. A couple of bullets are duplicated in Section 5.1 (or I cannot see
the differences)
</pre>
    </blockquote>
    Done.<br>
    <blockquote
cite="mid:[email-masked]"
      type="cite">
      <pre wrap="">
E3. In Section 5.2, (ii) - s/QoS G1000 [G1000]/QoS [G1000]/
</pre>
    </blockquote>
    Done.<br>
    <blockquote
cite="mid:[email-masked]"
      type="cite">
      <pre wrap="">
E4. In Section 5.3.2 - provide a reference for the G.107 R Factor
</pre>
    </blockquote>
    Done.<br>
    <blockquote
cite="mid:[email-masked]"
      type="cite">
      <pre wrap="">
E5. Expand BOF, NAT at first occurrences
</pre>
    </blockquote>
    Done.<br>
    <blockquote
cite="mid:[email-masked]"
      type="cite">
      <pre wrap="">
E6. Section 5.7 - ALL is not a keyword, do not capitalize it
</pre>
    </blockquote>
    Done.<br>
    <blockquote
cite="mid:[email-masked]"
      type="cite">
      <pre wrap="">
E7. Section 6.1 - 'Proposals SHOULD be vetted by the corresponding
protocol development WG ...' - I think that we should be more clear here
about what 'vetted' means. In most cases this means a charter item
and/or a deliverable in the WG plans
</pre>
    </blockquote>
    <br>
    That makes sense since we're proposing a directorate.<br>
    I added "Proposals SHOULD be deliverables of the corresponding
    protocol development WG charters. As such,...
    " in the text below<br>
    <br>
    OLD:<br>
    <pre>   Proposals SHOULD be prepared as Internet Drafts, describing the
   Performance Metric and conforming to the qualifications above as much
   as possible.
</pre>
    <pre>   Proposals SHOULD be vetted by the corresponding protocol development
   WG prior to discussion by the Performance Metrics Entity.  This
   aspect of the process includes an assessment of the need for the
   Performance Metric proposed and assessment of the support for their
   development in IETF.
</pre>
    NEW:<br>
    <br>
    <pre>   Proposals SHOULD be prepared as Internet Drafts, describing the
   Performance Metric and conforming to the qualifications above as much
   as possible.

   <u>Proposals SHOULD be deliverables of the corresponding protocol development
   WG charters. As such, </u>the Proposals SHOULD be vetted <u>by that 
   WG</u> prior to discussion by the Performance Metrics Entity.  This
   aspect of the process includes an assessment of the need for the
   Performance Metric proposed and assessment of the support for their
   development in IETF.
</pre>
    <br>
    <blockquote
cite="mid:[email-masked]"
      type="cite">
      <pre wrap="">
E8. There is some overlap between the 4th and 6th paragraphs in this
section when it comes to relating to other SDOs
</pre>
    </blockquote>
    Proposal: a combination of the 4th and 6th paragraphs<br>
    <br>
    OLD:<br>
    <pre>   Proposals SHOULD include an assessment of interaction and/or overlap
   with work in other Standards Development Organizations.

   Proposals SHOULD survey the existing standards work in the area and
   identify additional expertise that might be consulted, or possible
   overlap with other standards development orgs.

NEW:
   Proposals SHOULD include an assessment of interaction and/or overlap
   with work in other Standards Development Organizations.  Proposals SHOULD 
   identify additional expertise that might be consulted.

</pre>
    <blockquote
cite="mid:[email-masked]"
      type="cite">
      <pre wrap="">
E9. In section 6.4 I think that we need to mention that a formal review
is RECOMMENDED by the time the document is reviewed by the Area
Directors, or an IETF Last Call is being conducted - same as expert
reviews are being performed by other directorates
</pre>
    </blockquote>
    Do we actually need to say this, as we wrote <br>
    <pre>   "Proposals SHOULD be deliverables of the corresponding protocol development
   WG charters"</pre>
    <br>
    Regards, Benoit.<br>
    <blockquote
cite="mid:[email-masked]"
      type="cite">
      <pre wrap="">
Thanks and Regards,

Dan

_______________________________________________
PMOL mailing list
<a class="moz-txt-link-abbreviated" href="mailto:[email-masked]">[email-masked]</a>
<a class="moz-txt-link-freetext" href="https://www.ietf.org/mailman/listinfo/pmol">https://www.ietf.org/mailman/listinfo/pmol</a>
</pre>
    </blockquote>
    <br>
  </body>
</html>

--------------010600070907040005080106--

Content-class: urn:content-classes:message
Date: Tue, 11 Jan 2011 11:29:00 +0100
From: "Romascanu, Dan (Dan)" <[email-masked]>
To: <[email-masked]>
Subject: [PMOL] AD review of draft-ietf-pmol-metrics-framework-06

Please find below the AD review of draft-ietf-pmol-metrics-framework-06.
This document is in good shape. There is however at least one key issue
which I suggest to be clarified in a revised version of the document
before submitting it to IETF Last Call.=20

Meta-issue:

I believe that the document should make the recommendation that the
Performance Metrics Entity more clear. For this purpose I would
recommend a change on the following lines in Section 1.1:=20

OLD:=20

   This document refers to a Performance Metrics Entity, which will
   become a directorate, whose goal is to coordinate the Performance
   Metric development at the IETF.

NEW:=20

   This document refers to a Performance Metrics Entity, whose goal is=20
   to advice and support the Performance Metric development at the IETF.
   A recommendation about the Performance Metrics Entity is made in=20
   Section 6.6.=20

And NEW:=20

Section 6.6

    This document recommends that the Performance Metrics Entity be=20
    implemented as a directorate in one of the IETF Areas, providing
advice
    and support as described in this document to all areas in the IETF.=20

Editorial Issues:=20

E1. I suggest to add at the end of Section 1.1 a paragraph that lists
the intended audience for this document including but maybe not
restricted to IETF participants who write performance metrics documents
in the IETF, reviewers of such documents and members of the Performance
Metrics Entity.=20

E2. A couple of bullets are duplicated in Section 5.1 (or I cannot see
the differences)

E3. In Section 5.2, (ii) - s/QoS G1000 [G1000]/QoS [G1000]/

E4. In Section 5.3.2 - provide a reference for the G.107 R Factor

E5. Expand BOF, NAT at first occurrences

E6. Section 5.7 - ALL is not a keyword, do not capitalize it

E7. Section 6.1 - 'Proposals SHOULD be vetted by the corresponding
protocol development WG ...' - I think that we should be more clear here
about what 'vetted' means. In most cases this means a charter item
and/or a deliverable in the WG plans

E8. There is some overlap between the 4th and 6th paragraphs in this
section when it comes to relating to other SDOs

E9. In section 6.4 I think that we need to mention that a formal review
is RECOMMENDED by the time the document is reviewed by the Area
Directors, or an IETF Last Call is being conducted - same as expert
reviews are being performed by other directorates

Thanks and Regards,

Dan