Subject: Re: [codec] WGLC of draft-ietf-codec-requirements-03

From: Erik Norvell <[email-masked]>
Date: Thu, 5 May 2011 17:05:34 +0200

Hi,

I agree with Paul here, setting the requirement to "no worse than" for RF codec references would be setting the bar too low. Now it looks like the codec is not required to be better than any existing RF codec.

The section on complexity requirements now specifies a number in terms of percentage of CPU power. However, it is still unclear to me how this requirement would be measured.

Best regards,
Erik 

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] 
> On Behalf Of Paul Coverdale
> Sent: den 29 april 2011 00:49
> To: 'Cullen Jennings'; [email-masked]
> Subject: Re: [codec] WGLC of draft-ietf-codec-requirements-03
> 
> Hi Cullen,
> 
> I don't recall the discussion on this point, but it seems 
> that in section 5.2, the codec quality requirement is now "no 
> worse than" the reference codecs, whereas in the previous 
> version it was "better than". I suggest that we revert to 
> "better than", otherwise what have we achieved 
> performance-wise with the new codec?
> 
> Regards,
> 
> ...Paul
> 
> 
> >-----Original Message-----
> >From: [email-masked] [mailto:[email-masked]] 
> On Behalf 
> >Of Cullen Jennings
> >Sent: Tuesday, April 26, 2011 6:29 PM
> >To: [email-masked]
> >Cc: Jonathan Rosenberg
> >Subject: [codec] WGLC of draft-ietf-codec-requirements-03
> >
> >
> >The chairs believe the bulk of the changes agreed to at the 
> last IETF 
> >have been made to draft-ietf-codec-requirements-03, all of 
> which were a 
> >consequence of the previous working group last call for this 
> document.
> >Since then, there has been a good deal of list discussion 
> centered on 
> >requirements for codec comparison, and the document update 
> on April 13 
> >includes a proposal for such comparison. In order to clearly 
> evaluate 
> >consensus on this, we would like to start a second two week WGLC of 
> >this draft. Please review the draft and if you believe any 
> changes are 
> >needed before this is ready to sent to the IESG, please 
> propose the new 
> >text you would like in the draft to the list before May 10 
> along with 
> >the reason you think the text should be changed.
> >
> >Also note that this document does NOT cover the actual test 
> plan - that 
> >is out of scope. Here, we only consider requirements that.
> >
> >Jonathan & Cullen <CODEC WG Chairs>
> >
> >
> >_______________________________________________
> >codec mailing list
> >[email-masked]
> >https://www.ietf.org/mailman/listinfo/codec
> 
> _______________________________________________
> codec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/codec
>

---

From: Cullen Jennings <[email-masked]>
Date: Wed, 18 May 2011 21:33:36 -0600

I have reviewed the WGLC comments. Given all the people that commented supported revering to the original "better than" text, I am declaring there is currently consensus for Paul's suggested changed of 

Paul Coverdale commented that:

> in section
> 5.2, the codec quality requirement is now "no worse than" the reference
> codecs, whereas in the previous version it was "better than". I suggest
> that we revert to "better than", otherwise what have we achieved

To address Erik's comment on how to measure, I'm not sure we need to cover this in the draft. Most performance monitoring tools would allow some analysis of percentage of time being used by the codec - for example gperf. I realize that how the measurements are done, and how the codec was optimized, will result in variations but as long as the results are not close to the edge of the requirement, it doubt it will be contentious of if the codec meets the requirement or not. If the editors can add some simple text to address this issues, I'd be happy to see that change but given the comments, I believe we have rough consensus to progress the draft without addressing this issue if Erik and the editors can not figure out some simple text to address his comment.  

Once the editors have submitted an updated draft, Jonathan and I will start the AD write up and send the draft to the AD to request publication.

Just as FYI for folks that have not done this before, I'll give the abbreviated list of what happens after that. The process can vary but the typical process for a draft like this goes:
1) the AD typically does some review and check they are OK with the draft - that may result in some updates. 
2) Then the AD sends it out for a two week IETF Last Call which may result in changes 
3) Then the draft gets put forward for approval by the whole IESG
4) Various IESG member may have comments of discusses on it that need to be resolved and could result in changes to the draft
5) after approval the RFC editor creates the RFC and will make changes
6) the authors get a final chance to catch any mistakes
7) it gets published as an RFC

Thanks, Cullen <CODEC WG Co-chair>

On Apr 26, 2011, at 4:29 PM, Cullen Jennings wrote:

> 
> The chairs believe the bulk of the changes agreed to at the last IETF have been made to draft-ietf-codec-requirements-03, all of which were a consequence of the previous working group last call for this document. Since then, there has been a good deal of list discussion centered on requirements for codec comparison, and the document update on April 13 includes a proposal for such comparison. In order to clearly evaluate consensus on this, we would like to start a second two week WGLC of this draft. Please review the draft and if you believe any changes are needed before this is ready to sent to the IESG, please propose the new text you would like in the draft to the list before May 10 along with the reason you think the text should be changed. 
> 
> Also note that this document does NOT cover the actual test plan - that is out of scope. Here, we only consider requirements that.
>

---

From: Erik Norvell <[email-masked]>
Date: Thu, 19 May 2011 11:20:37 +0200

Hi Cullen,

Thank you for addressing our concerns. 

When it comes to the complexity I am fine if the measurement method is not covered in this document, as long as the method is excplained somewhere and that ambiguities are resolved. For instance, if the complexity is defined as percentage load of a CPU, how it that computed for a multi-core CPU? Is it percentage per core? Are we measuring peak or average load? Will we specify a platform for this measurement? Since there are many operating modes, which mode should be used for measurements and will we include frame errors?

Best,
Erik 

> On Behalf Of Cullen Jennings
> Sent: den 19 maj 2011 05:34
> To: [email-masked]
> Cc: Jonathan Rosenberg
> I have reviewed the WGLC comments. Given all the people that 
> commented supported revering to the original "better than" 
> text, I am declaring there is currently consensus for Paul's 
> suggested changed of 
> 
> > 5.2, the codec quality requirement is now "no worse than" the 
> > reference codecs, whereas in the previous version it was "better 
> > than". I suggest that we revert to "better than", otherwise 
> what have 
> > we achieved performance-wise with the new codec?
> 
> To address Erik's comment on how to measure, I'm not sure we 
> need to cover this in the draft. Most performance monitoring 
> tools would allow some analysis of percentage of time being 
> used by the codec - for example gperf. I realize that how the 
> measurements are done, and how the codec was optimized, will 
> result in variations but as long as the results are not close 
> to the edge of the requirement, it doubt it will be 
> contentious of if the codec meets the requirement or not. If 
> the editors can add some simple text to address this issues, 
> I'd be happy to see that change but given the comments, I 
> believe we have rough consensus to progress the draft without 
> addressing this issue if Erik and the editors can not figure 
> out some simple text to address his comment.  
> 
> Once the editors have submitted an updated draft, Jonathan 
> and I will start the AD write up and send the draft to the AD 
> to request publication.
> 
> Just as FYI for folks that have not done this before, I'll 
> give the abbreviated list of what happens after that. The 
> process can vary but the typical process for a draft like this goes:
> 1) the AD typically does some review and check they are OK 
> with the draft - that may result in some updates. 
> 4) Various IESG member may have comments of discusses on it 
> that need to be resolved and could result in changes to the draft
> last IETF have been made to draft-ietf-codec-requirements-03, 
> all of which were a consequence of the previous working group 
> last call for this document. Since then, there has been a 
> good deal of list discussion centered on requirements for 
> codec comparison, and the document update on April 13 
> evaluate consensus on this, we would like to start a second 
> two week WGLC of this draft. Please review the draft and if 
> you believe any changes are needed before this is ready to 
> sent to the IESG, please propose the new text you would like 
> in the draft to the list before May 10 along with the reason 
> you think the text should be changed. 
> > 
> plan - that is out of scope. Here, we only consider requirements that.
> >