Subject: Re: [codec] Adopt draft-valin-codec-results as a working group item

From: Jonathan Rosenberg <[email-masked]>
Date: Wed, 07 Sep 2011 14:46:37 -0400

There were only two responses to this mail - one raising an issue, and 
one saying they were OK adopting this. This does not represent 
sufficient consensus to adopt this document as a working group item.

I will reiterate that, adopting this document does NOT mean that you 
believe it is complete, merely that you believe it is a good start for 
meeting our milestone.

Please, if you have an opinion one way or another, speak up.

Thanks,
Jonathan R.

On 8/22/2011 4:55 PM, Jonathan Rosenberg wrote:
> The chairs would like to take a poll for consensus to adopt:
> http://datatracker.ietf.org/doc/draft-valin-codec-results/
>
> as a working group item, matching our deliverable of a testing document
> based on our updated charter.
>
> As with any other working group item, adoption means that the group
> feels that the document is a good starting point to achieving this
> milestone.
>
> Please reply with your opinion.
>
> Thanks,
> Jonathan R.

-- 
Jonathan D. Rosenberg, Ph.D.                   SkypeID: jdrosen
Skype Chief Technology Strategist
[email-masked]                              http://www.skype.com
[email-masked]                            http://www.jdrosen.net

---

From: Jean-Marc Valin <[email-masked]>
Date: Wed, 07 Sep 2011 15:05:46 -0400

I vote for adopting this as WG item (surprise!)

     Jean-Marc

---

From: Ralph Giles <[email-masked]>
Date: Wed, 7 Sep 2011 12:12:17 -0700

I would like to adopt draft-valin-codec-results as a working group item.

 -r

---

From: Gregory Maxwell <[email-masked]>
Date: Wed, 7 Sep 2011 13:31:11 -0700

> sufficient consensus to adopt this document as a working group item.

Sounds fine to me.

---

From: Paul Coverdale <[email-masked]>
Date: Wed, 7 Sep 2011 18:43:06 -0400

As I recall the discussion in Quebec City, Christian Hoene was to be the
editor of this WG document? Has that been confirmed?

...Paul

>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]] On Behalf
>Of Jonathan Rosenberg
>Sent: Wednesday, September 07, 2011 2:47 PM
>To: [email-masked]
>Subject: Re: [codec] Adopt draft-valin-codec-results as a working group
>item
>
>_______________________________________________
>codec mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/codec

---

From: Jean-Marc Valin <[email-masked]>
Date: Wed, 07 Sep 2011 18:59:40 -0400

> editor of this WG document? Has that been confirmed?

That's also what I recall and as an original author, I'm fine with this.

     Jean-Marc

---

From: Anisse Taleb <[email-masked]>
Date: Thu, 08 Sep 2011 09:54:27 +0000

Hi all,

If I recall right, there was a consensus at the meeting in Quebec that this document will be assigned to Christian as editor and that work will progress into making this document as good and as complete as possible. Thus, IMHO, the document is a good starting point for further editing. When it comes to the contents of the document, I think what the industry would like to see is a full characterization of the Opus codec and its features as an internet codec which is a core objective of this work. On the other hand, tests performed prior to the adoption (or freeze) of the codec do not offer much in terms of characterization since they relate to a earlier version of the codec and will not accurately reflect the performance of the codec. In summary, section 2) and 3) should be removed and the focus shifted towards a more detailed section 4).

Kind regards,
/Anisse

________________________________________
From: [email-masked] [[email-masked]] on behalf of Jonathan Rosenberg [[email-masked]]
Sent: 07 September 2011 20:46
To: [email-masked]
Subject: Re: [codec] Adopt draft-valin-codec-results as a working group item

Thanks,
Jonathan R.

---

From: Ron <[email-masked]>
Date: Fri, 9 Sep 2011 07:58:42 +0930

On Thu, Sep 08, 2011 at 09:54:27AM +0000, Anisse Taleb wrote:
> If I recall right, there was a consensus at the meeting in Quebec that this
> document will be assigned to Christian as editor and that work will progress
> into making this document as good and as complete as possible. Thus, IMHO,
> the document is a good starting point for further editing. When it comes to
> the contents of the document, I think what the industry would like to see is
> a full characterization of the Opus codec and its features as an internet
> codec which is a core objective of this work.

I missed being (tele)present for the Quebec meeting, so I can't comment on
what was agreed there, but I agree this document seems like a good starting
point for a WG item, that should be added to as further test results become
available which are accepted by the group as being meaningful and valid.

> On the other hand, tests performed prior to the adoption (or freeze) of the
> codec do not offer much in terms of characterization since they relate to a
> earlier version of the codec and will not accurately reflect the performance
> of the codec.

I don't think they are being misrepresented as such though, do you?
For CELT at least, the versions of the codec that were tested are still
available, and documenting these tests provides useful historical context
for people who may wish to repeat similar tests on the final codec and
gauge the level of improvement that has been made since this group began
its work on them.

> In summary, section 2) and 3) should be removed

... so I disagree with this as a blanket statement if your only reasoning
is "this documents history, not the present".

I think possibly the SILK tests could be clarified to identify exactly
which version of SILK was used for them.  And if you have reason to doubt
the validity of individual tests for reasons other than "they are old",
then that should be considered.  But otherwise, I think these sections
provide useful historical information of both the improvements that were
made and the process of testing that guided them, and that the value and
transparency of this document would not be improved by removing them.

One of the aims of this group was to demonstrate the benefits of using
an open and evolutionary methodology to create something better than
could be achieved by a "develop behind closed doors, winner takes all
shootout" process.  I think documenting the history of testing, and the
results that were seen, is a valuable part of describing that process
and its results for the people who will come after us, and that simply
discarding it now that it has already been compiled would only do them
a disservice.

I don't think this is inconsistent with the desire that you say was
expressed in Quebec for it to be "as good and complete as possible".
Discarding these sections would only make it less complete, without
actually making it better in any way.

But, that said ...

> and the focus shifted towards a more detailed section 4).

... I agree there isn't a lot to be gained by bikeshedding over these
old tests now, unless there are serious concerns about the validity
of the process they employed, that would make them bad examples to
follow, for people who wished to do later testing.

The focus of future editing should mainly be on the relevance and
validity of any new tests that people wish to add to it as they come.

I completely agree that it's highly desirable for us to avoid misleading
"the industry" about the characteristics of the final RFC defined codec,
so results should only be added there after WG scrutiny and consensus.

Cheers,
Ron

---

From: "Dr. Christian Hoene" <[email-masked]>
Date: Fri, 09 Sep 2011 10:54:22 +0200

Hi all,

> Hi all,
>
> If I recall right, there was a consensus at the meeting in Quebec  
> that this document will be assigned to Christian as editor and that  
> work will progress into making this document as good and as complete  
> as possible. Thus, IMHO, the document is a good starting point for  
> further editing.

Jean-Marc, may you send me the XML file. I would like to start editing  
to include the comments and to add the new results by Google and us.

> When it comes to the contents of the document, I think what the  
> industry would like to see is a full characterization of the Opus  
> codec and its features as an internet codec which is a core  
> objective of this work. On the other hand, tests performed prior to  
> the adoption (or freeze) of the codec do not offer much in terms of  
> characterization since they relate to a earlier version of the codec  
> and will not accurately reflect the performance of the codec. In  
> summary, section 2) and 3) should be removed and the focus shifted  
> towards a more detailed section 4).

Yes, makes sense for the final, numerical results that will be added  
to the document. Still, one could remain the references given in  
Section 2 and 3 for further reading.

With best regards,

  Christian Hoene

---

From: Erik Norvell <[email-masked]>
Date: Fri, 9 Sep 2011 11:27:00 +0200

Hi all,

I support excluding sections 2 and 3 from the document. Not because they are old, but because the codec has changed since the tests were conducted and the results cannot represent the Opus performance. I agree the document should only contain tests considered meaningful and valid to this WG and removing these sections would increase the quality of the document.

Best regards,
Erik 

> From: [email-masked] [mailto:[email-masked]] 
> On Behalf Of Ron
> Sent: den 9 september 2011 00:29
> To: [email-masked]
> > If I recall right, there was a consensus at the meeting in 
> Quebec that 
> > this document will be assigned to Christian as editor and that work 
> > will progress into making this document as good and as complete as 
> > possible. Thus, IMHO, the document is a good starting point for 
> > further editing. When it comes to the contents of the document, I 
> > think what the industry would like to see is a full 
> characterization 
> > of the Opus codec and its features as an internet codec 
> which is a core objective of this work.
> 
> I missed being (tele)present for the Quebec meeting, so I 
> can't comment on what was agreed there, but I agree this 
> document seems like a good starting point for a WG item, that 
> should be added to as further test results become available 
> which are accepted by the group as being meaningful and valid.
> 
> > On the other hand, tests performed prior to the adoption 
> (or freeze) 
> > of the codec do not offer much in terms of characterization 
> since they 
> > relate to a earlier version of the codec and will not accurately 
> > reflect the performance of the codec.
> 
> For CELT at least, the versions of the codec that were tested 
> are still available, and documenting these tests provides 
> useful historical context for people who may wish to repeat 
> similar tests on the final codec and gauge the level of 
> improvement that has been made since this group began its 
> work on them.
> 
> ... so I disagree with this as a blanket statement if your 
> only reasoning is "this documents history, not the present".
> 
> I think possibly the SILK tests could be clarified to 
> identify exactly which version of SILK was used for them.  
> And if you have reason to doubt the validity of individual 
> tests for reasons other than "they are old", then that should 
> be considered.  But otherwise, I think these sections provide 
> useful historical information of both the improvements that 
> were made and the process of testing that guided them, and 
> that the value and transparency of this document would not be 
> improved by removing them.
> 
> One of the aims of this group was to demonstrate the benefits 
> of using an open and evolutionary methodology to create 
> something better than could be achieved by a "develop behind 
> closed doors, winner takes all shootout" process.  I think 
> documenting the history of testing, and the results that were 
> seen, is a valuable part of describing that process and its 
> results for the people who will come after us, and that 
> simply discarding it now that it has already been compiled 
> would only do them a disservice.
> 
> I don't think this is inconsistent with the desire that you 
> say was expressed in Quebec for it to be "as good and 
> complete as possible".
> Discarding these sections would only make it less complete, 
> without actually making it better in any way.
> 
> But, that said ...
> 
> ... I agree there isn't a lot to be gained by bikeshedding 
> over these old tests now, unless there are serious concerns 
> about the validity of the process they employed, that would 
> make them bad examples to follow, for people who wished to do 
> later testing.
> 
> The focus of future editing should mainly be on the relevance 
> and validity of any new tests that people wish to add to it 
> as they come.
> 
> I completely agree that it's highly desirable for us to avoid 
> misleading "the industry" about the characteristics of the 
> final RFC defined codec, so results should only be added 
> there after WG scrutiny and consensus.
> 
> Cheers,
> Ron
> 
>

---

From: Peter Saint-Andre <[email-masked]>
Date: Fri, 09 Sep 2011 09:49:00 -0600

On 9/9/11 3:27 AM, Erik Norvell wrote:
> Hi all,
> 
> I support excluding sections 2 and 3 from the document. Not because
> they are old, but because the codec has changed since the tests were
> conducted and the results cannot represent the Opus performance. I
> agree the document should only contain tests considered meaningful
> and valid to this WG and removing these sections would increase the
> quality of the document.

I suggest that Sections 2 and 3 could be moved to an Appendix. The
information presented in those sections is of interest but doesn't
deserve to be the main focus.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Jean-Marc Valin <[email-masked]>
Date: Fri, 09 Sep 2011 11:56:07 -0400

> deserve to be the main focus.

Sounds reasonable.

     Jean-Marc

---

From: Jonathan Rosenberg <[email-masked]>
Date: Mon, 12 Sep 2011 15:47:42 -0400

To be clear, assignment of editors for working group documents is done 
by the chairs.

Thanks,
Jonathan R.