Subject: Re: [codec] A concrete proposal for requirements and testing

From: "Roni Even" <[email-masked]>
Date: Tue, 5 Apr 2011 20:54:54 +0300

Hi,

I would expect that a message that in my view calls for consensus, to come
from the WG chairs. 

As for your question, saying that a presentation by one tester addresses the
requirements is not sufficient in my view. I would expect to see a document
that summarized all tests done based on a common test plan by more than one
tester. The problem is that if there is no plan to comment on and to  use
you cannot compare between different results.

I think that Paul sent an example of how to draft a plan that can be used.

As for removing GSM-FR, G.722 and Speex-UWB I am OK. As far as I remember
the meeting there was no consensus on the reference codecs.

Roni Even

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Jean-Marc Valin
Sent: Thursday, March 31, 2011 4:54 PM
To: [email-masked]
Subject: [codec] A concrete proposal for requirements and testing

Hi,

Following the meeting and post-meeting discussions about requirements and
testing, we would like to make the following proposal which addresses the
opposing views which prevented consensus in the meeting today. 

First, we propose to remove the following codecs from the requirements:

- GSM-FR, based on consensus from the list
- G.722, based on being clearly out-performed by G.722.1
- Speex-UWB, based on the fact that the author himself does not recommend it
being used :-)

We can keep the other reference codecs as minimum quality requirement and
include being no worse than AMR-NB and AMR-WB as "objectives" that are "nice
to have", but not hard requirements.

>From there and based on the listening tests presented by Jan Skoglund today,
let's see what we can already conclude and what still needs more testing:

1) The narrowband test showed that Opus had higher quality than Speex at 11
kb/s. Does anyone disagree that this is sufficient to meet the Sec 4.2
requirement of out-performing Speex in narrowband mode?

2) The narrowband test showed that Opus had higher quality at 11 kb/s than
iLBC at 15 kb/s. Does anyone disagree that this is sufficient to meet the
Sec 4.2 requirement of out-performing iLBC.

3) There have been no formal comparison with AMR-NB yet. What do you think
would be sufficient to assess the quality of Opus compared to AMR-NB?

4) The wideband test showed that Opus at 19.85 kb/s had higher quality than
Speex-WB at 24 kb/s. Does anyone disagree that this is sufficient to meet
the Sec 4.2 requirement of out-performing Speex in wideband mode?

5) The wideband test showed that Opus at 19.85 kb/s had higher quality than
G.722.1 at 24 kb/s. Does anyone disagree that this is sufficient to meet the
Sec 4.2 requirement of out-performing G.722.1?

6) The wideband test showed that Opus at 19.85 kb/s had higher quality than
AMR-WB at 19.85 kb/s. Does anyone disagree that this is sufficient to
concluded that the proposed "nice to have" objective of "no worse than
AMR-WB" is met?

7) The fullband test showed that Opus at 32 kb/s had higher quality than
G.719 at 32 kb/s. Does anyone disagree that this is sufficient to meet the
Sec 4.2 requirement of out-performing G.722.1C, considering that G.719 has
already been shown to out-perform G.722.1C

If you disagree with any of the points above -- as may very well be the case
-- please do provide a concrete test proposal that would be sufficient to
convince you.

Cheers,

    Jean-Marc and Koen

---

From: Stephen Botzko <[email-masked]>
Date: Tue, 5 Apr 2011 14:00:54 -0400

My recollection was that everyone supported removing GSM-FR.  All other
removals had strong hums on both sides, so there was clearly no consensus.

Although the anchor codecs matter, I think another topic that needs closure
are the speech conditions.  Usually there is some clean speech, but also
samples with noisy or reverberant speech.  Sometimes multiple speakers are
also tested.

I would also suggest at least one tandeming test.

Steve B.

On Tue, Apr 5, 2011 at 1:54 PM, Roni Even <[email-masked]> wrote:

> As for your question, saying that a presentation by one tester addresses
> the requirements is not sufficient in my view. I would expect to see a
> document that summarized all tests done based on a common test plan by more
> than one tester. The problem is that if there is no plan to comment on and
> to  use you cannot compare between different results.
>
> *From:* [email-masked] [mailto:[email-masked]] *On Behalf
> Of *Jean-Marc Valin
> *Sent:* Thursday, March 31, 2011 4:54 PM
> *To:* [email-masked]
> *Subject:* [codec] A concrete proposal for requirements and testing
>
>
>
> Hi,
>
> - Speex-UWB, based on the fact that the author himself does not recommend
> it being used :-)
>
> From there and based on the listening tests presented by Jan Skoglund
> today, let's see what we can already conclude and what still needs more
> testing:
>
> If you disagree with any of the points above -- as may very well be the
> case -- please do provide a concrete test proposal that would be sufficient
> to convince you.
>
> Cheers,
>
>     Jean-Marc and Koen
>
> _______________________________________________
> codec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/codec
>
>

---

From: Jean-Marc Valin <[email-masked]>
Date: Tue, 5 Apr 2011 14:15:51 -0400

Hi Roni,

Sorry if this sounded like a call for consensus since that's not what it 
was meant to be. Rather, it was merely a *proposal* to get the discussion 
started. More precisely, this is about what needs to be answered. This is 
why we listed these 7 questions in the original email. Taking just the 
first one: "Does anyone disagree that this is sufficient to meet the Sec 
4.2 requirement of out-performing Speex in narrowband mode?", I'm fine if 
you disagree, but if that's the case I'd at least know what test it would 
take to convince you.

Cheers,

	Jean-Marc

On 11-04-05 01:54 PM, Roni Even wrote:
> *From:[email-masked] [mailto:[email-masked]] *On Behalf Of
> *Jean-Marc Valin
> include being no worse than AMR-NB and AMR-WB as "objectives" that are
> "nice to have", but not hard requirements.
>
> G.722.1 at 24 kb/s. Does anyone disagree that this is sufficient to meet
> the Sec 4.2 requirement of out-performing G.722.1?
>

---

From: Jean-Marc Valin <[email-masked]>
Date: Tue, 5 Apr 2011 14:24:44 -0400

> removals had strong hums on both sides, so there was clearly no consensus.

Correct. The only other codecs that were removed (G.722 and Speex-UWB) from 
that *proposal* had no hums on them, but we expected there would be no 
objections.

> samples with noisy or reverberant speech. Sometimes multiple speakers are

Makes sense. As a note, some of the speech samples in the Google tests were 
using reverberant with (IIRC) some office noise.

> I would also suggest at least one tandeming test.

What use case would you suggest that involves tandeming?

Cheers,

	Jean-Marc

> Steve B.
>
> On Tue, Apr 5, 2011 at 1:54 PM, Roni Even <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     Hi,
>
>     I would expect that a message that in my view calls for consensus, to
>     come from the WG chairs.
>
>     As for your question, saying that a presentation by one tester
>     addresses the requirements is not sufficient in my view. I would expect
>     to see a document that summarized all tests done based on a common test
>     plan by more than one tester. The problem is that if there is no plan
>     to comment on and to use you cannot compare between different results.
>
>     As for removing GSM-FR, G.722 and Speex-UWB I am OK. As far as I
>     remember the meeting there was no consensus on the reference codecs.
>
>     Roni Even
>
>     *From:[email-masked] <mailto:[email-masked]>
>     [mailto:[email-masked] <mailto:[email-masked]>] *On
>     Behalf Of *Jean-Marc Valin
>     *To:* [email-masked] <mailto:[email-masked]>
>     Following the meeting and post-meeting discussions about requirements
>     and testing, we would like to make the following proposal which
>     addresses the opposing views which prevented consensus in the meeting
>     today.
>
>     - Speex-UWB, based on the fact that the author himself does not
>     recommend it being used :-)
>
>     We can keep the other reference codecs as minimum quality requirement
>     and include being no worse than AMR-NB and AMR-WB as "objectives" that
>     are "nice to have", but not hard requirements.
>
>     1) The narrowband test showed that Opus had higher quality than Speex
>     at 11 kb/s. Does anyone disagree that this is sufficient to meet the
>     Sec 4.2 requirement of out-performing Speex in narrowband mode?
>
>     2) The narrowband test showed that Opus had higher quality at 11 kb/s
>     than iLBC at 15 kb/s. Does anyone disagree that this is sufficient to
>     meet the Sec 4.2 requirement of out-performing iLBC.
>
>     3) There have been no formal comparison with AMR-NB yet. What do you
>     think would be sufficient to assess the quality of Opus compared to AMR-NB?
>
>     4) The wideband test showed that Opus at 19.85 kb/s had higher quality
>     than Speex-WB at 24 kb/s. Does anyone disagree that this is sufficient
>     to meet the Sec 4.2 requirement of out-performing Speex in wideband mode?
>
>     5) The wideband test showed that Opus at 19.85 kb/s had higher quality
>     than G.722.1 at 24 kb/s. Does anyone disagree that this is sufficient
>     to meet the Sec 4.2 requirement of out-performing G.722.1?
>
>     6) The wideband test showed that Opus at 19.85 kb/s had higher quality
>     than AMR-WB at 19.85 kb/s. Does anyone disagree that this is sufficient
>     to concluded that the proposed "nice to have" objective of "no worse
>     than AMR-WB" is met?
>
>     7) The fullband test showed that Opus at 32 kb/s had higher quality
>     than G.719 at 32 kb/s. Does anyone disagree that this is sufficient to
>     meet the Sec 4.2 requirement of out-performing G.722.1C, considering
>     that G.719 has already been shown to out-perform G.722.1C
>
>     case -- please do provide a concrete test proposal that would be
>     sufficient to convince you.
>
>     Cheers,
>
>     Jean-Marc and Koen
>
>
>     [email-masked] <mailto:[email-masked]>

---

From: Jan Skoglund <[email-masked]>
Date: Tue, 5 Apr 2011 11:32:01 -0700

No, we only had clean speech in our tests.

Jan

---

From: Anisse Taleb <[email-masked]>
Date: Wed, 06 Apr 2011 08:11:50 +0000

Dear Jean Marc,

Thanks for the fruitful discussions we had at the meeting. Regarding the points you raise, I don't think we can conclude on quality yet.

>1) The narrowband test showed that Opus had higher quality than Speex at 11 kb/s. Does anyone disagree that this is sufficient to meet the Sec 4.2 requirement of out-performing Speex in narrowband mode?
>2) The narrowband test showed that Opus had higher quality at 11 kb/s than iLBC at 15 kb/s. Does anyone disagree that this is sufficient to meet the Sec 4.2 requirement of out-performing iLBC.

The tests conducted by Jan were performed with clean speech only (to the best of my knowledge), in addition the methodology used (MUSHRA) is admittedly limited when it comes to testing narrowband signals and especially speech. Moreover, I am not sure as to the use of different native language listeners.  The test is useful in itself, however, more is needed in order to reach the conclusion you claim.

>3) There have been no formal comparison with AMR-NB yet. What do you think would be sufficient to assess the quality of Opus compared to AMR-NB?

Not sure what the question means, as was discussed during the meeting, I think the inclusion and comparison to state of the art codecs, including AMR-NB, is a must in order to fully assess the quality of Opus.

>4) The wideband test showed that Opus at 19.85 kb/s had higher quality than Speex-WB at 24 kb/s. Does anyone disagree that this is sufficient to meet the Sec 4.2 requirement of out-performing Speex in wideband mode?
>5) The wideband test showed that Opus at 19.85 kb/s had higher quality than G.722.1 at 24 kb/s. Does anyone disagree that this is sufficient to meet the Sec 4.2 requirement of out-performing G.722.1?
>6) The wideband test showed that Opus at 19.85 kb/s had higher quality than AMR-WB at 19.85 kb/s. Does anyone disagree that this is sufficient to concluded that the proposed "nice to have" objective of "no worse than AMR-WB" is met?

Same comment as for 1) and 2), higher quality cannot be concluded on a such a restrictive test set.

>7) The fullband test showed that Opus at 32 kb/s had higher quality than G.719 at 32 kb/s. Does anyone disagree that this is sufficient to meet the Sec 4.2 requirement of out-performing G.722.1C, considering that G.719 has already been shown to out-perform G.722.1C

Not strongly against taking such shortcuts, however, please consider that G.719 has been tested with many signals including reverberant speech, background noise conditions etc...

Kind regards,
/Anisse

From: [email-masked] [mailto:[email-masked]] On Behalf Of Jean-Marc Valin
Sent: Thursday, March 31, 2011 4:54 PM
To: [email-masked]
Subject: [codec] A concrete proposal for requirements and testing

Hi,

Following the meeting and post-meeting discussions about requirements and testing, we would like to make the following proposal which addresses the opposing views which prevented consensus in the meeting today.

- GSM-FR, based on consensus from the list
- G.722, based on being clearly out-performed by G.722.1
- Speex-UWB, based on the fact that the author himself does not recommend it being used :-)

We can keep the other reference codecs as minimum quality requirement and include being no worse than AMR-NB and AMR-WB as "objectives" that are "nice to have", but not hard requirements.

>From there and based on the listening tests presented by Jan Skoglund today, let's see what we can already conclude and what still needs more testing:

1) The narrowband test showed that Opus had higher quality than Speex at 11 kb/s. Does anyone disagree that this is sufficient to meet the Sec 4.2 requirement of out-performing Speex in narrowband mode?

2) The narrowband test showed that Opus had higher quality at 11 kb/s than iLBC at 15 kb/s. Does anyone disagree that this is sufficient to meet the Sec 4.2 requirement of out-performing iLBC.

3) There have been no formal comparison with AMR-NB yet. What do you think would be sufficient to assess the quality of Opus compared to AMR-NB?

4) The wideband test showed that Opus at 19.85 kb/s had higher quality than Speex-WB at 24 kb/s. Does anyone disagree that this is sufficient to meet the Sec 4.2 requirement of out-performing Speex in wideband mode?

5) The wideband test showed that Opus at 19.85 kb/s had higher quality than G.722.1 at 24 kb/s. Does anyone disagree that this is sufficient to meet the Sec 4.2 requirement of out-performing G.722.1?

6) The wideband test showed that Opus at 19.85 kb/s had higher quality than AMR-WB at 19.85 kb/s. Does anyone disagree that this is sufficient to concluded that the proposed "nice to have" objective of "no worse than AMR-WB" is met?

7) The fullband test showed that Opus at 32 kb/s had higher quality than G.719 at 32 kb/s. Does anyone disagree that this is sufficient to meet the Sec 4.2 requirement of out-performing G.722.1C, considering that G.719 has already been shown to out-perform G.722.1C

If you disagree with any of the points above -- as may very well be the case -- please do provide a concrete test proposal that would be sufficient to convince you.

Cheers,

    Jean-Marc and Koen

---

From: Erik Norvell <[email-masked]>
Date: Wed, 6 Apr 2011 10:39:20 +0200

+1
 
The tests presented so far serve well in aiding the development work, but they are not mature enough to support general conclusions on the Opus performance. I think the examples from Paul are a good starting point for specifying the tests.

Best regards,
Erik

________________________________

	From: [email-masked] [mailto:[email-masked]] On Behalf Of Anisse Taleb
	Sent: den 6 april 2011 10:12
	To: Jean-Marc Valin; [email-masked]
	Subject: Re: [codec] A concrete proposal for requirements and testing

	Dear Jean Marc,

	 

	 

	Kind regards,

	/Anisse

	Hi,

	

	

	

	

	
	
	Cheers,
	
	    Jean-Marc and Koen

---

From: Anisse Taleb <[email-masked]>
Date: Wed, 06 Apr 2011 08:42:54 +0000

Dear Ben,

> Let's make sure not to let these application-specific tests delay the
> standardization process, though.  With the latest test results from
> Broadcom, Hydrogen Audio, and Google, we now have strong evidence that the
> codec will exceed the working group's compression requirements.  We should
> therefore focus our efforts on finalizing the standard without delay.

Testing is a integral part of codecs standardization process and should not be considered as delaying it. I agree with you that there is an indication that the codec delivers good quality at the tested rates. Qualifying these as "Strong" is arguable: So far, only a subset of signal types have been used and verifying the requirements requires a thorough  mirrored tests that includes many signal types encountered in the foreseen applications and usecases.

Kind regards,
/Anisse

---

From: Anisse Taleb <[email-masked]>
Date: Wed, 06 Apr 2011 08:47:30 +0000

> What use case would you suggest that involves tandeming?

I can see many, for instance a wideband call to a 3G mobile network subscriber would need to either use AMR-WB or trans-code to AMR-WB.

Mvh,
/Anisse

---

From: Paul Coverdale <[email-masked]>
Date: Wed, 6 Apr 2011 09:59:01 -0400

Yes, mobile network + Opus is one likely tandemming scenario, as well as
tandemming with fixed private networks which use non-G.711 codecs.

...Paul

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Anisse Taleb
Sent: Wednesday, April 06, 2011 4:48 AM
To: Jean-Marc Valin; Stephen Botzko
Cc: [email-masked]
Subject: Re: [codec] A concrete proposal for requirements and testing

> What use case would you suggest that involves tandeming?

I can see many, for instance a wideband call to a 3G mobile network
subscriber would need to either use AMR-WB or trans-code to AMR-WB.

Mvh,
/Anisse

---

From: "Benjamin M. Schwartz" <[email-masked]>
Date: Wed, 06 Apr 2011 10:21:08 -0400

On 04/06/2011 04:42 AM, Anisse Taleb wrote:
> So far, only a subset of signal types have been used

This will always be true.  There are infinitely many potential signal
types, and even in broad categories, there are a combinatorial explosion
of use cases.  Most of them will necessarily remain untested.  Testing
all, or even most, use cases is _not_ a standardization requirement for
this working group.

The experts in audio quality testing with whom I have spoken have
consistently recommended identifying a small number of core use cases and
focusing testing effort on them.  That is exactly what we have done.

> and verifying the requirements requires a thorough  mirrored tests that includes many signal types encountered in the foreseen applications and usecases.

I am not familiar with the term "mirrored test", but I presume it refers
to performing similar experiments at multiple sites.  I don't find this
appealing, but if you do then you may volunteer to administer such a site
and list the tests you are willing to perform.

If those who are calling here for more testing do not volunteer to perform
the tests, then we will have no logical choice but to standardize with the
testing we have.  The tests so far are statistically significant and show
Opus performing excellently in a wide range of applications.

Because the Opus encoder is non-normative, further pre-standardization
testing is highly unlikely to produce results that impact the
specification.  At this point we can let the market decide, and resolve
any remaining deficits by improving the encoder after standardization.

--Ben

---

From: Jean-Marc Valin <[email-masked]>
Date: Wed, 06 Apr 2011 14:32:10 -0400

Hi Anisse,

On 11-04-06 04:11 AM, Anisse Taleb wrote:
>>1) The narrowband test showed that Opus had higher quality than Speex at
> 11 kb/s. Does anyone disagree that this is sufficient to meet the Sec
> 4.2 requirement of out-performing Speex in narrowband mode?
>
> The tests conducted by Jan were performed with clean speech only (to the
> best of my knowledge), in addition the methodology used (MUSHRA) is
> admittedly limited when it comes to testing narrowband signals and
> especially speech. Moreover, I am not sure as to the use of different
> native language listeners. The test is useful in itself, however, more
> is needed in order to reach the conclusion you claim.

Thank you for answering my questions. So I understand that what you 
would like to see is a MOS-type listening test with 40-60 listeners 
(from memory, number quoted by Paul Coverdale at the codec dinner) in 
both English and (I assume) Chinese. Is that correct? Other things that 
should be included for narrowband in your opinion?

> Not sure what the question means, as was discussed during the meeting, I
> think the inclusion and comparison to state of the art codecs, including
> AMR-NB, is a must in order to fully assess the quality of Opus.

In the original email, Koen and I were proposing to include "no worse 
than AMB-NB" as a desirable *objective* (rather than a hard 
requirement), as proposed by Paul Coverdale during the meeting.

> than G.722.1 at 24 kb/s. Does anyone disagree that this is sufficient to
> meet the Sec 4.2 requirement of out-performing G.722.1?

So for wideband, my understanding is that you'd also like to see a 
MOS-type listening test with 40-60 listeners in both English and 
Chinese. Is that correct? Anything else on your list?

> G.719 at 32 kb/s. Does anyone disagree that this is sufficient to meet
> the Sec 4.2 requirement of out-performing G.722.1C, considering that
> G.719 has already been shown to out-perform G.722.1C
>
> Not strongly against taking such shortcuts, however, please consider
> that G.719 has been tested with many signals including reverberant
> speech, background noise conditions etc…

OK. So I understand that the main issues you have are with narrowband 
and wideband, right?

Cheers,

	Jean-Marc

---

From: Paul Coverdale <[email-masked]>
Date: Wed, 6 Apr 2011 18:41:48 -0400

Hi Jean-Marc,

Just to clarify a couple of points.

When I mentioned 40-60 listeners for an MOS test, this was the number that
we used to try to use in tests in Nortel. Of course, more listeners are
better, but often there are constraints on time and money. The ITU examples
I sent earlier specify 32 listeners for the NB/WB test and 24 for the SWB
test. I would suggest 24 listeners as a minimum.

I (and others) suggested the use of objectives ("nice to have") as well as
requirements ("must have") in the requirements document, but I didn't
specifically propose "no worse than AMR-NB" (or AMR-WB) as an objective. In
fact, my preference would be to include AMR-NB and AMR-WB as requirements in
section 4.2, since there is not much else of substance there.

Regards,

...Paul

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Jean-Marc Valin
Sent: Wednesday, April 06, 2011 2:32 PM
To: Anisse Taleb
Cc: [email-masked]
Subject: Re: [codec] A concrete proposal for requirements and testing

Hi Anisse,

> speech, background noise conditions etc.

Cheers,

	Jean-Marc

---

From: Jean-Marc Valin <[email-masked]>
Date: Wed, 06 Apr 2011 21:37:10 -0400

> test. I would suggest 24 listeners as a minimum.

Thanks. Noted.

> section 4.2, since there is not much else of substance there.

I strongly disagree with including AMR-NB and AMR-WB as strict 
requirements for several reasons. Those codecs are not credible 
alternatives to Opus within the charter of this WG, to "try to avoid 
encumbered technologies."

Also, I note from the ITU test plan for G.729.1 (then G.729EV) you sent 
that it does not even mention AMR-NB. As for AMR-WB (G.722.2), it is 
only included at a bit-rate of 8.85 kb/s even though the lowest G.729.1 
bit-rate is (AFAIK) 14 kb/s. Looking at this paper from Nokia: 
http://research.nokia.com/files/public/%5B11%5D_ICASSP2010_Voice%20Quality%20Evaluation%20of%20Various%20Codecs.pdf

we see that G.729.1 is clearly worse than AMR-WB. Despite that, it seems 
like the ITU-T has seen the value of this codec, which supports both 
narrowband and wideband at multiple rates. The codec we're designing 
here is meant to do not only narrowband and wideband, but also 
super-wideband and fullband, for both speech and music and for mono and 
stereo. This is something which -- to the best of my knowledge -- has 
not been done before for an interactive codec. Requiring Opus to also 
beat all codecs out there in all circumstances is thus totally 
unreasonable (despite the fact that the tests we did so far indicate we 
may be close to that).

Cheers,

	Jean-Marc

---

From: Stephen Botzko <[email-masked]>
Date: Wed, 6 Apr 2011 21:44:26 -0400

In-line-
BR, Stephen

On Wed, Apr 6, 2011 at 4:57 PM, Jean-Marc Valin <[email-masked]> wrote:

> samples with noisy or reverberant speech.
>
>
> What's your suggestion for this? Making one set of tests on clean
> conditions and one on noisy+reverberant?
>
> Yes.

>
> Sometimes multiple speakers are also tested.
>
>
> What do you mean by "multiple speakers". All the tests we've done so far
> have used a different speaker for each sample. Or do you mean something
> else?
>
> I was meaning a test with an interfering talker (as shown on page 13 of the
"Quality Assessment Characterisation/Optimisation step1 Test Plan for
the ITU-T G.729 based ..." document that Paul Coverdale posted).  Though the
other background noise conditions on that page are also worth testing I
think.

> What form do you see this taking?
>
> You could test OPUS -> AMR-NB with G.711 -> AMR->NB as the anchor (or G.729
-> AMR-NB) with (for instance) noisy speech.
and/or a similar test using OPUS -> AMR-WB, perhaps with G.719-> AMR-WB as
the anchor.

>> 11 kb/s. Does anyone disagree that this is sufficient to meet the Sec 4.2
>> than Speex-WB at 24 kb/s. Does anyone disagree that this is sufficient to
>> meet the Sec 4.2 requirement of out-performing Speex in wideband mode?
>>
>> than AMR-WB at 19.85 kb/s. Does anyone disagree that this is sufficient to
> codec mailing [email-masked]://www.ietf.org/mailman/listinfo/codec
>
>
>

---

From: Anisse Taleb <[email-masked]>
Date: Thu, 07 Apr 2011 06:33:23 +0000

Dear JM,
> should be included for narrowband in your opinion?

The languages used for testing will depend on the labs participating in the testing,
I would not explicitly suggest English and Chinese at this point. 

For the number of listeners, we need to balance the cost of testing and the required accuracy of the test and the methodology used. I would not venture in setting the exact number of listeners, but requiring a minimum of say 24 listeners could be reasonable.
 
> requirement), as proposed by Paul Coverdale during the meeting.

My understanding of the concept of "objectives" is similar to that of a "Bonus", i.e. we require the codec to have certain "requirements" and would like, as a "bonus", to have more than just fulfilling the requirements.
As you phrased it, what would be the requirement on Opus wrt. AMR if NWT is the objective?

> Chinese. Is that correct? Anything else on your list?

Correct, also all item types should be considered, i.e.  clean, noisy, reverberant speech with and without interfering talker, music, speech and music mixed.

> > speech, background noise conditions etc...
> 

Some of the issues apply equally on all bandwidths, so the issues are not only for wideband and narrowband.

Kind regards,
/Anisse

---

From: Ron <[email-masked]>
Date: Thu, 7 Apr 2011 22:23:45 +0930

Hi,

So I must say at the outset, that I'm greatly encouraged by the renewed
vigour and interest we've seen here in the last few weeks.  Though much
work has gone on in the background between the active developers, we
haven't really seen such a flurry of interested contributions, since,
well, since many of these same people first tried to veto the formation
of this group because they didn't believe that we could do it.  :>

I'm grateful to them for sticking with us, and helping us to sanity check
that really, we actually have.  Beyond the wildest expectations of both
the 'for' and 'against' crowds it would seem so far.

Good engineering is about finding faults that can be fixed.  If no other
thing binds us to a common purpose, I think that we can all agree on.
And faults are faults, whether pointed out by friend or foe, so to speak.

What I'd really like to point out at this stage though, is that there are
some very fundamental differences between the way that a collaborative
group such as this one tends to work, as compared to a competitive group
such as some other SDOs seem to prefer.

When you have a relatively closed organisation, comprised of companies
that each would like to have a monopoly hold over some technology, and
which aren't particularly interested in sharing their secrets with others
except when it is to their direct advantage over them, then it does seem
logical to lay down some rules in advance, let each group work in isolation
and then have a shootout at the end to see who "wins".

What we have here though, is both in-principle, and as we've seen, in
practice, quite different to that.  Instead of taking a dog-eat-dog
approach, all of the people with real technology to contribute have
instead banded together to create a single solution which is better
than what any of them initially had to offer on their own.

Even the testing that has occurred to date has been of quite a different
flavour to what many of the currently active voices here would probably
be used to.  When you have a competitive process, each group is naturally
going to try to advocate the tests for which their particular technology
is known (by them) to be superior at.

Much of the testing I've observed for Opus however has been quite the
opposite.  The developers have actively sought out the tests for which
the codec sucks the worst at, and compared it to the results of codecs
which outperform it (or should do given their relative constraints),
in order to find the things with the greatest scope for improvement.

We've already seen a number of published tests.  And we've in turn seen
some people challenge the validity of those tests.  What we haven't seen
so far is any tests which prove the validity of those challenges.

This is an open process.  Anyone is free to test anything they wish and
provide feedback to the group as to their findings.  So far the people
who have done tests seem to have indicated that they are satisfied with
the results they have seen, and have neither suggestions for things that
need to improve further, nor plans for further tests of their own, that
they haven't already shared.

Roni hinted at the Prague meeting that certain "internal testing" had
taken place which he was aware of.  That we haven't seen the results of
those tests I can only assume means they paint us in as favourable a
light as the disclosed tests have.

Stephan intimated that he was "not trying to obstruct this process,
anymore".  From which I wonder if he also has seen tests that prove
there is something valuable in this for his company too (but I won't
speculate, beyond inviting him to explain for himself the reason(s)
for this change of heart :)

On Wed, Apr 06, 2011 at 10:39:20AM +0200, Erik Norvell wrote:
> The tests presented so far serve well in aiding the development work, but
> they are not mature enough to support general conclusions on the Opus
> performance. I think the examples from Paul are a good starting point for
> specifying the tests.

I think that's a fair statement to make, but if Erik and Paul have
doubts they want allayed, then I don't think it's really within the
power of any of us to help them to their own conclusions, except to
invite them to present their own testing that they do find satisfactory
and have it peer reviewed here like the rest of the tests to date.

So I'd like to suggest something like the following:

Let's give people a week or so to formally propose the test plans which
*they* intend to conduct, along with a time when they plan to have them
completed.  Others can give input as to how the tests may be improved.
Then bring those results back to the group for an open discussion of
their relevance to assessing the codec.

Based on those results, we can then assess if the codec needs further work
or whether it is sufficient to release as is, for wider evaluation in the
real world roles that we're all waiting to deploy it in.

I think getting bogged down in devising an intractable number of tests
that nobody here is actually proposing to perform, isn't advancing the
goals of the WG.  Nobody doubts we'll beat G.711, so proposing to test
against it is pointless.  Likewise testing against other codecs that
*cannot* fill the requirements the WG initially set out, may be good
for bragging rights, but offers little or nothing in the way of a
meaningful and useful comparison.

The kind of tests and minimum performance requirements that we had
envisaged at the outset, did not forsee that we'd be outperforming
HE-AAC on an unconstrained listening test.  We certainly weren't when
the WG was first formed.  In the light of that result, and others like
it, it doesn't seem unreasonable at all to not waste time testing it
against lesser codecs from a former state of the art.  But if someone
wants to do those tests themselves and report on the results, then by
all means, please do.  Just don't expect us to wait for you if you
take a really long time to do that.

So please, let's avoid the shootout mentality.  We have a candidate
codec.  Go test it.  If you find something wrong, then please tell us
while there is still time to fix it.  If you don't, let's move on and
ship it.  The Real testing won't actually begin until we take that
next step into the real world, and no contrived test can model that
in a way that will satisfy everyone.  So throw your worst at it now,
because if you can't break it, then claims the LC is premature will
be rather hard to sustain for very much longer.

If you made it this far,
Thanks :)

Ron

---

From: Paul Coverdale <[email-masked]>
Date: Thu, 7 Apr 2011 09:33:37 -0400

Hi Marc,

I think that the issues of performance requirements and encumbered
technologies are separate. In fact, the WG charter states "The working group
cannot explicitly rule out the possibility of adopting encumbered
technologies". At the end of the day, people will make their own trade-off
between Opus performance and whether it is encumbered or not. So the
requirements need to include some reasonable performance targets, and AMR-NB
and AMR-WB provide those.

In any case, the test plans that I sent out earlier are examples, and need
to be viewed in relation to the terms of reference that were agreed for
G.729EV. The intention at the time was to provide scalable wideband on top
of G.729, so for NB the main comparison was with G.729A. For WB, the chosen
comparison was primarily G.722, with some G.722.2.

Regards,

...Paul

-----Original Message-----
From: Jean-Marc Valin [mailto:[email-masked]] 
Sent: Wednesday, April 06, 2011 9:37 PM
To: Paul Coverdale
Cc: 'Anisse Taleb'; [email-masked]
Subject: Re: [codec] A concrete proposal for requirements and testing

Also, I note from the ITU test plan for G.729.1 (then G.729EV) you sent 
that it does not even mention AMR-NB. As for AMR-WB (G.722.2), it is 
only included at a bit-rate of 8.85 kb/s even though the lowest G.729.1 
bit-rate is (AFAIK) 14 kb/s. Looking at this paper from Nokia: 
http://research.nokia.com/files/public/%5B11%5D_ICASSP2010_Voice%20Quality%2
0Evaluation%20of%20Various%20Codecs.pdf

Cheers,

	Jean-Marc

---

From: Stephen Botzko <[email-masked]>
Date: Thu, 7 Apr 2011 11:46:05 -0400

Hi Ron

This is not a debugging task, it is a codec characterization/quality
assessment that will be done on the finished product.

In my view, the proper way to go about it is to first get consensus on the
tests that need to be run, what results are needed, and how the tests need
to be conducted in order to meaningfully compare the results.  Then get
folks signed up to actually do the tests.

This is not about collaboration vs competition, rather it is about running
distributed tests with results that can integrated and scientifically
analyzed/compared.  And recording the test methods, in order to allow other
people in the future to re-do the tests and duplicate the results.

Regards,
Stephen Botzko

---

From: Ron <[email-masked]>
Date: Fri, 8 Apr 2011 02:18:17 +0930

Hi Stephen,

> assessment that will be done on the finished product.

Yes, I quite understand the formal, dare I say bureaucratic, process that
is being proposed here.  And whatever you call it, I do wholly respect the
scientific method that underpins it (and that was used to create and tune
this codec for that matter :).

But I like reality too.  And the reality is:  Have you listened to Opus?

Is there actually any doubt in your mind that this is not a wholly adequate
(to understate the case) codec for the purpose proposed, with a few minor
nits still to be worked out?

Since we don't actually have another candidate proposed for the WG to
endorse, what exactly would we be shooting off against?  The quality of
the codec seems among the least of our worries on the present facts.
We only have to be better than iLBC to be better than the IETF's current
best of breed, and nobody has suggested setting the bar that low.

Which isn't to say that I'm against further quality testing.  But I think
we have enough data to comfortably say it's Good Enough.  Better Than Most
even.  I don't think this should be a blocker for the group moving forward
to assess its other aims and publish the decoder spec.

Quality improvements can still be made in the encoder after that.

> folks signed up to actually do the tests.

I think you've just described a near impossible task.  This group is far
too disparate to share a tractable common set of requirements.
Interested parties should test their own use cases - and in the event of
trouble, report on those tests with the scientifically reproducible detail
that you describe.  So others can analyse them, and devise solutions.

We can equally build consensus on which of those reports are significant
and which are 'outliers' to the group charter.

> people in the future to re-do the tests and duplicate the results.

Well I'd call that collaboration, but I'm not going to compete over what
to call it.  My point is, doing that stuff is great.  And doing even more
of it is greater.  But I don't think it needs to hold us up from moving on
given the results we already have.

What test would Opus have to not pass for you to think it is unsuitable?

Best,
Ron

---

From: Anisse Taleb <[email-masked]>
Date: Thu, 07 Apr 2011 18:05:16 +0000

Dear Ron,

There are many individuals and organizations that may be willing to help the testing effort on a voluntary basis provided there is a test plan that is agreed and represents a consensus of the group of what is to be tested.  
The results we have are incomplete, there are many cases which needs to be tested and we are trying to build consensus over those and derive a test plan that can be agreed and be conducted by, hopefully, more than one organization.

Undermining this WG testing effort doesn't encourage organizations to chip in their resources and technical expertise to make this work go smoothly and according to plan. 

Kind regards,
/Anisse

> Sent: Thursday, April 07, 2011 6:48 PM
> To: Stephen Botzko
> Cc: [email-masked]

---

From: Paul Coverdale <[email-masked]>
Date: Thu, 7 Apr 2011 14:13:53 -0400

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Ron
Sent: Thursday, April 07, 2011 12:48 PM
To: Stephen Botzko
Cc: [email-masked]
Subject: Re: [codec] A concrete proposal for requirements and testing

>nits still to be worked out?

It doesn't matter what any single individual on this mailing list thinks
about Opus quality. What matters is the collective opinion of the population
of naïve users who may listen to speech/music over Opus in the future, and
how they rate it compared to listening to speech/music via other codecs (not
that they will know/care what a codec is...). A well-designed test plan and
statistical analysis can give a good estimate of this opinion.

...Paul

---

From: Peter Saint-Andre <[email-masked]>
Date: Thu, 07 Apr 2011 12:36:51 -0600

> statistical analysis can give a good estimate of this opinion.

I completely agree. That's why it is so important for us to publish
draft-ietf-codec-opus as a Proposed Standard.

The IETF tradition is one of "rough consensus and running code". What
this means is that we work hard on a new technology and publish it as an
RFC ("request for comments"). This is the rough consensus part.

Then we implement it and deploy it. Testing is something that happens
naturally when we run a technology on networks. This is the "running
code" part.

Implementation and deployment experience can lead to revisions and
refinements and better specification of the technology. Of course we'll
also seek rough consensus on any modifications, but it's not necessary,
or even desirable, to get that implementation and deployment experience
before we publish a specification as a Proposed Standard RFC.

Indeed, much discussion has occurred of late within the IETF about
simplifying the Internet Standards Process by making it easier, not
harder, to advance Internet-Drafts to Proposed Standard:

https://datatracker.ietf.org/doc/draft-housley-two-maturity-levels/

Let's get to rough consensus and then start running this code on the
network. That will be the true test.

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: Stephen Botzko <[email-masked]>
Date: Thu, 7 Apr 2011 14:49:17 -0400

Early on there was consensus on the need for codec characterization.

I see no reason to have that discussion again, I suggest it would be better
to get back on track and work through what a sensible test plan would
include.

Stephen Botzko

---

From: Jean-Marc Valin <[email-masked]>
Date: Thu, 7 Apr 2011 14:50:16 -0400

Hi Peter,

I totally agree with this view. In fact, this technology is currently being 
used by hundreds of millions of Skype users who are currently using SILK 
and hundreds of thousands of current CELT users. Making this codec advance 
to Proposed Standard should enable even more such testing. I don't mean to 
criticize the ITU-T methodology (the ITU's needs are different from the 
IETF's), but my understanding is that ITU codecs usually get nowhere near 
the amount of "in the field" testing that we've had so far.

Cheers,

	Jean-Marc

On 11-04-07 02:36 PM, Peter Saint-Andre wrote:
>> of naÃ¯ve users who may listen to speech/music over Opus in the future, and

---

From: Peter Saint-Andre <[email-masked]>
Date: Thu, 07 Apr 2011 13:06:09 -0600

:)

A: Because it messes up the order in which people normally read text.

Q: Why is top-posting so bad?

A: Top-posting.

Q: What is the most annoying behavior on email discussion lists?

> Early on there was consensus on the need for codec characterization.

(1) It's not clear to me that there is yet WG consensus about the need
for codec characterization. The topic is discussed in Section 3 of
draft-ietf-codec-guidelines-00 but the working group has not finalized
that document yet.

(2) Could you please explain what you mean by "codec characterization",
if it is different from what's in the guidelines document?

> I see no reason to have that discussion again, I suggest it would be
> better to get back on track and work through what a sensible test plan
> would include.

Although I think complete characterization and detailed testing might be
helpful at some point, I think the right time to do that is after the
RFC is published. This is completely consistent with how we test any
other specification we publish. At least within the Internet Standards
Process, I see no reason to set the bar higher on the specification of a
codec than on the specification of a protocol like SIP or XMPP or whatever.

Please see Section 4.1.1. of RFC 2026, which describes the expected
characteristics of specifications at the level of Proposed Standard
within the Internet Standards Process. The text, in full, is:

###

   The entry-level maturity for the standards track is "Proposed
   Standard".  A specific action by the IESG is required to move a
   specification onto the standards track at the "Proposed Standard"
   level.

   A Proposed Standard specification is generally stable, has resolved
   known design choices, is believed to be well-understood, has received
   significant community review, and appears to enjoy enough community
   interest to be considered valuable.  However, further experience
   might result in a change or even retraction of the specification
   before it advances.

   Usually, neither implementation nor operational experience is
   required for the designation of a specification as a Proposed
   Standard.  However, such experience is highly desirable, and will
   usually represent a strong argument in favor of a Proposed Standard
   designation.

   The IESG may require implementation and/or operational experience
   prior to granting Proposed Standard status to a specification that
   materially affects the core Internet protocols or that specifies
   behavior that may have significant operational impact on the
   Internet.

   A Proposed Standard should have no known technical omissions with
   respect to the requirements placed upon it.  However, the IESG may
   waive this requirement in order to allow a specification to advance
   to the Proposed Standard state when it is considered to be useful and
   necessary (and timely) even with known technical omissions.

   Implementors should treat Proposed Standards as immature
   specifications.  It is desirable to implement them in order to gain
   experience and to validate, test, and clarify the specification.
   However, since the content of Proposed Standards may be changed if
   problems are found or better solutions are identified, deploying
   implementations of such standards into a disruption-sensitive
   environment is not recommended.

###

Peter

-- 
Peter Saint-Andre
https://stpeter.im/

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Thu, 07 Apr 2011 12:13:00 -0700

> technologies are separate. In fact, the WG charter states "The working group

I strongly disagree with this. Groups like my employer, Mozilla, are 
actually interested in deploying this codec on the internet (you know, 
the purpose for which this working group was formed), in things like 
RTC-Web. Royalty-free is a requirement for us, as well as for the W3C, 
where part of that work is being done. The alternatives are things like 
Speex, iLBC, and G.722, not AMR-NB or AMR-WB (see, for example, 
https://bugzilla.mozilla.org/show_bug.cgi?id=476752#c20).

> and AMR-WB provide those.

This is pure fallacy. You don't need to quote the charter to know that 
the structure of the patent system makes any guarantees impossible. Yet 
if Opus were eventually found to be encumbered, we still couldn't fall 
back to AMR-NB or AMR-WB. If you don't care about such encumbrances, 
then there are plenty of good alternatives, but the lack of them for 
those who do care is one of the reasons this WG was formed. The charter 
states this explicitly.

The charter also mentions the possibility of co-publication of the 
completed codec by the ITU. Now, if you wanted to make such tests a 
requirement for co-publication, that would make perfect sense, but I 
don't believe it makes sense to prevent publication of the codec by this 
WG, regardless of their outcome.

---

From: Stephan Wenger <[email-masked]>
Date: Thu, 07 Apr 2011 12:24:05 -0700

At this point, there is no indication that opus is going to be royalty
free.  In fact, there is evidence to the contrary:
https://datatracker.ietf.org/ipr/1520/

Stephan

---

From: Monty Montgomery <[email-masked]>
Date: Thu, 7 Apr 2011 16:11:27 -0400

On Thu, Apr 7, 2011 at 3:24 PM, Stephan Wenger <[email-masked]> wrote:
> free.

Those most concerned with delivering royalty free status would appear to be
no more concerned than at any earlier point in the process.

>   In fact, there is evidence to the contrary:

How so?  This is evidence that things are working properly and as hoped
(and kudos to Qualcomm for acting in spirit).

With patent numbers of concern in hand, we can make certain those
patents either do not read on a final standard or are freely
licensed if used.  IPR declarations greatly enhance the certainty
of the project, they do not detract.

Monty
Xiph.Org

---

From: "Timothy B. Terriberry" <[email-masked]>
Date: Thu, 07 Apr 2011 13:19:49 -0700

> On Thu, Apr 7, 2011 at 3:24 PM, Stephan Wenger<[email-masked]>  wrote:

> of the project, they do not detract.

In addition to "what he said": if you read what I wrote, you'll see that 
your claim is irrelevant to my actual point.

---

From: Stephan Wenger <[email-masked]>
Date: Thu, 07 Apr 2011 13:46:23 -0700

Let me try to be more precise (and, unfortunately, more verbose).

The context of my statement, and yours (I believe) is the possible testing
of opus against other codecs.  Some of us believe that the selection of
the codecs tested against opus ought to be independent from their
licensing scheme.  My understanding is that you (and others) advocate
testing should focus on codecs known or believed as royalty free.  I don't
agree with this, but let's assume it anyway in a thought experiment: let's
test only codecs with comparable licensing schemes.

There are some codecs that are believed to be out of patent (1), others
are believed to be RF because the known rightholders (based on disclosures
received in the IETF, ITU, or elsewhere) have stated in public that they
do not demand royalties (2).  Still others never have received formal
standardization in an organization with a disclosure obligation, so one
cannot be as confident about their encumbrance or lack thereof; but there
may be statements from the lead development organization that they are not
aware of patents not available under RF terms (3).  The assorted open
source codecs and Silk appear to belong into this category.

According to Qualcomm, they will make six patents they believe read on
opus under terms that are not necessarily royalty free ("with possibly
royalty/fee").  Therefore, opus does not fit any of the three categories
mentioned above.  Regardless of the intentions of the proponents here, at
this point opus v5 cannot reasonably perceived to be royalty free.

The net result appears to me to be that requiring opus v5 to be tested
against only RF technologies seems unfair, as it tests technologies
believed to be RF (with all the technology drawbacks that may include)
against a technology that has six granted US patents (and a number of
international counterparts) disclosed against it, all of which may not be
available under RF terms.

So try your design around, hope that Qualcomm is amicable to submit an
updated disclosure against your (hopefully clean) solution, and only then
propose to test only against RF technologies.  Before that, compete fairly.

Stephan

---

From: Gregory Maxwell <[email-masked]>
Date: Thu, 7 Apr 2011 13:57:56 -0700

> before we publish a specification as a Proposed Standard RFC.
[snip]
> network. That will be the true test.

Peter's message succinctly describes much of what I've been thinking but having a difficult time expressing clearly.
Thank you.

High effort post-hoc characterization is very useful for "marketing" and somewhat useful for "best practices", but it doesn't appear to be very useful at driving forward a collaborative development process.  We've seen this already: by the time the developers first saw Broadcom's listening test results, we'd already significantly improved the codec in at least one of the two cases where their test showed lower performance,  mostly based on the feedback of some people on hydrogenaudio trying out the codec and reporting weaknesses.  Likewise, we've received considerable computational complexity and memory usage feedback from people trying to implement the codec on _real_ systems and encountering _real_ difficulties, and I think that this is far more useful than anything we could hope to achieve from a prolonged debate about abstract WMOPS figures.

In order to expand the testing further we need to move beyond testing only parts, and we need to start creating some interoperability rather than having everyone using different incompatible development bitstreams.

> Undermining this WG testing effort doesn't encourage organizations to chip in their resources and technical expertise to make this work go smoothly and according to plan.

Many people here are not especially picky about what testing is performed by other people.  Personally, I invite all comers to perform whatever testing they find interesting and helpful for their own purposes.  I'm sure the working group—as a whole—will find almost any results interesting.  Different groups have different requirements.  For example, some people are interested in Opus' performance in non-interactive applications. These are outside of the scope of the working group, but the results are still interesting.  Other people care about low bitrate performance, which I think is of less interest to many people here, or stereo support, which is interesting to a different and mostly non-overlapping set of people.  I expect that many groups will do the characterization which they find to be most applicable to their interests, and they'll report back to the working group.  I do not see why much consensus is required in advance of this kind of testing—except on the basis of ensuring basic scientific method (e.g. a non-blind test would likely be uninteresting to everyone), which we would probably find that we already agree on without even having a discussion.

What I find more concerning is this late in the process effort to raise the _requirements_: from requiring the WG product to be better than the nearest alternatives according to the WG charter (royalty free-ish things), to requiring it to be better than "best of breed" royalty-bearing codecs in each of the several broadly defined operating areas.  I believe this change is contrary to the intended purpose of the working group, as the charter acknowledges the existence of alternative standards which do not meet the intended licensing goals several times but never proposes the our output must have higher listening test performance then these codecs. Accepting this requirement risks the paradoxical outcome of being unable to finish a codec even if we have one which meets the explicit purpose of the WG's creation. Not being able to _guarantee_ a result, as a reality of the law, is not the same as willfully failing to achieve that result— so I don't find the argument that we may fail compelling.  The charter is abundantly clear about the purpose of the working group and I think it doesn't make any sense to change from optimizing for cases where the WG's purpose is successful to optimizing for cases where the working group has failed especially when such an optimization would increase the chance of failure!

In particular, I think the initial requirements were establishing a level below which collaborative development wouldn't work. If we put out something which underperformed Speex, for example, few people would deploy it and we'd gain little operational feedback from which we could further improve the codec and/or implementations. But at least beating Speex would ensure rapid deployment from at least the segment which is already using Speex and provide the operational feedback required to collaboratively advance the codec, as already we've seen from some users who were using Speex and who could tolerate a non-frozen non-interoperable codec.

By conflating two issues—characterization targets and requirements—we've taken what should be two fairly uncontroversial issues and converted it into a contentious one.  The testing should be uncontroversial because absent a requirement gate the working group will probably support whatever testing the participants find interesting and are able to perform, and the requirements issue should be uncontroversial because there are relatively few codecs which come anywhere near the WG's stated purpose.

---

From: Monty Montgomery <[email-masked]>
Date: Thu, 7 Apr 2011 17:20:23 -0400

> but let's assume it anyway in a thought experiment: let's
[...]
> propose to test only against RF technologies.  Before that, compete fairly.

This argument is based upon and presupposes WG failure.  If we do not
presuppose
failure, the argument is specious at best.

It also assumes prima facie applicability of the Qualcomm IPR
submission.  Please
don't assume my current lack of comment grants you this point.

Monty
Xiph.Org

---

From: Koen Vos <[email-masked]>
Date: Fri, 8 Apr 2011 02:24:02 +0200 (CEST)

Stephen Botzko wrote: 
> This is not a debugging task, it is a codec characterization/quality assessment that will be done on the finished product. 

Does such testing have a realistic chance of revealing something that should alter the course of the WG? 
If not, then the testing can be done separately. 
I f yes: how precisely? 

Is the concern that the codec may not be good enough to be worth publishing, given the other codecs already out there? 
--> In the tests so far, Opus has consistently outperformed iLBC, Speex, G.722.1, etc. To me it seems not realistic to expect that pattern to reverse with more or "better" testing. 

Or is the concern that for certain input signals or network conditions the codec produces results that are somehow unacceptable? 
--> Opus has gone through all kinds of testing during development. Feel free to do some more, you don't need test plans or consensus for that. Remember that there will always be untested cases (speech with dog barking in background over dial up hasn't been tried yet I believe), and that's ok. 

Characterization may be useful, but I don't see why it should be a deliverable of the WG. 

best, 
koen. 

----- Original Message -----
From: "Stephen Botzko" <[email-masked]> 
To: "Ron" <[email-masked]> 
Cc: [email-masked] 
Sent: Thursday, April 7, 2011 8:46:05 AM 
Subject: Re: [codec] A concrete proposal for requirements and testing 

Hi Ron 

This is not a debugging task, it is a codec characterization/quality assessment that will be done on the finished product. 

In my view, the proper way to go about it is to first get consensus on the tests that need to be run, what results are needed, and how the tests need to be conducted in order to meaningfully compare the results. Then get folks signed up to actually do the tests. 

This is not about collaboration vs competition, rather it is about running distributed tests with results that can integrated and scientifically analyzed/compared. And recording the test methods, in order to allow other people in the future to re-do the tests and duplicate the results. 

Regards, 
Stephen Botzko 

On Thu, Apr 7, 2011 at 8:53 AM, Ron < [email-masked] > wrote: 

Hi, 

If you made it this far, 
Thanks :) 

Ron

---

From: Roman Shpount <[email-masked]>
Date: Thu, 7 Apr 2011 20:50:06 -0400

Quoting Nokia's paper: "Codecs done without thorough standardization effort
like Speex and iLBC offer significantly reduced efficiency, probably due to
much lesser optimization, *listening tests* and IPR free design." Testing is
an important part of development and standardization process for a CODEC.

It is in everybody's interest to produce the best CODEC possible. Comments
that "we listen to it and we like it" and "you are free to run your own
tests", even though valid, are not very productive. I think the best course
possible is to treat testing the same way as we do development. We need to
collaborate in putting together a comprehensive test plan, and then in
actual testing. I am not sure I can contribute a lot in creating a test
plan, but I can definitely contribute in testing if such test plan exists. I
think this is a common situation for a lot of people in this group.
_____________
Roman Shpount

On Thu, Apr 7, 2011 at 8:24 PM, Koen Vos <[email-masked]> wrote:

> If yes: how precisely?
> Is the concern that the codec may not be good enough to be worth
> publishing, given the other codecs already out there?
> --> In the tests so far, Opus has consistently outperformed iLBC, Speex,
> G.722.1, etc.  To me it seems not realistic to expect that pattern to
> reverse with more or "better" testing.
>
> Or is the concern that for certain input signals or network conditions the
> codec produces results that are somehow unacceptable?
> --> Opus has gone through all kinds of testing during development.  Feel
> free to do some more, you don't need test plans or consensus for that.
> Remember that there will always be untested cases (speech with dog barking
> in background over dial up hasn't been tried yet I believe), and that's ok.
>
>
> ------------------------------
> *From: *"Stephen Botzko" <[email-masked]>
> *To: *"Ron" <[email-masked]>
> *Cc: [email-masked]
> *Sent: *Thursday, April 7, 2011 8:46:05 AM
>
> *Subject: *Re: [codec] A concrete proposal for requirements and testing
>
> Hi Ron
>

---

From: Koen Vos <[email-masked]>
Date: Fri, 8 Apr 2011 02:57:39 +0200 (CEST)

Roman: 

How could what Stephen Botzko describes as "a codec characterization/quality assessment that will be done on the finished product " possibly improve coding efficiency? 

best, 
koen. 

----- Original Message -----
From: "Roman Shpount" <[email-masked]> 
To: "Koen Vos" <[email-masked]> 
Cc: "Stephen Botzko" <[email-masked]>, [email-masked] 
Sent: Thursday, April 7, 2011 5:50:06 PM 
Subject: Re: [codec] A concrete proposal for requirements and testing 

Quoting Nokia's paper: "Codecs done without thorough standardization effort like Speex and iLBC offer significantly reduced efficiency, probably due to much lesser optimization, listening tests and IPR free design." Testing is an important part of development and standardization process for a CODEC. 

It is in everybody's interest to produce the best CODEC possible. Comments that "we listen to it and we like it" and "you are free to run your own tests", even though valid, are not very productive. I think the best course possible is to treat testing the same way as we do development. We need to collaborate in putting together a comprehensive test plan, and then in actual testing. I am not sure I can contribute a lot in creating a test plan, but I can definitely contribute in testing if such test plan exists. I think this is a common situation for a lot of people in this group. 
_____________ 
Roman Shpount 

On Thu, Apr 7, 2011 at 8:24 PM, Koen Vos < [email-masked] > wrote: 

best, 
koen. 

From: "Stephen Botzko" < [email-masked] > 
To: "Ron" < [email-masked] > 
Cc: [email-masked] 
Sent: Thursday, April 7, 2011 8:46:05 AM 

Hi Ron 

Regards, 
Stephen Botzko 

Hi, 

If you made it this far, 
Thanks :) 

Ron

---

From: "Benjamin M. Schwartz" <[email-masked]>
Date: Thu, 07 Apr 2011 21:18:13 -0400

On 04/07/2011 08:50 PM, Roman Shpount wrote:
> Quoting Nokia's paper: "Codecs done without thorough standardization
> effort like Speex and iLBC offer significantly reduced efficiency,
> probably due to much lesser optimization, *listening tests* and IPR free
> design." Testing is an important part of development and standardization
> process for a CODEC.

How do you know when you have done enough development testing to propose a
standard?

You know that you have done enough testing when you perform a battery of
tests, and the results don't make you want to change the specification.

An enormous amount of testing has been performed on Opus, as part of the
rapid development feedback loop.  Those tests are now obsolete, because
each of them provoked changes in the specification to improve
performance.[1]  The latest round of tests from Google, Hydrogen Audio,
and Broadcom total many thousands of quality assessments and a huge
expenditure of time and effort ... and the results had no effect.  None.
All of the Opus maintainers agree that the results do not suggest any
further change to the specification.

That means we have done enough development testing.  Now it is time to
propose a standard, so that we have something on which to perform formal
assessment.

[1] Here's one that shows an ancient version of CELT beating G.722.1C at
48kbps: http://www.celt-codec.org/comparison/

---

From: Paul Coverdale <[email-masked]>
Date: Thu, 7 Apr 2011 21:27:27 -0400

To me, the “finished product” means the final, frozen, design, complete with all of the inherent trade-offs that may have been made. Formal testing, based on an agreed test plan, can then be carried out to determine whether that design does indeed meet all of the initial requirements.

Regards,

…Paul

From: [email-masked] [mailto:[email-masked]] On Behalf Of Koen Vos
Sent: Thursday, April 07, 2011 8:58 PM
To: Roman Shpount
Cc: [email-masked]; Stephen Botzko
Subject: Re: [codec] A concrete proposal for requirements and testing

Roman:

How could what Stephen Botzko describes as "a codec characterization/quality assessment that will be done on the finished product" possibly improve coding efficiency?

best,
koen.

  _____  

best,
koen.

  _____  

Hi Ron

Regards,
Stephen Botzko

---

From: Jean-Marc Valin <[email-masked]>
Date: Thu, 07 Apr 2011 21:53:02 -0400

On 11-04-07 08:50 PM, Roman Shpount wrote:
> probably due to much lesser optimization, *listening tests* and IPR 
> free design." Testing is an important part of development and 
> standardization process for a CODEC.

I don't think I'm qualified to comment on iLBC, but when it comes to 
Speex, I think I know better than anyone why it does not perform as well 
as codecs like AMR. Probably the main reason is quite simple: it was 
written by a PhD student (me) with no prior experience in speech coding 
as a side project in during his spare time. Another reason is the fact 
that the original goal was just to get something decent and free out 
there, not to beat (or even equal) the state-of-the-art. Avoiding 
encumbered technology (e.g. ACELP) also had something to do with it, 
along with the fact that I did almost all the code by myself. Lack of 
listening test at the end of the process was totally irrelevant. Even if 
I had all the listening tests in the world at that point, it wouldn't 
have changed anything.

This is the exact opposite of what has happened so far with Opus. I'm 
obviously much more experienced than when I started Speex in 2002, but 
I'm also no longer working by myself. There have been other developers 
(e.g. Tim and Greg) involved right from the start, and since the codec 
WG was created, there's been a lot of collaboration between all the 
companies/developers involved in Opus. This has let to many improvements 
to the components that were originally the SILK and CELT codecs. The 
last difference is testing. But not the type of testing that's been 
discussed in this thread. Spending a lot of time to just get back a 
handful of numbers is mostly useless in terms of improving a codec. The 
kind of testing that has allowed us to make huge progress is of a 
different kind. More along the lines of "I tried playing live with my 
friends and all the instruments sounded great except for my bass guitar" 
(true story) or "I used this on this dong from my collection and at 
exactly this point in the file I hear this funny artefact". This is the 
kind of testing that has lead to *huge* improvements to Opus.

Cheers,

     Jean-Marc

---

From: Gregory Maxwell <[email-masked]>
Date: Thu, 7 Apr 2011 19:45:25 -0700

Roman Shpount [[email-masked]] wrote:
> It is in everybody's interest to produce the best CODEC possible. Comments that "we listen to it and we like it" and "you are free to run your own tests", even though valid, are not very productive.

Jean-Marc Valin [[email-masked]] wrote:
> The kind of testing that has allowed us to make huge progress is of a different kind.

Someone who hasn't been following all of the discussion but read Jean-Marc's and Roman's posts might reach an erroneous conclusion that controlled blind tests have not been conducted at all.

But this isn't the case, The  "Spending a lot of time to just get back a handful of numbers" tests of the controlled and blind large-panel type have also been conducted— not because they're especially useful in development, but because they are sanity checks, because they are useful to systems integrators and others in understanding where the codec fits, etc.

The most recent and applicable of these tests was Jan Skoglund's, which included some 346 listener*samples across a variety of relevant codecs and configurations (http://www.ietf.org/mail-archive/web/codec/current/msg02272.html)

Jean-Marc's description of the really helpful™ testing also may have made it sound rather unscientific.  Much of the work in development-tuning is finding the interesting cases and that simply works best if you use the codec in a lot of ways.  Anything that gets in the way of usage volume is probably bad.  After finding them the actual validation of the changes can be done with simple blind A/B testing with a small number of listeners (sometimes just the person doing the tuning themself). There have been hundreds of these tests conducted during development— Enough that one of the persons involved in Opus tuning wrote a new open source tool for blind A/B,A/B/X,X/X/Y testing (Squishyball, http://people.xiph.org/~xiphmont/demo/celt/demo.html#demo) because the available tools were intrusive and got in the way of high volume testing.

---

From: Roman Shpount <[email-masked]>
Date: Fri, 8 Apr 2011 00:22:21 -0400

Please, don't get me misunderstood: What I am trying to say is if there is a
clear test plan I, and probably a lot of other people on this list, would be
able to conduct more tests of the CODECs. Not having a test plan makes it a
lot more difficult for us to do tests.
_____________
Roman Shpount

On Thu, Apr 7, 2011 at 10:45 PM, Gregory Maxwell <[email-masked]>wrote:

> > It is in everybody's interest to produce the best CODEC possible.
> Comments that "we listen to it and we like it" and "you are free to run your
> own tests", even though valid, are not very productive.
>
> Someone who hasn't been following all of the discussion but read
> Jean-Marc's and Roman's posts might reach an erroneous conclusion that
> controlled blind tests have not been conducted at all.
>
> But this isn't the case, The  "Spending a lot of time to just get back a
> handful of numbers" tests of the controlled and blind large-panel type have
> also been conducted— not because they're especially useful in development,
> but because they are sanity checks, because they are useful to systems
> integrators and others in understanding where the codec fits, etc.
>
> The most recent and applicable of these tests was Jan Skoglund's, which
> included some 346 listener*samples across a variety of relevant codecs and
> configurations (
> http://www.ietf.org/mail-archive/web/codec/current/msg02272.html)
>
> Jean-Marc's description of the really helpful™ testing also may have made
> it sound rather unscientific.  Much of the work in development-tuning is
> finding the interesting cases and that simply works best if you use the
> codec in a lot of ways.  Anything that gets in the way of usage volume is
> probably bad.  After finding them the actual validation of the changes can
> be done with simple blind A/B testing with a small number of listeners
> (sometimes just the person doing the tuning themself). There have been
> hundreds of these tests conducted during development— Enough that one of the
> persons involved in Opus tuning wrote a new open source tool for blind
> A/B,A/B/X,X/X/Y testing (Squishyball,
> http://people.xiph.org/~xiphmont/demo/celt/demo.html#demo) because the
> available tools were intrusive and got in the way of high volume testing.
>
>
>

---

From: Ron <[email-masked]>
Date: Fri, 8 Apr 2011 19:06:12 +0930

+1

Peter speaks well exactly what went through my mind about the best way
to actually get a meaningful statistical analysis of what we have today,
from a real world mix of naïve, golden-eared, and technically savvy users,
over a huge variety of use cases.

Of course I wasn't suggesting that we should delegate Stephen to go have
a listen, then hinge the remainder of this process on his impressions.
I was suggesting that *everyone* proposing tests should go have a listen,
because if at G.711 (mono) data rates, you can't ABX (stereo) Opus from
the reference (as many good listeners apparently can't on many samples),
then clearly many of the tests that people have been proposing aren't
actually very meaningful at all.  It might be nice to have a number that
shows how much we are better than them, to brag about over dinner with
p < 0.05 confidence, but it's already self-evident that we've by far
exceeded the quality goals that we initially merely aspired to aim for.
The sky is blue.  Opus is almost transparent.  Most people who aren't
blind or deaf seem to have rough consensus on these things already.

Audio quality isn't the leading problem we have to solve at this stage.
The biggest problem now is the problems we still don't know about yet.
And the only way to find them is to give it to real users, and see what
they can break.  The developers can't break it anymore, and nobody here
has shown that they can either.  That says it's time to expand our circle
of benevolent wreckers to people with more imaginative (ab)uses than we
are able to contrive here in the lab from molecules of past experience.

Please don't stop testing.  But please don't stop everyone else while
you do.  There are many more people waiting to test this who are outside
this group than we currently have within it.  It's time to put them to
work for us :)  Implement your real use case and ship it.  Nothing else
will show us what bugs really do remain as efficiently as that will.

Cheers,
Ron

---

From: Koen Vos <[email-masked]>
Date: Fri, 8 Apr 2011 19:34:20 +0200 (CEST)

Roman Shpount wrote 
> What I am trying to say is if there is a clear test plan I, and probably a lot of other people on this list, would be able to conduct more tests of the CODECs. 

And we all welcome such tests. The more the better. 

It's just unclear how or why such testing should have any bearing on the WG. There are no other candidates to compete against; it won't retroactively improve the codec; quality has been shown to be good enough for the codec to be useful... 

Perhaps we can set up a separate mailing list for those who want to do ITU-T style codec testing? 

best, 
koen. 

----- Original Message -----
From: "Roman Shpount" <[email-masked]> 
To: "Gregory Maxwell" <[email-masked]> 
Cc: [email-masked], "Stephen Botzko" <[email-masked]> 
Sent: Thursday, April 7, 2011 9:22:21 PM 
Subject: Re: [codec] A concrete proposal for requirements and testing 

Please, don't get me misunderstood: What I am trying to say is if there is a clear test plan I, and probably a lot of other people on this list, would be able to conduct more tests of the CODECs. Not having a test plan makes it a lot more difficult for us to do tests. 
_____________ 
Roman Shpount 

On Thu, Apr 7, 2011 at 10:45 PM, Gregory Maxwell < [email-masked] > wrote: 

The most recent and applicable of these tests was Jan Skoglund's, which included some 346 listener*samples across a variety of relevant codecs and configurations ( http://www.ietf.org/mail-archive/web/codec/current/msg02272.html ) 

Jean-Marc's description of the really helpful™ testing also may have made it sound rather unscientific. Much of the work in development-tuning is finding the interesting cases and that simply works best if you use the codec in a lot of ways. Anything that gets in the way of usage volume is probably bad. After finding them the actual validation of the changes can be done with simple blind A/B testing with a small number of listeners (sometimes just the person doing the tuning themself). There have been hundreds of these tests conducted during development— Enough that one of the persons involved in Opus tuning wrote a new open source tool for blind A/B,A/B/X,X/X/Y testing (Squishyball, http://people.xiph.org/~xiphmont/demo/celt/demo.html#demo ) because the available tools were intrusive and got in the way of high volume testing.

---

From: Paul Coverdale <[email-masked]>
Date: Fri, 8 Apr 2011 14:32:09 -0400

Koen Vos wrote: 

>quality has been shown to be good enough for the codec to be useful...

I think that’s where some people have difficulty. There’s been no systematic attempt to evaluate Opus against the performance requirements given in the codec requirements document (as thin as it is) in a controlled and repeatable manner.

…Paul

---

From: "Roni Even" <[email-masked]>
Date: Fri, 8 Apr 2011 23:32:33 +0300

And as stated in the charter on deliverables:

" Specification of a codec that meets the agreed-upon requirements"

I think there are no agreed-upon requirements yet and this is what I see as the  current discussion and not if the codec has fulfilled the requirements.

Roni Even

From: [email-masked] [mailto:[email-masked]] On Behalf Of Paul Coverdale
Sent: Friday, April 08, 2011 9:32 PM
To: 'Koen Vos'; 'Roman Shpount'
Cc: [email-masked]
Subject: Re: [codec] A concrete proposal for requirements and testing

…Paul

---

From: Ron <[email-masked]>
Date: Sat, 9 Apr 2011 12:36:11 +0930

On Fri, Apr 08, 2011 at 11:32:33PM +0300, Roni Even wrote:
> I think there are no agreed-upon requirements yet and this is what I see as
> the  current discussion and not if the codec has fulfilled the requirements.

Thanks for the change of tack.  Since it seems unlikely that the WG
document draft-ietf-codec-requirements-02 would be amended to impose
a quality requirement of "Better than transparent", this current
thread has clearly been sailing a bit too close to the wind so far.

Which parts of that document do you still dispute or think need
improving?

That seems like precisely the sort of thing we should be finalising
now, rather than embarking upon arduous additional quality testing
that all indications are will only tell us things we already know.

> I think that’s where some people have difficulty. There’s been no systematic
> attempt to evaluate Opus against the performance requirements given in the
> codec requirements document (as thin as it is) in a controlled and repeatable
> manner.

I think that's a bit disingenuous to the careful methodology that was
employed by the developers, and to the people who so far have done
blind and scientific tests of their results, but without dwelling on
that slur:

There's also been no, even informal, results presented to suggest that
it does not exceed even the most ambitious performance requirement
expectations held at the outset, by a rather surprising margin.

There has certainly been a systematic attempt to cast doubt upon the
results that we do have, but so far not a shred of even questionable
evidence to back those doubts up.

Do you have some results to share, that back up the claims of the
people "having difficulty" accepting what has been achieved to date?

While I respect your adherence to processes that are indeed necessary
for the ITU to licence a technology and deploy it in the telephone
network, one of the very reasons for forming this group under the
auspices of the IETF was that the reality of developing and deploying
internet services is quite different.  We don't have just a handful
of companies responsible for signing off on a spec, and making do if
it later proves insufficient - we have millions of them, most of which
haven't even heard of this group yet.  And the best way to engage them
is to provide a proposed standard which they can work to and assess
for their own uses.

If we'd had to wait for DNS and IPv6 to mature before continuing that
process, our jaws wouldn't be dropping at what this group has produced
for its first proposed release, like they are today.

We have some very clever and very experienced people here.  If none of
them can present us with actual engineering faults (as opposed to minor
procedural details), then it's time to put our work to the real test.

Every day that goes by in which such faults aren't found and presented
is further evidence to support the appropriateness of the Last Call.

Cheers,
Ron

---

From: Kavan Seggie <[email-masked]>
Date: Sat, 9 Apr 2011 07:15:08 +0100

> We don't have just a handful

+1

I am the Founder and CEO of a startup who has been quietly observing this WG
and the development of CELT for almost two years.

It would be great if we could progress to real world testing and get the
product out to the people. For us, SILK is an amazing codec, better than any
voice codec we have tried including proprietary technology from the best
vendors. CELT, sounds excellent too. We do not have the resources to run a
formal test plan so we trust the ears of our team, friends and family and we
think they are really solid.

IMHO the next step is to provide a release version of Opus to the wider
community for real world testing. They'll let you know if there are any real
issues far quicker than more formal testing.

Regards

Kavan

-- 
Kavan Seggie
SayMama <http://www.saymama.com> - The video chat social network.

Find us on Facebook: facebook.com/officialsaymama
Follow us on Twitter: twitter.com/saymama

---

From: "Roni Even" <[email-masked]>
Date: Sat, 9 Apr 2011 10:14:06 +0300

Hi,

According to the proponents there is nothing preventing you from using OPUS
for testing or in your product. What is being discussed is if the current
version of the codec answers the current charter of the CODEC WG.

I see in the email discussion two issues which may need to be resolved.

The first is if the codec meets the requirements, but the requirements are
not finalized yet. I can see that agreement can be reached if there is a
focus to finish the requirement document. Note that the objectives are "
Designing for use in interactive applications (examples include, but are not
limited to, point-to-point voice calls, multi-party voice conferencing,
telepresence, teleoperation, in-game voice chat, and live music
performance)" so there are claims that you need to compare OPUS to codecs
used in this application spaces. Another way to address it is to limit the
applications in the charter. One example may be to remove the telepresence
application case.

The second issue is the objective that  "The goal of this working group is
to ensure the existence of a single high-quality audio codec that is
optimized for use over the Internet and that can be widely implemented and
easily distributed among application developers, service operators, and end
users." Here again we may need to address the new IPR claims. It can be done
by re-charter or by changing the codec around the IPR. In the mean time some
view that in the current state, the OPUS codec should be compared to
existing royalty baring codecs.
 
Regards
Roni Even

From: [email-masked] [mailto:[email-masked]] On Behalf Of
Kavan Seggie
Sent: Saturday, April 09, 2011 9:15 AM
To: Ron
Cc: [email-masked]
Subject: Re: [codec] A concrete proposal for requirements and testing

+1

Regards

Kavan

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
Paul Coverdale

> I think that's where some people have difficulty. There's been no
systematic

Cheers,
Ron

-- 
Kavan Seggie

---

From: Ron <[email-masked]>
Date: Sat, 9 Apr 2011 19:27:21 +0930

Hi Roni,

> version of the codec answers the current charter of the CODEC WG.

Well the RF IPR grants only permit us to do that once we've reached the
stage of publishing a definitive RFC.  The "internal" testing of course
has already taken place, within the bounds that Koen and others clarified
for interim use of their technology.  That we aren't seeing any more real
feedback from that testing says to me that it has achieved what it is
likely to in the short term.  And now we should put out a request for
comments to a broader audience, to solicit feedback from newer eyes.

But yes, the question now is indeed, have we fallen short of some part
of the charter, that so far has been missed.  I'm no real expert on
IETF procedure, but I see the "last call" as exactly that.  Time to
point out whatever you think others have so far missed, while there is
still time to fix the draft documents.

> focus to finish the requirement document.

We do have a draft requirements document.  So I guess it would be helpful
if people can point out exactly what sections they don't like, and what
they'd like to replace it with, if anything.

I'm not aware of the editors being behind with any proposed changes
that there is consensus for, are they?

> Note that the objectives are "
> used in this application spaces.

Are you aware of, and/or proposing an alternate codec, that you think
meets the group charter better than our candidate codec?  I'd have
thought the time to mention that would now be long past, but if you
have one that we've missed, then please, I'd love to know, what is
its name, and where do I download its reference implementation?

That would certainly be an important piece of new information that
the group should consider.

But since I'm not aware of one, I can only read that as saying that
the candidate must indeed be suitable "for use in interactive etc."

I think the charter also says that if it isn't, we may need to make
another codec.  But I'm not aware of anyone else seeing the need for
that just yet, are you?

> Another way to address it is to limit the applications in the charter.

Given that we're now rumbling toward last call, could you please be
explicit about which of the applications you think we have failed at,
and what evidence you have to support that.  It will be much easier
to address them that way.

> One example may be to remove the telepresence application case.

For instance I'm not sure I follow how you think we may have failed
at that use case?

requirements 2.3 says:

 ...
 In addition, telepresence applications require super-wideband and
 full-band audio capability with useful bit-rates in the 32 - 80 kb/s
 range.  While voice is still the most important signal to be encoded,
 it must be possible to obtain good quality (even if not transparent)
 music.
 ...

Given that Opus just trumped HE-AAC in an unconstrained blind test
at 64kb/s (stereo), got mistaken for the reference, and can do 2.5ms
latency, which bit do you think we need to improve for telepresence?

> users." Here again we may need to address the new IPR claims.

I think Stephan said he'd prefer people to talk about that elsewhere,
and afaik they are.  That's still a goal, and to the best of my knowledge
we are still on target to achieve it by the time we've dotted the t's and
crossed the i's on the rest of this paperwork.

> It can be done

I think it's on the record under note well that xiph will pull its IPR if
RF status cannot be achieved, so I don't think we need to worry about that.
We can't guarantee there will not be further vexatious claims, but we can
be sure that isn't a possibility this group will need to consider in the
context of finalising the Opus candidate we have.

Opus only need be compared to other codecs that this group could endorse
as being superior to it at meeting the charter objectives.  If you really
have one, then maybe we do have a contest after all, but otherwise, the
only question remaining is did we fail at a goal, and what can we do
about fixing that.

It doesn't matter if moonrocks are prettier than the ones I have in my
garden, they're still never going to get used there.  So please, if
we've failed at the charter, I *really* want to know.  Starting with
paragraph and verse.  But there's no point comparing apples and oranges.

The answer to that is simple.  Oranges always win. :)

Cheers,
Ron

---

From: "Roni Even" <[email-masked]>
Date: Sat, 9 Apr 2011 14:39:13 +0300

Hi,
There are license free codecs like G.722.1 and G.722.1C. I am not sure what
is the status of G.719

As for telepresence HE-AAC is not the codec currently in use so comparing to
it does not qualify as getting better quality
Roni Even

> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Ron
> Sent: Saturday, April 09, 2011 12:57 PM
> To: [email-masked]

---

From: Ron <[email-masked]>
Date: Sat, 9 Apr 2011 22:08:52 +0930

> There are license free codecs like G.722.1 and G.722.1C. I am not sure what

On Thu, Mar 31, 2011 at 10:53 AM, Jean-Marc Valin wrote:

 7) The fullband test showed that Opus at 32 kb/s had higher quality than G.719
    at 32 kb/s. Does anyone disagree that this is sufficient to meet the Sec 4.2
    requirement of out-performing G.722.1C, considering that G.719 has already
    been shown to out-perform G.722.1C

Did I miss a reasoned argument that disagreed with this?  Or can we consider
that tested?  Even if they were comparable to Opus in that very narrow band
they still cannot nearly compete with it across its full range.

Is anyone seriously prepared to stake their reputation on nominating them
as candidate codecs to replace Opus here ;?

> it does not qualify as getting better quality

Well it's a little hard for me to offer you a comparison to a codec you won't
name ;)  But the question remains, what exactly about telepresence do you think
we fail at?  So far as I can see we have that fairly amply covered, no?

Cheers,
Ron

---

From: Stephen Botzko <[email-masked]>
Date: Sat, 9 Apr 2011 09:27:53 -0400

Ericsson and Polycom are the two known IPR holders on G.719, and both offer
royalty-free licenses.

Regards,
Stephen Botzko

---

From: Jean-Marc Valin <[email-masked]>
Date: Sat, 09 Apr 2011 09:35:56 -0400

Hi Roni,

> it does not qualify as getting better quality

Thanks very much for pointing out that HE-AAC is not a telepresence 
codec. As you know, when we design a codec for interactive applications, 
we have to make sacrifices (e.g. in quality) to achieve a low delay and 
Opus is no exception. On the other hand HE-AAC makes no such sacrifices 
and is considered by many (including the people who ran that test) as 
the best non-interactive codec for music at 64 kb/s.

The people who ran the HA listening test are only interested in highest 
quality at a given bitrate, which is why they selected HE-AAC and 
Vorbis. We're just glad they accepted to include Opus in their 
comparison, despite the test not being fair to Opus (other codecs have a 
much higher delay). Despite all that, it turned out that Opus 
out-performed all HE-AAC encoders on a wide variety of audio samples. 
Considering that low-delay flavors of AAC encoders (which we have never 
been able to obtain to test against) have to make the quality/delay 
trade-offs I mentioned, I'm pretty confident that Opus would do well 
against them -- despite this not being a requirement.

Cheers,

	Jean-Marc

---

From: "Roni Even" <[email-masked]>
Date: Sat, 9 Apr 2011 17:42:59 +0300

Jean Marc,
G.722.1 and G.722.1C were mentioned as royalty free codecs. Is OPUS one?
Roni

> Sent: Saturday, April 09, 2011 3:39 PM
> To: [email-masked]
> than
> than G.719

---

From: Ron <[email-masked]>
Date: Sun, 10 Apr 2011 00:51:47 +0930

> G.722.1 and G.722.1C were mentioned as royalty free codecs. Is OPUS one?

Has anyone requested or obtained a royalty from anyone using it?
That should give as black and white an answer to that question as there is.

Now can we please stop repeating this.  There was another list set up
specifically for telling patent ghost stories.

Thanks.

---

From: Jean-Marc Valin <[email-masked]>
Date: Sat, 09 Apr 2011 11:27:32 -0400

> G.722.1 and G.722.1C were mentioned as royalty free codecs. Is OPUS one?

The goal is still to have Opus be royalty-free and I don't see any 
reason that would not be the outcome. I assume you're again making 
reference to Qualcomm, which has just made our job easier by bringing 
some patents to our attention rather than having us dig for them.

Cheers,

	Jean-Marc

---

From: Stephan Wenger <[email-masked]>
Date: Sat, 09 Apr 2011 08:35:41 -0700

With hat:

Wrong answer.

In the IETF, as a group, we look at information made available by the
rightholders (to the IETF, or, when there are other SDOs involved, at
their disclosure system and/or their patent policy).  Those of us who feel
qualified can also look at objectively verifiable (by anyone!) data such
as patent expiration dates.

We do not look at known, rumored, or unknown licensing deals.  We do not
solicit such information.

Let's stay out of antitrust trouble.

Stephan

---

From: Kat Walsh <[email-masked]>
Date: Sat, 9 Apr 2011 13:56:36 -0400

> With hat:

The same hat we're all wearing, the one that indicates "simply
speaking your own opinion"?

> solicit such information.

In fact, per BCP 79, it seems the opposite is true. We do look at
licensing: "Although the IETF can make no actual determination of
validity, enforceability or applicability of any particular IPR claim,
it is reasonable that a working group will take into account on their
own opinions of the validity, enforceability or applicability of
Intellectual Property Rights in their evaluation of alternative
technologies." And we do solicit such information: "Since IPR
disclosures will be used by IETF working groups during their
evaluation of alternative technical solutions, it is helpful if an IPR
disclosure includes information about licensing of the IPR in case
Implementing Technologies require a license."

Not only is this information explicitly asked for, but I've never seen
limitations on what we're all allowed to look at and consider, even if
we have to dance around public discussion of some things that some
participants or their employers misguidedly believe there is more
safety in ignoring.

I can't see any justification for discouraging individuals from
looking at as much information as is necessary to make their own
informed decisions, even if the information is of no official interest
to the process. If people are broadly unwilling to deploy the
technology, regardless of their reasons, then the goals of the charter
are not met.

> Let's stay out of antitrust trouble.

What kind of antitrust trouble are you thinking of? (You can't
possibly be suggesting that choosing to use or not use a technology
based on its patent licensing status has antitrust implications.)

Cheers,
Kat Walsh

---

From: Stephan Wenger <[email-masked]>
Date: Sat, 09 Apr 2011 13:36:37 -0700

Hi Kat,
Inline.
Regards, Stephan

>speaking your own opinion"?

The official job title was something like "Advisor to IESG re IPR matters
related to Codec WG".

>Implementing Technologies require a license."

Huh?  Let's sort this out:

"validity, enforceability, or applicability" surely have nothing to do
with licensing conditions, right?  They may have something to do whether
one may need a license to practice the technology, but not the conditions
under which the license is (or is not) granted...  So the first sentence
you cited appears to be unrelated to the issue in question.

The second sentence simply suggests to those making disclosures that a
disclosure is particularly helpful when it contains licensing information.
 The practice in the IETF is that ALL disclosures (except third party
disclosures) contain such information.  Many third party disclosures are
followed up by disclosures of the rightholders, providing licensing info.
I have absolutely no issue with the WG taking such information into
account.

Note, though, that I was talking about "licensing deals", and that the
context of this discussion has been that someone was arguing about
knowledge (or lack thereof) of royalty-bearing licensing deals.  This has
nothing to do with disclosures received.  (I don't have the exact
quotation readily available, and I'm too lazy to look it up.)

Note also that the IETF does not have a mechanism that allows someone to
state that there is no (known) patent right associated with a technology.
If someone dares to make such a statement, I would suggest to use any
random Blog that deals with IPR matters.  I'm sure the statement would
make headlines rather sooner than later :-)

>participants or their employers misguidedly believe there is more

There are such limitations, both in the policy and in our typical conduct
of work in the IETF.  I could write pages about each.  However, I don't
see a value in doing so on this technical mailing list.  Also, let me
suggest that "misguidedly" may perhaps be a slightly misleading word...
Most big-corp employers base their policy (employees and patents) on
matters like this on legal experience that can easily be orders of
magnitude larger than yours and mine combined.

Please also note that I never advocate to ignore any information related
to IPR that comes into ones way.  It's IMO stupid to do so.  Talking about
such information makes a difference, though.  Business case for publishing
information and risks have to be carefully weighted.  Quite obviously, a
group of open source developers have very different tradeoffs to observe
than other entities.

>to the process. 

Me not, either.  I never have argued against that point.  In fact, I have
quite often stated the same in the past.  Note, though, that researching
and talking in public about the results of such research are very
different things.  

>If people are broadly unwilling to deploy the

Absolutely.

>based on its patent licensing status has antitrust implications.)

I'm unwilling to discuss such matters on a technical mailing list.  If you
want to discuss it in private, please let me know.  It'll probably be by
phone...

---

From: Stefan Hacker <[email-masked]>
Date: Sun, 10 Apr 2011 00:27:38 +0200

Hello,

I am writing this on behalf of the developers of the OSS VoIP tool Mumble. Since the 1.2.0 release of our software in December 2009, which integrated CELT as the new default codec to complement Speex, hundreds of thousands of users have sent and listened to millions of hours of CELT VoIP audio of different versions using Mumble. The response was hugely positive, with the most recent versions of CELT reportedly outperforming Speex even at low bit-rates. CELT, despite not even being finalized, has also been picked up by a number of other well known VoIP tools. It goes without saying that SILK has seen an even greater amount of usage, and is well known for its quality.

We have been closely following the development of Opus as well as the tests being performed by others and are excited about the prospect of being able to utilize Opus in future versions of our software. From the tests we have seen and participated in so far, we expect Opus in its current state to outperform pure CELT in our specific use-case and want to integrate and distribute it to our users sooner rather than later. We will do so as soon as licensing allows us to, which will generate millions of hours of real-world listening tests.

It is the reality of most Internet based applications that they are in testing and production at the same time. Publishing a proposed standard does not set it in stone forever, it just opens the field for testing on a much, much broader scale. We believe in the current state of Opus and are eager to show that belief by delivering it to all of our users.

Thank you all for your great work.

Regards,
Thorvald Natvig,
Mikkel Krautz,
Benjamin Jemlich,
Stefan Hacker

---

From: Ron <[email-masked]>
Date: Sun, 10 Apr 2011 08:38:57 +0930

> The official job title was something like "Advisor to IESG re IPR matters

Which part of your advise here was directed to the IESG?

> quotation readily available, and I'm too lazy to look it up.)

That would have been Roni, repeating your query about whether Opus was
royalty-free, I guess.  After we'd already agreed, by your request and
hatted advice, that we not talk about that on this list.

But FTR, the answer is quite clearly still, yes it is.  You don't seriously
think that Xiph would have kept it a secret if some party had triggered the
termination clause in their own IPR grant, do you?

In the absence of news about that (not to mention other assurances given
directly to you on this list and at the Prague meeting), I think we can
safely conclude that there has been no such claim to date and Opus remains
as RF as any thing that royalties have never been collected on can be.
We don't have to pry into people's backroom deals at all to see that.

So let's drop the FUD about Scary Claims, and the bizarre claims about
transitive, questionable IPR states obligating us to test against codecs
that are clearly unsuitable for meeting the WG charter, and just get on
with the tedious task of trying to find out if any of the people currently
wasting our time with spurious circular arguments, actually have a point
that this group really needs to consider.

Because so far, we have a flood of vague handwavy claims about supposed
'alternative codecs', which aren't backed up by even the most rudimentary
standards of proof, and cries that the WG documents aren't satisfactory -
which aren't backed up by even pointing at the parts that are considered
unsatisfactory, let alone an explanation of what is actually supposed
to be unsatisfactory about them.  Which try as I might, I don't seem to
be able to glean even a single, simple, contribution to the advancement
of this group's charter from.  No matter how many times they are repeated
by the same people to the point of going "oh, uh, nevermind".  Again.

The last call is rolling.  If you have something of substance to add to
the work we have, please detail it precisely.  If you can't, the only
logical conclusion is that we have consensus from the non-obstructing
people on the current documents in their present form and nobody knows
how to improve them further yet.

I sincerely hope this isn't actually representative of the way that some
of these people work within the other SDO's they are a part of, because
that would be acutely embarrassing for both them, and their organisations.

When the supposed amateurs appear more professional than you do, making
sure you have the right hat on is the least of your problems.  It's too
late to pretend that Opus hasn't wiped the floor with pretty much every
other codec, and that the achievement of its developers isn't distinctly
remarkable.  It may not be too late to suck up your initial disdain and
disbelief in this group and salvage some self respect for both yourselves
and the organisations you inevitably represent to the world.

Remember all this is becoming a part of your permanent record, and part
of the internet history even.  So who wants to look like an engineer in
that story, and who wants to look like a cheap, inept saboteur?  It's
your last call to make that decision too. :>

Choose wisely,
Ron

---

From: Paul Coverdale <[email-masked]>
Date: Sat, 9 Apr 2011 20:30:27 -0400

>blind and scientific tests of their results, but without dwelling on

I'm not trying to be disingenuous. I just haven't seen details of the "careful methodology that was employed". There may well be a lot of results floating around behind the scenes, but the only official information that I have seen so far is the slide deck "Opus Testing" presented during IETF80. This did not provide a lot of experimental detail, and the number of subjects was quite small. Anyone familiar with subjective testing knows that the results can be heavily influenced by the methodology employed.

>expectations held at the outset, by a rather surprising margin.

We need a test plan before we can conduct any useful tests.

>people "having difficulty" accepting what has been achieved to date?

Many people have indicated they will be willing to conduct tests, and share results, when a test plan is available.

>is to provide a proposed standard which they can work to and assess

ITU doesn't license technologies, that's a matter between individual organizations who may want to implement a specific standard which incorporates IPR. But in any case, the processes they follow are not there because of licensing. They are there to provide a logical progression from determining the requirements, developing solution(s), and testing the solution(s) against those requirements. 

Regards,

...Paul

---

From: Ron <[email-masked]>
Date: Sun, 10 Apr 2011 12:03:45 +0930

On Sat, Apr 09, 2011 at 08:30:27PM -0400, Paul Coverdale wrote:
> I'm not trying to be disingenuous. I just haven't seen details of the
> "careful methodology that was employed". There may well be a lot of results
> floating around behind the scenes, but the only official information that I
> have seen so far is the slide deck "Opus Testing" presented during IETF80.
> This did not provide a lot of experimental detail, and the number of subjects
> was quite small. Anyone familiar with subjective testing knows that the
> results can be heavily influenced by the methodology employed.

I presume you've seen enough of codec development though to know that you
don't arrive at the sort of results that Opus has achieved through randomly
stabbing in the dark.  At every step things have been evaluated in an
appropriately controlled and repeatable manner.  The HA folk also take
their testing very seriously, which I believe was described in Prague.

Their complete set of results should be available soon, and my understanding
at this stage is that they are not notably different from the small sample
that was available at the time of Prague.

> We need a test plan before we can conduct any useful tests.

Except we've already established that not all groups are actually interested
in, or find significant, or satisfactory, the same test plan.  We don't need
consensus from the whole group for further testing to take place, only between
the people actually planning to conduct, and interested in, that test.

The group can reach consensus on the relevance of these additional tests
if and when they ever get conducted and results are presented.

> Many people have indicated they will be willing to conduct tests, and
> share results, when a test plan is available.

This has been repeated often, and yet weeks have passed and still there
is no sign of even a fledgling plan that these people are proposing
to commit to ...

> ITU doesn't license technologies, that's a matter between individual
> organizations who may want to implement a specific standard which
> incorporates IPR. But in any case, the processes they follow are not there
> because of licensing. They are there to provide a logical progression from
> determining the requirements, developing solution(s), and testing the
> solution(s) against those requirements. 

Yes, bad choice of overloaded terms, I meant to say "give licence to",
in the sense of grant its imprimatur to the result.  I wasn't talking
about IPR there, since we weren't supposed to be talking about that here ;)

When you're about to deploy a stable technology to (many) thousands of
fixed exchanges, with the reliability demanded of the telephone network,
that sort of rigour is essential.  Here though we are talking about
deploying a proposed technology, to many thousands of implementers,
any or all of which may provide us useful feedback for improving that
technology further.  And since it's the internet, we can do that and
not only get away with it, but evolve a superior result faster too.

I don't see that the testing which will be conducted by those people
is any less valuable than the testing which is yet to emerge, from
people who say they'll probably do some, if only there was a plan.
We have people who say they can begin tests with thousands of users,
in real application use cases, tomorrow if we can give them an RFC
by then.  The complaint that sample spaces are too small is easily
remedied by unleashing those people to go do that.  You're offering
a test with ~24 people, they're offering tests on 24 million+.
Which should we choose?

We have a perfectly good plan.  Everyone go test the things that are
important to *you*, that's the only way we, and you, can ensure that
you'll actually be satisfied by the testing regime to undertake.
When you have results, share them, and we'll assess them.  Many people
have already satisfied themselves with such tests.  Those who haven't
really only have themselves to blame for leaving it until the last
moment to wonder what they should test.  The people who know what they
want from this codec are satisfied, it's not clear to me what exactly
we expect to learn from people who don't already have a use case in
mind, and can't even begin to figure out what sort of test would
actually satisfy them, or even be meaningful to apply to a codec of
this calibre.

If groups of people want to agree on a common regime, that's excellent.
But if they can't that's no reason to hold up deploying something that
even to people with golden ears, is basically transparent.

As I suggested previously, we should give people a few more days to
see if they actually do come up with more test plans, and a sensible
timeline in which their results will be presented, and if they can't
do that, then we should just move on to the next phase asap.

Best,
Ron

---

From: Stephen Botzko <[email-masked]>
Date: Sun, 10 Apr 2011 08:46:02 -0400

-inline
Stephen Botzko

> their testing very seriously, which I believe was described in Prague.

No one is criticizing the developers here.  Let's take that off the table.
People want to see this codec tested/characterized because they think it is
good, not because they think it is poor. There is no need to test a bad
codec.

And no one has accused the testers of not being "serious". However,
repeatable controlled testing means that anyone can re-run the tests [using
the same test methodology] and get the the same results.  The documentation
of the procedures was not complete enough to allow this; the Q&A in Prague
made that pretty clear.  And there are traps - small mistakes in methodology
will produce misleading results.  So describing the details on precisely how
the speech samples were prepared, how the tests were conducted, what
equipment was used, etc. actually does matter.

And since the testing can be a lot of work, at least some folks want to see
a test plan developed so the work can be split up.  This is not being
obstructive, it is a belief (based on past experience) that ad-hoc tests are
not sufficient.

> that was available at the time of Prague.
Will their results be submitted as a contribution to this working group?

> the people actually planning to conduct, and interested in, that test.

And many folks are interested in such a plan, and have said so since the
beginning (at least since the Maastricht meeting).   Obviously you disagree,
but you seem to be working very hard to block any progress on a concrete
test plan; repeatedly arguing it is a waste of time [to a thread which was
specifically started to develop a concrete test plan].  Which is very
different from saying you are not interested.

> not only get away with it, but evolve a superior result faster too.

ITU-T codecs are of course used on the internet, many of them are not used
on the public telephone network, and were never intended for that use.  MPEG
also characterizes/verifies their codecs, using somewhat different test
procedures, but still using a unified test plan.  Anyway, framing this
discussion as "ITU  against IETF" is disingenuous and not constructive, we
are all contributing to the IETF by posting here, and AFAIK, the individuals
and the companies they work for have been involved in many IETF WGs for many
years.

One reason for an up-front testing focus is that (despite the "proposed
standard" RFC  comments), it is very difficult to upgrade codecs once they
are widely deployed.  Once OPUS is released, it will be put in PCs, and
appliances like tablets, mobile phones, and voip phones.  SIP devices using
it will need to support the first version for many years.

And [assuming success] Opus will be implemented in *silicon*, making the
upgrade situation much more problematic and expensive than a typical RFC.

So I don't agree with the "since its the internet we can..." line of
reasoning.

> actually satisfy them, or even be meaningful to apply to a codec of
Honestly, this is a non-plan.

> \

We are not at WG last call, there is ongoing development still happening.
Framing this as a "last minute" surprise, which is holding up publication of
the RFC is incorrect.  Discussions on the need and nature of
characterization and testing have been happening all along.  The working
group perhaps did not close on it (though honestly, I recall no objections
to the need for characterization in any of the previous meetings; perhaps
people should re-visit the recordings). But the topic is not new, and I hope
we can do better than simply "agreeing to disagree", and releasing a codec
for wide deployment without the normal verification.

---

From: Ron <[email-masked]>
Date: Mon, 11 Apr 2011 03:36:27 +0930

> good, not because they think it is poor. There is no need to test a bad

How good do they think it is?  Almost transparent?  That's what many
people appear to think.  To how many decimal places of accuracy do
you think we need to quantise that before it will be suitable for
broader release?

> equipment was used, etc. actually does matter.

What small mistake in methodology causes people to mistake the encoded
version for the reference?  Forgetting to actually use the codec at all?

> obstructive, it is a belief (based on past experience) that ad-hoc tests are

Well to call a spade a spade, so far the only real interest I see in this
testing call is from people whose only confirmed concern is that it must
be a lot of work.  And who aside from repeatedly claiming that it must be
done, haven't actually gone and done anything at all to make it any closer
to really happening.  How long has the bitstream been frozen now?

  obstructive
      adj 1: preventing movement;

  belief n.
     1. Assent to a proposition or affirmation, or the acceptance
        of a fact, opinion, or assertion as real or true, without
        immediate personal knowledge;

People with immediate personal knowledge of actual test results tell us
that Opus is almost transparent.  People who say they don't have that
knowledge want to talk about testing against irrelevant other references.

Who should we believe?

I won't make an accusation of malice, but if you still haven't even got
to the first step of knowing what you want to test by now, then I'll
leave you to select the alternative for yourself.

> Will their results be submitted as a contribution to this working group?

I believe everything about their previous tests have been made publicly
available, and the same will occur for this one.  Or do you mean something
more formal?

The test closed just a few hours ago, anyone here could have taken it.
All it would have taken from the many people waiting to start testing
to do that was enough initiative to say "hey that test you said was
still running, can you give me a lmgtfy link so I can take it?".

Want to guess how many people have?

> different from saying you are not interested.

I'm not blocking more testing at all.  I've said in nearly every mail now,
"Please go test.  As much as you can.  As professionally as you can."

The goals we had at Maastricht have largely turned out to be gross
underestimations of what was actually achieved.  Go take the HA test
for yourself, just for kicks.  Then we can have an *informed* discussion.
Not one based upon the testing needs of poorer codecs from the past.

If you want to go back to what was said at that time, several of the
people who are now (coincidentally?) insisting upon this lot-of-work
test, also said this group didn't need to be formed at all, because
G.711 was perfectly adequate to meet the needs we described ...

Well, we've given you a codec that can do stereo at G.711 mono rates,
that beats HE-AAC and largely can't be distinguished from the reference.
What possible challenge do they now have to the perfect adequacy of that?

I also don't see myself starving the oxygen from some other (sub)thread
that actually is fleshing out the details of this mythical test.  Or are
we waiting on me to devise the test, because nobody else has?  I think we
can let that accusation fall on its own face without further help from me.

Jean-Marc posed a list of very simple questions to quantify what testing
people considered was sufficiently complete and what needs further scrutiny.
I even reposted what boiled down to the salient two questions from the
meandering discussion that did follow.

Nobody has commented on that.  They just keep repeating, we need a plan,
it must be a lot of work, without offering anything that might be actually
considered one.  I won't return the accusation of thread hijacking.  The
list archive can speak for itself on that question.

If nobody disagrees that the original testing requirements are in fact met,
then that agenda item is complete right?  If you do, then please go back
to the top and answer Jean-Marc's query.  With something more concrete than
vague concerns about things you haven't even done an informal test of your
own to gauge its applicability to what we have today.

So let's move on.  What else are we missing from the draft requirements
to be able to sign off on that document now?

Cheers,
Ron

---

From: Serge Smirnoff <[email-masked]>
Date: Mon, 11 Apr 2011 13:48:21 +0200

Hi,

If there is still a necessity to test Opus codec against some other 
codecs or on different sound samples I'm ready to place SoundExpert (SE) 
testing platform at your disposal:
     - any nine sound samples
     - any number of contender codecs
     - one-two days to setup a test
     - simple participation through web interface
     - instantly available results

As the most interesting area for Opus testing is below 128kbit/s and 
artifacts are clearly audible in most cases:
     - no need to use controversial SE artifacts amplification, encoded 
samples will be used as is
     - simplified SE way of presenting two samples - reference and 
encoded (in random order) will be sufficient; annoyance of artifacts is 
graded according to 5-grade impairment scale

Unfortunately there is no way to control listener's skills and equipment 
used. We recently implemented post-screening based on "Median+/-1.5" 
rule but all submitted grades (with time-stamps and IP addressees) will 
be also available from MySQL dump for additional analysis.

If any interest and/or questions - I'm here.

Cheers,
Serge.

---

From: Anisse Taleb <[email-masked]>
Date: Mon, 11 Apr 2011 18:26:08 +0000

Hi all,

I feel this thread has somewhat diverged from its initial purpose which was to have a concrete plan for testing and requirements. I have also a feeling that there are "cultural" differences and divergences as to what "testing" means.

That aside, I would like to inform that a proposal for test plan is currently being drafted and will be communicated this week.

Kind regards,
/Anisse

---

From: Ron <[email-masked]>
Date: Tue, 12 Apr 2011 14:49:48 +0930

Hi Anisse,

On Mon, Apr 11, 2011 at 06:26:08PM +0000, Anisse Taleb wrote:
> Hi all,
> 
> I feel this thread has somewhat diverged from its initial purpose which was
> to have a concrete plan for testing and requirements.

Well, its initial purpose, as clearly seen in its first post, was to resolve
the disagreement (or possibly confusion?) that resulted from trying to take
a hum on those questions, without adequate discussion, at Prague.

It diverged from that almost immediately, and has steadfastly refused to
return to it.  If we can take that as meaning that nobody actually does
disagree with the points that Jean-Marc asked to clarify, then I think
we can take that as meaning this (original) thread has actually achieved
its purpose and we now do have consensus on those points.

Is there some formal IETF thing we need to do to acknowledge that now?
(as separate from this subthread about an additional planned test)

> I have also a feeling
> that there are "cultural" differences and divergences as to what "testing"
> means.

I don't think anyone disagrees on what testing means.  I do think we have
some disagreement on what tests are relevant though.  Some people think
testing against G.711, or against codecs that cannot possibly fill the
requirements this group set out to achieve, is a fairly pointless exercise.
At least in the context of assessing whether we have met the requirements
of the WG, and to what standard.  And in the face of the test results that
we do already have.

Such tests may still be interesting to see (I'd be interested in seeing
them for one), but I strongly disagree that they are at all serving the
aims of the WG, and don't think we should delay its progress if we don't
have them in time.

> That aside, I would like to inform that a proposal for test plan is currently
> being drafted and will be communicated this week.

Thank you.  I'm very glad to hear that the people who weren't satisfied
by the results we've already seen are taking the initiative to resolve
that to their full satisfaction.  As only they can do.

I look forward to seeing your plan, and its results.  As I have all of
the other tests performed to date.

Best,
Ron

---

From: Stephen Botzko <[email-masked]>
Date: Tue, 12 Apr 2011 07:16:26 -0400

in-line
Stephen Botzko

> its purpose and we now do have consensus on those points.

I don't see how anyone can think we've achieved consensus.  That is just
plain silly.  BTW, isn't up to *chairs* to call for consensus?

On the whole, the thread started reasonably, with good discussions on adding
some speech conditions, choosing the number of subjects, etc. The thread
diverged when AMR was proposed as an anchor in section 4.2. Jean-Marc
"strongly disagreed" to testing requirements against AMR-NB and AMR-WB,
saying it was "not fair", and suggesting that he thought Opus would fail in
that comparison (though he thought it would be close).  He suggested that
only unencumbered codecs be used as anchors for strict requirements. [He did
*not* object to the use of encumbered codecs in testing objectives, just in
testing "strict requirements"].

However, the idea of limiting IPR in the anchors was clearly not accepted by
others on the thread, and as is often the case on IETF lists, the mention of
IPR took us off-topic.  Then the idea of systematic testing itself became
the predominant hot topic, with many (like me) thinking it is essential, and
others (like yourself) thinking it is a waste of time.  Since the value of
systematic testing is a more basic discussion than the details of the test
plan, the thread never got back on track.

Starting over by contributing draft systematic test plan seems like the best
approach.

> aims of the WG, and don't think we should delay its progress if we don't

I think you and Anisse have very different views on what it means to run a
proper test.

---

From: Jean-Marc Valin <[email-masked]>
Date: Tue, 12 Apr 2011 13:35:40 -0400

Hi Stephen,

On 11-04-12 07:16 AM, Stephen Botzko wrote:
> On the whole, the thread started reasonably, with good discussions on
> adding some speech conditions, choosing the number of subjects, etc. The
> thread diverged when AMR was proposed as an anchor in section 4.2.
> Jean-Marc "strongly disagreed" to testing requirements against AMR-NB
> and AMR-WB, saying it was "not fair",

Correct so far.

> and suggesting that he thought
> Opus would fail in that comparison (though he thought it would be
> close).

Not quite. I actually believe Opus sounds better than AMR-WB since 
that's the conclusion from both the Google test and the earlier Dynastat 
test on SILK. I have no opinion on AMR-NB because I haven't listened to 
that codec enough to have an opinion on how good it sounds. It still 
remains that there's no point in having "no worse than AMR-*" as a 
strict requirement, not only because of the IPR, but also because the 
codec we're working on does much more than each of these codecs do. I 
assume that's the same reason why G.729.1 didn't compare with AMR-NB at 
all and used a *much* lower rate for its AMR-WB comparison.

> He suggested that only unencumbered codecs be used as anchors
> for strict requirements. [He did _not_ object to the use of encumbered
> codecs in testing objectives, just in testing "strict requirements"].

Correct.

	Jean-Marc

---

From: Stephen Botzko <[email-masked]>
Date: Tue, 12 Apr 2011 13:57:55 -0400

in-line
Stephen Botzko

On Tue, Apr 12, 2011 at 1:35 PM, Jean-Marc Valin <[email-masked]> wrote:

> Not quite. I actually believe Opus sounds better than AMR-WB since that's
> the conclusion from both the Google test and the earlier Dynastat test on
> SILK. I have no opinion on AMR-NB because I haven't listened to that codec
> enough to have an opinion on how good it sounds. It still remains that
> there's no point in having "no worse than AMR-*" as a strict requirement,
> not only because of the IPR, but also because the codec we're working on
> does much more than each of these codecs do. I assume that's the same reason
> why G.729.1 didn't compare with AMR-NB at all and used a *much* lower rate
> for its AMR-WB comparison.

Sorry if I misunderstood your original comment, it was not intentional.
     Requiring Opus to also beat all codecs out there in all circumstances
is thus totally unreasonable (despite the fact that the tests we did so far
indicate we may be close to that).
BTW, I do agree that the WG was not chartered to create a codec that beats
all codecs out there at all operating points.  As I recall, we wanted a good
codec, but were prepared to lose some quality to [hopefully] avoid
encumbrances.

ITU-T test plans don't cover all possible anchors (it simply costs too
much).   And the bit rates of the anchors can be lower than the codec under
test.  Quality is only one of the requirements, complexity is also a
requirement - so a codec with lower quality but also lower complexity can be
proposed and developed there.  It is possible to apply that same idea here -
agreeing on an anchor that may out-perform Opus for a narrow range of
operating points, but also agreeing to give Opus a reasonable bit-rate
advantage for the "not worse" test condition.

---

From: Ron <[email-masked]>
Date: Wed, 13 Apr 2011 05:04:08 +0930

> plain silly.  BTW, isn't up to *chairs* to call for consensus?

That would have been why I didn't prefix that by "with hat".

Do you actually have some things you would like to dispute about the
queries which Jean-Marc raised?  *I'm* not seeing any.

Should we ask the chairs if they see any, and think we can finalise
that part of the discussion before moving on to the next big one?
What is the correct procedure for this?

If a test is going to be proposed, clarifying what tests have already
been proven to satisfaction, seems like a relevant precursor to that.

 Ron

---

From: Stephen Botzko <[email-masked]>
Date: Tue, 12 Apr 2011 17:06:19 -0400

in line
Stephen Botzko

> queries which Jean-Marc raised?  *I'm* not seeing any.

I did provide some input already to the thread on the need for more speech
conditions and tandeming tests.

I personally do not object to removing GSM-FR and Speex-UWB.

I personally don't see much point in testing G.722 and Opus at the same
rate, though there could be some value in testing Opus at a lower rate
against G.722.  This might require adding an objective (or if we are gutsy a
new requirement).

on (4) I'd be interested in hearing Jean-Marc's view on what bit rate he
thinks provides roughly comparable quality.

I am not prepared to draw the conclusions in 1-7 yet (4 is not a conclusion,
but a question).  I am not trying to be contentious, but I think at least
the additional speech conditions need to be tested as well as clean speech.

Also, generally, I am thinking we can review the draft test plan w/o
worrying about what the previous tests have shown.  If there is overlap, it
still ought to confirm the older results, correct?  If the overall amount of
testing is too much, then we can talk about what to drop.

---

From: Anisse Taleb <[email-masked]>
Date: Wed, 13 Apr 2011 07:00:02 +0000

Dear Ron,

> If we can take that as meaning that nobody actually does

With all the "background noise" you may have missed my reply to JM. I do not think we have consensus.

> I don't think anyone disagrees on what testing means.

We definitely do. I have heard many voices stating the codec has been extensively tested, and consider the tests conducted during the development of the codec as representative of what a "codec standardization test" really means.  

> And in the face of the test results that
> we do already have

What test results you speak of ? 
- Tests that have been conducted on various versions of the codec and hence not allowing to look at the results side by side.
- Admittedly inadequate methodologies for certain tests
- Objective measurements (oh please !)
- Low number of subjects
- Lack of items 
- No cross-checks

Don't get me wrong, I really appreciate the efforts of the developers and the testers in their continuous effort to improve and bug-fix the codec, but this is a natural part of codec development and a necessary burden to produce a decent codec. However, I strongly disagree that the results seen so far represent more than a simple indication of a solid performance that qualify the codec to be considered for requirement testing. 

> aims of the WG, and don't think we should delay its progress if we don't

And what does serve the aims of the WG ?

The charter clearly states the goals of the WG. Producing an un-encumbered high quality audio codec that is optimized for internet applications. How can you speak of serving the aims of the WG by rushing an encumbered codec of questionable testing with no consideration of FER and delay jitter subjective tests into WGLC. 

Kind regards,
/Anisse

---

From: Cullen Jennings <[email-masked]>
Date: Thu, 14 Apr 2011 18:54:55 -0600

> What is the correct procedure for this?

When consensus is not obvious one way or the other, the chairs do need to make a determination of consensus. Typically for hard things the chairs will try and ask some clear questions and after reasonable discussion declare their read of the consensus. If they make a mistake on that, one can address it with the chairs and if that does not resolve it, push it to the AD. If anyone thinks there is something we need to make a call for consensus on and sort out, let us know. Much of discussion I am seeing right now seems to be on topics where there is no consensus yet or where if there is or is not consensus is not really all that critical to the goals of the WG. 

Cullen <with my co-chair hat on>