Subject: [codec] Characterization and Testing

From: Cullen Jennings <[email-masked]>
Date: Mon, 21 Mar 2011 22:52:19 -0600

Can the people who are planning on, or wiling to do, some characterization and testing of opus send an email to the list or chairs and give us a heads up what is being planned for this?

Thanks, Cullen

---

From: "Christian Hoene" <[email-masked]>
Date: Tue, 22 Mar 2011 09:00:19 +0100

Hi Cullen and all,

a test plan is given in draft-hoene-codec-quality at Section 10.2.
http://tools.ietf.org/html/draft-hoene-codec-quality-00#section-10.2

As stated before we can contribute with formal MUSHRA testing, with
objective tests using on PESQ, PEAQ, PEMOQ, and POLQA, and with runtime
measurements to the characterization phase.

With best regards,

 Christian Hoene

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Cullen Jennings
> Sent: Tuesday, March 22, 2011 5:52 AM
> To: [email-masked]
> Subject: [codec] Characterization and Testing
> 
> 
> Can the people who are planning on, or wiling to do, some characterization
> and testing of opus send an email to the list or chairs and give us a
heads up
> what is being planned for this?
> 
> Thanks, Cullen
> 
> _______________________________________________
> codec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/codec

---

From: "Paul Coverdale" <[email-masked]>
Date: Tue, 22 Mar 2011 09:15:22 +0100

With all due respect, Section 10.2 of this document does not specify a
detailed enough test plan to be able to draw valid conclusions, especially
where tests are conducted in different laboratories.

Regards,

Paul Coverdale

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Christian Hoene
Sent: Tuesday, March 22, 2011 9:00 AM
To: 'Cullen Jennings'; [email-masked]
Subject: Re: [codec] Characterization and Testing

Hi Cullen and all,

With best regards,

 Christian Hoene

heads up

---

From: "Christian Hoene" <[email-masked]>
Date: Tue, 22 Mar 2011 10:35:19 +0100

Hi Paul,

Could you please be more precise, on which details are missing? Are you
referring open issues on how the samples are degraded? Which tools are used,
which distortions are enforced, which modes are used? If so, we could easily
define them.

With best regards,

 Christian Hoene

> From: Paul Coverdale [mailto:[email-masked]]
> Sent: Tuesday, March 22, 2011 9:15 AM
> To: 'Christian Hoene'; 'Cullen Jennings'; [email-masked]
> Subject: RE: [codec] Characterization and Testing
> 
> Of

---

From: "Paul Coverdale" <[email-masked]>
Date: Tue, 22 Mar 2011 11:46:06 +0100

Hi Christian,

Well, the goal is to minimize all sources of variance, except those in the
experimental conditions under test. So normally a test plan would carefully
specify factors such as the exact set of conditions (codecs, bit-rates,
packet loss, reference codecs), input speech level and associated spectral
shaping, additive background noise (SNR, type), listening level,
experimental randomization, voting scale, instructions to subjects,
statistical analysis methodology, etc.

Regards,

Paul Coverdale

-----Original Message-----
From: Christian Hoene [mailto:[email-masked]] 
Sent: Tuesday, March 22, 2011 10:35 AM
To: 'Paul Coverdale'; [email-masked]
Subject: RE: [codec] Characterization and Testing

Hi Paul,

With best regards,

 Christian Hoene

---

From: "Christian Hoene" <[email-masked]>
Date: Tue, 22 Mar 2011 14:47:32 +0100

Hi Paul,

> statistical analysis methodology, etc.
[Christian Hoene] 

Yes, you are right. These all have to be considered. If I have not mentioned
all the details is was because they were mentioned in ITU documents, which I
was referring too.

Luckily, the ITU Study Group 12 has written some excellent tutorial on how
to do the tests http://www.itu.int/md/T09-SG12-110118-TD-GEN-0477/en 
I thought it would be best to stick to the well proven procedures of the
ITU, if not Opus specific features and Internet specific requirements must
be tested.
Anyhow, one should repeat the detail testing procedures and specifications
in the test plan.

Beyond that, one must define precise codec parameters to test and a
schedule. Last not least, the testing houses must be willing to contribute.
I am looking forward to see which companies want to participate...

With best regards,
 Christian

---

From: Jean-Marc Valin <[email-masked]>
Date: Tue, 22 Mar 2011 11:03:18 -0400

Christian,

My main concern with your draft is actually Section 10.2.4. Degradation 
Factors, which to me look more like a wish list than a test plan. At the 
end of the list, you conclude that "Already, these are 48 different tests 
that need to be conducted." Not only is this number *really* high for 
listening tests, but it seems to result from merely adding the number of 
cases listed above, rather than *multiplying* them. For example, 5 
different bitrates with 2 different frame sizes is 10 tests, not 7. 
Considering this, we end up with *thousands* of tests.

Also, I think I remember (but I could be wrong) that you were interested in 
getting involved in the actual testing. I wanted to know how that was 
progressing and how you were applying draft-hoene-codec-quality-00 to these 
tests?

Cheers,

	Jean-Marc

---

From: "Paul Coverdale" <[email-masked]>
Date: Tue, 22 Mar 2011 17:11:45 +0100

Hi Christian,

Yes, the general approach for subjective testing of codecs is well covered
in ITU-T Study Group 12 documentation, but for any given codec under
evaluation a specific detailed test plan is required. One thing I forgot to
mention in my earlier post is that several languages are generally used.
This, of course, requires the need for listeners who are completely
comfortable in the source language (eg, allows bi-lingual Canadians).

Regards,

...Paul

-----Original Message-----
From: Christian Hoene [mailto:[email-masked]] 
Sent: Tuesday, March 22, 2011 2:48 PM
To: 'Paul Coverdale'; [email-masked]
Subject: RE: [codec] Characterization and Testing

Hi Paul,

[Christian Hoene] 

With best regards,
 Christian

---

From: "Christian Hoene" <[email-masked]>
Date: Tue, 22 Mar 2011 17:30:15 +0100

Hi Jean-Marc,

> Factors, which to me look more like a wish list than a test plan. 

[Christian Hoene] Yes, as usual in negotiations, it is meant as a starting
point and open to be changed after discussion has proceeded. Actual it
contains features that are not yet implemented (or ever will be?) such as
time stretching and shrinking. These features, which not listed in the
requirements draft but in the codec testing, are however important for IP
based transmission. (You don't believe it? Read the entire document.)

Nevertheless, the selection of the 48 parameters test is a first step to
bring the number of tests down to a reasonable amount. 

> At the

[Christian Hoene] Thousands of tests can only be done by instrumental means
with the help of computing clusters. Such things have been done before and
can be done again for Opus.

To limit the number of listening-only tests, the idea is just to test the
select a low number of important parameter sets, which then are be used to
calibrate the instrumental results. 

Also, I think it makes only sense to test those parameter tests, which are
important for transmissions on the Internet.  For example, it does not make
sense to test a low-latency and high quality CELT modes together with a FEC
or low complexity modes because high quality CELT demands already high
performance devices and goods links that have low loss rates. Instead, FEC
should be tested with narrow-band Silk mode.

> progressing and how you were applying draft-hoene-codec-quality-00 to
> these tests?

[Christian Hoene] The code base is still moving. Last time I had a look on
the source code (roughly 4 weeks ago) there were many unfinished things.

I think a reasonable plan for the characterization still is to do them in
second half of 2011, after the code base has become more stable.

With best regards,

  Christian Hoene

---

From: "Paul Coverdale" <[email-masked]>
Date: Tue, 22 Mar 2011 17:37:03 +0100

This is a key issue in developing a realistic test plan - there are always
more conditions that you would like to test than is reasonably possible. Of
course, you can segregate into individual experiments, but there is always a
trade-off required which needs discussion and agreement.

Paul Coverdale

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Jean-Marc Valin
Sent: Tuesday, March 22, 2011 4:03 PM
To: Christian Hoene
Cc: [email-masked]
Subject: Re: [codec] Characterization and Testing

Christian,

Cheers,

	Jean-Marc

---

From: Stephan Wenger <[email-masked]>
Date: Tue, 22 Mar 2011 18:26:53 +0100

Please understand that cost-sensitive third parties (including my
employer) are willing to contribute to a test only after they are
convinced that the test conditions make sense.  Therefore, we should
perhaps spend a (considerable) amount of time (during the WG session to
rally support, and outside to get the work done) in advancing the test
plan.  

In the spirit of previous discussions on this list, I would be willing to
agree reducing the listening test to the bare minimum.  To me, a
restriction to a single language would be OK.  Voluntary "community based"
subjective tests in more than one language are of course encouraged, but I
think we can forego a formal MUSHRA style setting for those tests.

Needless to say, I don't believe that there is a point in last-calling a
codec specification document before the results of a characterization are
in and matched against requirements (I have said that before).

Stephan

On 3.22.2011 16:03 , "Jean-Marc Valin" <[email-masked]> wrote:

>these

---

From: Jean-Marc Valin <[email-masked]>
Date: Tue, 22 Mar 2011 23:16:25 +0000

Hi,

I'm aware of two tests being conducted at the moment on Opus. One is an internal test being done by Google. The other is being conducted on the HydrogenAudio forum: 
http://listening-tests.hydrogenaudio.org/igorc/Public%20Multiformat%20Listening%20Test%20@%2064kbps.htm

The HydrogenAudio test is comparing Opus (CELT-only mode) to *non-interactive* (>100 ms delay) codecs HE-AAC and Vorbis. So while it's not an apples-to-apples comparison, it should be interesting to see how Opus does compared to state-of-the-art codecs that do not have latency constraints.

Hopefully, both these tests will get enough participants by the next IETF meeting.

Cheers,

      Jean-Marc

________________________________________
From: [email-masked] [[email-masked]] on behalf of Cullen Jennings [[email-masked]]
Sent: Tuesday, March 22, 2011 12:52 AM
To: [email-masked]
Subject: [codec] Characterization and Testing

Thanks, Cullen

---

From: Anisse Taleb <[email-masked]>
Date: Tue, 29 Mar 2011 12:35:04 +0000

Dear Stephan,
> rally support, and outside to get the work done) in advancing the test

I support this viewpoint, not only because of the costs, but that there needs to be an agreement 
what these tests are. The performance of the Opus codec need to be known and characterized with reliable enough test results which are reproducible. Therefore, at least 2 different test labs (with different languages) should participate for every test. 

So first, we need an "agreed" test plan that fully specifies what methodology and what tests are to be conducted. It will in addition help Organizations (including my employer) to either commit or not to these tests.

> think we can forego a formal MUSHRA style setting for those tests.

There are examples on which certain (unnamed) codecs performed very well for English while they show clear weaknesses for other languages. An "Internet" codec addressing a global market should not shy away from being tested in the many languages that the internet community speaks. I would not feel at ease to WGLC a codec if it is tested in one language only.

When it comes to the size of the test, and the relevant test points, we need to be pragmatic: One idea could be to list -in order of importance- the use cases (relevant bitrates, frame lengths, modes, etc...) of the Opus codec. For the most important use cases (maybe speech one-to-one call @12kbps or 16kbps, 20ms, silk mode?), the tests need to be performed in at least two languages (i.e. > 2 labs) with clearly established reference codecs (AMR-WB, G.718, ...?). For other, say less relevant or niche, use cases a relaxed testing could be envisioned.

Kind regards,
/Anisse