Subject: [codec] Opus comparision test plan

From: "Christian Hoene" <[email-masked]>
Date: Tue, 16 Apr 2013 09:43:25 +0200

Hello,

as for next week, we place to compare Opus against AMR-WB (with noisy
signals) and AAC-eLD. Attached our preliminary test plan. Feel free to
criticize it.

In addition to the comparison, we will do some Opus characterization (Opus
vs. Opus). More to about this test later.

With best regards,

Christian Hoene

Symonics GmbH
Sand 13
72076 Tübingen
Tel +49 7071 5681302
Fax +49 7071 5681309
Email: [email-masked]

Geschäftsführer/Presidents: Michael Haun, Dr. Christian Hoene, Patrick
Schreiner
Sitz der Gesellschaft/Place of Business: Tübingen
Registereintrag/Commercial Register: Amtsgericht Stuttgart, HRB 739918

---

From: Jean-Marc Valin <[email-masked]>
Date: Tue, 16 Apr 2013 14:36:16 -0400

Christian,

I was only able to look at your plan quickly but still found many issues
with it. I think you should take your time to fix those and also all the
ones I'm sure I missed. So far...

== AMR-WB test ==

- You should use 8 kHz bandwidth for Opus. Opus it not restricted to 7
kHz and low-pass filtering will hurt quality.
- For AMR-WB the 8.85 mode is "intended to be used only temporarily
during severe radio channel conditions or during network congestion", so
you should probably test the 18.25 kbps mode instead.
- It's not clear from your document, but Opus should use VBR
- If you're going to test packet loss, there should be at least one test
with FEC -- probably the 23.85 one
- Make sure to tell the Opus encoder what the percentage of loss is so
it can optimize the encoding for it.

== AAC-ELD test ==

- Some of the rates in the test are in the bitrates where both codecs
are expected to be "falling apart", so we will not learn much there
(really bad quality vs absolutely terrible doesn't help because people
don't want either).
- Some of the rates are way too high for listeners to be able to
reliably judge. These are guaranteed to end up with everything tied
unless there's a problem with the test.
- Using SBR at 128 kb/s for ELD is useless and probably harming quality
- There is no reason to test music at 24/32 kHz because music is almost
always available at 48 kHz, so it would hurt to artificially limit the
signal.
- When considering CBR vs VBR you have to consider the delay, but also
the different meanings these have between codecs. I'm not familiar with
the details, but what most codecs like AAC-LC and MP3 call "CBR"
actually uses a bit reservoir and is equivalent to Opus "Constrained VBR".
- The choice of music samples is critical. Just that choice can
sometimes determine the outcome of a music test. You need a wide variety
of samples to get something statistically significant. It's also
probably a good idea to use "normal" samples at low rates and "hard"
samples at high rate.
- I'd need to recheck the MUSHRA spec, but I'm pretty sure mixing mono
and stereo in the same session is highly discouraged.
- Not sure in what form, but it might be worth testing the pre-1.1
encoder of Opus.

In general, for mono music, I'd recommend restricting yourself to rates
between 40 and 64 kb/s. For stereo music, I'd stay within 56-96 kb/s.
Outside of this you're either in the falling apart region or in the
"nearly transparent for most listeners".

Also, I hope you will fix your methodology from the previous test to
actually tell the listeners about the hidden reference.

In any case, this is only what I've been able to find with a quick look.
It's probably good to have wider review once you fix everything.

	Jean-Marc

On 04/16/2013 03:43 AM, Christian Hoene wrote:
> _______________________________________________
> codec mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/codec
>

---

From: Paul Coverdale <[email-masked]>
Date: Tue, 16 Apr 2013 19:27:37 -0400

Hi Christian,

Yes, I think there is still some work to be done on the plan, I noticed for
example that input level and listening level still need to be defined. And
why are you using MUSHRA? It may be quite time-consuming with a large number
of conditions.

But just to comment on some of the points raised by Jean-Marc:
- we need to be consistent about the input filtering. We can't use different
bandwidths for Opus and AMR-WB.
- I don't understand the point about making sure to tell the Opus encoder
what the percentage of loss is so it can optimize the encoding for it. What
happens in practice when the loss percentage isn't known?

Cheers,

...Paul

>-----Original Message-----
>From: [email-masked] [mailto:[email-masked]] On Behalf
>Of Jean-Marc Valin
>Sent: Tuesday, April 16, 2013 2:36 PM
>To: Christian Hoene
>Cc: [email-masked]; [email-masked];
>[email-masked]
>Subject: Re: [codec] Opus comparision test plan
>
>Christian,
>
>> In addition to the comparison, we will do some Opus characterization
>> (Opus vs. Opus). More to about this test later.
>>
>> With best regards,
>>
>> Christian Hoene
>>
>>
>>
>>
>> Symonics GmbH
>> Sand 13
>> 72076 Tübingen
>> Tel +49 7071 5681302
>> Fax +49 7071 5681309
>> Schreiner Sitz der Gesellschaft/Place of Business: Tübingen

---

From: Jean-Marc Valin <[email-masked]>
Date: Tue, 16 Apr 2013 20:40:23 -0400

On 04/16/2013 07:27 PM, Paul Coverdale wrote:
> Yes, I think there is still some work to be done on the plan, I
> noticed for example that input level and listening level still need
> to be defined. And why are you using MUSHRA? It may be quite
> time-consuming with a large number of conditions.

MUSHRA is actually the least time-consuming and it is designed for
"intermediate quality level", which is mostly what we're after here.
Technically the 12 kbps content may be in the MOS area and the 96 kb/s
content would be in the BS.1116-1 area, but I suspect MUSHRA is
probably goo enough for everything.

> But just to comment on some of the points raised by Jean-Marc: - we
> need to be consistent about the input filtering. We can't use
> different bandwidths for Opus and AMR-WB.

Well, then make it 8 kHz for everything. There's no point in degrading
the input on purpose. In fact, technically it should be 48 kHz and
then each codec gets to decide what it does to it.

> - I don't understand the point about making sure to tell the Opus
> encoder what the percentage of loss is so it can optimize the
> encoding for it. What happens in practice when the loss percentage
> isn't known?

Well, that's what RTCP and others are for. Sure you don't necessarily
know that you have 4.9% or 5% loss, but you can have a pretty good idea.

	Jean-Marc

> 
> Cheers,
> 
> ...Paul
> 
> 
>> -----Original Message----- From: [email-masked]
>> [mailto:[email-masked]] On Behalf Of Jean-Marc Valin 
>> Sent: Tuesday, April 16, 2013 2:36 PM To: Christian Hoene Cc:
>> [email-masked]; [email-masked]; 
>> [email-masked] Subject: Re: [codec] Opus
>> comparision test plan
>> 
>> Christian,
>> 
>> I was only able to look at your plan quickly but still found many
>> issues with it. I think you should take your time to fix those
>> and also all the ones I'm sure I missed. So far...
>> 
>> == AMR-WB test ==
>> 
>> - You should use 8 kHz bandwidth for Opus. Opus it not restricted
>> to 7 kHz and low-pass filtering will hurt quality. - For AMR-WB
>> the 8.85 mode is "intended to be used only temporarily during
>> severe radio channel conditions or during network congestion",
>> so you should probably test the 18.25 kbps mode instead. - It's
>> not clear from your document, but Opus should use VBR - If you're
>> going to test packet loss, there should be at least one test with
>> FEC -- probably the 23.85 one - Make sure to tell the Opus
>> encoder what the percentage of loss is so it can optimize the
>> encoding for it.
>> 
>> == AAC-ELD test ==
>> 
>> - Some of the rates in the test are in the bitrates where both
>> codecs are expected to be "falling apart", so we will not learn
>> much there (really bad quality vs absolutely terrible doesn't
>> help because people don't want either). - Some of the rates are
>> way too high for listeners to be able to reliably judge. These
>> are guaranteed to end up with everything tied unless there's a
>> problem with the test. - Using SBR at 128 kb/s for ELD is useless
>> and probably harming quality - There is no reason to test music
>> at 24/32 kHz because music is almost always available at 48 kHz,
>> so it would hurt to artificially limit the signal. - When
>> considering CBR vs VBR you have to consider the delay, but also 
>> the different meanings these have between codecs. I'm not
>> familiar with the details, but what most codecs like AAC-LC and
>> MP3 call "CBR" actually uses a bit reservoir and is equivalent to
>> Opus "Constrained VBR". - The choice of music samples is
>> critical. Just that choice can sometimes determine the outcome of
>> a music test. You need a wide variety of samples to get something
>> statistically significant. It's also probably a good idea to use
>> "normal" samples at low rates and "hard" samples at high rate. -
>> I'd need to recheck the MUSHRA spec, but I'm pretty sure mixing
>> mono and stereo in the same session is highly discouraged. - Not
>> sure in what form, but it might be worth testing the pre-1.1 
>> encoder of Opus.
>> 
>> In general, for mono music, I'd recommend restricting yourself to
>> rates between 40 and 64 kb/s. For stereo music, I'd stay within
>> 56-96 kb/s. Outside of this you're either in the falling apart
>> region or in the "nearly transparent for most listeners".
>> 
>> Also, I hope you will fix your methodology from the previous test
>> to actually tell the listeners about the hidden reference.
>> 
>> In any case, this is only what I've been able to find with a
>> quick look. It's probably good to have wider review once you fix
>> everything.
>> 
>> Jean-Marc
>> 
>>> as for next week, we place to compare Opus against AMR-WB (with
>>> noisy signals) and AAC-eLD. Attached our preliminary test plan.
>>> Feel free to criticize it.
>>> 
>>> In addition to the comparison, we will do some Opus
>>> characterization (Opus vs. Opus). More to about this test
>>> later.
>>> 
>>> With best regards,
>>> 
>>> Christian Hoene
>>> 
>>> 
>>> 
>>> 
>>> Symonics GmbH Sand 13 72076 Tübingen Tel +49 7071 5681302 Fax
>>> +49 7071 5681309 Email: [email-masked]
>>> 
>>> Geschäftsführer/Presidents: Michael Haun, Dr. Christian Hoene,
>>> Patrick Schreiner Sitz der Gesellschaft/Place of Business:
>>> Tübingen Registereintrag/Commercial Register: Amtsgericht
>>> Stuttgart, HRB 739918
>>> 
>>> 
>>> 
>>> 
>>> _______________________________________________ codec mailing
>>> list [email-masked] 
>> _______________________________________________ codec mailing
>> list [email-masked] https://www.ietf.org/mailman/listinfo/codec
> 
> _______________________________________________ codec mailing list 
> [email-masked] https://www.ietf.org/mailman/listinfo/codec
>

---

From: Koen Vos <[email-masked]>
Date: Wed, 17 Apr 2013 00:47:48 +0000

Hi Paul,

> bandwidths for Opus and AMR-WB.

Nobody uses a low-pass filter to remove the frequencies between 7 and 8 kHz in practice.  So applying such a filter to test material reveals little about real-life performance.

Philosophically, filtering with the smaller frequency response makes little sense either.  Let's say someone built a codec that encodes only frequencies from 1000 to 1100 Hz.  When comparing that codec, should we apply a similar narrow bandpass filter to the inputs of other codecs?  How about if two codecs encoded mutually-non-overlapping parts of the spectrum?

> happens in practice when the loss percentage isn't known?

Here I agree more with you.  There may be poor implementations that neglect to report packet loss back to the encoder.  For characterization purposes it would be interesting to measure the performance hit from such implementations as compared to ideal implementations.

best,
koen.

________________________________________
From: [email-masked] on behalf of Paul Coverdale
Sent: Tuesday, April 16, 2013 4:27 PM
To: 'Jean-Marc Valin'; 'Christian Hoene'
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [codec] Opus comparision test plan

Hi Christian,

Cheers,

...Paul

---

From: <[email-masked]>
Date: Wed, 17 Apr 2013 04:04:22 +0200

Hi Christian,

For the test it should be ensured that the used codec implementation fulfills  high quality requirements. I don't know the AAC-ELD fdk-aac library.  The link in your document did not work for me. It does not make sense to test an AAC-ELD encoder that does not provide state-of-the-art encoding quality.
Concerning SBR: AFAIK know AAC-ELD standard does not have SBR specified as a tool.

I think, 20 % Frame erasure rate makes no sense as it is unrealistic. I would prefer frame erase rates like 1%, 3% or  6%. These rates should be used both in speech and music experiments. 

For music testing I would recommend  the bitrates 24, 32, 48 kbit/s for mono at 32 kHz sample rate. For stereo 32, 48, 64 and 96 kbit/s could be used. I think these working points would be used most likely in applications where low delay is a major requirement.
It is more important to test a higher number of contents than to tests the very high bitrates.

Mono and stereo should not be mixed in one experiment to avoid the influence of this factor on the assessment of the coding quality. 
 
What is a MOS 4 anchor in the context of a MUSHRA test?

Best regards,
Bernhard

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Paul Coverdale
Sent: Mittwoch, 17. April 2013 01:28
To: 'Jean-Marc Valin'; 'Christian Hoene'
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [codec] Opus comparision test plan

Hi Christian,

Yes, I think there is still some work to be done on the plan, I noticed for example that input level and listening level still need to be defined. And why are you using MUSHRA? It may be quite time-consuming with a large number of conditions.

But just to comment on some of the points raised by Jean-Marc:
- we need to be consistent about the input filtering. We can't use different bandwidths for Opus and AMR-WB.
- I don't understand the point about making sure to tell the Opus encoder what the percentage of loss is so it can optimize the encoding for it. What happens in practice when the loss percentage isn't known?

Cheers,

...Paul

>issues with it. I think you should take your time to fix those and also 
>all the ones I'm sure I missed. So far...
>
>== AMR-WB test ==
>
>during severe radio channel conditions or during network congestion", 
>so you should probably test the 18.25 kbps mode instead.
>- If you're going to test packet loss, there should be at least one 
>test with FEC -- probably the 23.85 one
>sometimes determine the outcome of a music test. You need a wide 
>variety of samples to get something statistically significant. It's 
>also probably a good idea to use "normal" samples at low rates and "hard"
>samples at high rate.
>> signals) and AAC-eLD. Attached our preliminary test plan. Feel free 
>> to criticize it.
>>
>> Patrick Schreiner Sitz der Gesellschaft/Place of Business: Tübingen

---

From: Jean-Marc Valin <[email-masked]>
Date: Tue, 16 Apr 2013 22:20:38 -0400

On 04/16/2013 10:04 PM, [email-masked] wrote:
> For the test it should be ensured that the used codec
> implementation fulfills  high quality requirements. I don't know
> the AAC-ELD fdk-aac library.  The link in your document did not
> work for me. It does not make sense to test an AAC-ELD encoder that
> does not provide state-of-the-art encoding quality.

Well, last I heard when Jan Skoglund was organizing a music, there was
just no way for him to obtain any LD or ELD implementation (IIRC FhG
refused to send him one). That being said, I've no idea what the
quality of tdk-aac looks like.

> Concerning SBR: AFAIK know AAC-ELD standard does not have SBR

Actually SBR is the main improvement that ELD has over LD.

> I think, 20 % Frame erasure rate makes no sense as it is
> unrealistic. I would prefer frame erase rates like 1%, 3% or  6%.
> These rates should be used both in speech and music experiments.

I tend to agree on that one.

> For music testing I would recommend  the bitrates 24, 32, 48
> kbit/s for mono at 32 kHz sample rate.

This is way too low. You end up in a zone where neither codec will
provide acceptable quality music and where the result is highly
sensitive on the exact way in which the encoder implementation decided
to mitigate the "falling apart" case.

> For stereo 32, 48, 64 and 96 kbit/s could be used.

48 kb/s stereo is pushing it. 32 kb/s is just ridiculously low for
stereo and requires parametric stereo, which neither AAC-ELD nor Opus
has (not that PS sounds particularly good in the first place).

> I think these working points would be used most likely in
> applications where low delay is a major requirement.

For low-delay applications, it's useless to try saving 8 kb/s when
you're spending 32+ kb/s transmitting headers.

> tests the very high bitrates.

Totally agreed.

> What is a MOS 4 anchor in the context of a MUSHRA test?

Not sure what you mean by "MOS 4 anchor", but MUSHRA specifies that
you must have a 3.5 kHz low-pass anchor and optionally anchors with a
low-pass at 7 or 10 kHz (in this case 7 is more appropriate IMO).

Cheers,

	Jean-Marc

---

From: <[email-masked]>
Date: Wed, 17 Apr 2013 04:45:21 +0200

>Actually SBR is the main improvement that ELD has over LD.
Ok. You are right. Sorry for that.

>> For music testing I would recommend  the bitrates 24, 32, 48 kbit/s 
>> for mono at 32 kHz sample rate.
>This is way too low. You end up in a zone where neither codec will provide acceptable quality music and where the result is highly sensitive on the exact way in which the encoder implementation decided to mitigate the "falling >apart" case.
We are interested in a codec for telephony applications that also provides good audio quality. It has been shown that AAC-ELD might fulfill this in the range  of 24  to 32 kbit/s.  (AES 129, Ulf Wüstenhagen et. 
all, "Evaluation of Super-Wideband Speech and Audio Codecs"). Now it would be of interest how Opus behaves in this range.

>For low-delay applications, it's useless to try saving 8 kb/s when you're spending 32+ kb/s transmitting headers.
Headers might be compressed

Best regards,
Bernhard

-----Original Message-----
From: Jean-Marc Valin [mailto:[email-masked]] 
Sent: Mittwoch, 17. April 2013 04:21
To: Feiten, Bernhard
Cc: [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: ***CAUTION_Invalid_Signature*** Re: [codec] Opus comparision test plan

On 04/16/2013 10:04 PM, [email-masked] wrote:
> For the test it should be ensured that the used codec implementation 
> fulfills  high quality requirements. I don't know the AAC-ELD fdk-aac 
> library.  The link in your document did not work for me. It does not 
> make sense to test an AAC-ELD encoder that does not provide 
> state-of-the-art encoding quality.

Well, last I heard when Jan Skoglund was organizing a music, there was just no way for him to obtain any LD or ELD implementation (IIRC FhG refused to send him one). That being said, I've no idea what the quality of tdk-aac looks like.

> I think, 20 % Frame erasure rate makes no sense as it is unrealistic. 
> I would prefer frame erase rates like 1%, 3% or  6%.

> For music testing I would recommend  the bitrates 24, 32, 48 kbit/s 
> for mono at 32 kHz sample rate.

This is way too low. You end up in a zone where neither codec will provide acceptable quality music and where the result is highly sensitive on the exact way in which the encoder implementation decided to mitigate the "falling apart" case.

> For stereo 32, 48, 64 and 96 kbit/s could be used.

48 kb/s stereo is pushing it. 32 kb/s is just ridiculously low for stereo and requires parametric stereo, which neither AAC-ELD nor Opus has (not that PS sounds particularly good in the first place).

> I think these working points would be used most likely in applications 
> where low delay is a major requirement.

For low-delay applications, it's useless to try saving 8 kb/s when you're spending 32+ kb/s transmitting headers.

Totally agreed.

> What is a MOS 4 anchor in the context of a MUSHRA test?

Not sure what you mean by "MOS 4 anchor", but MUSHRA specifies that you must have a 3.5 kHz low-pass anchor and optionally anchors with a low-pass at 7 or 10 kHz (in this case 7 is more appropriate IMO).

Cheers,

	Jean-Marc

---

From: "Christian Hoene" <[email-masked]>
Date: Wed, 17 Apr 2013 11:09:45 +0200

Hello Jean-Marc,

> Well, last I heard when Jan Skoglund was organizing a music, there was
just no way for him to obtain any LD or ELD implementation (IIRC FhG refused
to send him one). That being said, I've no idea what the quality of tdk-aac
looks like.

The AAC-eLD is available for Android:
http://www.iis.fraunhofer.de/en/bf/amm/implementierungen/fdkaaccodec.html

With best regards,

 Christian

---

From: Paul Coverdale <[email-masked]>
Date: Wed, 17 Apr 2013 10:30:03 -0400

>MUSHRA is actually the least time-consuming...

This isn't what my colleagues in ITU-T and 3GPP, who are heavily involved in
codec subjective testing, are telling me. Typically, you can only test about
12 conditions at a time.

...Paul

---

From: "Christian Hoene" <[email-masked]>
Date: Wed, 17 Apr 2013 16:37:20 +0200

Long time ago, I collected some information about durations and no of sample
items:
http://tools.ietf.org/html/draft-hoene-codec-quality-01#page-8
For MUSHRA: In 30 minutes, about 42 stimuli can be tested.
For ACS : Listeners can judge about 54 stimuli within 30 minutes.

That's what I have been told.

ACS is faster but less precise.

With best regards,

 Christian

-----Ursprüngliche Nachricht-----
Von: Paul Coverdale [mailto:[email-masked]] 
Gesendet: Mittwoch, 17. April 2013 16:30
An: 'Jean-Marc Valin'
Cc: 'Jean-Marc Valin'; 'Christian Hoene'; [email-masked];
[email-masked]; [email-masked]
Betreff: RE: [codec] Opus comparision test plan

...Paul

---

From: Paul Coverdale <[email-masked]>
Date: Wed, 17 Apr 2013 11:22:22 -0400

>From: Koen Vos [mailto:[email-masked]]
>Sent: Tuesday, April 16, 2013 8:48 PM
>To: Paul Coverdale; 'Jean-Marc Valin'; 'Christian Hoene'
>Subject: RE: [codec] Opus comparision test plan
>
>Hi Paul,
>
>> - we need to be consistent about the input filtering. We can't use
>Nobody uses a low-pass filter to remove the frequencies between 7 and 8
>kHz in practice.  So applying such a filter to test material reveals
>little about real-life performance.
>
>Philosophically, filtering with the smaller frequency response makes
>little sense either.  Let's say someone built a codec that encodes only
>frequencies from 1000 to 1100 Hz.  When comparing that codec, should we
>apply a similar narrow bandpass filter to the inputs of other codecs?
>How about if two codecs encoded mutually-non-overlapping parts of the
>spectrum?

In the past, a low-pass filter was typically used in codec subjective tests
to mask any differences in anti-aliassing filters inherent in the codecs
under test, so that the subjective opinion would be based on distortion and
artifacts alone, and not confounded with bandwidth. If this isn't
appropriate any more, we can take it out.

>> encoder what the percentage of loss is so it can optimize the encoding
>> for it. What happens in practice when the loss percentage isn't known?
>
>Here I agree more with you.  There may be poor implementations that
>neglect to report packet loss back to the encoder.  For characterization
>purposes it would be interesting to measure the performance hit from
>such implementations as compared to ideal implementations.

Yes, I think we need to include the situation where packet loss is not
reported back to the encoder.

....Paul